All nodes that are supposed to be managed using this repository must have valid DNS settings and be able to reach each other using FQDN.
Before start adding new Kafka servers there should be at least one ZooKeeper server that will maintain Kafka cluster. For this reason, be sure to add "zookeeper_node" to one or more Chef-managed nodes and wait for chef-client run.
Start adding role "kafka_node" to Chef-managed nodes. It is not mandatory, but recommended to avoid simultaneous provision of Kafka nodes. General recommendation is: add role "kafka_node" to new server, wait for successful ending of chef-client run on that server, add role "kafka_node" to next server, etc.
Ssh into any of your Kafka servers. Use your zookeeper server FQDN and replication factor that is less or equal to the number of Kafka nodes in a cluster.
cd /opt/kafka
bin/kafka-topics.sh --create --zookeeper zookeeper.example.com:2181 --replication-factor 3 --partitions 1 --topic testStart producer on the same server and type some messages.
bin/kafka-console-producer.sh --broker-list this-server-fqdn.example.com:9092 --topic test
This is a message
This is another messageSsh into another Kafka server and start consumer.
cd /opt/kafka
bin/kafka-console-consumer.sh --zookeeper zookeeper.example.com:2181 --topic test --from-beginningYou will see the output:
This is a message
This is another message