Environment

node1 node2 node3
Role Server Producer Consumer
Mem 2048 MB 2048 MB 2048 MB
OS Ubuntu 14.04 64 Ubuntu 14.04 64 Ubuntu 14.04 64
Public IP 121.40.83.134 - -
Internal IP 192.168.6.18 192.168.6.15 192.168.6.16


Apache Kafka: kafka_2.11-0.8.2.2.tgz

Roles

img

  • node1: Kafka cluster
  • node2: Producer
  • node3: Comsumer

Launch

Preparation

Download kafka_2.11-0.8.2.2.tgz and extract to each node.

scp kafka_2.11-0.8.2.2.tgz [user]@[node]:[path]
tar xvzf kafka_2.11-0.8.2.2.tgz

Start Kafka Server

Note

If you don’t have Zoopkeeper running in your cluster, don’t forget to lauch Zoopkeeper before starting Kafka server.

# In the root directory of Kafka
./bin/zookeeper-server-start.sh config/zookeeper.properties

Launch Kafka server in node1

./bin/kafka-server-start.sh ./config/server.properties

You can check the status of the default port for Kafka Server

netstat -npl 9092

Setup Producer

I developed a simple log generator web service with Node.js, you can clone the repo for a quick demo.

git clone https://github.com/HaohanStacks/haohan_data_producer.git generator && cd generator
npm i && npm start

For more details about this repo, please check the document here: https://github.com/HaohanStacks/haohan_data_producer.

In node2, logs will be generated in ./generator/logs/access.log. Now we need to send this log content to Kafka Server.

tail -n 0 -f ../haohan_data_producer/logs/access.log | bin/kafka-console-producer.sh --broker-list 192.168.6.18:9092 --topic haohan

Pay attention to the path of log file and binary files. 192.168.6.18 is node1’s ip address, haohan is the topic of this producer. Now we have pushed logs to Kafka Server.

Setup Comsumer

In node3, go to the root directory of Kafka

bin/kafka-console-consumer.sh --zookeeper 192.168.6.18:2181 --topic haohan --from-beginning

2181 is the default port for zookeeper, pay attention to the topic name, it should be the same as what you have named when you setup the producer in node2.

Test

Open another terminal, send http request to your web service (This web service is launched by Node.js, the default port is 3000).

curl 192.168.6.15:3000

And voilà, in node3 you will see the received log text. :D