# Message Broker using Kafka

*Originally published on Feb 4, 2024 on* [*my previous blog*](https://somesh-banerjee.github.io/blogs/blog/14)*.*

---

This is the second part of the message brokers tutorial. In the first part, we discussed RabbitMQ. In this part, we will discuss Kafka.

## What is Kafka?

Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming applications. It is designed to handle high throughput and provides fault tolerance and scalability. Kafka is often used for use cases like real-time analytics, log aggregation, event sourcing, and more.

### Key Features

1. **Producer**: A producer is a component that publishes messages to Kafka topics. Topics are logical channels for organizing and categorizing messages.
2. **Consumer**: A consumer is a component that subscribes to Kafka topics and reads messages from them.
3. **Broker**: Kafka runs as a cluster of servers, and each server in the cluster is called a broker. Brokers handle storage, replication, and serving of messages.
4. **Topic**: A topic is a category or feed name to which messages are published by producers and from which consumers consume messages.
5. **Partition**: Each topic can be split into multiple partitions, which allows Kafka to scale and distribute data across multiple brokers.
6. **Offset**: Each message within a partition has a unique identifier called an offset, which denotes the position of the message in the partition.
7. **Consumer Group**: Consumers reading from a Kafka topic can be organized into consumer groups. Each message in a topic is consumed by only one member of a consumer group.

### Setting up Kafka

We will use Docker to set up Kafka. If you don't have Docker installed, you can follow the instructions [here](https://docs.docker.com/get-docker/) to install it.

Once Docker is installed, we can follow the docker-compose file to set up Kafka.

```yaml
version: '3'

services:
  zoo1:
    image: confluentinc/cp-zookeeper:7.3.2
    hostname: zoo1
    container_name: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888

  kafka1:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
      - "29092:29092"
      - "9999:9999"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1
```

## Hello World

Now that we have Kafka up and running, let's write a simple Hello World program to get a feel for how Kafka works.

We will try to create a simple log aggregation system using Kafka. In this system multiple services will send logs to Kafka and a consumer will read the logs and print them in one place.

We will have two components in our Hello World program: a producer and a consumer. The producer will send log to a queue and the consumer will receive the log from the queue and print it to the console.

### Producer

```js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'log-aggregator-app',
  brokers: ['kafka1:9092']
});

const producer = kafka.producer();

async function sendLogMessage(message) {
  await producer.connect();
  await producer.send({
    topic: 'logs',
    messages: [
      { value: JSON.stringify(message) }
    ]
  });
  await producer.disconnect();
}

setInterval(() => {
    sendLogMessage({
        message: 'This is a log message',
        timestamp: new Date().toISOString(),
        service: process.env.SERVICE_NAME
    });
}, 4000);

producer.connect().then(() => {
    console.log('Connected to Kafka');
}).catch((error) => {
    console.error('Error connecting to Kafka', error);
});
```

### Consumer

```js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'log-aggregator-app',
  brokers: ['kafka1:9092']
});


const consumer = kafka.consumer({
  groupId: 'log-aggregator-group',
  allowAutoOffsetReset: true,
  autoOffsetReset: 'earliest'
});

async function consumeLogMessages() {
  await consumer.connect();
  await consumer.subscribe({
    topic: 'logs',
    fromBeginning: true
  });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const logMessage = JSON.parse(message.value.toString());
      console.log(`Received log message: ${JSON.stringify(logMessage)}`);

      // send log message to log aggregation service
      // e.g. using http request or websocket
    }
  });
}

consumeLogMessages()
    .then(() => {
        console.log('Consuming log messages');
    })    
    .catch((error) => {
        console.error('Error consuming log messages', error);
    });
```

### Running the Program

I have created a docker-compose file to run the producer and consumer along with kafka. You can run the following command to start the producer and consumer.

```bash
docker compose up 
```

This is a simple example of how Kafka works using the application of log aggregation.
I have also created a repository which contains all the code. You can find the repository [here](https://github.com/somesh-banerjee/message-brokers-tutorial/tree/main/kafka)

