Skip to main content

Message Broker using Kafka

A guide to message broker using Kafka by creating a simple real-time loggin system using python and Kafka.

Updated 4 min readMarkdown

Originally published on Feb 4, 2024 on my previous blog.


This is the second part of the message brokers tutorial. In the first part, we discussed RabbitMQ. In this part, we will discuss Kafka.

What is Kafka?

Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming applications. It is designed to handle high throughput and provides fault tolerance and scalability. Kafka is often used for use cases like real-time analytics, log aggregation, event sourcing, and more.

Key Features

  1. Producer: A producer is a component that publishes messages to Kafka topics. Topics are logical channels for organizing and categorizing messages.
  2. Consumer: A consumer is a component that subscribes to Kafka topics and reads messages from them.
  3. Broker: Kafka runs as a cluster of servers, and each server in the cluster is called a broker. Brokers handle storage, replication, and serving of messages.
  4. Topic: A topic is a category or feed name to which messages are published by producers and from which consumers consume messages.
  5. Partition: Each topic can be split into multiple partitions, which allows Kafka to scale and distribute data across multiple brokers.
  6. Offset: Each message within a partition has a unique identifier called an offset, which denotes the position of the message in the partition.
  7. Consumer Group: Consumers reading from a Kafka topic can be organized into consumer groups. Each message in a topic is consumed by only one member of a consumer group.

Setting up Kafka

We will use Docker to set up Kafka. If you don't have Docker installed, you can follow the instructions here to install it.

Once Docker is installed, we can follow the docker-compose file to set up Kafka.

version: '3'

services:
  zoo1:
    image: confluentinc/cp-zookeeper:7.3.2
    hostname: zoo1
    container_name: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888

  kafka1:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
      - "29092:29092"
      - "9999:9999"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1

Hello World

Now that we have Kafka up and running, let's write a simple Hello World program to get a feel for how Kafka works.

We will try to create a simple log aggregation system using Kafka. In this system multiple services will send logs to Kafka and a consumer will read the logs and print them in one place.

We will have two components in our Hello World program: a producer and a consumer. The producer will send log to a queue and the consumer will receive the log from the queue and print it to the console.

Producer

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'log-aggregator-app',
  brokers: ['kafka1:9092']
});

const producer = kafka.producer();

async function sendLogMessage(message) {
  await producer.connect();
  await producer.send({
    topic: 'logs',
    messages: [
      { value: JSON.stringify(message) }
    ]
  });
  await producer.disconnect();
}

setInterval(() => {
    sendLogMessage({
        message: 'This is a log message',
        timestamp: new Date().toISOString(),
        service: process.env.SERVICE_NAME
    });
}, 4000);

producer.connect().then(() => {
    console.log('Connected to Kafka');
}).catch((error) => {
    console.error('Error connecting to Kafka', error);
});

Consumer

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'log-aggregator-app',
  brokers: ['kafka1:9092']
});


const consumer = kafka.consumer({
  groupId: 'log-aggregator-group',
  allowAutoOffsetReset: true,
  autoOffsetReset: 'earliest'
});

async function consumeLogMessages() {
  await consumer.connect();
  await consumer.subscribe({
    topic: 'logs',
    fromBeginning: true
  });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      const logMessage = JSON.parse(message.value.toString());
      console.log(`Received log message: ${JSON.stringify(logMessage)}`);

      // send log message to log aggregation service
      // e.g. using http request or websocket
    }
  });
}

consumeLogMessages()
    .then(() => {
        console.log('Consuming log messages');
    })    
    .catch((error) => {
        console.error('Error consuming log messages', error);
    });

Running the Program

I have created a docker-compose file to run the producer and consumer along with kafka. You can run the following command to start the producer and consumer.

docker compose up 

This is a simple example of how Kafka works using the application of log aggregation. I have also created a repository which contains all the code. You can find the repository here

No comments yet. Be the first to comment.