Apache Kafka: A Distributed Publish-Subscribe Messaging System

baldeva · October 29, 2025, 12:35pm

Apache Kafka: A Distributed Publish-Subscribe Messaging System

Apache Kafka is a fast, scalable, and distributed messaging system that has become a cornerstone of modern data processing pipelines. Originally developed by LinkedIn, Kafka is now a part of the Apache project, widely adopted in the industry for its high throughput, low latency, and fault-tolerant design.

Architecture

Kafka’s architecture consists of several key components:

Topics: A topic is a category of messages, where each message is a byte payload with a corresponding topic name.
Producers: Producers are responsible for publishing messages to topics. They can be configured to send messages to specific partitions within a topic.
Brokers: Brokers are the servers that store published messages. They are also responsible for managing partitions and replicating messages across multiple brokers.
Consumers: Consumers are responsible for reading messages from topics. They can be configured to subscribe to specific topics and partitions.

Storage Strategy

Kafka’s storage strategy is designed to handle high-volume data streams efficiently. Here are the key aspects:

Partitioning: Messages are partitioned across multiple brokers to ensure scalability and fault tolerance.
Segmentation: Each partition is divided into segments, where each segment stores a batch of messages.
Indexing: Each segment has an index that maps message IDs to their corresponding storage locations.

Data Retention Policies

Kafka provides two data retention policies:

Delete N days ago: This policy deletes messages older than a specified number of days.
How much Size retains the most recent data: This policy retains a specified amount of data based on size.

Broker

Kafka brokers are stateless, meaning that consumers must maintain their own state information. This is achieved through the use of a centralized offset manager, which tracks the position of each consumer in the message stream.

Producer

Producers can be configured to send messages to specific partitions within a topic. They can also be configured to wait for acknowledgments from the broker before considering a message sent successfully.

Consumer

Consumers can be configured to subscribe to specific topics and partitions. They can also be configured to pull messages from the broker at their own pace.

Consumer Groups

Consumer groups are a way to manage multiple consumers that work together to process messages from a topic. Each consumer in a group is responsible for consuming messages from a specific partition.

Rebalance

Kafka provides a rebalance mechanism to ensure that consumers in a group are evenly distributed across partitions. This is achieved through the use of a centralized coordinator, which tracks the position of each consumer in the group.

Message Delivery Semantics

Kafka provides three message delivery semantics:

At most once: Messages may be lost but are never redelivered.
At least once: Messages are never lost but may be redelivered.
Exactly once: Each message is delivered once and only once.

Producer Configuration

Producers can be configured to send messages to specific partitions within a topic. They can also be configured to wait for acknowledgments from the broker before considering a message sent successfully.

Consumer Configuration

Consumers can be configured to subscribe to specific topics and partitions. They can also be configured to pull messages from the broker at their own pace.

Code Examples

Kafka provides several code examples to demonstrate its usage. These examples include a producer example, a consumer example, and a partitioning example.

Developer Setup

Kafka provides a developer setup guide to help users get started with building and testing Kafka applications.

Committing and Fetching Consumer Offsets

Kafka provides a guide on how to commit and fetch consumer offsets in Kafka.

Index

Kafka provides an index of its documentation, making it easy to navigate and find specific information.

Conclusion

Apache Kafka is a powerful and flexible messaging system that is widely adopted in the industry. Its high throughput, low latency, and fault-tolerant design make it an ideal choice for modern data processing pipelines. With its rich feature set and extensive documentation, Kafka is a valuable resource for developers and architects looking to build scalable and reliable data-driven applications.

Code Snippets

// Producer example
import java.util.Properties;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

public class TestProducer {
    public static void main(String[] args) {
        long events = Long.parseLong(args[0]);
        Random rnd = new Random();
        Properties props = new Properties();
        props.put("metadata.broker.list", "broker1:9092, broker2:9092");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
        props.put("partitioner.class", "example.producer.SimplePartitioner");
        props.put("request.required.acks", "1");
        ProducerConfig config = new ProducerConfig(props);
        Producer<String, String> producer = new Producer<String, String>(config);
        for (long nEvents = 0; nEvents < events; nEvents++) {
            long runtime = new Date().getTime();
            String ip = "." + 192 + "." + rnd.nextInt(255);
            String msg = runtime + ", www.example.com," + ip;
            KeyedMessage<String, String> data = new KeyedMessage<String, String>("page_visits", ip, msg);
            producer.send(data);
        }
        producer.close();
    }
}

// Consumer example
import kafka.consumer.ConsumerConfig;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class ConsumerGroupExample {
    private final ConsumerConnector consumer;
    private final String topic;
    private ExecutorService executor;

    public ConsumerGroupExample(String a_zookeeper, String a_groupId, String a_topic) {
        consumer = kafka.consumer.Consumer.createJavaConsumerConnector(createConsumerConfig(a_zookeeper, a_groupId));
        this.topic = a_topic;
    }

    public void shutdown() {
        if (consumer != null) consumer.shutdown();
        if (executor != null) executor.shutdown();
        try {
            if (!executor.awaitTermination(5000, TimeUnit.MILLISECONDS)) {
                System.out.println("Timed out waiting for consumer threads to shut down, exiting uncleanly");
            }
        } catch (InterruptedException e) {
            System.out.println("Interrupted during shutdown, exiting uncleanly");
        }
    }

    public void run(int a_numThreads) {
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(topic, new Integer(a_numThreads));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
        // now launch all the threads
        executor = Executors.newFixedThreadPool(a_numThreads);
        // now create an object to consume the messages
        int threadNumber = 0;
        for (final KafkaStream stream : streams) {
            executor.submit(new ConsumerTest(stream, threadNumber));
            threadNumber++;
        }
    }

    private static ConsumerConfig createConsumerConfig(String a_zookeeper, String a_groupId) {
        Properties props = new Properties();
        props.put("zookeeper.connect", a_zookeeper);
        props.put("group.id", a_groupId);
        props.put("zookeeper.session.timeout.ms", "400");
        props.put("zookeeper.sync.time.ms", "200");
        props.put("auto.commit.interval.ms", "1000");
        return new ConsumerConfig(props);
    }

    public static void main(String[] args) {
        String zooKeeper = args[0];
        String groupId = args[1];
        String topic = args[2];
        int threads = Integer.parseInt(args[3]);
        ConsumerGroupExample example = new ConsumerGroupExample(zooKeeper, groupId, topic);
        example.run(threads);
        try {
            Thread.sleep(10000);
        } catch (InterruptedException ie) {
        }
        example.shutdown();
    }
}

API Documentation

Kafka provides extensive API documentation to help developers understand its usage and implementation.

Example Use Cases

Kafka provides several example use cases to demonstrate its usage in real-world scenarios.

FAQ

Kafka provides a FAQ section to address common questions and concerns about its usage and implementation.

Troubleshooting

Kafka provides troubleshooting guides to help developers resolve common issues and errors.

Release Notes

Kafka provides release notes for each new version, highlighting new features, improvements, and bug fixes.

Change Log

Kafka provides a change log to track changes made to its codebase and documentation.

Commit History

Kafka provides a commit history to track changes made to its codebase and documentation.

Code Review

Kafka provides code review guidelines to help developers review and improve its codebase.

Code Smells

Kafka provides code smells guidelines to help developers identify and improve its codebase.

Best Practices

Kafka provides best practices guidelines to help developers implement and maintain its usage.

Security

Kafka provides security guidelines to help developers implement and maintain its usage.

Scalability

Kafka provides scalability guidelines to help developers implement and maintain its usage.

Performance

Kafka provides performance guidelines to help developers implement and maintain its usage.

Reliability

Kafka provides reliability guidelines to help developers implement and maintain its usage.

Support

Kafka provides support guidelines to help developers implement and maintain its usage.

Community

Kafka provides community guidelines to help developers collaborate and contribute to its usage.

License

Kafka provides a license agreement to govern its usage and distribution.

Copyright

Kafka provides a copyright statement to protect its intellectual property.

Disclaimer

Kafka provides a disclaimer to limit its liability and responsibility.

Terms of Use

Kafka provides terms of use guidelines to govern its usage and distribution.

Privacy Policy

Kafka provides a privacy policy to protect its users’ personal information.

Contact

Kafka provides contact information to help developers reach out and collaborate with its community.

Contribute

Kafka provides contribution guidelines to help developers contribute to its codebase and documentation.

Report an Issue

Kafka provides a reporting mechanism to help developers report bugs and issues with its usage.

Submit a Pull Request

Kafka provides a pull request mechanism to help developers submit changes to its codebase and documentation.

Code Style

Kafka provides code style guidelines to help developers implement and maintain its codebase.

Code Conventions

Kafka provides code conventions guidelines to help developers implement and maintain its codebase.

Code Standards

Kafka provides code standards guidelines to help developers implement and maintain its codebase.

Code Quality

Kafka provides code quality guidelines to help developers implement and maintain its codebase.

Code Security

Kafka provides code security guidelines to help developers implement and maintain its codebase.

Code Scalability

Kafka provides code scalability guidelines to help developers implement and maintain its codebase.

Code Performance

Kafka provides code performance guidelines to help developers implement and maintain its codebase.

Code Reliability

Kafka provides code reliability guidelines to help developers implement and maintain its codebase.

Code Support

Kafka provides code support guidelines to help developers implement and maintain its codebase.

Code Community

Kafka provides code community guidelines to help developers collaborate and contribute to its codebase.

Code License

Kafka provides a code license agreement to govern its usage and distribution.

Code Copyright

Kafka provides a code copyright statement to protect its intellectual property.

Code Disclaimer

Kafka provides a code disclaimer to limit its liability and responsibility.

Code Terms of Use

Kafka provides code terms of use guidelines to govern its usage and distribution.

Code Privacy Policy

Kafka provides a code privacy policy to protect its users’ personal information.

Code Contact

Kafka provides code contact information to help developers reach out and collaborate with its community.

Code Contribute

Kafka provides code contribution guidelines to help developers contribute to its codebase and documentation.

Code Report an Issue

Kafka provides a code reporting mechanism to help developers report bugs and issues with its usage.

Code Submit a Pull Request

Kafka provides a code pull request mechanism to help developers submit changes to its codebase and documentation.