Apache Kafka

Apache Kafka is a distributed event streaming platform. Jay Kreps, Neha Narkhede, and Jun Rao built it at LinkedIn in 2011 to handle the company's need for low-latency ingestion of billions of events per day. It graduated as an Apache top-level project in 2012 and is now used by the majority of Fortune 100 companies for real-time data pipelines and event-driven architectures.

The current release is Kafka 4.2. Messages are organized into topics, which are split into partitions for parallel processing. Consumer groups distribute partition reads across multiple consumers. Starting with Kafka 4.0, the cluster metadata layer uses KRaft (Kafka's built-in Raft consensus) exclusively, removing the previous ZooKeeper dependency. Kafka Streams is a client library for building stream processing applications, and Kafka Connect provides a framework for integrating Kafka with external databases and services.

The official documentation covers broker configuration, client APIs, and operations. The source code is on GitHub under the Apache 2.0 license.