Apache Kafka Architecture: Distributed Commit Log and High-Throughput Storage
Kafka uses ordered per-partition logs, retention policies, and consumer offsets. Loss, global ordering, and exactly-once behavior depend on configuration and processing boundaries.
Publication & corrections: infostoa · Editorial standards
Individual author and specialist reviewer information is not provided for this article.
Kafka partition logs, offsets, retention, and consumer commits
An Apache Kafka topic is divided into partitions, and each partition is an ordered log. Ordering is defined within a partition; Kafka does not provide one global order across partitions. A key commonly influences partition placement so records with the same key can share a partition, but changing partition counts or partitioning logic can change placement assumptions. An offset identifies a position in a partition log. After retention or compaction, a record at a prior offset may no longer exist, so an offset is best understood as a consumer position, not a permanent identifier for a currently retained record.
Practitioner perspective: Kafka is a retained log, not a delete-on-read queue
Jay Kreps, Kafka’s initial developer and Confluent co-founder, argues that Kafka can be used for retained data logs and discusses replay, event sourcing, disk persistence, and replication. This is an architectural perspective from a project creator and vendor executive. It is not a guarantee that any deployment retains data forever. Actual retention depends on time- or size-based cleanup, compaction settings, replication, storage capacity, and operational limits. Compaction asynchronously preserves selected latest values by key; it is not an archive of every historical event. Tombstones can also be removed after the configured delete-retention period.
Durability depends on producer and broker configuration
Kafka does not promise that every record is processed without loss under all configurations and failures. Producer acknowledgements, retry behavior, replication factor, in-sync replica requirements, leader-election policy, and storage behavior determine the practical failure envelope. acks=0, leader acknowledgement, and acknowledgement from the configured in-sync replicas have different meanings. Document which event a successful produce response confirms. Consumer commits are a separate boundary: committing before processing finishes can lose work after a crash; committing after processing can lead to duplicate work. Neither pattern makes an arbitrary external side effect exactly-once by itself.
Boundaries of parallelism, ordering, and replay
Within a consumer group, partitions are normally assigned among group members, which gives partition-level parallelism. Adding more consumers than partitions does not create more assigned partitions for that group. To preserve per-partition processing order while using worker pools, applications must track completion and commit only a safe next position. Replay is possible only while the needed records remain available. A Kafka transaction can atomically coordinate Kafka input offsets and Kafka output records in supported processing patterns; it does not automatically include an external database or payment service. Plan retention, replica acknowledgements, consumer commit mode, lag and disk alerts, and tested replay procedures together.