Kafka Explained: Partitions, Offsets & Consumer Groups

The most useful reframe about Kafka is that it isn't a message queue. A queue deletes a message once a consumer acknowledges it. Kafka is an append-only log: consuming changes nothing, and only the retention policy (time or size based) removes data. That's why several independent consumer groups can read the same topic at completely different positions without competing.

A topic is split into partitions, each an ordered sequence where every message gets an increasing offset. Ordering is guaranteed within a partition, never across a topic. Messages sharing a key hash to the same partition, which is how per-key ordering survives. Inside one consumer group each partition goes to exactly one consumer, so consumers beyond the partition count sit idle, and every group commits its own offsets.

Partition count is both your parallelism ceiling and your ordering boundary, and it can't be reduced later. Choose generously up front.