Skip to main content
Kafka beginner Lesson 2 of 10

Kafka Topics, Partitions, and Offsets

Watch a topic split across partitions, see how offsets are assigned per partition, and learn why Kafka only guarantees ordering inside a single partition.

A topic is not one log. It is a set of independent logs called partitions, and almost every surprising Kafka behaviour traces back to that one fact. This lesson makes the partitions visible.

Creating a topic with three partitions

bin/kafka-topics.sh --create \
  --topic events \
  --partitions 3 \
  --replication-factor 1 \
  --bootstrap-server localhost:9092

bin/kafka-topics.sh --describe --topic events --bootstrap-server localhost:9092
Created topic events.
Topic: events	TopicId: mK4pQ8vXRt2yLs9nZw3BdA	PartitionCount: 3	ReplicationFactor: 1	Configs:
	Topic: events	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: events	Partition: 1	Leader: 1	Replicas: 1	Isr: 1
	Topic: events	Partition: 2	Leader: 1	Replicas: 1	Isr: 1

Three separate logs, numbered 0, 1 and 2. Each will get its own offset sequence starting at 0.

Watching records land in different partitions

The console consumer can print the partition and offset of every record, which turns the abstract structure into something you can read. First produce six messages:

for i in 1 2 3 4 5 6; do echo "event-$i"; done | \
  bin/kafka-console-producer.sh --topic events --bootstrap-server localhost:9092

Now consume with formatting turned on:

bin/kafka-console-consumer.sh \
  --topic events \
  --from-beginning \
  --property print.partition=true \
  --property print.offset=true \
  --property print.value=true \
  --bootstrap-server localhost:9092
Partition:0	Offset:0	event-1
Partition:0	Offset:1	event-4
Partition:1	Offset:0	event-2
Partition:1	Offset:1	event-5
Partition:2	Offset:0	event-3
Partition:2	Offset:1	event-6

Two things to notice, and both matter.

The output is not in production order. You produced event-1 through event-6 in sequence; you got them back grouped by partition. The consumer fetches from each partition independently and hands you whatever has arrived — there is no global sort.

Each partition has its own offset sequence. Offset 0 appears three times, once per partition. An offset is only meaningful as the pair (partition, offset).

Within a partition, though, order is exact: partition 0 gave you event-1 then event-4, which is the order they were written there. That guarantee never breaks.

Why the messages spread out

These records had no key. When a producer sends a keyless record, the default partitioner assigns it using a sticky round-robin: it fills one batch for a partition, then switches. With one message per batch here, the effect is plain round-robin across 0, 1, 2.

Give the records a key and the behaviour changes completely:

printf 'alice:login\nbob:login\nalice:logout\nbob:logout\nalice:purchase\n' | \
  bin/kafka-console-producer.sh \
    --topic events \
    --property "parse.key=true" \
    --property "key.separator=:" \
    --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh \
  --topic events --from-beginning \
  --property print.partition=true \
  --property print.offset=true \
  --property print.key=true \
  --bootstrap-server localhost:9092
Partition:0	Offset:0	null	event-1
Partition:0	Offset:1	null	event-4
Partition:1	Offset:0	null	event-2
Partition:1	Offset:1	null	event-5
Partition:1	Offset:2	bob	login
Partition:1	Offset:3	bob	logout
Partition:2	Offset:0	null	event-3
Partition:2	Offset:1	null	event-6
Partition:2	Offset:2	alice	login
Partition:2	Offset:3	alice	logout
Partition:2	Offset:4	alice	purchase

Every alice record went to partition 2. Every bob record went to partition 1. That is not luck — the partitioner computes murmur2(key) % partition_count, so the same key always resolves to the same partition.

And that is the mechanism behind the only ordering guarantee Kafka offers you in practice: all records with the same key land in the same partition, so they are strictly ordered relative to each other. Alice’s login, logout, purchase arrive in that order, always, no matter how busy the cluster is.

Offsets are positions, not IDs

Ask for the end offset of each partition:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic events
events:0:2
events:1:4
events:2:5

Partition 2 holds five records (offsets 0-4) and will assign offset 5 next. Partition 0 holds two. The partitions are at different lengths because the key hashing sent more records to some than others — this is normal, and one reason a hot key can create a lopsided topic.

You can also start reading from a specific position rather than the beginning or end:

bin/kafka-console-consumer.sh \
  --topic events --partition 2 --offset 2 \
  --property print.key=true \
  --bootstrap-server localhost:9092
alice	login
alice	logout
alice	purchase

Skipping the two keyless records that occupy offsets 0 and 1 of that partition. Being able to point a consumer at an arbitrary offset is what makes reprocessing possible — you are seeking within a retained log, not re-requesting a delivery.

Practice

1. Produce ten keyless messages to events and check the end offsets. Are the partitions even? 
for i in $(seq 1 10); do echo "m-$i"; done | \
  bin/kafka-console-producer.sh --topic events --bootstrap-server localhost:9092
bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
  --broker-list localhost:9092 --topic events
events:0:12
events:1:7
events:2:8

Roughly even but not exact. The sticky partitioner batches records per partition before switching, so short bursts can land unevenly. Over a large volume it evens out.

2. Send three records keyed order-42. Which partition do they land in, and why can you predict it? 
printf 'order-42:created\norder-42:paid\norder-42:shipped\n' | \
  bin/kafka-console-producer.sh --topic events \
    --property parse.key=true --property key.separator=: \
    --bootstrap-server localhost:9092
Partition:0	Offset:12	order-42	created
Partition:0	Offset:13	order-42	paid
Partition:0	Offset:14	order-42	shipped

All three in one partition, in order. The partition is murmur2("order-42") % 3, which is deterministic — the same key on the same partition count always resolves the same way.

3. Increase events to 5 partitions, then produce order-42 again. Did it stay in the same partition? 
bin/kafka-topics.sh --alter --topic events --partitions 5 \
  --bootstrap-server localhost:9092
printf 'order-42:delivered\n' | bin/kafka-console-producer.sh --topic events \
  --property parse.key=true --property key.separator=: \
  --bootstrap-server localhost:9092
Partition:3	Offset:0	order-42	delivered

It moved. The partitioner divides by the partition count, so changing that count remaps every key. order-42’s history is now split across two partitions with no ordering between them — which is exactly why adding partitions to a keyed topic is a breaking change, not a routine scale-up.

4. Read only partition 1, starting at offset 2, and stop after 2 messages. 
bin/kafka-console-consumer.sh --topic events \
  --partition 1 --offset 2 --max-messages 2 \
  --property print.offset=true \
  --bootstrap-server localhost:9092
Offset:2	login
Offset:3	logout
Processed a total of 2 messages

--max-messages makes the consumer exit instead of waiting, which is useful in scripts.

Next: producing from application code rather than the console, and what the producer actually does with your record before it reaches the broker.

Frequently Asked Questions

Does Kafka guarantee message ordering?
Only within a single partition. Records in one partition are strictly ordered by offset and always delivered in that order. Across partitions there is no ordering guarantee at all, because partitions are independent logs that may be read at different speeds.
How many partitions should a topic have?
Enough to support your peak consumer parallelism, since one partition can be read by at most one consumer in a group. Start with a small number you can reason about and increase it later. Note that increasing partitions changes key-to-partition mapping, which breaks per-key ordering for existing keys.
Can I reduce the number of partitions on a topic?
No. Kafka only supports increasing partition count. Reducing it would mean deleting logs that hold data and would break offset continuity, so the operation is not offered. To shrink, create a new topic and migrate.
Are offsets unique across a topic?
No. Offsets are per-partition. A topic with three partitions has three independent offset sequences that all start at 0, so 'offset 5' is only meaningful when paired with a partition number.