mrvitej.in

Archive · DevOps

Kafka: Consumer Groups, Replication and Retention

How consumers share work, how data survives a broker failure, and the settings that matter.

1 min read

  • Apache Kafka 3.7
  • Consumer groups
  • Replication

Consumer groups: consumers with the same group.id split a topic’s partitions between them. Three partitions can feed at most three consumers in one group; extra consumers sit idle. Different groups each receive all messages.

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing

The LAG column shows how far behind a group is; alert on lag that keeps growing.

Replication: with three brokers, create topics with replication factor 3:

bin/kafka-topics.sh --create --topic payments --partitions 6 \
  --replication-factor 3 --config min.insync.replicas=2 --bootstrap-server broker1:9092

Pair min.insync.replicas=2 with the producer setting acks=all: a write succeeds only when two replicas have it, so losing one broker loses no acknowledged data.

Retention:

bin/kafka-configs.sh --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=259200000 --bootstrap-server localhost:9092

Use cleanup.policy=compact to keep only the latest value per key. Make consumers idempotent, because delivery can repeat after a rebalance.