mrvitej.in

Archive · DevOps

Self-Hosted Kafka 4: A Three-Node KRaft Cluster

Apache Kafka 4.0 has no ZooKeeper at all. Build a three-node cluster with combined controller and broker roles.

1 min read

  • Apache Kafka 4.0
  • KRaft
  • OpenJDK 17
  • Ubuntu 24.04 LTS

Kafka 4.0 (March 2025) removed ZooKeeper completely, so every cluster uses KRaft. It requires Java 17 for brokers.

Three small nodes can each act as both controller and broker:

# config/server.properties on node 1 (change node.id and advertised listener per node)
process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka1:9093,2@kafka2:9093,3@kafka3:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://kafka1:9092
controller.listener.names=CONTROLLER
log.dirs=/var/lib/kafka/data
num.partitions=3
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3

Format storage once with the same cluster ID everywhere, then start:

CID=$(bin/kafka-storage.sh random-uuid)          # run once, reuse the value
bin/kafka-storage.sh format -t $CID -c config/server.properties
bin/kafka-server-start.sh -daemon config/server.properties
bin/kafka-metadata-quorum.sh --bootstrap-server kafka1:9092 describe --status

Run it as a systemd service under a kafka user, give it dedicated disks, and open only ports 9092 and 9093 between the nodes. Kafka 4.1 followed in September 2025, so use the newest patch of your chosen line.