Self-Hosted Kafka: Security, Monitoring and Day-Two Operations

A cluster on a private network still needs authentication and encryption.

Listeners with TLS and SCRAM users:

listeners=SASL_SSL://:9092,CONTROLLER://:9093
advertised.listeners=SASL_SSL://kafka1:9092
sasl.enabled.mechanisms=SCRAM-SHA-512
security.inter.broker.protocol=SASL_SSL
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
ssl.keystore.location=/etc/kafka/tls/kafka1.p12
ssl.truststore.location=/etc/kafka/tls/ca.p12
authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer

bin/kafka-configs.sh --bootstrap-server kafka1:9092 --alter --add-config \
  'SCRAM-SHA-512=[password=change-me]' --entity-type users --entity-name orders-svc
bin/kafka-acls.sh --bootstrap-server kafka1:9092 --add --allow-principal User:orders-svc \
  --operation Read --operation Write --topic orders

Monitoring. Attach the Prometheus JMX exporter to each broker and alert on: under-replicated partitions above zero, offline partitions above zero, controller count not equal to one, consumer lag growing, and disk above 75 percent.

Operations. Upgrade by rolling one node at a time and checking under-replicated partitions between nodes. Reassign partitions with kafka-reassign-partitions.sh when adding brokers, and set retention by bytes as well as time so disks cannot fill.