mrvitej.in
Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

Archive · Kubernetes

Redis Pod Recovery on MicroK8s: Corrupt AOF, Full Disk and Permissions

Recover a Redis pod that will not start on a MicroK8s cluster: read the failure, repair or restore the AOF and RDB files on the hostpath volume, and harden the deployment so it recovers by itself.

4 min read

  • MicroK8s 1.32
  • Redis 7.2
  • StatefulSet
  • hostpath-storage
  • redis-check-aof

Redis keeps its data in memory and writes it to disk through two mechanisms: RDB snapshots and the AOF (append-only file) log. When a Redis pod on MicroK8s will not start after a node crash, power loss or full disk, the cause is nearly always in one of those files or in the volume they sit on. This post shows how to find out which, and how to recover.

Versions in use on this date. MicroK8s 1.32 (released December 2024, current stable) with the hostpath-storage add-on. Redis 7.2.x is the last BSD-licensed line, so it is the one a self-hosted cluster should stay on (Redis 7.4 moved to a source-available licence in 2024; Valkey 8 is the open fork). Since Redis 7 the AOF is split into several files inside an appendonlydir folder with a manifest.

Where the data lives

flowchart TB
  P[Redis pod<br/>StatefulSet redis-0] ==>|mounts| PVC[PVC data-redis-0<br/>microk8s-hostpath]
  PVC ==> H[(Node folder<br/>/var/snap/microk8s/common/default-storage)]
  H --> R[dump.rdb snapshot]
  H --> A[appendonlydir<br/>manifest and incr files]
  P -.->|fails on load| L[Pod log<br/>kubectl logs --previous]

The hostpath provisioner stores each volume as a plain folder on the node. That makes recovery easy, because you can look at the files directly, and risky, because there is no replication: one node, one copy.

Step 1: see why it is failing

microk8s kubectl get pods -n data
microk8s kubectl describe pod redis-0 -n data | grep -A8 'Last State'
microk8s kubectl logs redis-0 -n data --previous --tail=50

Match the message:

Log message Cause
Bad file format reading the append only file or Unexpected end of file AOF truncated or corrupt after a crash
Can't open the append only file: Permission denied Folder owner differs from the container user
No space left on device Volume or node disk is full
MISCONF Redis is configured to save RDB snapshots Writes are failing, usually the disk
Exit 137, OOMKilled maxmemory or pod limit too high or missing

Step 2: stop the pod and work on the files

A crash-looping pod cannot be inspected, and Redis must not run while you repair files. Scale it down, then back the data up before touching anything:

microk8s kubectl scale statefulset redis -n data --replicas=0
sudo ls -la /var/snap/microk8s/common/default-storage/
# the folder is named like default-data-redis-0-pvc-<uid>
VOL=$(sudo ls -d /var/snap/microk8s/common/default-storage/*redis-0*)
sudo cp -a "$VOL" /root/redis-backup-$(date +%F-%H%M)

Always keep that copy. Repairs can discard data, and with a copy you can try again a different way.

Step 3a: repair a corrupt AOF

Run the checker from the same image, mounting the volume, or from a Redis install on the node:

cd "$VOL/appendonlydir"
sudo redis-check-aof --fix appendonly.aof.1.incr.aof

redis-check-aof shows the offset of the bad entry, and --fix truncates the file at the last good command. You lose only the writes after that point (usually the last second before the crash). If the manifest itself is damaged, run it on the manifest file: redis-check-aof --fix appendonly.aof.manifest. The RDB file can be checked the same way with redis-check-rdb dump.rdb.

Step 3b: fix permissions

The Redis image runs as user 999. If a folder was restored or copied as root, Redis cannot write to it:

sudo chown -R 999:999 "$VOL"

Better, make Kubernetes do it. In the pod spec set securityContext: { fsGroup: 999, runAsUser: 999 } so the mount is correct on every start.

Step 3c: disk full

df -h /var/snap/microk8s/common
sudo du -sh "$VOL"/*

Free space on the node first (old logs, unused images with microk8s ctr images prune). Redis needs room for a rewrite of the AOF, which temporarily needs extra space, so keep at least the size of the dataset free.

Step 4: start it and check

microk8s kubectl scale statefulset redis -n data --replicas=1
microk8s kubectl logs -f redis-0 -n data
microk8s kubectl exec redis-0 -n data -- redis-cli -a "$REDIS_PASSWORD" --no-auth-warning dbsize
microk8s kubectl exec redis-0 -n data -- redis-cli -a "$REDIS_PASSWORD" --no-auth-warning info persistence

Wait until the log shows Ready to accept connections. Large datasets show Loading for a while, and info persistence has loading:1 until done. Compare dbsize with what the application expects.

If nothing can be repaired

Restore the last good dump.rdb from a backup into the volume, delete the appendonlydir folder, and start Redis; it then builds a fresh AOF from the snapshot. Without a backup the data is gone, which is why the next section matters.

Make it recover by itself

  • Probes: a startupProbe with a long allowance so slow loading is not killed, then a livenessProbe using redis-cli ping.
  • Memory: set maxmemory to about 75% of the pod limit and a sensible maxmemory-policy.
  • Persistence: appendonly yes and appendfsync everysec limit loss to about a second.
  • Backups: a CronJob that copies dump.rdb (after BGSAVE) to object storage or another machine. A hostpath volume on one node is not a backup.
  • Replication: for anything important, run a replica with Sentinel (see the earlier Redis post) on a different node.