MySQL’s InnoDB engine repairs most crashes by itself: it replays the redo log when it starts. When the pod still loops, something prevents that replay, such as bad permissions, a full disk, a missing file or a damaged page. This post goes from the harmless fixes to the last resort, innodb_force_recovery, and shows how to get the data out.
Versions in use on this date. MySQL 8.4 LTS (8.4.5 was released in April 2025) on MicroK8s 1.32 with the hostpath-storage add-on. MySQL 8.0 reaches end of life in April 2026, so plan the move to 8.4. The steps below are the same on 8.0.
Reading the failure
microk8s kubectl get pods -n data
microk8s kubectl logs mysql-0 -n data --previous --tail=60
microk8s kubectl describe pod mysql-0 -n data | grep -E 'Reason|Exit Code|Message'flowchart TB
S[Pod restarting] --> L[Previous log]
L --> E{First ERROR line}
E -->|Operating system error 13| P[Permissions]
E -->|No space left| D[Disk full]
E -->|Cannot create redo log files| R[Redo log or disk]
E -->|Database page corruption| C[Corrupt page]
E -->|data directory has files in it| I[Wrong init on old volume]
E -->|Exit 137| O[Out of memory]
C --> F[innodb_force_recovery then dump]
Always find the first error in the log. Later lines are usually consequences of it.
| Message | Cause | Fix |
|---|---|---|
Operating system error number 13 |
Folder not owned by the mysql user | chown -R 999:999, set fsGroup: 999 |
No space left on device |
Volume or node disk full | Free space, then restart |
Cannot create redo log files |
No space or no permission in the data folder | Same as above |
--initialize specified but the data directory has files in it |
Init ran on a non-empty volume | Remove the init flag, or use a new empty volume |
Database page corruption on disk |
Damaged data page | Forced recovery and dump |
| Exit 137 | OOMKilled | Lower innodb_buffer_pool_size, raise the limit |
Step 1: stop MySQL and take a copy
microk8s kubectl scale statefulset mysql -n data --replicas=0
VOL=$(sudo ls -d /var/snap/microk8s/common/default-storage/*mysql-0*)
sudo df -h /var/snap/microk8s/common
sudo cp -a "$VOL" /root/mysql-backup-$(date +%F-%H%M)Make that copy before any forced-recovery attempt. If there is no room for a full copy, free space or copy to another machine with rsync. Repairs that go wrong cannot be undone without it.
Step 2: the cheap fixes
Permissions and ownership. The official image runs as user 999:
sudo chown -R 999:999 "$VOL"Disk. Free node space, then scale up again. InnoDB needs room for the redo log and temporary tables.
Memory. On a pod with a 2 GiB limit, a default buffer pool can push it over the limit. In the config map set innodb_buffer_pool_size=1G (about half the limit) and max_connections to something realistic.
If the pod now reaches ready for connections, you are done; go to “After recovery”.
Step 3: forced recovery to rescue the data
When InnoDB reports corruption and cannot start, start it in a mode that skips parts of the recovery so you can read the data and dump it. Add the setting with a config map or a temporary command argument:
microk8s kubectl patch statefulset mysql -n data --type=json -p \
'[{"op":"add","path":"/spec/template/spec/containers/0/args","value":["--innodb-force-recovery=1"]}]'
microk8s kubectl scale statefulset mysql -n data --replicas=1Start at 1 and raise it one level at a time only if the server still fails:
| Level | What it does | Risk |
|---|---|---|
| 1 | Continues past corrupt pages | Low |
| 2 | Stops the master background thread | Low |
| 3 | Skips rolling back transactions | Some uncommitted data may appear |
| 4 | Skips insert buffer merge | Possible permanent index damage |
| 5 | Ignores the undo logs | Uncommitted data stays |
| 6 | Skips redo log roll-forward | Data may be inconsistent |
At level 1 to 3 the data is usually fine. From 4 up the data may be inconsistent, so treat what you get with suspicion. Treat any non-zero level as read-only in practice: writes are restricted or unsafe, so you cannot repair the instance in place, only read from it and dump.
Step 4: dump everything
microk8s kubectl exec mysql-0 -n data -- sh -c 'mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" \
--all-databases --routines --triggers --events --single-transaction' > /root/rescue-$(date +%F).sqlIf one table is damaged, the dump stops at it. Skip it with --ignore-table=db.tablename, and dump the rest, then try to recover that table separately with SELECT ... INTO ranges by primary key. Check the file size and the last line (-- Dump completed).
Step 5: rebuild clean
Never keep running on a forced-recovery instance. Create a fresh volume, start MySQL normally without the flag, and import the dump:
microk8s kubectl delete pvc data-mysql-0 -n data # only after the dump is verified and copied away
microk8s kubectl scale statefulset mysql -n data --replicas=1
microk8s kubectl exec -i mysql-0 -n data -- sh -c 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < /root/rescue-2025-06-17.sqlThen compare row counts for the main tables with what the application or a replica shows, and run CHECK TABLE on them.
After recovery
- Take a logical backup every night (a CronJob running
mysqldumpto storage off the node) and test a restore. - Use
innodb_flush_log_at_trx_commit=1andsync_binlog=1if you cannot afford to lose a committed transaction. The defaults are safe, but some tuning guides tell you to turn them down. - Set requests and limits, a startup probe with a generous delay (crash recovery can take minutes), and a disk space alert.
- For anything that matters, keep a replica on another node. A single hostpath volume on one machine means one disk failure ends the story.