mrvitej.in

Archive · DevOps

Self-Hosted MongoDB: Backups, Monitoring and Upgrades

Backups you can actually restore, the metrics and alerts that matter, and a safe rolling upgrade procedure for a MongoDB 8.0 replica set.

6 min read

  • MongoDB 8.0
  • mongodump
  • Percona mongodb_exporter
  • Prometheus
  • Grafana
  • Amazon S3

Running a replica set is the easy part. Keeping it recoverable and visible is the work. This post covers three things for the replica set built in the previous article: backups, monitoring and upgrades.

Part 1: Backups

Replication is not a backup. A dropped collection or a bad migration replicates to every member within seconds. You need copies that live elsewhere and can be restored to a point before the mistake.

Decide RPO and RTO first

Term Question Drives
RPO (recovery point objective) How much data can we lose? Backup frequency, oplog capture
RTO (recovery time objective) How long can we be down? Backup method, restore speed

Choose a method

Method Good for Trade-off
mongodump --oplog Up to a few tens of GB Slow to restore large data; logical copy
Filesystem or LVM snapshot of a secondary Large data sets, fast restore Needs LVM or cloud volumes; whole-node copy
Percona Backup for MongoDB (PBM) Scheduled backups plus point-in-time recovery Another agent to run on each member 
flowchart LR
  S[Hidden or normal secondary] --> D[mongodump with oplog<br/>or volume snapshot]
  D --> L[Local staging disk]
  L --> O[Object storage<br/>versioned, lifecycle rules]
  O --> R[Monthly restore drill<br/>on a test instance]

A logical backup script

/usr/local/bin/mongo-backup.sh on a host that can reach the replica set:

#!/usr/bin/env bash
set -euo pipefail

STAMP=$(date +%F-%H%M)
DEST=/backup/mongo/$STAMP
URI="mongodb://backup:${MONGO_BACKUP_PASS}@mongo1,mongo2,mongo3/?replicaSet=rs0&authSource=admin&tls=true&tlsCAFile=/etc/ssl/mongo-ca.pem&readPreference=secondary"

mkdir -p "$DEST"
mongodump --uri "$URI" --oplog --gzip --out "$DEST"

# copy off the host, then keep only 3 local copies
aws s3 sync "$DEST" "s3://acme-db-backups/mongo/$STAMP/" --only-show-errors
ls -1dt /backup/mongo/* | tail -n +4 | xargs -r rm -rf

echo "backup $STAMP ok"
  • --oplog captures writes that happen during the dump, so the restore is consistent to one moment.
  • readPreference=secondary keeps the load off the primary.
  • The password comes from the environment (a systemd EnvironmentFile readable only by root), not from the script.

Schedule it with a systemd timer, which logs to the journal and runs missed jobs after downtime:

# /etc/systemd/system/mongo-backup.service
[Unit]
Description=MongoDB logical backup

[Service]
Type=oneshot
EnvironmentFile=/etc/mongo-backup.env
ExecStart=/usr/local/bin/mongo-backup.sh

# /etc/systemd/system/mongo-backup.timer
[Unit]
Description=Nightly MongoDB backup

[Timer]
OnCalendar=*-*-* 01:30:00
Persistent=true

[Install]
WantedBy=timers.target
sudo systemctl daemon-reload && sudo systemctl enable --now mongo-backup.timer
systemctl list-timers mongo-backup.timer

On the bucket, turn on versioning, block public access, and add a lifecycle rule (for example move to an infrequent-access class after 30 days and expire after 180).

Snapshot backups for large data

For hundreds of GB, mongodump is too slow to restore. Snapshot a secondary instead. With journaling on the same volume, an LVM or cloud-volume snapshot is consistent on its own; locking is extra safety if data and journal are on separate volumes:

mongosh "$URI_SECONDARY" --eval 'db.fsyncLock()'
sudo lvcreate --size 50G --snapshot --name mongo-snap /dev/vg_data/lv_mongo
mongosh "$URI_SECONDARY" --eval 'db.fsyncUnlock()'

sudo mount -o ro,nouuid /dev/vg_data/mongo-snap /mnt/mongo-snap
sudo tar -C /mnt/mongo-snap -czf - . | aws s3 cp - s3://acme-db-backups/snap/$(date +%F).tgz
sudo umount /mnt/mongo-snap && sudo lvremove -y /dev/vg_data/mongo-snap

Restore drills

A backup is proven only by a restore. Once a month:

# start a throwaway mongod
docker run -d --name restore-test -p 37017:27017 mongo:8.0
aws s3 sync s3://acme-db-backups/mongo/2025-11-16-0130/ ./restore/
mongorestore --uri "mongodb://localhost:37017" --oplogReplay --gzip ./restore
mongosh --port 37017 --eval 'db.getSiblingDB("shop").orders.countDocuments()'

Record how long it took. That number is your real RTO, not the one in the plan.

If you need point-in-time recovery (restore to 10:42, just before the bad deploy), use Percona Backup for MongoDB, which saves oplog slices continuously, or keep your own oplog archive. Plain nightly dumps only give you last night.

Part 2: Monitoring

Exporter and scrape

Run Percona’s mongodb_exporter beside each member, using the exporter user created earlier:

# docker-compose.yml on each MongoDB host
services:
  mongodb-exporter:
    image: percona/mongodb_exporter:0.44
    restart: unless-stopped
    command:
      - --mongodb.uri=mongodb://exporter:${EXPORTER_PASS}@localhost:27017/?authSource=admin&tls=true&tlsCAFile=/certs/ca.pem&directConnection=true
      - --collect-all
      - --compatible-mode
    volumes: ["/etc/mongodb/tls/ca.pem:/certs/ca.pem:ro"]
    ports: ["9216:9216"]
# prometheus.yml
scrape_configs:
  - job_name: mongodb
    static_configs:
      - targets: ["mongo1:9216", "mongo2:9216", "mongo3:9216"]
  - job_name: node
    static_configs:
      - targets: ["mongo1:9100", "mongo2:9100", "mongo3:9100"]

Pair it with node_exporter for disk, CPU and memory on the hosts.

What to alert on

Signal Why Example threshold
Exporter or member down Nothing else matters if it is down up == 0 for 2 min
No primary Writes are failing No member in PRIMARY state for 1 min
Replication lag Secondaries falling behind; failover would lose writes Above 30 s for 5 min
Oplog window Secondary may need full resync after an outage Below 24 h
Connections Pool exhaustion Above 80% of the limit
WiredTiger cache dirty % Write pressure, stalls Above 20% for 10 min
Disk usage MongoDB stops when the disk fills Above 75%
Slow operations Missing indexes after a release Sustained rise

Alert rules (metric names depend on the exporter version and mode, so confirm them on http://mongo1:9216/metrics first):

groups:
  - name: mongodb
    rules:
      - alert: MongoDBDown
        expr: mongodb_up == 0
        for: 2m
        labels: { severity: critical }
        annotations: { summary: "MongoDB on {{ $labels.instance }} is down" }

      - alert: MongoDBReplicationLag
        expr: mongodb_mongod_replset_member_replication_lag > 30
        for: 5m
        labels: { severity: warning }
        annotations: { summary: "Replication lag above 30s on {{ $labels.name }}" }

      - alert: MongoDBDiskFilling
        expr: (1 - node_filesystem_avail_bytes{mountpoint="/data/mongodb"} / node_filesystem_size_bytes{mountpoint="/data/mongodb"}) > 0.75
        for: 10m
        labels: { severity: warning }

Import a MongoDB dashboard into Grafana for the replica-set overview, and add a panel for the oplog window, which most dashboards miss.

Find slow queries

With slowOpThresholdMs: 100 set in the config, slow operations are written to the log and the profiler:

db.getSiblingDB("shop").system.profile.find().sort({ millis: -1 }).limit(5)
db.getSiblingDB("shop").orders.find({ status: "PENDING" }).explain("executionStats")

A COLLSCAN stage with a large totalDocsExamined means a missing index.

Part 3: Upgrades

Patch releases (8.0.x to 8.0.y)

Rolling, one member at a time, primary last:

# on a secondary
sudo apt-mark unhold mongodb-org mongodb-org-server mongodb-org-mongos mongodb-org-tools
sudo apt install -y mongodb-org=8.0.Y mongodb-org-server=8.0.Y mongodb-org-mongos=8.0.Y mongodb-org-tools=8.0.Y
sudo apt-mark hold mongodb-org mongodb-org-server mongodb-org-mongos mongodb-org-tools
sudo systemctl restart mongod

Wait until rs.status() shows the member as SECONDARY and caught up before moving to the next. For the primary, run rs.stepDown() first, then upgrade it as a secondary.

Major releases (for example 7.0 to 8.0)

flowchart LR
  A[Check FCV is the old version] --> B[Upgrade secondaries one by one]
  B --> C[Step down primary and upgrade it]
  C --> D[Run on new binaries<br/>for days, watch metrics]
  D --> E[Raise featureCompatibilityVersion]

db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })   // before
// ... rolling binary upgrade as above ...
db.adminCommand({ setFeatureCompatibilityVersion: "8.0", confirm: true }) // after soak time

Rules that prevent bad days:

  • Never skip a major version; go 6.0 to 7.0 to 8.0.
  • Keep the old feature compatibility version until you are sure, because downgrading after raising it is much harder.
  • Take a fresh backup and check driver compatibility before starting.
  • Read the release notes for removed parameters; a removed option in mongod.conf stops the server from starting.

Summary

Area Minimum standard
Backups Nightly, off-host, versioned, restored monthly
Monitoring Member up, primary present, lag, oplog window, disk
Upgrades Rolling, secondaries first, FCV raised last

With these in place, a self-hosted replica set is something you can run calmly for years.