mrvitej.in

Archive · DevOps

Self-Hosted MongoDB 8.0: A Production Replica Set

Design, prepare and build a three-member MongoDB 8.0 replica set on Ubuntu 24.04, with TLS, keyfile authentication, users and a failover test.

7 min read

  • MongoDB 8.0
  • Ubuntu 24.04 LTS
  • mongosh 2.x
  • XFS
  • OpenSSL

Managed MongoDB (Atlas, DocumentDB) is the easy route, but there are good reasons to run it yourself: data has to stay on servers you control, the workload is steady enough that managed pricing is hard to justify, or you need an on-premises copy of a cloud stack. This post builds a replica set that is fit for production, not just a lab.

MongoDB 8.0 (October 2024) is the release line to deploy. The 8.1 and 8.2 releases are short-lived rapid releases, so self-hosted production stays on the 8.0 line and its patch releases.

The target architecture

flowchart LR
  subgraph APP[Application tier]
    A1[API pods or VMs]
  end
  subgraph RS[Replica set rs0]
    P[mongo1<br/>primary]
    S1[mongo2<br/>secondary]
    S2[mongo3<br/>secondary]
  end
  A1 -- writes and reads --> P
  A1 -. secondary reads optional .-> S1
  P -- oplog replication --> S1
  P -- oplog replication --> S2
  S2 --> B[Backup job<br/>reads from a secondary]
  B --> O[Object storage]
  • Three data-bearing members. Any one can fail and the remaining two still form a majority, elect a primary and accept majority writes. Avoid arbiters: a primary plus a secondary plus an arbiter cannot satisfy majority writes when the secondary is down.
  • Separate failure domains. Put the three members on different physical hosts, racks or availability zones. Three VMs on one hypervisor is one failure domain.
  • Backups read from a secondary, so they do not compete with the primary for I/O.

Sizing

Resource Guidance
RAM Enough for the working set (frequently used data and indexes). WiredTiger’s cache defaults to 50 percent of (RAM minus 1 GB).
CPU 4 vCPUs minimum for production; WiredTiger uses one ticket per concurrent operation.
Disk SSD or NVMe, dedicated to data, XFS. Size for data plus indexes plus 30 to 40 percent headroom for compaction and growth.
Network Low latency between members; replication lag grows with round-trip time.

1. Prepare each host

Do this on all three servers. Hostnames must resolve consistently, because the replica set configuration stores them.

# names that every member and every client can resolve (DNS is better than hosts files)
echo "10.10.1.11 mongo1
10.10.1.12 mongo2
10.10.1.13 mongo3" | sudo tee -a /etc/hosts

# time sync: elections and logs depend on it
sudo timedatectl set-ntp true

# dedicated XFS data volume, mounted with noatime
sudo mkfs.xfs /dev/nvme1n1
sudo mkdir -p /data/mongodb
echo "/dev/nvme1n1 /data/mongodb xfs defaults,noatime 0 0" | sudo tee -a /etc/fstab
sudo mount -a

Kernel and limits:

cat <<'EOF' | sudo tee /etc/sysctl.d/90-mongodb.conf
vm.swappiness = 1
vm.max_map_count = 262144
net.core.somaxconn = 4096
net.ipv4.tcp_keepalive_time = 120
EOF
sudo sysctl --system

Transparent Huge Pages changed in 8.0. Older MongoDB versions asked you to disable THP. MongoDB 8.0 ships an updated TCMalloc that performs better with THP enabled, and its production notes also recommend disabling glibc’s restartable sequences for it. Check the production notes for your exact patch release, then apply:

# /etc/systemd/system/mongodb-thp.service
[Unit]
Description=THP settings for MongoDB 8.0
Before=mongod.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo always > /sys/kernel/mm/transparent_hugepage/enabled'
ExecStart=/bin/sh -c 'echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag'
ExecStart=/bin/sh -c 'echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none'

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload && sudo systemctl enable --now mongodb-thp

2. Install MongoDB 8.0

curl -fsSL https://www.mongodb.org/static/pgp/server-8.0.asc | \
  sudo gpg --dearmor -o /usr/share/keyrings/mongodb-server-8.0.gpg
echo "deb [arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse" | \
  sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list
sudo apt update && sudo apt install -y mongodb-org
sudo chown -R mongodb:mongodb /data/mongodb

Pin the packages so a routine apt upgrade cannot move you to a new release unplanned:

sudo apt-mark hold mongodb-org mongodb-org-server mongodb-org-mongos mongodb-org-tools

Raise limits and set the glibc tunable through a systemd drop-in:

# sudo systemctl edit mongod
[Service]
LimitNOFILE=64000
LimitNPROC=64000
Environment="GLIBC_TUNABLES=glibc.pthread.rseq=0"

3. Keyfile and TLS

Members authenticate to each other. A keyfile is the simplest method; x.509 certificates are stronger and suit larger estates. Clients should always use TLS.

Create the keyfile once and copy it to all three hosts:

openssl rand -base64 756 | sudo tee /etc/mongodb/keyfile > /dev/null
sudo chown mongodb:mongodb /etc/mongodb/keyfile && sudo chmod 400 /etc/mongodb/keyfile

A private CA and one certificate per member (in production use your organisation’s CA):

# CA
openssl req -x509 -newkey rsa:4096 -nodes -days 1825 -subj "/CN=mongo-ca" \
  -keyout ca.key -out ca.pem

# member certificate with SANs for the hostname and IP
for n in 1 2 3; do
  openssl req -newkey rsa:2048 -nodes -subj "/CN=mongo$n" -keyout mongo$n.key -out mongo$n.csr
  printf "subjectAltName=DNS:mongo$n,IP:10.10.1.1$n\nextendedKeyUsage=serverAuth,clientAuth" > san$n.ext
  openssl x509 -req -in mongo$n.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
    -days 825 -extfile san$n.ext -out mongo$n.crt
  cat mongo$n.key mongo$n.crt > mongo$n.pem
done

Copy mongoN.pem and ca.pem to /etc/mongodb/tls/ on each host, owned by mongodb, mode 400. Keep ca.key offline.

4. Configure mongod

/etc/mongod.conf on mongo1 (change the certificate file name on the others):

storage:
  dbPath: /data/mongodb
  wiredTiger:
    engineConfig:
      cacheSizeGB: 6            # roughly 50% of (RAM - 1 GB); set explicitly on shared hosts

systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
  logRotate: reopen

net:
  port: 27017
  bindIp: 0.0.0.0              # the firewall restricts who can reach it
  tls:
    mode: requireTLS
    certificateKeyFile: /etc/mongodb/tls/mongo1.pem
    CAFile: /etc/mongodb/tls/ca.pem

security:
  authorization: enabled
  keyFile: /etc/mongodb/keyfile

replication:
  replSetName: rs0
  oplogSizeMB: 20480           # size for at least 24-72 hours of writes

operationProfiling:
  mode: slowOp
  slowOpThresholdMs: 100

The oplog window is how long a secondary can be offline and still catch up without a full resync. Measure it after a few days with rs.printReplicationInfo() and grow it if the window is short.

Firewall: only the other members and the application subnet may connect.

sudo ufw allow from 10.10.1.0/28 to any port 27017 proto tcp    # members
sudo ufw allow from 10.20.0.0/24 to any port 27017 proto tcp    # app subnet
sudo ufw enable
sudo systemctl enable --now mongod

5. Initiate the replica set

Connect on mongo1 through the localhost exception, which allows creating the first user while no users exist:

mongosh --tls --tlsCAFile /etc/mongodb/tls/ca.pem --host mongo1
rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "mongo1:27017", priority: 2 },
    { _id: 1, host: "mongo2:27017", priority: 1 },
    { _id: 2, host: "mongo3:27017", priority: 1 }
  ]
})

// wait for PRIMARY in the prompt, then create the first admin
db.getSiblingDB("admin").createUser({
  user: "admin",
  pwd: passwordPrompt(),
  roles: [ { role: "root", db: "admin" } ]
})

The higher priority makes mongo1 the preferred primary during normal operation, which keeps routing predictable.

Then create least-privilege users:

db.getSiblingDB("admin").auth("admin", passwordPrompt())

db.getSiblingDB("admin").createUser({
  user: "shop_app", pwd: passwordPrompt(),
  roles: [ { role: "readWrite", db: "shop" } ]
})

db.getSiblingDB("admin").createUser({
  user: "backup", pwd: passwordPrompt(),
  roles: [ "backup", "restore", "clusterMonitor" ]
})

db.getSiblingDB("admin").createUser({
  user: "exporter", pwd: passwordPrompt(),
  roles: [ "clusterMonitor", { role: "read", db: "local" } ]
})

6. Connect applications correctly

mongodb://shop_app:<password>@mongo1:27017,mongo2:27017,mongo3:27017/shop?replicaSet=rs0&tls=true&tlsCAFile=/etc/ssl/mongo-ca.pem&authSource=admin&w=majority&retryWrites=true
  • List all members, so the driver can find the new primary after a failover.
  • w=majority confirms a write only once a majority holds it, so a failover cannot roll it back.
  • retryWrites=true lets the driver retry once across an election.
  • Keep readPreference=primary (the default) unless a query can tolerate slightly stale data.

7. Verify and test failover

rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health }))
rs.printReplicationInfo()            // oplog window
rs.printSecondaryReplicationInfo()   // lag per secondary

Then rehearse a failover while an application writes in a loop:

rs.stepDown(60)     // on the primary: steps down for 60 seconds

A new primary should be elected within about 10 to 12 seconds, and the application should log, at most, a brief retry. If it errors out instead, fix the connection string before going live.

Production checklist

  • Three members on separate failure domains, no arbiter.
  • XFS with noatime, THP and glibc settings from the 8.0 production notes, limits raised.
  • requireTLS, keyfile or x.509 between members, authorisation on, least-privilege users.
  • Firewall restricted to members and application subnets.
  • Oplog window measured and at least a day long.
  • Packages held at a known patch version.
  • Failover rehearsed with the real application.

The next post covers what keeps it healthy afterwards: backups, monitoring and upgrades.