mrvitej.in

DevOps · CNCF daily

Chaos Mesh: Cloud-Native Chaos Engineering for Kubernetes

CNCF Incubating. A Kubernetes-native platform that injects faults into pods, networks, file systems, kernels and cloud APIs through custom resources, so teams can test resilience before an outage does.

7 min read

CNCF Incubating

  • Chaos Mesh
  • Kubernetes
  • Prometheus
  • Grafana
  • Argo CD
CNCF project page

Chaos Mesh is a chaos engineering platform that runs on Kubernetes. Each fault is declared as a custom resource, such as a network delay, a pod kill, a file system error or a stressed CPU, and Chaos Mesh injects it into selected workloads for a defined time. It addresses a common gap in reliability work: retries, timeouts, failover and autoscaling are usually designed on paper and exercised for the first time in production. Controlled experiments let a team confirm that these mechanisms work and find the ones that do not. Chaos Mesh grew out of the internal chaos platform that PingCAP built to test TiDB, and was open sourced on December 31, 2019. It was accepted into the CNCF Sandbox on July 28, 2020 and moved to Incubating in February 2022. The 2.8 release line is current, with 2.8.4 published in August 2026.

CNCF daily · Sprint 1, day 7 · Chaos Engineering

At a glance
Category Chaos Engineering
CNCF status Incubating. Sandbox accepted July 2020; moved to Incubating in February 2022
Written in Go, with a React web dashboard
License Apache 2.0
Fault coverage Pods, network, I/O and file system, kernel, stress, time, JVM and HTTP, plus cloud API and physical machine faults
Orchestration Workflows for multi-step scenarios and Schedules for recurring experiments
Interfaces Kubernetes custom resources, Chaos Dashboard and the kubectl CLI
Release line 2.8 (2.8.4, August 2026); 2.7 is the previous line

Architecture

flowchart TB
  USER[Engineer or pipeline] -->|kubectl or dashboard| API[Kubernetes API server]
  DASH[Chaos Dashboard<br/>UI, API, RBAC] --> API
  API --> CM
  subgraph CP[Chaos Mesh control plane]
    CM[Controller Manager<br/>fault, workflow and schedule controllers]
    WH[Admission webhooks<br/>validate and mutate]
  end
  CM -->|instructs| D1[Chaos Daemon<br/>node A privileged DaemonSet]
  CM -->|instructs| D2[Chaos Daemon<br/>node B privileged DaemonSet]
  D1 -->|enters namespaces| P1[Target pods<br/>network, I/O, CPU, kernel faults]
  D2 -->|enters namespaces| P2[Target pods]
  CM -->|cloud faults| CLOUD[Cloud APIs<br/>AWS, Azure, GCP]
  P1 --> OBS[Prometheus and Grafana<br/>steady state signals]
  P2 --> OBS
Component Responsibility
Controller Manager The control plane. It reconciles fault resources, runs the Workflow and Schedule controllers, and hosts the admission webhooks.
Chaos Daemon A privileged DaemonSet on every node. It enters the namespaces of target pods to inject and remove faults. Its privileges can be restricted.
Chaos Dashboard A web interface and API for designing, running and watching experiments, with role-based access control.
Fault resources One custom resource per fault family, for example PodChaos, NetworkChaos, IOChaos, StressChaos and HTTPChaos.
Workflow Chains several experiments, with serial and parallel steps and status checks, into one scenario.
Schedule Runs an experiment on a recurring cron schedule.

Design principle. Chaos Mesh treats a fault as declarative Kubernetes configuration. An experiment is a reviewed manifest in Git with a selector, a duration and a mode, so it can be applied, scheduled, audited and removed like any other resource, and a delete or timeout reverts the fault.

Production reference design

A common pattern on EKS, GKE, AKS or on-premises Kubernetes: install it in a dedicated namespace, run experiments first in staging, and enable them in production only for narrow, approved cases.

  1. Install with Helm, pinning a version and setting the container runtime that your nodes use: 
    helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
  -n chaos-mesh --create-namespace \
  --version <version> \
  --set chaosDaemon.runtime=containerd \
  --set chaosDaemon.socketPath=/run/containerd/containerd.sock
    Check the runtime and socket path against your node image, and the chart values against the chart version you install.
  2. Limit where chaos can run. Enable authorisation and namespace filtering so faults only reach namespaces that are annotated for chaos, and give engineers RBAC for specific namespaces rather than cluster-wide access.
  3. Define the steady state first. Pick the signals that show the service is healthy, such as request success rate and latency, and put them on a Grafana dashboard before injecting anything.
  4. Write the first experiment as a manifest, with a small blast radius: 
    apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata: { name: orders-latency, namespace: chaos-testing }
spec:
  action: delay
  mode: one
  selector:
    namespaces: [orders]
    labelSelectors: { app: orders-api }
  delay: { latency: 200ms, jitter: 50ms }
  duration: 5m
  5. Compose scenarios. Use a Workflow to combine faults, for example a network delay followed by a pod kill, with a status check that aborts the run if an error-rate threshold is crossed.
  6. Schedule and automate. Use a Schedule for recurring checks in staging, and trigger experiments from CI after a deployment. Keep manifests in Git and apply them with Argo CD or Flux.
  7. Close the loop. Record each experiment’s hypothesis, result and fix, and rerun it after the fix.

Typical production uses include validating retries and timeouts, rehearsing node and zone failures, testing autoscaling and failover of databases and message brokers, and running regular game days.

Operational considerations

  • The daemon is privileged. It needs access to the container runtime and node namespaces. Restrict who can create chaos resources, enable namespace filtering, and review its security context against your cluster policy.
  • Match the runtime. The daemon talks to containerd, CRI-O or Docker through a socket, so a wrong runtime or path in the install is the most common cause of failed injections.
  • Control the blast radius. Start with one pod, a short duration and a clear abort condition. Use mode, selectors and duration to bound each run, and know how to delete the resource to stop it.
  • Avoid unreviewed production runs. Run in production only with an agreed hypothesis, a named owner, an on-call engineer watching and an off switch.
  • Measure, do not guess. An experiment without steady state metrics only shows that something broke. Wire experiments to the same dashboards and alerts that production uses.
  • Keep versions aligned. Upgrade the CRDs with the chart, read the release notes for each minor release, and avoid running mixed versions of the controller and daemon.

Adoption

  • Coverage of the move to Incubating named DataStax, Percona and Prudential among more than 50 organisations using Chaos Mesh in production at that time, and reported that Digital China combines it with its DevOps platform.
  • Microsoft Azure Chaos Studio integrates Chaos Mesh to run faults on Kubernetes workloads.
  • PingCAP created the project from its work testing TiDB and continues to contribute, and the project keeps a public adopters list in its repository.

Alternatives

Solution Model Best suited to
Litmus CNCF Incubating, Kubernetes chaos with an experiment hub and workflows Teams that want a shared library of ready-made experiments
Chaosblade CNCF Sandbox, fault injection for hosts, containers and applications Mixed estates that include non-Kubernetes hosts
AWS Fault Injection Service and Azure Chaos Studio Managed fault injection in a single cloud Teams that want cloud-level faults with provider IAM and audit
Gremlin Commercial chaos platform Organisations that want a supported product and reporting
Chaos Toolkit and chaoskube Scriptable experiments and random pod termination Simple experiments driven from CI or scripts

Strengths and limitations

Strengths Limitations
Declarative, Git-friendly experiments through Kubernetes custom resources Kubernetes-focused, with less coverage for non-containerised hosts
Broad fault coverage: pods, network, I/O, kernel, stress, time and cloud APIs The privileged daemon widens the attack surface and needs governance
Workflows, schedules and a dashboard cover one-off and recurring use Needs good steady state metrics, or results are hard to interpret
Incubating CNCF project with an active release cadence Runtime and kernel specifics can make injection fail on unusual nodes
Works with Prometheus, Grafana and GitOps tooling A fault can outlast an experiment if cleanup fails, so abort paths matter

Recommendation

Adopt Chaos Mesh for Kubernetes teams that want declarative, repeatable fault injection in staging and approved production game days. Start with a small blast radius, define steady state metrics first, restrict the namespaces and people allowed to inject faults, and keep experiments in Git. Assess Litmus alongside it if a shared experiment library matters more than fault breadth, and a managed cloud fault service if most risk sits in cloud infrastructure.

References: CNCF project page · Chaos Mesh joins CNCF as a Sandbox project · PingCAP: Announcing Chaos Mesh as a CNCF Sandbox project · FOSS Force: CNCF moves Chaos Mesh to its incubator · Chaos Mesh documentation · Chaos Mesh on GitHub

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.