Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps · CNCF daily

Litmus: Chaos Engineering Workflows with Probes and Resilience Scoring

CNCF Incubating. A Kubernetes chaos engineering platform that packages fault injection as reusable experiments, orchestrates them as workflows, and measures the outcome with probes and resilience scores.

6 min read

CNCF Incubating

  • Litmus
  • Kubernetes
  • Argo Workflows
  • Prometheus
  • MongoDB
CNCF project page

Litmus is an open-source chaos engineering platform for Kubernetes. It provides a library of ready-made fault experiments, such as deleting pods, adding network latency or exhausting CPU, a portal for composing them into scheduled workflows, and probes that check whether the system stayed healthy while the fault was active. The result is a pass or fail verdict and a resilience score per experiment, rather than only a fault injected and left for the engineer to interpret. It addresses the gap between assuming a system is resilient and demonstrating it under controlled failure. Litmus was started by MayaData, the company behind OpenEBS, and was accepted into the CNCF Sandbox on June 25, 2020. It moved to Incubating on January 11, 2022.

CNCF daily · Sprint 2, day 8 · Chaos Engineering

At a glance

Category Observability and Analysis › Chaos Engineering
CNCF status Incubating. Accepted June 25, 2020; incubating January 11, 2022
Written in Go (control plane and operators); TypeScript and React (portal)
License Apache 2.0
Core resources ChaosExperiment, ChaosEngine and ChaosResult custom resources
Experiment catalog ChaosHub, with Kubernetes, AWS, Azure, GCP, VMware and application-level faults
Validation Probes (HTTP, command, Kubernetes and Prometheus) evaluated during the experiment

Architecture

flowchart TB
  USER[Engineers and CI<br/>portal or API] ==> CC[ChaosCenter<br/>portal and server]
  HUB[ChaosHub<br/>experiment catalog] -.->|experiment definitions| CC
  CC ==>|stores workflows and results| DB[MongoDB]
  CC ==>|workflow request| SUB[Subscriber<br/>in target cluster]
  SUB ==> WF[Workflow controller<br/>Argo Workflows]
  WF ==> OP[Chaos operator]
  OP ==>|reads ChaosEngine| RUN[Chaos runner<br/>and experiment pod]
  RUN ==>|injects fault| APP[Application under test]
  RUN -.->|probes check health| APP
  RUN ==>|ChaosResult| OP
  OBS[Prometheus and<br/>Grafana] -.->|steady state metrics| RUN
  classDef actor fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  classDef ctrl fill:#ede7f6,stroke:#5e35b1,stroke-width:1.5px,color:#1d1c1a
  class USER actor
  class CC,SUB,OP,RUN core
  class APP,DB data
  class HUB,WF,OBS ctrl

Key: teal blocks are Litmus components, blue are the workload and the database, purple are supporting systems, grey are people. Thick arrows show the execution path, dashed arrows catalog, verification and monitoring.

Component Responsibility
ChaosCenter Web portal and API server for creating workflows, scheduling runs, managing projects and users, and viewing results and resilience scores.
ChaosHub Catalog of experiment definitions that teams can use directly or extend.
Subscriber and workflow controller The in-cluster agent that receives workflows from ChaosCenter and runs them, using Argo Workflows to sequence experiments.
Chaos operator and runner Watch ChaosEngine resources, start the experiment pod that applies the fault, and write a ChaosResult.
Probes Checks that run before, during and after a fault and decide whether the experiment passes.

Design principle. A chaos experiment is only useful if its hypothesis is explicit. Litmus makes the steady-state check, the probe, part of the experiment definition, so each run answers a specific question and produces a recorded verdict that CI and dashboards can act on.

Production reference design

A typical setup runs ChaosCenter in one management cluster and agents in each target cluster, starting in non-production.

  1. Install ChaosCenter. Use the project Helm chart:
    helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
    helm install chaos litmuschaos/litmus --namespace litmus --create-namespace
  2. Connect target clusters. Register each cluster as an infrastructure environment in the portal and apply the generated agent manifest, which installs the subscriber and operator.
  3. Define an experiment. A ChaosEngine selects a target and an experiment, here deleting pods from a labelled deployment:
    apiVersion: litmuschaos.io/v1alpha1
    kind: ChaosEngine
    metadata:
      name: checkout-pod-delete
      namespace: shop
    spec:
      appinfo:
        appns: shop
        applabel: app=checkout
        appkind: deployment
      chaosServiceAccount: pod-delete-sa
      experiments:
        - name: pod-delete
          spec:
            components:
              env:
                - name: TOTAL_CHAOS_DURATION
                  value: "60"
                - name: PODS_AFFECTED_PERC
                  value: "50"
  4. Add probes. Attach an HTTP probe against the service health endpoint or a Prometheus probe on the error rate, so the experiment fails if user-facing behaviour degrades.
  5. Automate. Schedule workflows, trigger them from CI after deployment, and track resilience scores over time.

Typical uses include validating autoscaling and failover, testing dependency outages, running game days, and gating releases on resilience checks.

Operational considerations

  • Blast radius. Limit experiments by namespace, labels and percentage affected, and use dedicated service accounts with minimal RBAC for each experiment.
  • Start outside production. Run in staging first, define steady state clearly, and have an abort path before any production use.
  • Control plane footprint. ChaosCenter needs MongoDB and a persistent volume. Back them up and secure the portal with authentication and TLS.
  • Probe quality. A weak probe produces false passes. Base probes on user-facing signals such as latency and error rate, not only on pod status.
  • Version alignment. Keep the agent, operator and experiment images at compatible versions, and review the release notes on upgrade, as the 3.x architecture differs from 1.x.

Adoption

  • MayaData (now part of DataCore) created Litmus, and ChaosNative, founded by its maintainers, built a commercial offering on it.
  • Harness acquired ChaosNative in 2022 and built Harness Chaos Engineering on Litmus.
  • The project keeps an ADOPTERS file in its repository listing organisations that use Litmus; teams should consult it for current names.

Alternatives

Solution Model Best suited to
Chaos Mesh (CNCF Incubating) Kubernetes-native fault injection through custom resources Rich fault types and a dashboard; see the Chaos Mesh brief
Gremlin Commercial chaos engineering platform Managed service, support and safety controls
AWS Fault Injection Service Cloud-native fault injection AWS workloads that need provider-level faults
Chaos Toolkit Open-source, scripted experiments Custom, code-driven experiments across platforms
Argo Workflows with scripts General workflow engine Teams building their own chaos pipeline; see the Argo brief

Strengths and limitations

Strengths Limitations
Probes and resilience scores make experiments verifiable More components to operate than a single-operator tool
Large catalog of experiments across Kubernetes and clouds ChaosCenter depends on MongoDB and Argo Workflows
Workflows, scheduling and CI integration Learning curve for CRDs, probes and RBAC
Open source with an active CNCF community Experiment quality varies across the catalog
Commercial support available through Harness Version differences between 1.x, 2.x and 3.x need care

Recommendation

Trial. Litmus suits teams that want chaos experiments to be repeatable, verified and part of delivery pipelines, not occasional manual tests. Begin in staging with one or two experiments, such as pod delete and network latency, define probes on user-facing metrics, and expand once results are trusted. Teams that want the simplest fault injection may compare it with Chaos Mesh.

References: CNCF project page · Litmus website · Litmus repository · CNCF Landscape

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.