Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps · CNCF daily

Kuma: A Universal Envoy Service Mesh for Kubernetes and VMs, in One or Many Zones

CNCF Sandbox. An Envoy-based service mesh with mutual TLS, traffic policies and observability that runs on Kubernetes and virtual machines, and connects multiple zones through a global control plane.

6 min read

CNCF Sandbox

  • Kuma
  • Envoy
  • Kubernetes
  • mTLS
  • Kong Mesh
CNCF project page

Kuma is a service mesh built on Envoy. A control plane configures an Envoy proxy beside each service, and those proxies encrypt traffic with mutual TLS, enforce which services may call which, and apply retries, timeouts, circuit breaking and traffic splitting, while reporting metrics, traces and access logs. Its distinguishing features are that it runs on both Kubernetes and virtual machines and that it has a first-class multi-zone mode, in which a global control plane coordinates separate zones such as a cloud cluster and edge sites. It addresses the need for one consistent networking and security layer across mixed and distributed infrastructure. Kuma was created by Kong, donated to the CNCF, and accepted as a Sandbox project on June 25, 2020.

CNCF daily · Sprint 3, post 6 of 21 · Service Mesh

At a glance

Category Orchestration & Management › Service Mesh
CNCF status Sandbox. Accepted June 25, 2020
Written in Go (control plane); Envoy as the data plane
License Apache 2.0
Modes Kubernetes (custom resources) and Universal (VMs and bare metal, with a PostgreSQL store or in-memory)
Deployment topologies Single zone, or multi-zone with a global control plane and a control plane per zone
Policies MeshTrafficPermission, MeshTLS, MeshTimeout, MeshRetry, MeshCircuitBreaker, MeshLoadBalancingStrategy, MeshFaultInjection, MeshAccessLog, MeshMetric, MeshTrace
Gateways A built-in gateway (MeshGateway), and support for delegated gateways such as Kong
Commercial distribution Kong Mesh, from Kong

Architecture

flowchart TB
  OPS[Operator<br/>kubectl or kumactl] ==>|policies and Mesh| GCP[Global control plane<br/>policy source of truth]
  GCP ==>|sync policies<br/>KDS over gRPC| ZCA[Zone control plane<br/>cloud zone]
  GCP ==>|sync policies| ZCB[Zone control plane<br/>edge zone]
  ZCA ==>|xDS config and certificates| DPA[Envoy sidecars<br/>zone A services]
  ZCB ==>|xDS config and certificates| DPB[Envoy sidecars<br/>zone B services]
  DPA ==>|mTLS through zone ingress| ZIB[Zone ingress<br/>zone B]
  ZIB ==>|mTLS to service| DPB
  DPA -.->|metrics and traces| OBS[Prometheus and tracing backend]
  DPB -.->|metrics and traces| OBS
  ZCA -.->|status and resources| GCP
  ZCB -.->|status and resources| GCP
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  classDef actor fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
  class GCP,ZCA,ZCB,ZIB core
  class DPA,DPB,OBS data
  class OPS actor

Key: teal blocks are Kuma control plane components, blue are Envoy proxies and observability tools, grey is the operator. Thick arrows show configuration and traffic, dashed arrows show telemetry and status.

Component Responsibility
kuma-cp (control plane) Holds policies, builds Envoy configuration, acts as the certificate authority for mTLS and serves it over xDS.
kuma-dp (data plane) Starts and configures Envoy next to each service; on Kubernetes it is injected as a sidecar.
Global control plane In multi-zone mode, the single place where policies are written and synchronised to every zone.
Zone control plane Manages the data planes in one zone and keeps running if the global control plane is unavailable.
Zone ingress and egress Gateways that carry cross-zone traffic and, optionally, outbound traffic, so zones need not expose every pod.

Design principle. One policy model for every environment. Services are identified by labels and tags rather than by where they run, so the same permission or timeout policy can apply to a pod in the cloud and a virtual machine at an edge site.

Production reference design

A typical multi-zone design uses one global control plane and a zone control plane for each cluster or data centre.

  1. Install the global control plane in a management cluster:
    helm repo add kuma https://kumahq.github.io/charts
    helm install kuma kuma/kuma --namespace kuma-system --create-namespace \
      --set controlPlane.mode=global
  2. Install a zone control plane in each workload cluster, pointing at the global control plane and enabling the ingress:
    helm install kuma kuma/kuma --namespace kuma-system --create-namespace \
      --set controlPlane.mode=zone \
      --set controlPlane.zone=edge-vizag \
      --set controlPlane.kdsGlobalAddress=grpcs://<global-address>:5685 \
      --set ingress.enabled=true
  3. Enable sidecar injection for application namespaces:
    kubectl label namespace shop kuma.io/sidecar-injection=enabled
  4. Turn on mutual TLS with a mesh resource, created on the global control plane:
    apiVersion: kuma.io/v1alpha1
    kind: Mesh
    metadata: { name: default }
    spec:
      mtls:
        enabledBackend: ca-1
        backends:
          - name: ca-1
            type: builtin
  5. Allow only intended calls. After enabling mTLS, add explicit permissions:
    apiVersion: kuma.io/v1alpha1
    kind: MeshTrafficPermission
    metadata:
      name: orders-to-inventory
      namespace: kuma-system
      labels: { kuma.io/mesh: default }
    spec:
      targetRef:
        kind: MeshSubset
        tags: { kuma.io/service: inventory_shop_svc_8080 }
      from:
        - targetRef:
            kind: MeshSubset
            tags: { kuma.io/service: order_shop_svc_8080 }
          default: { action: Allow }
    Service tag formats and targetRef kinds differ between Kuma releases, so check the documentation for yours.
  6. Wire observability. Enable the Prometheus metrics and tracing policies and send data to your existing backends.

Typical uses include zero-trust service communication, mixing Kubernetes and VM workloads in one mesh, connecting cloud and edge zones, and gradual mesh adoption.

Operational considerations

  • Global control plane availability. Zones keep serving traffic if the global control plane is down, but policy changes stop. Run it highly available with a database for state.
  • Certificate authority. The built-in CA is simple, but production setups often use an external CA or secret store. Plan rotation and test it before it is needed.
  • Resource cost. Every service gets an Envoy proxy. Measure memory and CPU per sidecar, and tune for small edge nodes.
  • Policy model changes. Kuma moved from older policies to the targetRef policies. Use the newer policies for new work and plan migrations.
  • Cross-zone latency and failure. Traffic across zones passes through ingress gateways over wide-area links. Use locality-aware load balancing so requests stay local when healthy.

Adoption

  • Kong created Kuma and builds Kong Mesh and its connectivity platform on it.
  • The project maintains an ADOPTERS file in its repository naming organisations that use Kuma; consult it for current, first-party names.
  • As a Sandbox project, Kuma has a smaller community than Istio or Linkerd, so confirm that its features and support options fit before standardising on it.

Alternatives

Solution Model Best suited to
Linkerd (CNCF Graduated) Lightweight Rust-proxy mesh for Kubernetes Simple, low-overhead mesh on Kubernetes only; see the Linkerd brief
Istio (CNCF Graduated) Feature-rich Envoy mesh, sidecar or ambient Large Kubernetes estates needing the broadest feature set; see the Istio brief
Cilium service mesh eBPF-based networking and mesh Teams already using Cilium who want fewer sidecars
Consul Service discovery and mesh HashiCorp-centred environments
No mesh, mTLS in the application Libraries and frameworks Small systems with few services

Strengths and limitations

Strengths Limitations
Runs on Kubernetes and on VMs with one policy model Sandbox project with a smaller community
Multi-zone mode with global and zone control planes Operating global and zone control planes adds complexity
Envoy data plane with a wide feature base Sidecars use resources on small edge nodes
Built-in gateway and ingress for cross-zone traffic Policy model changed between releases
Commercial support available through Kong Mesh Fewer integrations and tutorials than Istio

Recommendation

Assess, and Trial where you must connect Kubernetes and VM workloads or several zones in one mesh. Start with a single zone and mTLS in permissive mode, move to explicit traffic permissions, then add a second zone to test cross-zone failover and latency. If you only need a mesh inside one Kubernetes cluster, Linkerd or Istio will usually be easier to operate.

References: CNCF project page · Kuma website · Kuma repository · CNCF Landscape

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.