Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps · CNCF daily

Antrea: Kubernetes Networking and Network Policy on Open vSwitch

CNCF Sandbox. A Kubernetes-native CNI that uses Open vSwitch as its data plane to provide pod networking, tiered network policy, egress control and flow visibility.

7 min read

CNCF Sandbox

  • Antrea
  • Open vSwitch
  • Kubernetes
  • CNI
  • NetworkPolicy
CNCF project page

Antrea is a Kubernetes networking solution that implements the Container Network Interface (CNI) and Kubernetes NetworkPolicy using Open vSwitch (OVS) as the data plane. It operates at layers 3 and 4, gives every pod an address and connectivity across nodes, and adds security and visibility features beyond the standard NetworkPolicy API. It addresses the need for a CNI that is programmable, well suited to mixed Linux and Windows clusters, and familiar to teams that already operate virtual switches. Antrea was created at VMware (now Broadcom) and open-sourced in 2019. It joined the CNCF as a sandbox project on 28 April 2021, and the CNCF Landscape still lists it at the sandbox level.

CNCF daily · Sprint 2, day 5 · Cloud Native Network

At a glance

Category Runtime › Cloud Native Network
CNCF status Sandbox. Accepted April 28, 2021
Written in Go for the control and agent components; Open vSwitch (C) provides the data plane
License Apache 2.0
Data plane Open vSwitch, with Geneve, VXLAN or GRE tunnels, or no encapsulation on a routed network
Platforms Linux and Windows nodes; tested with Kubernetes 1.23 and later
Current release 2.7 (v2.7.0 is the latest tagged release at the time of writing)

Architecture

flowchart TB
  KAPI[Kubernetes API server] ==>|watches Pods Namespaces and policies| CTRL[Antrea Controller<br/>computes policies]
  CTRL ==>|policy per node over gRPC| AG
  subgraph NODE[Every worker node]
    CNI[kubelet and antrea-cni] ==>|CNI add and delete| AG[Antrea Agent]
    AG ==>|programs flows| OVS[Open vSwitch<br/>bridge and flow tables]
    OVS ==> POD[Pod interfaces]
  end
  OVS ==>|Geneve or VXLAN tunnel| PEER[Open vSwitch on other nodes]
  AG -.->|flow records| FA[Flow Aggregator]
  FA -.->|export| OBS[Flow collector or Theia]
  classDef actor fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef ctrl fill:#ede7f6,stroke:#5e35b1,stroke-width:1.5px,color:#1d1c1a
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  classDef store fill:#fff3e0,stroke:#e65100,stroke-width:1.5px,color:#1d1c1a
  classDef sec fill:#fce4ec,stroke:#ad1457,stroke-width:1.5px,color:#1d1c1a
  classDef obs fill:#e8f5e9,stroke:#2e7d32,stroke-width:1.5px,color:#1d1c1a
  class KAPI ctrl
  class CTRL,AG,OVS,CNI core
  class POD,PEER data
  class FA,OBS obs
  style NODE fill:#0f766e14,stroke:#0f766e,stroke-width:2px

Key: teal blocks are the project itself, blue are workloads and proxies, purple are control and configuration, amber are storage, red are security and identity, green are observability, grey are external actors. Thick arrows show the primary path, thin solid arrows other data flows, dashed arrows control, discovery and telemetry.

Component Responsibility
Antrea Controller A Deployment that watches Pods, Namespaces and policy resources, computes the effective policies, and sends each node only the policies that apply to its pods. It keeps its state in memory and needs no datastore.
Antrea Agent A DaemonSet on every node that sets up pod interfaces, allocates addresses, creates tunnels to other nodes and installs OVS flows.
Open vSwitch The ovs-vswitchd and ovsdb-server daemons that forward packets and enforce policy in the kernel-based data path.
antrea-cni The CNI plugin binary that kubelet calls; it passes each request to the local Agent.
Flow Exporter and Flow Aggregator Export connection records from each node and aggregate them with Kubernetes metadata for analysis, for example in Theia.
antctl Command-line tool for runtime information, troubleshooting and packet tracing.

Design principle. Antrea uses Kubernetes machinery wherever possible: its Controller extends the Kubernetes API server library, authenticates through ServiceAccounts, and exposes its policies as custom resources. The data plane is a general-purpose programmable switch, so new features (egress, multicast, encryption) are added as flow rules rather than new kernel code.

Production reference design

A common deployment is a kubeadm, VMware or on-premises cluster, or a managed cluster where the provider’s default CNI is replaced or complemented.

  1. Check prerequisites. The Open vSwitch kernel module must be available on every node, and the cluster must allocate pod CIDRs per node (for kubeadm, set --pod-network-cidr).
  2. Install a pinned release with the manifest or the Helm chart:
    helm repo add antrea https://charts.antrea.io && helm repo update
    helm install antrea antrea/antrea -n kube-system --version 2.7.0
    kubectl -n kube-system rollout status ds/antrea-agent
  3. Enable features deliberately in the antrea-config ConfigMap. Examples are AntreaProxy to handle Services without kube-proxy, traffic encryption with IPsec or WireGuard between nodes, Egress and FlowExporter.
  4. Apply cluster-wide guardrails with tiered policy. Antrea’s ClusterNetworkPolicy evaluates in ordered tiers ahead of namespace-level NetworkPolicy:
    apiVersion: crd.antrea.io/v1beta1
    kind: ClusterNetworkPolicy
    metadata: { name: block-dev-to-data }
    spec:
      priority: 5
      tier: securityops
      appliedTo:
        - namespaceSelector: { matchLabels: { tier: data } }
      ingress:
        - action: Drop
          from:
            - namespaceSelector: { matchLabels: { env: dev } }
  5. Give selected workloads a stable source address with an Egress resource that points at an ExternalIPPool, so firewalls outside the cluster can allow-list traffic from specific applications:
    apiVersion: crd.antrea.io/v1beta1
    kind: Egress
    metadata: { name: egress-payments }
    spec:
      appliedTo:
        podSelector: { matchLabels: { app: payments } }
      externalIPPool: egress-pool

Typical uses include multi-tenant clusters that need platform-owned security rules, clusters with Windows worker nodes, on-premises clusters where applications need predictable source IPs for external firewalls, and teams that want connection-level flow records without a service mesh.

Operational considerations

  • Controller availability. The Controller runs as a single replica. Agents keep enforcing the last policies they received if it is down, but new or changed policies wait until it returns, so monitor it and let Kubernetes reschedule it quickly.
  • Kernel and node requirements. OVS needs its kernel module on every node image. Check this in your OS baseline and test node upgrades, because a kernel change can break the module.
  • Encapsulation and MTU. Tunnels reduce the usable MTU, and encryption reduces it further. Set the MTU explicitly, or use no-encapsulation mode on a routed network where the underlay allows it.
  • Policy ordering. Tiers and priorities decide which rule wins. Document the tier model (for example, security operations above application teams) and test changes with the Antrea policy tools before applying them cluster-wide.
  • Migration risk. Changing the CNI on a running cluster is disruptive. Plan it as a new cluster or a node-by-node migration with a rollback path, and do not mix two CNIs on one node.

Adoption

  • The project’s ADOPTERS.md lists Glasnostic, which uses Antrea’s Open vSwitch support to shape how services interact; Transwarp, which applies Antrea policies to protect big-data software for each tenant of its Kubernetes platform and uses the Egress feature to preserve source addresses; and TeraSky, which uses Antrea network policies and Egress in its internal clusters and for customers.
  • Antrea originated at VMware, and VMware’s Kubernetes offerings have used it as a default CNI, which gives it a large installed base in enterprise environments.
  • The published adopter list is short compared with larger CNI projects. Teams should check recent project activity and the support options of their platform vendor before choosing it.

Alternatives

Solution Model Best suited to
Cilium (CNCF Graduated) eBPF data plane with network policy, service mesh and observability The default choice for most new clusters; see the Cilium brief
Calico Routed networking with iptables or eBPF data planes and rich policy Teams wanting BGP-based, non-overlay networking and a large ecosystem
Kube-OVN (CNCF Sandbox) Open Virtual Network based, with VPC-style subnets Multi-tenant clusters that want per-tenant subnets and networks
Flannel Simple overlay with no policy engine Small clusters and labs that need basic connectivity only
Cloud VPC CNIs (AWS VPC CNI, Azure CNI) Pods receive addresses from the cloud network Managed clusters that need native VPC addressing and routing

Strengths and limitations

Strengths Limitations
Tiered, cluster-scoped policy gives platform teams guardrails above namespace rules Single-replica Controller; policy changes wait while it is down
Mature Windows node support alongside Linux Requires the OVS kernel module on every node
Egress, multicast, encryption and flow export built in Smaller community and adopter base than Cilium or Calico
Kubernetes-native design with in-memory state and no extra datastore A sandbox project, so governance and long-term support depend on its maintainers
Apache 2.0 with regular releases and good documentation Layer 3 and 4 focus; no built-in layer 7 policy or service mesh

Recommendation

Assess. Antrea is a capable CNI with strong policy features and clear value in clusters that mix Linux and Windows nodes, need tiered security rules owned by a platform team, or need predictable egress addresses. For most new general-purpose clusters, Cilium has the larger ecosystem and is the safer default. Evaluate Antrea in a test cluster against your policy and observability needs before committing production workloads.

References: CNCF project page · Antrea documentation · Antrea architecture · Antrea adopters · CNCF Landscape

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.