mrvitej.in
Happy Dussehra! May good always win. Happy Diwali! Wishing you a bright year ahead. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps · CNCF daily

Submariner: Direct Pod and Service Networking Between Kubernetes Clusters

CNCF Sandbox. Connects the pod and service networks of separate Kubernetes clusters through encrypted gateway tunnels, with cross-cluster service discovery, independent of the CNI.

7 min read

CNCF Sandbox

  • Submariner
  • Calico
  • Lighthouse
  • WireGuard
  • Kubernetes
CNCF project page

Submariner connects the networks of separate Kubernetes clusters so that a pod in one cluster can reach pods and Services in another by IP address, whether the clusters run on-premises, in different clouds or at the edge. It addresses a gap in multi-cluster Kubernetes: each cluster has its own pod and service network that is invisible to the others, and workarounds such as public ingress for every internal service add cost, latency and attack surface. Submariner adds encrypted tunnels between clusters, routing on each node and, through its Lighthouse component, DNS-based discovery of services exported from other clusters. It works alongside the cluster’s existing CNI rather than replacing it. Submariner originated at Rancher Labs and was accepted into the CNCF as a Sandbox project on 28 April 2021.

CNCF daily · Sprint 3, post 12 of 21 · Cloud Native Network

At a glance
Category Runtime › Cloud Native Network
CNCF status Sandbox. Accepted April 28, 2021
Written in Go
License Apache 2.0
Components Broker, Gateway Engine, Route Agent, Lighthouse (service discovery), optional Globalnet
Tunnel drivers IPsec (Libreswan), WireGuard and VXLAN
Service discovery ServiceExport and ServiceImport, reachable at <service>.<namespace>.svc.clusterset.local
Ports UDP 4500 (tunnel), UDP 4490 (NAT discovery), UDP 4800 (VXLAN from nodes to the gateway)
CLI subctl for installing, joining, exporting and diagnosing

Architecture

Submariner architecture: two clusters each run a Lighthouse service discovery component, a Gateway Engine, Route Agents on every node and ordinary workloads; the gateways publish their endpoints and the Lighthouse components publish service exports to a Broker in one cluster; the two gateways build an encrypted tunnel; pod traffic goes from a pod through the Route Agent to the gateway, across the tunnel, and through the remote gateway and Route Agent to the remote pod

Purple is Submariner, slate the broker and grey the workloads. The broker only holds metadata: pod traffic never passes through it. Open the zoomable diagram, or download it as SVG or PNG.

Component Responsibility
Broker A set of custom resources in one cluster that clusters use to exchange gateway endpoints and service information. It does not carry traffic.
Lighthouse Publishes exported services to the broker, creates ServiceImport objects from other clusters and answers clusterset.local DNS queries.
Gateway Engine Runs on a gateway node in each cluster, finds the other gateways through the broker and maintains the encrypted tunnels.
Route Agent Runs on every node and routes traffic for remote clusters to the active gateway.
Pods and Services Ordinary workloads that need no changes to reach remote pod and service addresses.

Design principle. Use the existing CNI inside each cluster and add a thin layer between clusters. Only the gateways terminate tunnels and the broker carries only metadata, so the data path is short and the cluster networks stay independent.

Request flow

Submariner flow in seven steps: each gateway publishes its endpoint to the broker, a pod sends traffic for a remote pod address, the route agent forwards it to the local gateway, the gateway sends it through the encrypted tunnel to the remote gateway, the remote gateway hands it to the route agent on the node, and the packet reaches the remote pod

The service is assumed to be exported already: Lighthouse has published a ServiceExport and each cluster holds the matching ServiceImport, so a name such as shop.default.svc.clusterset.local resolves to an address that is reachable across the tunnel.

  1. Publish gateway info. The gateway in cluster A publishes its endpoint and subnets to the broker.
  2. Discover the peer. The gateway in cluster B does the same, and each gateway reads the other’s entry to build the tunnel.
  3. Send pod traffic. A pod sends a packet to a remote pod or service address.
  4. Route to the gateway. The Route Agent on the node forwards packets for the remote cluster’s range to the active gateway.
  5. Cross the tunnel. The gateway sends the traffic through the encrypted tunnel to the remote gateway.
  6. Deliver to the node. The remote gateway hands the packet to the Route Agent on the destination side.
  7. Reach the pod. The packet reaches the remote pod, and the reply returns along the same path.

Open the zoomable flow diagram, or download it as SVG or PNG.

Production reference design

This Sprint 3 design joins a cloud cluster on Google Compute Engine (self-managed Kubernetes with Calico, not GKE) and a bare-metal edge cluster.

  1. Plan the addresses. Give each cluster a unique pod CIDR and service CIDR, or enable Globalnet for overlapping ranges. Check that the CNI is on Submariner’s tested list; Calico is.
  2. Choose gateways. Label one node per cluster as a gateway (submariner.io/gateway=true). On GCP, give it a reachable address, or use the --load-balancer option so the gateway does not need a public IP.
  3. Open the firewall. (GCP) Create VPC firewall rules that allow UDP 4500 and 4490 between the gateways, UDP 4800 inside the cluster, and the pod and service ranges between sites. Do the same on the edge firewall.
  4. Deploy the broker and join clusters: 
    subctl deploy-broker --kubeconfig cloud.kubeconfig
subctl join broker-info.subm --kubeconfig cloud.kubeconfig --clusterid cloud --cable-driver wireguard
subctl join broker-info.subm --kubeconfig edge.kubeconfig --clusterid edge --cable-driver wireguard
    Use subctl cloud prepare generic for clusters that are not OpenShift installer-provisioned.
  5. Export a service and call it from the other cluster: 
    subctl export service --namespace shop catalog
    The service is then reachable as catalog.shop.svc.clusterset.local.
  6. Verify. Run subctl show connections and subctl verify between the two contexts, and subctl diagnose all when something fails.

Typical uses include edge and cloud clusters that share services, database replication between sites, disaster recovery, and gradual migration of workloads between clusters while they still talk to each other.

Operational considerations

  • Address planning. Overlapping CIDRs require Globalnet, which adds NAT and some limits. Planning unique ranges early is simpler.
  • CNI compatibility. Submariner lists the CNIs it is tested with. Using another CNI, or a managed cluster’s native CNI, may need extra setup and is outside the tested path.
  • Gateway availability. Run more than one gateway node per cluster for failover, and monitor tunnel status; a failed tunnel splits the clusters’ networks.
  • Firewalls and NAT. The main cause of failures is blocked UDP ports or asymmetric routing. Use subctl diagnose firewall to check before debugging further.
  • Security. Prefer WireGuard or IPsec, restrict who can read the broker information, and use network policies, because pod addresses become reachable between clusters.

Adoption

  • Red Hat is a major contributor and packages Submariner for its OpenShift multi-cluster management.
  • SUSE and Rancher originated the project and contributed to its early development.
  • The repository keeps a list of other users and contributors; consult it for current organisations. Public adopter lists are shorter than for larger networking projects such as Cilium.

Alternatives

Solution Model Best suited to
Cilium Cluster Mesh (CNCF Graduated) CNI-integrated multi-cluster networking Estates that already use Cilium; see the Cilium brief
Istio multi-cluster (CNCF Graduated) Service mesh across clusters L7 policy and mTLS between services, not pod-to-pod IP routing
Skupper Application-layer connections between sites Connecting a few services without changing the network or firewall
Karmada (CNCF Graduated) Multi-cluster workload placement Deciding where workloads run, with Submariner as the network; see the Karmada brief
k8gb (CNCF Incubating) DNS-based global load balancing Sending external users to a healthy cluster; see the k8gb brief

Strengths and limitations

Strengths Limitations
Works with several CNIs, with no CNI replacement Only tested CNIs are supported; others need care
Encrypted tunnels with IPsec or WireGuard Needs unique CIDRs, or Globalnet with its limits
Cross-cluster service discovery through clusterset.local Gateway nodes and UDP ports must be reachable between sites
Broker holds metadata only and stays off the data path Sandbox project with a smaller community
subctl simplifies install, verify and diagnose Managed Kubernetes services can need extra setup

Recommendation

Trial for teams that need direct pod and service connectivity between a small number of clusters and cannot adopt a single CNI everywhere. Start with two clusters, unique CIDRs, WireGuard and one exported service, and use subctl verify and diagnose before relying on it. Estates that already run Cilium should compare Cluster Mesh first, and teams that need only a few service-to-service links may prefer an application-layer tool.

References: CNCF project page · Submariner documentation · Submariner repository · CNCF Landscape

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.