Sprint 3 project: A Multi-Cluster Edge Platform on Google Cloud

Sprint 3 covered 21 categories and 21 tools, and one of them, OpenEBS, came with its backup companion K8up. This post connects them into one platform that can be built, operated and broken on purpose. The theme is the edge: a retail chain runs the same shop application in a cloud cluster and in a store’s back room, and the store keeps selling when the link to the cloud is down. A customer places an order, the order is priced by a Java service, an event goes to Kafka, and an inventory service reserves stock in TiKV. Karmada places the workloads, Submariner and KubeEdge connect the sites, k8gb and MetalLB decide where traffic lands, and Kuma, Keycloak and Pinniped decide who may talk to what. The business logic is deliberately trivial; the value is in how the tools meet.

The cloud side runs on Google Cloud in Mumbai (asia-south1) on Compute Engine VMs with self-managed Kubernetes and Calico, not GKE. That was a decision made at the start of the sprint: Submariner’s tested network plugins include Calico but not GKE’s native one, so every tool here runs on a path its own project documents. The edge site is a small bare-metal cluster. Steps that are specific to Google Cloud are labelled GCP.

Four honest notes before the table, because several tools overlap and each needs a clear job:

Two ways to reach the edge. Karmada places workloads on whole clusters. KubeEdge attaches single machines and devices (kiosks, shelf sensors) as nodes of the cloud cluster. They solve different problems and are not alternatives here.
Two ways to cross sites. Submariner gives pod-to-pod and DNS connectivity between the clusters for things like database replication and clusterset.local names. Kuma’s multi-zone mesh carries application traffic with mutual TLS. Keep the CIDR ranges apart and give each a different job.
Two RPC styles. Dapr provides the event and invocation APIs for the Go services. Dubbo is used for one thing, the Java pricing service, whose Triple protocol is also callable with plain gRPC.
Two identity jobs. Keycloak is the identity provider for customers and staff. Pinniped uses Keycloak to log engineers in to every cluster with kubectl. TiKV has no Dapr component in this design, so the inventory service uses TiKV’s client library directly.

The platform at a glance

Layer	Tool	Role in the platform	Brief
Application	Dapr	Pub/sub and service invocation APIs through a sidecar for the Go services	Dapr
RPC	Apache Dubbo	Java pricing service over the Triple protocol, with Nacos as its registry (not a CNCF project)	Apache Dubbo
Streaming	Apicurio Registry	Schema rules for events on the Kafka `orders` topic	Apicurio Registry
Database	TiKV	Transactional stock ledger with three replicas	TiKV
Storage	OpenEBS and K8up	Volumes for TiKV and Kafka; restic backups of volumes to Cloud Storage	OpenEBS and K8up
Orchestration	Karmada	Places and overrides workloads across the cloud and edge clusters	Karmada
Delivery	OpenKruise	CloneSets with in-place updates and image pre-pulls	OpenKruise
Service mesh	Kuma	Multi-zone mesh with mutual TLS and traffic permissions	Kuma
API gateway	Higress	Public entry at each site: routing, JWT checks, rate limits	Higress
Global routing	k8gb	DNS-based failover between the cloud and the edge	k8gb
Service proxy	MetalLB	Gives the bare-metal edge a load-balancer address	MetalLB
Cloud network	Submariner	Encrypted tunnel and cross-cluster service discovery	Submariner
Runtime	WasmEdge	Wasm node pool at the edge for small promo-rule modules	WasmEdge
Security	Keycloak	Single sign-on and tokens for customers and staff	Keycloak
Registry	zot	Central registry in the cloud and a caching registry at each edge	zot
Automation	KubeEdge	Connects kiosks and sensors as edge nodes that keep working offline	KubeEdge
Key management	Pinniped	One `kubectl` login to every cluster through Keycloak (not a CNCF project)	Pinniped
Observability	Fluentd	Per-node log collection with file buffering and forwarding	Fluentd
Chaos engineering	ChaosBlade	Fault experiments on nodes, pods and the link between sites	ChaosBlade
Optimization	Kruize	Right-sizing recommendations from Prometheus history (not a CNCF project)	Kruize
Feature flagging	GO Feature Flag	Flags as a file in Git, served by a relay proxy through OpenFeature (not a CNCF project)	GO Feature Flag

Supporting pieces that are not Sprint 3 tools: Kubernetes with kubeadm and Calico, containerd, Kafka (from Strimzi in Sprint 2), PostgreSQL (from CloudNativePG) for Keycloak and Apicurio, Nacos for Dubbo, OpenSearch for log search, Prometheus and Grafana, cert-manager, Cloud DNS, Cloud Load Balancing, Cloud Storage and Terraform. They are inexpensive to replace and are named wherever they appear.

Architecture

One image, every tool with its logo. The numbers 1 to 6 follow a customer order into the services. Open the zoomable diagram (scroll or pinch to zoom, drag to pan), or download it as SVG or PNG. The left side is the customer path, the dashed box is the Google Cloud cluster, and the right side is the edge site with its own copy of the stack. Only the cross-site links are drawn between them.

The platform is easier to read as three views of one system: the request and data path, the sites, connectivity and delivery, and operations and data protection. Every tool appears in at least one view, and each arrow is labelled with what travels over it.

View 1: request and data path

Blue is the edge, orange Google Cloud, purple mesh and routing, red identity, grey the services and teal the data tier. The edge site runs the same stack behind MetalLB instead of Cloud Load Balancing. Open the zoomable diagram, or download it as SVG or PNG.

View 2: sites, connectivity and delivery

Amber is placement and delivery, purple the links between sites, red identity. The numbers 1 to 7 follow a change from a merged pull request to a running pod at the edge. Pinniped’s Concierge runs in each cluster; one arrow stands for all of them. Open the zoomable diagram, or download it as SVG or PNG.

View 3: operations and data protection

Green is operations, teal the data tier, red chaos. Logs, flag events and volume backups all land in Cloud Storage; the sizing advice and the flag file live in Git. Open the zoomable diagram, or download it as SVG or PNG.

Wiring reference

Every integration point in the three views, with what is configured at each one.

From	To	Protocol and port	What is configured
Customer	Cloud DNS, k8gb	DNS, UDP and TCP 53	A parent zone in Cloud DNS delegates `gslb.example.com` to each cluster’s CoreDNS address, where k8gb answers with healthy addresses.
Cloud Load Balancing	Higress	TCP 443	A passthrough network load balancer to the nodes’ Higress NodePort. At the edge, MetalLB gives the Higress service an address instead.
Higress	Keycloak	HTTPS	The JWT plugin uses the realm’s JWKS endpoint to validate tokens before routing.
Higress	Order service	HTTP	An Ingress with `ingressClassName: higress`, routes by path, rate limits per route.
Order service	Pricing service	gRPC, Triple, port 50051	The Go service calls the Dubbo provider’s Triple port with a gRPC client. Nacos is the provider’s registry.
Services	Dapr	HTTP and gRPC on localhost 3500 and 50001	The sidecar exposes publish and invoke APIs; components are defined per namespace.
Dapr	Kafka	Kafka protocol 9092	`pubsub.kafka` component, CloudEvents JSON on topic `orders`.
CI	Apicurio Registry	HTTPS	Schemas are registered on merge with a `BACKWARD` compatibility rule; auto-registration is off in production.
Inventory service	TiKV	gRPC to PD 2379	The transactional client with the PD endpoints; three PD and three TiKV pods on OpenEBS local volumes.
OpenEBS	K8up, Cloud Storage	PVC, HTTPS	K8up `Schedule` backs up the application namespaces to a bucket through the S3-compatible API.
Kuma sidecars	Kuma control plane	KDS, port 5685	A global control plane in the cloud cluster and a zone control plane in each cluster, with mutual TLS and permissions.
Karmada	Member clusters	Kubernetes API, HTTPS	Push mode to the cloud cluster; pull mode (agent) at the edge, so the edge makes outbound connections only.
Karmada	OpenKruise	Kubernetes API	`PropagationPolicy` for `CloneSet`; the interpreter for it must exist in your Karmada version.
OpenKruise	Nodes	Kubelet and containerd	`ImagePullJob` pre-pulls from the local zot before each rollout.
Edge zot	Central zot	HTTPS 5000	`sync` extension with `onDemand`, so the edge fetches an image once and serves it locally.
Submariner gateways	Each other	UDP 4500 and 4490, WireGuard	A broker in the cloud cluster, `subctl join` on both, non-overlapping CIDRs, firewall rules on both sides.
KubeEdge EdgeCore	CloudCore	WebSocket, TCP 10000, 10002	Outbound only from the edge; the cloud address is a stable static IP.
MetalLB	Edge network	ARP or BGP	An `IPAddressPool` outside the DHCP range with an `L2Advertisement`.
Pinniped	Keycloak	OIDC	A Supervisor with a `FederationDomain` and an `OIDCIdentityProvider`; a Concierge and `JWTAuthenticator` in each cluster.
Fluentd (edge)	Fluentd (cloud)	Forward protocol, TCP 24224 with `shared_key`	File buffer sized for the longest outage; the cloud aggregator writes to OpenSearch and Cloud Storage.
Kruize	Prometheus	HTTP	Metric and metadata profiles, one experiment per container.
GO Feature Flag	Git or ConfigMap	HTTPS or API	A retriever polls the flag file; usage events go to an exporter.
ChaosBlade	Nodes, pods	Kubernetes API	`ChaosBlade` resources, one per experiment, with narrow matchers.

Build order

Each step depends on the one before it. Build and verify one layer at a time, cloud first, then the edge.

Google Cloud foundation (GCP). Create the project, VPC, subnets, firewall rules, a service account with least privilege, a Cloud DNS zone and the Cloud Storage buckets.
Cloud cluster. Build three or more Compute Engine VMs, run kubeadm, install Calico and containerd, and join workers. Pick pod and service ranges that will not overlap with the edge.
Edge cluster. Build the bare-metal cluster with the same Kubernetes version and Calico, with its own non-overlapping ranges. Install MetalLB.
Registries. Run zot in the cloud and at the edge, then point containerd at them.
Links between sites. Deploy the Submariner broker and join both clusters. Install KubeEdge CloudCore in the cloud cluster.
Placement. Install the Karmada control plane in its own small cluster or namespace and join the cloud cluster (push) and the edge cluster (pull).
Identity. Install PostgreSQL, Keycloak and Pinniped (Supervisor in the cloud, Concierge in each cluster).
Mesh and edge entry. Install Kuma (global control plane, then zones) and Higress; configure Cloud Load Balancing and k8gb.
Storage and data. Install OpenEBS and K8up, then TiKV, Kafka and Apicurio Registry.
Applications. Install Dapr, Nacos, then the pricing, order and inventory services as OpenKruise CloneSets propagated by Karmada.
Operations. Install Fluentd, Kruize and the GO Feature Flag relay proxy.
Edge extras. Join a KubeEdge device node and add a WasmEdge node pool.
Break it. Run the ChaosBlade game day below.

Build guide: Google Cloud, Mumbai (asia-south1)

GCP. Terraform with the Google provider describes the foundation. This sketch shows the shape; review the provider documentation and pin versions before applying:

provider "google" {
  project = var.project_id
  region  = "asia-south1"
}

resource "google_compute_network" "platform" {
  name                    = "sprint3"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "nodes" {
  name          = "sprint3-nodes"
  network       = google_compute_network.platform.id
  region        = "asia-south1"
  ip_cidr_range = "10.0.0.0/24"
}

# Submariner gateways and KubeEdge CloudCore, from the edge site's public address
resource "google_compute_firewall" "cross_site" {
  name    = "sprint3-cross-site"
  network = google_compute_network.platform.name
  allow {
    protocol = "udp"
    ports    = ["4500", "4490"]
  }
  allow {
    protocol = "tcp"
    ports    = ["10000", "10002", "5685"]
  }
  source_ranges = [var.edge_public_cidr]
}

resource "google_compute_instance" "cp" {
  name           = "cloud-cp-1"
  machine_type   = "e2-standard-4"
  zone           = "asia-south1-a"
  can_ip_forward = true   # gateway nodes forward packets for pod networks
  boot_disk { initialize_params { image = "ubuntu-os-cloud/ubuntu-2204-lts" size = 80 } }
  network_interface {
    subnetwork = google_compute_subnetwork.nodes.id
    access_config {}
  }
}

resource "google_storage_bucket" "backups" {
  name          = "${var.project_id}-sprint3-backups"
  location      = "ASIA-SOUTH1"
  force_destroy = false
}

resource "google_dns_managed_zone" "parent" {
  name     = "example-com"
  dns_name = "example.com."
}

Then initialise Kubernetes with ranges that will not overlap the edge cluster, and install Calico:

# cloud cluster
kubeadm init --pod-network-cidr=10.10.0.0/16 --service-cidr=10.11.0.0/16 \
  --control-plane-endpoint=<cloud-api-address>:6443
# edge cluster uses 10.20.0.0/16 and 10.21.0.0/16

Three Google Cloud details need care on self-managed clusters:

Load balancers. kubeadm does not install a cloud controller manager, so a Service of type LoadBalancer will not create a Google load balancer by itself. Either install the Google cloud controller manager, or run Higress behind a NodePort and create an external passthrough Network Load Balancer yourself that points at the nodes and checks the health of that port.
Identity for pods. GKE’s Workload Identity is a GKE feature. On self-managed nodes, give each tool a dedicated service account with only the roles it needs and store its key as a Secret, or use Workload Identity Federation with the cluster’s OIDC issuer if your policy forbids long-lived keys. K8up uses HMAC keys for the S3-compatible interface.
k8gb and Cloud DNS. Follow the k8gb Google Cloud guide: ExternalDNS manages the delegation records in the parent zone, each cluster has its own txtOwnerId and txtPrefix, and UDP 53 must reach each cluster’s CoreDNS load balancer. Leftover records may need to be deleted by hand at cleanup.

Check: kubectl get nodes shows the cloud nodes Ready, a test Service is reachable through the load balancer, and gcloud storage ls lists the bucket.

Key integration config

The snippets below are the points where tools meet. Resource versions change between releases, so check the API versions in each tool’s documentation for the release you install.

Karmada: place the order service on both clusters, weighted towards the cloud.

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata: { name: order, namespace: shop }
spec:
  resourceSelectors:
    - { apiVersion: apps.kruise.io/v1alpha1, kind: CloneSet, name: order }
  placement:
    clusterAffinity:
      clusterNames: [cloud-gcp, edge-vizag]
    replicaScheduling:
      replicaSchedulingType: Divided
      replicaDivisionPreference: Weighted
      weightPreference:
        staticWeightList:
          - { targetCluster: { clusterNames: [cloud-gcp] }, weight: 2 }
          - { targetCluster: { clusterNames: [edge-vizag] }, weight: 1 }

Karmada needs a resource interpreter to divide replicas of a custom workload type such as a CloneSet. Check that your release ships one for OpenKruise, or add your own, before you depend on this.

OpenKruise: an in-place canary with the image pre-pulled from the edge registry.

apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata: { name: order, namespace: shop }
spec:
  replicas: 6
  selector: { matchLabels: { app: order } }
  template:
    metadata:
      labels: { app: order }
      annotations: { dapr.io/enabled: "true", dapr.io/app-id: "order", dapr.io/app-port: "8080" }
    spec:
      containers:
        - name: order
          image: registry.edge.local/shop/order:1.4.0
  updateStrategy:
    type: InPlaceIfPossible
    partition: 5
    maxUnavailable: 20%

Submariner: join, then export the pricing service across sites.

subctl deploy-broker --kubeconfig cloud.kubeconfig
subctl join broker-info.subm --kubeconfig cloud.kubeconfig --clusterid cloud-gcp --cable-driver wireguard
subctl join broker-info.subm --kubeconfig edge.kubeconfig  --clusterid edge-vizag --cable-driver wireguard
subctl export service --namespace shop pricing
# callable from the other cluster as pricing.shop.svc.clusterset.local

Dapr: pub/sub on the Kafka orders topic.

apiVersion: dapr.io/v1alpha1
kind: Component
metadata: { name: orderpubsub, namespace: shop }
spec:
  type: pubsub.kafka
  version: v1
  metadata:
    - { name: brokers, value: "shop-kafka-bootstrap.data:9092" }
    - { name: authType, value: "none" }

Kuma: only the order service may call inventory.

apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: orders-to-inventory
  namespace: kuma-system
  labels: { kuma.io/mesh: default }
spec:
  targetRef: { kind: MeshSubset, tags: { kuma.io/service: inventory_shop_svc_8080 } }
  from:
    - targetRef: { kind: MeshSubset, tags: { kuma.io/service: order_shop_svc_8080 } }
      default: { action: Allow }

Pinniped: log in to every cluster through Keycloak. The Supervisor lives in the cloud cluster; each cluster has a Concierge with its own audience.

apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: FederationDomain
metadata: { name: platform, namespace: pinniped-supervisor }
spec:
  issuer: https://login.example.com/platform
  tls: { secretName: supervisor-tls }
---
apiVersion: authentication.concierge.pinniped.dev/v1alpha1
kind: JWTAuthenticator
metadata: { name: platform-supervisor }
spec:
  issuer: https://login.example.com/platform
  audience: edge-vizag-<unique-random-value>

The Supervisor also needs an OIDCIdentityProvider that points at the Keycloak realm and is listed under the FederationDomain; check its fields against the Pinniped documentation for your version.

zot at the edge: serve locally, fetch from the cloud once.

"sync": { "registries": [ { "urls": ["https://registry.example.com"], "onDemand": true, "tlsVerify": true } ] }

MetalLB at the edge: a pool outside the DHCP range.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata: { name: edge-pool, namespace: metallb-system }
spec: { addresses: ["192.168.10.240-192.168.10.250"] }
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata: { name: edge-l2, namespace: metallb-system }
spec: { ipAddressPools: ["edge-pool"] }

k8gb: prefer the edge for local customers, fail over to the cloud.

apiVersion: k8gb.io/v1beta1
kind: Gslb
metadata: { name: shop, namespace: shop }
spec:
  resourceRef:
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    matchLabels: { app: shop }
  strategy:
    type: failover
    primaryGeoTag: edge-vizag

KubeEdge: join a kiosk machine as an edge node.

keadm init --advertise-address="<CLOUDCORE-ADDRESS>" --kubeedge-version=v1.23.1 --kube-config=/root/.kube/config
keadm gettoken
keadm join --cloudcore-ipport="<CLOUDCORE-ADDRESS>":10000 --token=<TOKEN> --kubeedge-version=v1.23.1

K8up: nightly backups of the shop namespaces to Cloud Storage.

apiVersion: k8up.io/v1
kind: Schedule
metadata: { name: shop, namespace: shop }
spec:
  backend:
    s3:
      endpoint: https://storage.googleapis.com
      bucket: <project>-sprint3-backups
      accessKeyIDSecretRef: { name: backup-keys, key: access-key }
      secretAccessKeySecretRef: { name: backup-keys, key: secret-key }
    repoPasswordSecretRef: { name: backup-repo, key: password }
  backup: { schedule: '0 2 * * *' }
  check: { schedule: '0 1 * * 1' }
  prune: { schedule: '0 1 * * 0', retention: { keepLast: 5, keepDaily: 14 } }

Fluentd at the edge: buffer to disk, forward to the cloud aggregator.

<match kubernetes.**>
  @type forward
  <server>
    host fluentd.cloud.example.internal
    port 24224
  </server>
  <security>
    self_hostname edge-vizag
    shared_key <secret>
  </security>
  <buffer>
    @type file
    path /var/log/fluentd-buffers/forward
    total_limit_size 4GB
    retry_max_interval 60s
    overflow_action block
  </buffer>
</match>

GO Feature Flag: a flag file in a ConfigMap, a 20% rollout of new pricing.

new-pricing:
  variations: { on: true, off: false }
  targeting:
    - query: email co "@example.com"
      percentage: { on: 100, off: 0 }
  defaultRule:
    percentage: { on: 20, off: 80 }

Kruize: one experiment per container, then read the advice.

POST /createMetricProfile
POST /dsmetadata
POST /createExperiment
POST /generateRecommendations?experiment_name=order
GET  /listRecommendations?experiment_name=order

The other tools are configured as in their briefs: TiKV with the TiDB Operator, Dubbo with Nacos and the Triple protocol, WasmEdge with a RuntimeClass named wasm, Keycloak with its operator, Higress with an Ingress class, and Apicurio Registry with a BACKWARD compatibility rule.

Request walk-through

The customer opens https://shop.example.com. Cloud DNS delegates the gslb zone to the clusters, and k8gb answers with the address of a healthy site, normally the nearest store.
The request reaches Cloud Load Balancing (or MetalLB at the edge) and then Higress, which checks the customer’s Keycloak token and applies rate limits.
Higress routes /orders to the order service. Kuma’s sidecars encrypt the hop, and a permission allows only this path.
The order service calls the pricing service over Triple. The result depends on the new-pricing flag served by the GO Feature Flag relay proxy.
The order service publishes an order.created event through Dapr to Kafka. The schema was checked in CI by Apicurio Registry.
The inventory service consumes the event and reserves stock in TiKV inside a transaction. If the edge site has lost its link to the cloud, orders still complete locally, and events are replicated when the link returns.
Fluentd ships the logs of every hop to the cloud aggregator and OpenSearch.

Deployment walk-through

A developer merges a change to the fleet repository: a new image tag for the order service and an updated flag file.
Karmada sees the changed CloneSet and propagates it: directly to the cloud cluster, and through its pull-mode agent to the edge cluster.
At the edge, an ImagePullJob asks containerd to pre-pull the image from the local zot. On a miss, zot fetches it once from the central registry.
OpenKruise updates the pods in place, starting with the canary partition, so the containers restart without rescheduling the pods.
The new version starts behind a feature flag. The flag file is enabled for staff first and then raised from 20% in steps.
Kruize reads the new usage and, after enough history, proposes smaller or larger requests as a pull request.
If signals worsen, the flag is switched off first and a Git revert makes Karmada roll the cluster back.

Incident walk-through

Scenario: the Vizag store’s link to the cloud drops during a sale, and orders there are slow.

An alert fires on the Submariner connection status, and k8gb health checks begin to fail for the cloud cluster from the edge’s point of view.
Customers at the store still reach the edge Higress through MetalLB, since k8gb prefers the local site. Orders continue, and TiKV at the edge keeps its local ledger.
Fluentd’s buffer on the edge nodes grows, because the forward to the cloud is failing. The team checks the buffer queue and disk use against the planned outage window.
KubeEdge nodes stay up: devices keep running their pods without the cloud, and reconnect when the link returns.
Once the link is back, Submariner reconnects, the buffered logs drain, and the order events replicate. The team checks lag before declaring recovery.
In the review, the team adds a larger buffer and an alert at 70% of it, and writes a ChaosBlade experiment that reproduces the outage in staging.

Game day with ChaosBlade

The goal is to prove that stores keep selling when a site is cut off from the cloud. Run in staging first, with a written hypothesis: orders at the edge succeed throughout, the Fluentd buffer does not overflow in 30 minutes, and replication catches up within ten minutes of the link returning.

apiVersion: chaosblade.io/v1alpha1
kind: ChaosBlade
metadata: { name: cpu-load-edge-node-1 }
spec:
  experiments:
  - scope: node
    target: cpu
    action: fullload
    desc: "CPU load on one edge node"
    matchers:
    - { name: names, value: ["edge-node-1"] }
    - { name: cpu-percent, value: ["60"] }

Cut the link itself by blocking the Submariner and KubeEdge ports on the edge firewall for 30 minutes (a manual step, since it is outside the cluster), and use ChaosBlade’s network experiments to add delay and loss on a node’s interface; check the project documentation for the parameters in your version. During each run, watch the order success rate in the dashboards and the buffer sizes. Follow with a node CPU load on the TiKV nodes and a pod kill of the Kuma control plane to confirm the data planes keep working. Write down what failed, fix it, and repeat before widening the blast radius.

Cost and cleanup

Largest costs (GCP). The Compute Engine VMs (control plane and workers for the cloud cluster, plus Karmada’s), persistent disks for TiKV and Kafka, load balancer forwarding rules, Cloud NAT and egress between regions. Edge hardware is a fixed cost outside Google Cloud.
Control with the platform itself. Kruize recommendations right-size requests so fewer nodes are needed, K8up prunes old backups, and Cloud Storage lifecycle rules expire archived logs.
Teardown. Remove the Karmada propagation policies first so workloads are removed from the member clusters. Delete the k8gb Gslb objects and then any leftover records in Cloud DNS, remove the load balancer, empty the buckets, then destroy the VMs, network and firewall rules with the same Terraform you used to create them. Check for forwarding rules, reserved addresses and disks that remain.
Cheaper variant. For a learning setup, use one small cloud cluster and a single-node edge, one TiKV replica, skip KubeEdge and WasmEdge, and keep one registry. The wiring is unchanged.

AWS and Azure equivalents

Need	Google Cloud	AWS	Azure
Virtual machines	Compute Engine	EC2	Virtual Machines
Kubernetes	kubeadm on Compute Engine	kubeadm on EC2 (or EKS, with care for Submariner)	kubeadm on VMs (AKS has no Submariner guide)
DNS	Cloud DNS	Route 53	Azure DNS
Load balancer	External passthrough Network Load Balancer	Network Load Balancer	Azure Load Balancer
Object storage	Cloud Storage (HMAC, S3 API)	S3	Blob Storage
Firewall	VPC firewall rules	Security groups	Network security groups
Cloud identity for tools	Service account, Workload Identity Federation	IAM roles, IRSA on EKS	Managed identity, Workload Identity
k8gb DNS provider	Cloud DNS with ExternalDNS	Route 53	Azure DNS

Two points change by cloud. The Kubernetes layers (Karmada, Kuma, Dapr, TiKV, Kafka, OpenKruise and the rest) are identical, because they are Kubernetes resources. The provider steps (network, DNS, bucket access, load balancer and identity) differ, as do the firewall rules for Submariner and KubeEdge.

What to take from the sprint

Decide what each overlapping tool is for. Karmada places clusters and KubeEdge attaches devices; Submariner links networks and Kuma secures calls; Dapr gives event APIs and Dubbo serves one Java service.
Make the edge outbound-only. Karmada’s pull mode, KubeEdge’s WebSocket and Fluentd’s buffer all assume the edge calls the cloud, never the reverse, which keeps store firewalls simple.
Treat the link as a failure mode. Local registries, local ledgers, buffered logs and flags in files mean a site keeps selling when the cloud is out of reach.
Keep configuration in Git. Flags, placement policies, sizing advice and image tags all move through pull requests, so one history explains every change.
Test the cut. A game day that really breaks the link tells you more than a diagram does.

Sprint 3 briefs: Dapr · OpenKruise · TiKV · Apicurio Registry · Karmada · Kuma · Higress · k8gb · MetalLB · Apache Dubbo · OpenEBS and K8up · Submariner · WasmEdge · Keycloak · zot · KubeEdge · Pinniped · Fluentd · ChaosBlade · Kruize · GO Feature Flag

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.