Sprint 2 project: An Event-Driven Order Platform on AWS

Sprint 2 covered 21 categories and 25 tools, including a few paired and bonus picks. This post connects them into one platform that can be built, operated and broken on purpose. The theme is event-driven: a customer places an order, the order is stored in PostgreSQL, an event is published to Kafka, and an inventory service scales out to consume it. Tekton and Flux build and deliver the code, Buildpacks produce the images, Linkerd secures the traffic, and Karpenter adds nodes when KEDA adds pods. The business logic is deliberately trivial; the value is in how the tools meet. The infrastructure runs on AWS in the Mumbai region (ap-south-1) because EKS, Route 53, S3, ECR and IAM cover every external need with the fewest moving parts. A mapping table for GCP and Azure follows the build order, and provider-specific steps are labelled AWS.

Three tools need an honest note before the table. etcd is the Kubernetes control-plane database, and on EKS AWS operates it for you, so you cannot reach it; its role is real but invisible there, and it becomes hands-on on a self-managed cluster. CRI-O is not the default runtime on EKS, which uses containerd; it runs here on an optional, separately built node pool. Athenz and Linkerd both issue workload identities, so the project gives them different jobs. Each is explained below rather than forced into a role it does not fit.

The platform at a glance

Layer	Tool	Role in the platform	Brief
Application definition	Buildpacks	Turn source into OCI images with no Dockerfile, plus an SBOM	Buildpacks
CI/CD	Tekton	Pipeline that clones, builds, pushes, signs and records the new image tag	Tekton
CI/CD	Flux	Reconciles the cluster from Git, applies the new tag	Flux
Database	CloudNativePG	PostgreSQL cluster for orders, stock and Unleash, with failover and backups	CloudNativePG
Streaming	Strimzi and CloudEvents	Kafka on Kubernetes, and a standard envelope for every event	Strimzi
Scheduling	KEDA	Scales the inventory consumer on Kafka lag, down to zero	KEDA
Service mesh	Linkerd	Mutual TLS, per-request load balancing and golden metrics	Linkerd
API gateway	Kgateway	Public entry point on the Gateway API, auth and rate limiting	Kgateway
Coordination	etcd	Control-plane store; snapshot target on self-managed clusters	etcd
Service proxy	Contour	Internal ingress for operator tools, with IP allow lists	Contour
RPC	Connect RPC	Browser-friendly and gRPC-compatible APIs from one handler	Connect RPC
Storage	Longhorn and Velero	Replicated volumes and snapshots; cluster backup to S3	Longhorn
Network	Antrea	Network policy enforcement and flow visibility	Antrea
Container runtime	CRI-O	Runtime for the hardened node pool	CRI-O
Security	Kyverno and Falco	Admission policy and image verification; runtime threat detection	Kyverno
Registry	Dragonfly	Peer-to-peer image distribution in front of Amazon ECR	Dragonfly
Automation	Cloud Custodian	Governs the AWS account: tags, idle and orphaned resources	Cloud Custodian
Key management	Athenz	Service authorization, X.509 identities and temporary cloud credentials	Athenz
Observability	Jaeger	Distributed traces across gateway, services and Kafka	Jaeger
Chaos engineering	Litmus	Experiments with probes against the data tier	Litmus
Optimization	Karpenter	Launches and consolidates nodes (not a CNCF project)	Karpenter
Feature flagging	Unleash	Runtime flags and gradual rollout (not a CNCF project)	Unleash

Supporting pieces that are not Sprint 2 tools: Amazon EKS, ECR, S3, Route 53, an AWS Network Load Balancer with an ACM certificate, IAM roles for service accounts, and Prometheus and Grafana for metrics (installed with the Linkerd viz extension or a standard monitoring chart). They are inexpensive to replace and are named wherever they appear.

Architecture

One image, every tool with its logo. The numbers 1 to 8 follow a customer order. Open the zoomable diagram (scroll or pinch to zoom, drag to pan), or download it as SVG or PNG.

The platform is also easier to read as three views of one system: the request and data path that serves customers, the delivery and identity plane that builds, ships and authorises workloads, and the operations plane that observes, protects, scales and stress-tests them. Every Sprint 2 tool appears in at least one view, and each arrow is labelled with what travels over it.

View 1: request and data path

flowchart TB
  USER[Customer browser] -->|HTTPS 443| R53[Route 53 DNS]
  R53 -->|alias record| NLB[AWS NLB<br/>TLS with ACM certificate]
  subgraph EKS[EKS cluster, Antrea policy, Linkerd mesh]
    NLB -->|TCP to Envoy proxy| KGW[Kgateway<br/>Gateway API and Envoy]
    KGW -->|Connect protocol over HTTP<br/>mTLS by Linkerd| ORD[Order service<br/>Connect RPC]
    ORD -->|gRPC protocol<br/>per request balancing| INV[Inventory service<br/>Connect RPC]
    ORD -->|flag evaluation HTTPS| UNL[Unleash<br/>flag service]
    ORD -->|SQL 5432 to rw service| PG[CloudNativePG<br/>PostgreSQL primary]
    ORD -->|publish CloudEvent<br/>topic orders| KAF[Kafka by Strimzi<br/>KRaft brokers]
    KAF -->|consume at least once<br/>consumer group inventory| INV
    INV -->|SQL 5432 stock update| PG
    UNL -->|SQL 5432 own database| PG
    PG -->|PVC| LH[Longhorn volumes<br/>2 replicas]
    KAF -->|PVC| LH
  end
  KEDA[KEDA] -.->|reads consumer lag<br/>scales 0 to N| INV
  classDef edge fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  class USER,R53,NLB edge
  class KGW,ORD,INV,UNL,KEDA core
  class PG,KAF,LH data

Key: grey is the edge, teal are services and controllers, blue are the data tier. Each arrow label names the protocol or payload; the dashed arrow is a control signal rather than traffic.

View 2: delivery and identity plane

flowchart TB
  DEV[Developer] -->|git push| GIT[Git repository<br/>app and manifests]
  GIT -->|webhook| TRG[Tekton Triggers<br/>EventListener]
  TRG ==>|creates PipelineRun| TKN[Tekton pipeline<br/>clone, build, push, sign]
  TKN ==>|Buildpacks task<br/>builds layered image| ECR[Amazon ECR<br/>registry]
  TKN -->|Chains signs image<br/>and attestations| ECR
  TKN -->|commits new image tag| GIT
  GIT ==>|poll or webhook| FLX[Flux controllers<br/>source, kustomize, helm]
  FLX ==>|server side apply| API[Kubernetes API<br/>etcd behind it]
  API -->|admission review| KYV[Kyverno<br/>verifyImages and policy]
  KYV -->|verifies signature| ECR
  API -->|schedules pod| KLT[kubelet on node<br/>containerd or CRI-O]
  KLT -->|pull through local proxy| DFD[Dragonfly dfdaemon<br/>peer to peer layers]
  DFD -->|back to source on miss| ECR
  ATH[Athenz ZMS and ZTS] -.->|X.509 identity and role tokens| SVC[Platform services<br/>and operators]
  ATH -.->|temporary AWS credentials| AWS[AWS IAM roles<br/>Velero, Custodian]
  CUS[Cloud Custodian<br/>Lambda policies] -->|tag, stop, delete| ACC[AWS account<br/>resources]
  classDef actor fill:#eceff1,stroke:#455a64,stroke-width:1.5px,color:#1d1c1a
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  classDef sec fill:#fce4ec,stroke:#ad1457,stroke-width:1.5px,color:#1d1c1a
  class DEV actor
  class TRG,TKN,FLX,DFD,CUS core
  class GIT,ECR,API,KLT,ACC,AWS,SVC data
  class KYV,ATH sec

Key: grey is the person, teal are Sprint 2 delivery tools, blue are stores and platforms, pink are security and identity. Thick arrows are the build to deploy path.

View 3: operations plane

flowchart TB
  subgraph APPS[Workloads in the mesh]
    GW[Kgateway proxy]
    SV[Order and inventory]
    DB[PostgreSQL and Kafka]
  end
  SV -->|OTLP spans 4317| JAE[Jaeger v2<br/>collector and query]
  GW -->|OTLP spans| JAE
  SV -->|proxy metrics scrape| PRM[Prometheus and Grafana<br/>Linkerd viz]
  FAL[Falco on each node<br/>eBPF syscall rules] -->|alerts via Falcosidekick| KAF2[Kafka topic<br/>security alerts]
  FAL -.->|events from| DB
  PRM -->|consumer lag via exporter| KEDA2[KEDA]
  KEDA2 -->|scales Deployment| SV
  SV -->|pending pods| KRP[Karpenter<br/>NodePool]
  KRP -->|launches or consolidates EC2| NODES[EKS nodes]
  LIT[Litmus<br/>ChaosEngine and probes] -.->|pod delete, node drain| DB
  LIT -->|HTTP probe on API| GW
  VEL[Velero] -->|backups and CSI snapshots| S3[S3 bucket]
  LHB[Longhorn backups] -->|volume backups| S3
  PGB[CloudNativePG] -->|base backups and WAL| S3
  ANT[Antrea] -.->|policy and flow records| APPS
  ETC[etcd snapshots<br/>self managed only] -->|etcdctl snapshot save| S3
  classDef core fill:#0f766e,stroke:#064e4a,stroke-width:2px,color:#ffffff
  classDef data fill:#e3f2fd,stroke:#1565c0,stroke-width:1.5px,color:#1d1c1a
  classDef sec fill:#fce4ec,stroke:#ad1457,stroke-width:1.5px,color:#1d1c1a
  classDef ctrl fill:#ede7f6,stroke:#5e35b1,stroke-width:1.5px,color:#1d1c1a
  class JAE,PRM,KEDA2,KRP,LIT,VEL,LHB,PGB,ANT core
  class GW,SV,DB,NODES,S3,KAF2 data
  class FAL sec
  class ETC ctrl

Key: teal are Sprint 2 operations tools, blue are workloads and stores, pink is runtime security, purple is the optional self-managed item. KEDA gets the lag either directly from Kafka or from a metrics exporter.

Wiring reference

Every integration point in the three views, with what is configured at each one.

From	To	Protocol and port	What is configured
Git	Tekton Triggers	HTTPS webhook	EventListener with an interceptor that validates the signature and filters the branch.
Tekton	Buildpacks	Task in the pipeline	The Buildpacks task runs the pack lifecycle with a cache workspace and writes the image digest as a result.
Tekton	ECR	HTTPS, OCI	Image pushed with an IAM role for the pipeline service account. Tekton Chains signs the image and records provenance.
Tekton	Git	HTTPS	A final task commits the new image digest into the environment manifests.
Flux	Git, Kubernetes API	HTTPS and API	GitRepository plus Kustomization objects with dependsOn ordering, pruning and health checks.
Kyverno	Kubernetes API, ECR	Admission webhook, HTTPS	Policies verify the Chains signature, require resource limits and restrict registries to ECR.
kubelet	Dragonfly	HTTP on loopback 4001	containerd or CRI-O mirror configuration sends pulls through the local dfdaemon.
Dragonfly	ECR	HTTPS	Seed peers pull from ECR on a miss; peers share layers after that. ECR credentials are supplied to the daemon.
Karpenter	EC2	HTTPS to AWS APIs	NodePool and EC2NodeClass, interruption queue, IAM role through Pod Identity.
KEDA	Kafka, Deployment	Kafka protocol 9092, API	ScaledObject on consumer lag for the inventory Deployment, with a zero minimum.
Strimzi	Kafka, S3	Operator to cluster	Kafka, KafkaNodePool and KafkaTopic resources, KRaft mode, Longhorn-backed volumes.
Services	Kafka	Kafka protocol 9092	CloudEvents in structured JSON mode on topic orders, keyed by order id.
Services	CloudNativePG	PostgreSQL 5432	The -rw service for writes, credentials from the generated application secret.
CloudNativePG	Longhorn, S3	PVC, HTTPS	Three instances on longhorn volumes, base backups and WAL archive to S3 through IAM.
Longhorn	S3	HTTPS	Backup target for volume backups, with a recurring job.
Velero	S3, Longhorn	HTTPS, CSI	Backup storage location in S3, scheduled backups of platform namespaces with CSI snapshots.
Kgateway	Services	Gateway API, HTTP	Gateway, HTTPRoute for the Connect RPC paths, in the Linkerd mesh so the next hop is mTLS.
Kgateway	Athenz	gRPC ext_authz (adapter)	A small authorization service validates an Athenz access token and checks policy before routing.
Contour	Operator tools	HTTP	HTTPProxy for Tekton Dashboard, Jaeger UI and Unleash UI with ipAllowPolicy.
Linkerd	Services	Proxy to proxy, mTLS	Namespace injection, Server and AuthorizationPolicy resources, trust anchor kept outside the cluster.
Antrea	Pods	CNI policy mode	networkPolicyOnly alongside the VPC CNI, default deny per namespace, explicit allows.
Unleash	CloudNativePG	PostgreSQL 5432	Own database in the shared cluster, connection string from the CNPG secret.
Services	Unleash	HTTPS	Server-side SDK with an API token per environment.
Services	Jaeger	OTLP 4317 and 4318	OpenTelemetry SDKs export to the Jaeger collector.
Falco	Kafka	Falcosidekick output	Detections published to a security-alerts topic for routing and storage.
Litmus	Workloads	Kubernetes API	ChaosEngine resources with HTTP and Prometheus probes.
Athenz	AWS	HTTPS	ZTS issues temporary credentials for roles used by Velero and Cloud Custodian.
Cloud Custodian	AWS	HTTPS	Policies run as Lambda functions on a schedule, in a dedicated IAM role.

Build order

Each step depends on the one before it. Build and verify one layer at a time.

1. Account and cluster (AWS). Create the VPC and EKS cluster with a small managed node group for system components. This is also where etcd exists, managed by AWS.
2. Network policy. Install Antrea in networkPolicyOnly mode beside the VPC CNI, then apply default-deny in each application namespace.
3. Node autoscaling. Install Karpenter and a first NodePool, so every later step can create capacity.
4. GitOps. Bootstrap Flux against the repository. From here, everything else is a commit.
5. Registry acceleration. Install Dragonfly and point containerd at it, with ECR as the source.
6. Storage. Install Longhorn, set the default StorageClass, and configure the S3 backup target. Install Velero.
7. Policy. Install Kyverno with registry and resource policies in audit mode, then enforce.
8. Identity and mesh. Install Linkerd with a trust anchor you control. Install Athenz if you will use it for authorisation.
9. Data. Install the CloudNativePG operator and a three-instance cluster; install Strimzi and a three-broker Kafka cluster.
10. Edge. Install Kgateway and Contour, create the NLB, certificate and DNS records.
11. Delivery. Install Tekton, Triggers and Chains; create the build pipeline.
12. Applications. Deploy Unleash, then the order and inventory services, then the KEDA ScaledObject.
13. Operations. Install Jaeger, Falco and Litmus; deploy the Cloud Custodian policies.
14. Optional hardened pool. Build the CRI-O AMI and add a second Karpenter NodePool for it.

Build guide: AWS Mumbai region (ap-south-1)

AWS. The cluster and node autoscaling can be created with the community Terraform or OpenTofu modules. This sketch shows the shape; pin module versions and review the module documentation before applying:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "sprint2"
  cluster_version = "1.33" # use a version EKS supports in ap-south-1
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.private_subnets

  eks_managed_node_groups = {
    system = {
      instance_types = ["m6i.large"]
      min_size       = 2
      max_size       = 3
      desired_size   = 2
      labels         = { "workload" = "system" }
    }
  }
}

module "karpenter" {
  source  = "terraform-aws-modules/eks/aws//modules/karpenter"
  version = "~> 20.0"

  cluster_name = module.eks.cluster_name
  enable_pod_identity             = true
  create_pod_identity_association = true
}

Then bootstrap Flux, which installs the rest from Git:

flux bootstrap github \
  --owner=<org> --repository=<fleet-repo> \
  --branch=main --path=clusters/sprint2 --personal

A fleet repository layout keeps the order explicit:

clusters/sprint2/
  infrastructure.yaml   # antrea, karpenter, longhorn, velero, kyverno, linkerd, dragonfly
  data.yaml             # cloudnativepg, strimzi, unleash
  edge.yaml             # kgateway, contour
  delivery.yaml         # tekton, chains
  apps.yaml             # order, inventory, keda scaledobject
  operations.yaml       # jaeger, falco, litmus, custodian

Check: flux get kustomizations shows each layer Ready in turn, kubectl get nodes shows the system group, and a pending test pod causes Karpenter to launch a node.

Key integration config

The snippets below are the points where two tools meet. Resource versions change between releases, so check the API versions in each tool’s documentation for the release you install.

Flux: ordered layers with health gates. The data layer waits for storage and policy, and the apps wait for the data.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: { name: data, namespace: flux-system }
spec:
  interval: 10m
  path: ./data
  prune: true
  sourceRef: { kind: GitRepository, name: fleet }
  dependsOn:
    - name: infrastructure
  healthChecks:
    - apiVersion: postgresql.cnpg.io/v1
      kind: Cluster
      name: shop-db
      namespace: data

Tekton to Buildpacks: the build step of the pipeline. The Buildpacks task is installed from the Tekton catalog; pin its version.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata: { name: build-and-record }
spec:
  params:
    - { name: repo-url, type: string }
    - { name: image, type: string }
  workspaces: [ { name: source }, { name: cache } ]
  tasks:
    - name: clone
      taskRef: { name: git-clone }
      params: [ { name: url, value: $(params.repo-url) } ]
      workspaces: [ { name: output, workspace: source } ]
    - name: build
      runAfter: [clone]
      taskRef: { name: buildpacks }
      params:
        - { name: APP_IMAGE, value: $(params.image) }
        - { name: BUILDER_IMAGE, value: paketobuildpacks/builder-jammy-base }
      workspaces:
        - { name: source, workspace: source }
        - { name: cache, workspace: cache }

Kyverno: only signed images from the platform registry. Tekton Chains signs the image; Kyverno verifies the signature before a pod is admitted.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-images }
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: signed-by-pipeline
      match: { any: [ { resources: { kinds: [Pod] , namespaces: [shop] } } ] }
      verifyImages:
        - imageReferences: ["<account>.dkr.ecr.ap-south-1.amazonaws.com/*"]
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      <chains cosign public key>
                      -----END PUBLIC KEY-----

Dragonfly: send pulls through the local daemon. On containerd nodes, a hosts.toml for the ECR registry host. CRI-O nodes use a mirror entry in registries.conf.

# /etc/containerd/certs.d/<account>.dkr.ecr.ap-south-1.amazonaws.com/hosts.toml
server = "https://<account>.dkr.ecr.ap-south-1.amazonaws.com"

[host."http://127.0.0.1:4001"]
  capabilities = ["pull", "resolve"]

# /etc/containers/registries.conf.d/dragonfly.conf  (CRI-O nodes)
[[registry]]
location = "<account>.dkr.ecr.ap-south-1.amazonaws.com"

  [[registry.mirror]]
  location = "127.0.0.1:4001"
  insecure = true

ECR tokens expire after 12 hours, so the daemon needs a credential refresh mechanism. This is the most common cause of mysterious pull failures with this pattern.

Strimzi and CloudEvents: a Kafka cluster and the event envelope.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: shop
  namespace: data
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 4.0.0 # use a version your Strimzi release supports
    listeners:
      - { name: tls, port: 9093, type: internal, tls: true }
    config:
      default.replication.factor: 3
      min.insync.replicas: 2
  entityOperator:
    topicOperator: {}
    userOperator: {}
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: orders
  namespace: data
  labels: { strimzi.io/cluster: shop }
spec:
  partitions: 6
  replicas: 3

An event published by the order service, as a structured CloudEvent:

{
  "specversion": "1.0",
  "id": "9f1c7e6a-0b57-4a39-8f2e-6d1e2f4b7c11",
  "source": "/shop/order-service",
  "type": "com.example.shop.order.created",
  "time": "2026-10-21T06:30:00Z",
  "datacontenttype": "application/json",
  "subject": "order/10234",
  "data": { "orderId": 10234, "items": [ { "sku": "A-100", "qty": 2 } ] }
}

KEDA: scale the inventory consumer on Kafka lag.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: inventory, namespace: shop }
spec:
  scaleTargetRef: { name: inventory }
  minReplicaCount: 0
  maxReplicaCount: 12
  cooldownPeriod: 120
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: shop-kafka-bootstrap.data:9092
        consumerGroup: inventory
        topic: orders
        lagThreshold: "50"

Keep maxReplicaCount no larger than the number of partitions, since extra consumers in a group sit idle.

CloudNativePG: a three-instance cluster on Longhorn with backups to S3 (AWS).

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: shop-db, namespace: data }
spec:
  instances: 3
  storage: { size: 20Gi, storageClass: longhorn }
  bootstrap:
    initdb: { database: shop, owner: shop }
  backup:
    retentionPolicy: "14d"
    barmanObjectStore:
      destinationPath: s3://<bucket>/shop-db
      s3Credentials: { inheritFromIAMRole: true }
      wal: { compression: gzip }

The generated secret shop-db-app carries the connection details used by the services and by Unleash. Longhorn replication and PostgreSQL replication overlap, so the lab setting uses two Longhorn replicas, and a production design should weigh the cost of storage replication against database replicas.

Linkerd and Antrea: encrypt and then restrict.

apiVersion: v1
kind: Namespace
metadata:
  name: shop
  annotations: { linkerd.io/inject: enabled }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: shop }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Add explicit allows for the gateway to the order service, the services to Kafka, PostgreSQL and Unleash, and DNS. Linkerd authorisation policies then limit which identities may call which port.

Kgateway and Contour: a public path and an operator path.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: orders, namespace: shop }
spec:
  parentRefs: [ { name: public, namespace: gateway } ]
  hostnames: ["shop.example.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /shop.v1.OrderService/ }
      backendRefs: [ { name: order, port: 8080 } ]
---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata: { name: tools, namespace: tools }
spec:
  virtualhost: { fqdn: tools.internal.example.com }
  routes:
    - conditions: [ { prefix: /jaeger } ]
      ipAllowPolicy:
        - { cidr: 10.0.0.0/8, source: Peer }
      services: [ { name: jaeger-query, port: 16686 } ]

Athenz: authorisation at the edge, plus cloud credentials. The public path validates an Athenz access token at the gateway, and the policy decides whether a caller may act on an order resource. Athenz’s own policy form:

{
  "name": "shop:policy.order-writers",
  "assertions": [
    { "role": "shop:role.order-writers", "action": "create",
      "resource": "shop:orders", "effect": "ALLOW" }
  ]
}

Kgateway calls an authorisation service through its external authorisation support, and that small service is the piece you write: it validates the token against the ZTS keys and evaluates the policy. Linkerd handles pod-to-pod identity; Athenz handles who may do what at the edge and issues temporary AWS credentials to Velero and Cloud Custodian roles. Do not run both for the same job.

Unleash: a flag guards a new code path. Unleash uses a database in the shared cluster:

env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef: { name: unleash-db-app, key: uri }

const unleash = initialize({
  url: 'http://unleash.platform:4242/api/',
  appName: 'order-service',
  customHeaders: { Authorization: process.env.UNLEASH_TOKEN },
});
const useNewPricing = unleash.isEnabled('new-pricing', { userId });

Falco: detections as events. Route alerts to Kafka through Falcosidekick:

falcosidekick:
  enabled: true
  config:
    kafka:
      hostport: shop-kafka-bootstrap.data:9092
      topic: security-alerts

Cloud Custodian: remove orphaned volumes after a grace period (AWS).

policies:
  - name: ebs-unattached-mark
    resource: aws.ebs
    filters:
      - Attachments: []
      - "tag:custodian_cleanup": absent
    actions:
      - type: mark-for-op
        tag: custodian_cleanup
        op: delete
        days: 7
  - name: ebs-unattached-delete
    resource: aws.ebs
    filters:
      - type: marked-for-op
        tag: custodian_cleanup
        op: delete
    actions: [ delete ]

Karpenter: a general pool and an optional CRI-O pool. The pool described in the Karpenter brief serves the general workloads. The hardened pool uses a node class that points at your CRI-O image and a taint, so only workloads that tolerate it land there:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: crio }
spec:
  template:
    spec:
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: crio }
      taints: [ { key: runtime, value: crio, effect: NoSchedule } ]
      requirements:
        - { key: karpenter.sh/capacity-type, operator: In, values: ["on-demand"] }
  limits: { cpu: "64" }

The crio node class selects a custom AMI that you build with CRI-O installed and configured; this is the one genuinely manual step in the build.

Litmus: a probe-backed experiment on the database (see the game day below).

Request walk-through

1. The customer opens https://shop.example.com. Route 53 resolves to the AWS Network Load Balancer, which terminates TLS with the ACM certificate.
2. Kgateway’s Envoy proxy receives the request, applies the authorisation check against the Athenz token and rate limits, and routes by path to the order service.
3. Linkerd proxies encrypt the hop with mutual TLS. Antrea policy allows only this path into the namespace.
4. The order service checks the new-pricing flag in Unleash, calculates the order, and writes it to the CloudNativePG primary through the read-write service.
5. The service publishes a CloudEvent to the orders topic and calls the inventory service over the gRPC protocol of Connect RPC. Linkerd balances those calls per request.
6. The inventory service reads events from Kafka, updates stock in PostgreSQL, and acknowledges. Its replica count follows the consumer lag.
7. Every hop exports spans to Jaeger, so one trace shows gateway, services, database and Kafka producer in a single timeline.

Deployment walk-through

1. A developer pushes to the application repository. The webhook starts a Tekton PipelineRun through Triggers.
2. The pipeline clones the code and Buildpacks build a layered image and an SBOM, with no Dockerfile. The image is pushed to ECR.
3. Tekton Chains signs the image and records provenance. The last task commits the new image digest to the manifests in Git.
4. Flux detects the commit and reconciles. The API server asks Kyverno to admit the pod; Kyverno verifies the signature and policies.
5. A new node appears if needed. Karpenter launches it, and Dragonfly speeds the image pull by sharing layers between peers rather than hitting ECR from every node.
6. The new version starts behind an Unleash flag, is enabled for internal users first, then for a growing percentage while Jaeger and Linkerd metrics are watched.
7. If the signal worsens, the flag is switched off first, and a Git revert makes Flux roll the cluster back.

Incident walk-through

Scenario: order confirmations are delayed during a sale, and a deployment of the inventory service is suspected.

1. An alert fires on consumer lag for group inventory. KEDA has already raised replicas to the maximum, and Karpenter has added nodes.
2. The trace for a slow order shows the delay is after the Kafka publish, so the order path is healthy and the consumer side is behind.
3. Linkerd metrics show inventory pods with a high failure rate calling PostgreSQL. Jaeger shows long database spans on the same query.
4. The CloudNativePG dashboard shows the primary under heavy write load on one table. The team turns off the expensive stock-history feature through its Unleash flag, and lag begins to fall.
5. Falco shows no suspicious activity during the incident, and Antrea flow records show no unexpected traffic, which rules out a security cause.
6. After the incident the team adds an index through a reviewed migration, lowers the lag alert threshold, and writes a Litmus experiment that reproduces the load on the primary in staging.

Game day with Litmus

The goal is to prove that orders keep flowing when the database primary disappears. Run in staging first, with a written hypothesis: orders succeed within five seconds of the failure, no order is lost, and consumer lag returns to zero within ten minutes.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata: { name: kill-db-primary, namespace: data }
spec:
  appinfo:
    appns: data
    applabel: cnpg.io/instanceRole=primary
    appkind: ""
  chaosServiceAccount: pod-delete-sa
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - { name: TOTAL_CHAOS_DURATION, value: "60" }
            - { name: FORCE, value: "true" }

Attach an HTTP probe to the order endpoint that must keep returning success, and a Prometheus probe on the consumer lag. Follow with a second experiment that drains a Kafka broker node and a third that kills the Kyverno webhook pods to confirm the failure policy behaves as designed. After each run, record the resilience result and fix what failed before moving to production.

Cost and cleanup

• Largest costs (AWS). The EKS control plane, the NAT gateways, the load balancer and the nodes that carry Kafka and PostgreSQL dominate a small deployment. Replicated storage multiplies disk costs.
• Control with the platform itself. Karpenter consolidation removes idle nodes, KEDA scales consumers to zero, and Cloud Custodian removes orphaned volumes and stale resources. Review the node bill after the first sale-day load.
• Teardown. Delete the Flux Kustomizations first with pruning enabled, so load balancers and volumes are released. Delete the Velero and Longhorn backups only if you no longer need them, empty the S3 buckets, then destroy the cluster and VPC with the same Terraform or OpenTofu you used to create it. Check for leftover load balancers, security groups and EBS volumes.
• Cheaper variant. For a learning environment, run one small node group with fewer brokers and one database instance, skip the CRI-O pool, and use the default storage class instead of Longhorn. The wiring is unchanged.

GCP and Azure equivalents

Need	AWS	GCP	Azure
Kubernetes	EKS	GKE	AKS
Node autoscaling	Karpenter	Karpenter provider or GKE node auto-provisioning	AKS Node Auto Provisioning (Karpenter based)
Registry	Amazon ECR	Artifact Registry	Azure Container Registry
Object storage	S3	Cloud Storage	Blob Storage
DNS	Route 53	Cloud DNS	Azure DNS
Load balancer and certificate	NLB with ACM	Network load balancer with Google-managed certificate	Azure Load Balancer with Key Vault certificate
Workload cloud identity	Pod Identity or IRSA	Workload Identity	Workload Identity
Local disks for storage nodes	i4i and m6id families	Local SSD	Lsv3 family
Custodian execution	Lambda	Cloud Functions	Azure Functions
Athenz cloud integration	AWS provider and credentials	GCP provider	Azure provider

Two points change by cloud. The Kyverno, Linkerd, Kafka, PostgreSQL and delivery layers are identical, because they are Kubernetes resources. The provider steps (cluster, DNS, registry, bucket permissions, node images) differ, and so does the way the CRI-O image is built.

What to take from the sprint

• Make the event the contract. CloudEvents on a Kafka topic let the order and inventory services change independently, and KEDA turns the topic’s backlog into capacity.
• Let scaling be a chain, not a setting. Lag raises pods, pods raise nodes through Karpenter, and Dragonfly keeps the new nodes from becoming a registry bottleneck.
• Separate identity jobs. Linkerd authenticates pods to pods, Athenz decides what a caller may do and hands out cloud credentials, and Kyverno decides what the cluster accepts.
• Back up the data tier in three places that match its three failure modes: database backups and WAL for point-in-time recovery, Longhorn for volumes, Velero for cluster objects.
• Be clear about where a tool does not fit. etcd is invisible on a managed control plane, and CRI-O belongs on a purpose-built pool. Saying so is better than inventing a role for them.

Sprint 2 briefs: Buildpacks · Tekton · Flux · CloudNativePG · Strimzi and CloudEvents · KEDA · Linkerd · Kgateway · etcd · Contour · Connect RPC · Longhorn and Velero · Antrea · CRI-O · Kyverno · Falco · Dragonfly · Cloud Custodian · Athenz · Jaeger · Litmus · Karpenter · Unleash