Happy Dussehra! May good always win. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps

Sprint 5 project, Part 4: Access, Secrets and Evidence

Let carriers and customers sign in through their own identity providers with Dex, show each only its own shipments with OpenFGA, keep secrets in Secrets Manager and SOPS with KMS, admit only our images with Gatekeeper, test the partner API against its contract with Microcks, and keep a hash-chained custody record in S3 Object Lock with a tamper and access-denial test.

73 min read

  • Dex
  • OpenFGA
  • SOPS
  • external-secrets
  • OPA Gatekeeper
  • Microcks
  • CloudNativePG
  • Argo CD
  • Knative
  • Amazon RDS for PostgreSQL
  • S3 Object Lock
  • AWS KMS
  • AWS Secrets Manager
  • Amazon ECR
  • EKS Pod Identity
  • OpenTofu
  • Python
  • Kubernetes

This is Part 4 of the five-part build in the Sprint 5 project overview. Part 3 stores readings, opens excursion cases in RDS and records geofence events. Now outsiders arrive. A carrier should see its own shipments and nobody else’s. A hospital group should see what is being delivered to it. Nobody should hold a password in a file in Git, and nobody, including an administrator in a hurry, should be able to quietly change the record of what happened to a vaccine consignment. This part builds those four things: sign-in (Dex), per-shipment permission (OpenFGA on CloudNativePG), secrets (external-secrets with AWS Secrets Manager, and SOPS with KMS) with Gatekeeper admitting only our images, a contract for the partner API (Microcks), and a hash-chained custody record in S3 Object Lock.

Where this part fits. This part builds access, secrets and evidence (marked Part 4 on the overall diagram). Open the zoomable Sprint 5 architecture or read the Overview.

Overall Sprint 5 architecture

What this is and is not. This is engineering guidance, not legal or regulatory advice, and nothing here is validated or compliant on its own. The validation checklist in section 9 lists the evidence a quality unit usually asks for; deciding what a regulated product needs, and signing it off, is the company’s job. A hash chain and a locked bucket make tampering detectable and deletion hard, which is useful evidence, but they are not an electronic-signature system. If you find a mismatch with your versions, use the “Suggest a correction” link on this page.

What you build, and in what order

Architecture: carriers and customers sign in through their own identity providers, which Dex federates, and call the partner API with an ID token. The partner API verifies the token, asks OpenFGA whether the caller may see the shipment and reads shipments from RDS for PostgreSQL. OpenFGA keeps its tuples in a CloudNativePG database. Microcks tests the partner API against its contract. An evidence job reads cases, geofence events and access denials from RDS and writes one object each to an S3 bucket with Object Lock, each record naming the hash of the one before. Secrets Manager values reach the cluster through external-secrets with Pod Identity; SOPS files in Git are decrypted by Argo CD with a KSOPS plugin and the KMS platform key; Gatekeeper admits only images from our ECR.

One image, the layers of this part. Open the zoomable diagram, or download it as SVG or PNG.

Flow in eight steps: external-secrets serving Kafka and database values; SOPS with KMS and the Argo CD plugin; Gatekeeper in dry run then deny; OpenFGA with its database, model and tuples; Dex with lab users and connectors; the partner API answering 200 and 403 as expected; Microcks mock and contract test; and the evidence chain with its verifier and tamper test.

  1. Prerequisites and versions
  2. Secrets have one home each: external-secrets and Secrets Manager
  3. SOPS with KMS for values that must live in Git, and Argo CD
  4. Gatekeeper: only our images in react
  5. OpenFGA on CloudNativePG: the model and the tuples
  6. Dex: partner sign-in
  7. The partner API
  8. Microcks: mock and contract test
  9. Evidence: a hash-chained custody record, and the validation checklist
  10. The access-denial and tamper tests
  11. Troubleshooting, cost, cleanup, design notes and checklist

The decisions behind it

Every secret has one home. Values a pod reads and that we can rotate (the Kafka user, the database password, OpenFGA’s key) live in AWS Secrets Manager and reach the cluster through external-secrets. Values a partner gives us and that have no AWS home (their identity provider’s client secret) live in Git, encrypted with SOPS under the platform KMS key. Nothing is in both places, and no plain value is in either Git or a shell history.

Dex authenticates; OpenFGA decides. A partner’s token says who the caller is (a verified email). Which shipments that person may see is data in OpenFGA, not a claim in a token, so a membership change takes effect without waiting for a token to expire and without asking the partner to change their identity provider. The Overview’s wiring table mentions a groups claim: this build does not use one, for that reason.

The API says no by default. If OpenFGA, the database or the token keys cannot be reached, the API returns 503, never 200. A shipment that does not exist and a shipment you may not see both give 403, so the API does not reveal which shipment numbers exist.

Evidence is a chain, not a pile. Every custody record (an excursion case opened or closed, a geofence event, an access denial) is one small JSON object. Each names the SHA-256 of the record before it. The log of hashes sits in RDS, the objects in a locked bucket; changing either one is detectable from the other, and a verifier job checks both.

1. Prerequisites and versions

Item Version or detail Why
Parts 1 to 3 complete Every checkpoint passes The hub, RDS, the evidence bucket and the shipments exist
Dex v2.46.1 (chart dex/dex) Dex brief
OpenFGA v1.18.x; chart openfga/openfga OpenFGA brief
CloudNativePG The current operator release from cloudnative-pg/charts CloudNativePG brief
SOPS v3.13.3 or newer SOPS brief
KSOPS A tag from viaductoss/ksops on Docker Hub (set KSOPS_TAG) Argo CD has no built-in SOPS support
external-secrets The current minor, pinned (helm search repo external-secrets/external-secrets) external-secrets brief
Gatekeeper v3.23.1 or the chart you used in Sprint 4 OPA brief, Sprint 4 policies
Microcks 1.15.0 chart; microcks-cli image from quay.io/microcks/microcks-cli Microcks brief
Tools on your workstation sops, kubectl, helm, htpasswd, jq, Docker with buildx, Python 3.12

The new OpenTofu file reads module.eks, aws_kms_key.platform, aws_route53_zone.main and the pod_trust policy document from Part 1, and var.events_lb_hostname from Part 3. Set the shell variables this part uses:

export AWS_REGION=ap-south-1
export DOMAIN=coldchain.example.com
export EVIDENCE_BUCKET=coldchain-evidence-abc123          # the Object Lock bucket from Part 1
cd cold-chain/infra/foundation
export KMS_ARN=$(tofu output -raw kms_key_arn)
export PG_HOST=$(tofu output -raw pg_endpoint)
cd ../..
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
mkdir -p infra platform secrets contracts config lib partner evidence ops scripts db
kubectl --context hub get nodes
kubectl --context hub -n react get ksvc,cronjob

Expected: three Ready nodes and the Part 3 Knative services and the mkt CronJob in react. Add the repositories used below:

helm repo add external-secrets https://charts.external-secrets.io
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo add openfga https://openfga.github.io/helm-charts
helm repo add dex https://charts.dexidp.io
helm repo add microcks https://microcks.io/helm
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm repo update

Checkpoint 1. The exports print, the nodes are Ready, the Part 3 workloads are in react, and helm search repo openfga finds a chart.

2. Secrets have one home each: external-secrets and Secrets Manager

2.1 The AWS side

One OpenTofu file adds: the Secrets Manager entries (the database password’s home and OpenFGA’s key), four Pod Identity roles (external-secrets, Argo CD’s repo server for KMS decryption, an evidence writer and an evidence verifier), and one ACM certificate for the two new public names, login and api. The evidence writer can add and read objects under custody/ and nothing else: it has no delete right and no right to bypass a retention lock. The verifier can only read.

# infra/foundation/access.tf
variable "evidence_bucket" { type = string }   # the Object Lock bucket from Part 1, for example coldchain-evidence-abc123

# --- Secrets Manager: values that pods read through external-secrets --------------------
resource "aws_secretsmanager_secret" "pg_app" {
  name       = "${var.name}/pg-app"
  kms_key_id = aws_kms_key.platform.arn
}

resource "random_password" "fga_key" {
  length  = 40
  special = false
}

resource "aws_secretsmanager_secret" "fga_key" {
  name       = "${var.name}/fga-key"
  kms_key_id = aws_kms_key.platform.arn
}

resource "aws_secretsmanager_secret_version" "fga_key" {
  secret_id     = aws_secretsmanager_secret.fga_key.id
  secret_string = random_password.fga_key.result
}

# --- Pod Identity roles ------------------------------------------------------------------
locals {
  sm_arns = [
    "arn:aws:secretsmanager:${var.region}:${data.aws_caller_identity.me.account_id}:secret:AmazonMSK_${var.name}_*",
    "arn:aws:secretsmanager:${var.region}:${data.aws_caller_identity.me.account_id}:secret:${var.name}/*",
  ]
}
data "aws_caller_identity" "me" {}

resource "aws_iam_role" "eso" {
  name               = "${var.name}-external-secrets"
  assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "eso" {
  role = aws_iam_role.eso.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Effect = "Allow", Action = ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"], Resource = local.sm_arns },
      { Effect = "Allow", Action = ["kms:Decrypt"], Resource = aws_kms_key.platform.arn },
    ]
  })
}
resource "aws_eks_pod_identity_association" "eso" {
  cluster_name    = module.eks.cluster_name
  namespace       = "external-secrets"
  service_account = "external-secrets"
  role_arn        = aws_iam_role.eso.arn
}

# Argo CD's repo server decrypts SOPS files with the platform key. It can decrypt; it cannot encrypt or manage the key.
resource "aws_iam_role" "argocd_sops" {
  name               = "${var.name}-argocd-sops"
  assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "argocd_sops" {
  role = aws_iam_role.argocd_sops.id
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{ Effect = "Allow", Action = ["kms:Decrypt", "kms:DescribeKey"], Resource = aws_kms_key.platform.arn }]
  })
}
resource "aws_eks_pod_identity_association" "argocd_sops" {
  cluster_name    = module.eks.cluster_name
  namespace       = "argocd"
  service_account = "argocd-repo-server"
  role_arn        = aws_iam_role.argocd_sops.arn
}

# Evidence: the writer can add objects and read them; it has no delete and no bypass of retention. The verifier only reads.
resource "aws_iam_role" "evidence_writer" {
  name               = "${var.name}-evidence-writer"
  assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "evidence_writer" {
  role = aws_iam_role.evidence_writer.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Effect = "Allow", Action = ["s3:PutObject", "s3:GetObject", "s3:GetObjectVersion"], Resource = "arn:aws:s3:::${var.evidence_bucket}/custody/*" },
      { Effect = "Allow", Action = ["kms:GenerateDataKey", "kms:Decrypt"], Resource = aws_kms_key.platform.arn },
    ]
  })
}
resource "aws_eks_pod_identity_association" "evidence_writer" {
  cluster_name    = module.eks.cluster_name
  namespace       = "react"
  service_account = "evidence-sync"
  role_arn        = aws_iam_role.evidence_writer.arn
}

resource "aws_iam_role" "evidence_verifier" {
  name               = "${var.name}-evidence-verifier"
  assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "evidence_verifier" {
  role = aws_iam_role.evidence_verifier.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      { Effect = "Allow", Action = ["s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectRetention"], Resource = "arn:aws:s3:::${var.evidence_bucket}/custody/*" },
      { Effect = "Allow", Action = ["s3:ListBucket"], Resource = "arn:aws:s3:::${var.evidence_bucket}" },
      { Effect = "Allow", Action = ["kms:Decrypt"], Resource = aws_kms_key.platform.arn },
    ]
  })
}
resource "aws_eks_pod_identity_association" "evidence_verifier" {
  cluster_name    = module.eks.cluster_name
  namespace       = "react"
  service_account = "evidence-verify"
  role_arn        = aws_iam_role.evidence_verifier.arn
}

# --- Public names: login (Dex) and api (partner API) share one certificate -------------------
resource "aws_acm_certificate" "access" {
  domain_name               = "login.${var.domain}"
  subject_alternative_names = ["api.${var.domain}"]
  validation_method         = "DNS"
}
resource "aws_route53_record" "access_validation" {
  for_each = { for o in aws_acm_certificate.access.domain_validation_options : o.domain_name => o }
  zone_id  = aws_route53_zone.main.zone_id
  name     = each.value.resource_record_name
  type     = each.value.resource_record_type
  records  = [each.value.resource_record_value]
  ttl      = 60
  allow_overwrite = true
}
resource "aws_acm_certificate_validation" "access" {
  certificate_arn         = aws_acm_certificate.access.arn
  validation_record_fqdns = [for r in aws_route53_record.access_validation : r.fqdn]
}
resource "aws_route53_record" "access" {
  for_each = var.events_lb_hostname == "" ? toset([]) : toset(["login", "api"])
  zone_id  = aws_route53_zone.main.zone_id
  name     = "${each.key}.${var.domain}"
  type     = "CNAME"
  ttl      = 60
  records  = [var.events_lb_hostname]
}

output "access_cert_arn"   { value = aws_acm_certificate_validation.access.certificate_arn }
output "pg_app_secret_id"  { value = aws_secretsmanager_secret.pg_app.id }
cp infra/access.tf infra/foundation/access.tf
export TF_VAR_events_lb_hostname=$(kubectl --context hub -n kourier-system get svc kourier-public -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
(cd infra/foundation && tofu apply -var "evidence_bucket=$EVIDENCE_BUCKET")
export ACCESS_CERT=$(cd infra/foundation && tofu output -raw access_cert_arn)
aws eks list-pod-identity-associations --cluster-name coldchain-hub --query 'associations[].[namespace,serviceAccount]' --output text

Keep TF_VAR_events_lb_hostname set on every apply from now on: the Route 53 record for events is created only while the variable has a value, and an apply without it would remove that record. Expected: the apply succeeds (the certificate waits for its DNS validation record, a few minutes), and the association list now includes external-secrets, argocd-repo-server, evidence-sync and evidence-verify.

2.2 Move the database password into Secrets Manager

The password for the coldchain_app role was generated in Part 3 and sits in the Kubernetes Secret pg-app. Put the same value in Secrets Manager. Nothing in PostgreSQL changes, so nothing can break:

aws secretsmanager put-secret-value --secret-id coldchain/pg-app --secret-string \
  "$(kubectl --context hub -n react get secret pg-app -o jsonpath='{.data.password}' | base64 -d)"
aws secretsmanager describe-secret --secret-id coldchain/pg-app --query '[Name,KmsKeyId]' --output text

Rotation later means changing it in two places in order: ALTER ROLE coldchain_app PASSWORD ... as the administrator, then put-secret-value. Within the refresh interval (15 minutes here) the Kubernetes Secret follows, and pods pick the new value up at their next start. Automating that is Secrets Manager’s rotation feature with a Lambda, which this build does not use.

2.3 Install external-secrets and the store

helm upgrade --install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace --wait
kubectl --context hub -n external-secrets get pods
kubectl --context hub create namespace access

The store has no auth block on purpose: the controller then uses the AWS credentials its pod receives, which is the role from the Pod Identity association, so no access key exists anywhere. The ExternalSecrets that follow use creationPolicy: Merge for Secrets that Parts 2 and 3 already created by hand: external-secrets fills the named keys of the existing Secret instead of replacing it, so there is no moment without a Secret, and keys like host and bootstrap stay.

# platform/external-secrets.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: { name: aws-sm }
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-south-1              # no auth block: the controller uses the role from its EKS Pod Identity association
---
# Kafka credentials, one secret in Secrets Manager (AmazonMSK_coldchain_ingest, keys username and password), three shapes.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: msk-scram, namespace: ingest }
spec:
  refreshInterval: 15m
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target:
    name: msk-scram
    creationPolicy: Merge             # the hand-made Secret from Part 2 already exists; take it over without a gap
  data:
    - { secretKey: username, remoteRef: { key: AmazonMSK_coldchain_ingest, property: username } }
    - { secretKey: password, remoteRef: { key: AmazonMSK_coldchain_ingest, property: password } }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: msk-kn, namespace: react }
spec:
  refreshInterval: 15m
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target: { name: msk-kn, creationPolicy: Merge }
  data:
    - { secretKey: user, remoteRef: { key: AmazonMSK_coldchain_ingest, property: username } }
    - { secretKey: password, remoteRef: { key: AmazonMSK_coldchain_ingest, property: password } }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: msk-broker-auth, namespace: knative-eventing }
spec:
  refreshInterval: 15m
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target: { name: msk-broker-auth, creationPolicy: Merge }
  data:
    - { secretKey: user, remoteRef: { key: AmazonMSK_coldchain_ingest, property: username } }
    - { secretKey: password, remoteRef: { key: AmazonMSK_coldchain_ingest, property: password } }
---
# The application's database password. The Secret in Secrets Manager holds the same value as the role in PostgreSQL.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-app, namespace: react }
spec:
  refreshInterval: 15m
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target: { name: pg-app, creationPolicy: Merge }
  data:
    - { secretKey: password, remoteRef: { key: coldchain/pg-app } }
---
# OpenFGA's pre-shared key, for the server and for its clients.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: fga-key, namespace: access }
spec:
  refreshInterval: 15m
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target: { name: fga-keys, creationPolicy: Owner, template: { data: { keys: "{{ .key }}" } } }
  data:
    - { secretKey: key, remoteRef: { key: coldchain/fga-key } }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: fga-key, namespace: react }
spec:
  refreshInterval: 15m
  secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
  target: { name: fga-token, creationPolicy: Owner, template: { data: { token: "{{ .key }}" } } }
  data:
    - { secretKey: key, remoteRef: { key: coldchain/fga-key } }
kubectl --context hub apply -f platform/external-secrets.yaml
kubectl --context hub get clustersecretstore aws-sm
kubectl --context hub get externalsecret -A

Expected: the store Valid and Ready, and each ExternalSecret SecretSynced with status Ready. Prove the pipeline by changing a value in Secrets Manager and watching it arrive without touching the cluster:

kubectl --context hub -n react annotate externalsecret pg-app force-sync=$(date +%s) --overwrite
kubectl --context hub -n react get secret pg-app -o jsonpath='{.data}' | jq 'keys'
kubectl --context hub -n access get secret fga-keys -o jsonpath='{.data.keys}' | base64 -d | wc -c

Expected: pg-app still has host and password, and fga-keys holds a 40-character value. Restart the consumers once so they read the Secrets from the store (kubectl -n ingest rollout restart deploy/vector, and the Knative services in react get a new revision on their next deploy).

Checkpoint 2. The store is valid, every ExternalSecret is synced, and the same password is in Secrets Manager and in pg-app.

3. SOPS with KMS for values that must live in Git, and Argo CD

Some values arrive from outside and have no AWS home: the client ID and secret that a carrier’s identity provider gives us for the Dex connector, and the secret of Dex’s own API client. They belong in Git, because Git is how this platform is deployed, but encrypted. SOPS encrypts only the values of the file, so a pull request still shows which keys changed. The rule file says which key encrypts what:

# .sops.yaml
creation_rules:
  - path_regex: secrets/.*\.enc\.yaml$
    encrypted_regex: '^(data|stringData)$'     # keys and metadata stay readable in review; only values are encrypted
    kms: 'KMS_ARN'
sed -i "s#KMS_ARN#$KMS_ARN#" .sops.yaml

Write the plain file once, encrypt it, and delete the plain copy. Generate the one value that is ours (API_CLIENT_SECRET) now; the partner values stay as placeholders until a partner gives you real ones:

# secrets/dex-secrets.plain.yaml
# Encrypt this file, then delete the plain copy:  sops --encrypt secrets/dex-secrets.plain.yaml > secrets/dex-secrets.enc.yaml
apiVersion: v1
kind: Secret
metadata: { name: dex-secrets, namespace: access }
type: Opaque
stringData:
  API_CLIENT_SECRET: REPLACE_WITH_RANDOM_32_CHARS
  CARRIER_A_CLIENT_ID: REPLACE_WITH_VALUE_FROM_THE_PARTNER
  CARRIER_A_CLIENT_SECRET: REPLACE_WITH_VALUE_FROM_THE_PARTNER
  CUSTOMER_PUNE_CLIENT_ID: REPLACE_WITH_VALUE_FROM_THE_PARTNER
  CUSTOMER_PUNE_CLIENT_SECRET: REPLACE_WITH_VALUE_FROM_THE_PARTNER
export API_CLIENT_SECRET=$(openssl rand -hex 24)
sed -i "s#REPLACE_WITH_RANDOM_32_CHARS#$API_CLIENT_SECRET#" secrets/dex-secrets.plain.yaml
sops --encrypt secrets/dex-secrets.plain.yaml > secrets/dex-secrets.enc.yaml
shred -u secrets/dex-secrets.plain.yaml 2>/dev/null || rm -f secrets/dex-secrets.plain.yaml
head -12 secrets/dex-secrets.enc.yaml
sops --decrypt secrets/dex-secrets.enc.yaml | grep -c REPLACE_WITH

Expected: the encrypted file shows stringData keys with ENC[AES256_GCM,...] values and a sops: block listing the KMS ARN; the decrypt prints 4 (the four placeholders). Check that SOPS protects integrity as well as secrecy: change one character inside an ENC[...] value and decrypt again. SOPS refuses with a MAC error.

3.1 Argo CD decrypts it

Argo CD renders manifests with kustomize in its repo server. KSOPS is a kustomize plugin that calls SOPS during the render. The repo server needs the plugin binary, kustomize options that allow exec plugins, and rights on the KMS key (the Pod Identity association from section 2.1 gives kms:Decrypt only).

# platform/argocd-ksops-values.yaml
# Adds the KSOPS kustomize plugin to Argo CD's repo server. Pick KSOPS_TAG from the project's releases and read its README
# for the exact init-container recipe of that tag. Apply with:
#   helm upgrade argocd argo/argo-cd -n argocd --reuse-values -f platform/argocd-ksops-values.yaml
configs:
  cm:
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
repoServer:
  env:
    - { name: XDG_CONFIG_HOME, value: /.config }
    - { name: KUSTOMIZE_PLUGIN_HOME, value: /.config/kustomize/plugin }
    - { name: AWS_REGION, value: ap-south-1 }          # the role comes from the Pod Identity association
  volumes:
    - { name: custom-tools, emptyDir: {} }
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:KSOPS_TAG
      command: ["/bin/sh", "-c"]
      args: ["mv ksops /custom-tools/ && mv $GOPATH/bin/kustomize /custom-tools/"]
      volumeMounts: [{ mountPath: /custom-tools, name: custom-tools }]
  volumeMounts:
    - { mountPath: /usr/local/bin/kustomize, name: custom-tools, subPath: kustomize }
    - { mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops, name: custom-tools, subPath: ksops }
export KSOPS_TAG=v4.3.2          # replace with the tag you chose from the project's releases
sed "s#KSOPS_TAG#$KSOPS_TAG#" platform/argocd-ksops-values.yaml > /tmp/ksops-values.yaml
helm upgrade argocd argo/argo-cd -n argocd --reuse-values -f /tmp/ksops-values.yaml --kube-context hub --wait
kubectl --context hub -n argocd rollout status deploy/argocd-repo-server

The generator and kustomization tell kustomize which encrypted file to decrypt; the Application points Argo CD at the folder:

# secrets/ksops-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: coldchain-secrets
  annotations: { config.kubernetes.io/function: "exec:\n  path: ksops\n" }
files:
  - ./dex-secrets.enc.yaml
# secrets/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
  - ksops-generator.yaml
# gitops/apps/secrets.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: secrets, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR-ORG/cold-chain.git
    targetRevision: main
    path: secrets
  destination: { server: https://kubernetes.default.svc, namespace: access }
  syncPolicy: { automated: { prune: true, selfHeal: true } }
cp platform/argocd-app-secrets.yaml gitops/apps/secrets.yaml
git add .sops.yaml secrets gitops/apps/secrets.yaml && git commit -m "Dex secrets, SOPS-encrypted" && git push
kubectl --context hub -n argocd get application secrets
kubectl --context hub -n access get secret dex-secrets -o jsonpath='{.data}' | jq 'keys'

Expected: the Application Synced and Healthy, and the Secret dex-secrets with its five keys. Argo CD wrote the plain Secret into the cluster, so keep Kubernetes encryption at rest and RBAC on Secrets in mind: SOPS protects Git, not etcd. If the Application shows a plugin or kustomize error instead, kubectl -n argocd logs deploy/argocd-repo-server says whether it is the plugin path, the build options or the KMS call.

Checkpoint 3. The file in Git has only encrypted values, a changed value fails the integrity check, and dex-secrets appears in the cluster through Argo CD.

4. Gatekeeper: only our images in react

The Sprint 4 project used Gatekeeper and a registry allow-list the same way. Install it if the hub does not have it, then add a template that allows an image only when its name starts with an allowed prefix.

helm upgrade --install gatekeeper gatekeeper/gatekeeper -n gatekeeper-system --create-namespace --kube-context hub --wait
kubectl --context hub -n gatekeeper-system get pods

The allow-list is short and explicit. Knative adds a sidecar from gcr.io/knative-releases/, and the one-off psql pods in this series use the pinned postgres:17-alpine. Everything else in react must come from our ECR repositories. A default memory limit for containers goes in the same file as a LimitRange, because the resource rule in the Overview is simpler as a namespace default than as a policy. Start in dry run:

# platform/gatekeeper.yaml
# Same style as the Sprint 4 policies (see Sprint 4 project, Part 4, section 6). This one allows images from our ECR only.
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata: { name: coldallowedregistry }
spec:
  crd:
    spec:
      names: { kind: ColdAllowedRegistry }
      validation:
        openAPIV3Schema:
          type: object
          properties:
            prefixes:
              type: array
              items: { type: string }
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package coldallowedregistry

        containers[c] { c := input.review.object.spec.containers[_] }
        containers[c] { c := input.review.object.spec.initContainers[_] }

        allowed(img) { startswith(img, input.parameters.prefixes[_]) }

        violation[{"msg": msg}] {
          c := containers[_]
          not allowed(c.image)
          msg := sprintf("image %v is not on the allow-list for this namespace", [c.image])
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: ColdAllowedRegistry
metadata: { name: react-images-from-ecr }
spec:
  enforcementAction: dryrun           # change to deny after the audit shows no violations
  match:
    kinds: [{ apiGroups: [""], kinds: ["Pod"] }]
    namespaces: [react]
  parameters:
    prefixes:
      - ACCOUNT.dkr.ecr.ap-south-1.amazonaws.com/coldchain/
      - gcr.io/knative-releases/         # Knative's queue-proxy sidecar
      - postgres:17-alpine               # the one-off psql pods in this series
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: react }
spec:
  limits:
    - type: Container
      default: { memory: 256Mi }
      defaultRequest: { cpu: 50m, memory: 64Mi }
sed "s#ACCOUNT#$ACCOUNT#" platform/gatekeeper.yaml | kubectl --context hub apply -f -
sleep 60
kubectl --context hub get coldallowedregistry react-images-from-ecr -o jsonpath='{.status.totalViolations}{"\n"}'
kubectl --context hub get coldallowedregistry react-images-from-ecr -o jsonpath='{range .status.violations[*]}{.name}: {.message}{"\n"}{end}'

Expected: a violation count and, for each violation, the pod and the image. Fix every real violation before you enforce: an image you built moves to the ECR path; an upstream image you need is added to the list on purpose. When the list is empty, switch to deny and try a pod from outside:

kubectl --context hub patch coldallowedregistry react-images-from-ecr --type merge -p '{"spec":{"enforcementAction":"deny"}}'
sleep 10
kubectl --context hub -n react run bad --image=nginx:1.27 --restart=Never

Expected: the last command is refused by the admission webhook with the message that the image is not on the allow-list for this namespace. A pod using postgres:17-alpine or one of our digests is admitted.

Checkpoint 4. The dry-run list is empty, the constraint is deny, and an image from outside is refused.

5. OpenFGA on CloudNativePG: the model and the tuples

OpenFGA stores its tuples in PostgreSQL. The business data is in RDS (Part 1), but OpenFGA’s data is a small, separate, operator-managed database, so a CNCF project keeps a CNCF project’s data and the two failure domains stay apart. Install the operator and create the cluster:

helm upgrade --install cnpg cnpg/cloudnative-pg -n cnpg-system --create-namespace --kube-context hub --wait
# platform/cnpg.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: fga-pg, namespace: access }
spec:
  instances: 2                          # one primary and one replica, spread by the operator's defaults
  storage: { size: 5Gi, storageClass: gp3 }
  bootstrap:
    initdb: { database: app, owner: app }
  resources: { requests: { cpu: 100m, memory: 256Mi }, limits: { memory: 512Mi } }
kubectl --context hub apply -f platform/cnpg.yaml
kubectl --context hub -n access wait --for=condition=Ready cluster/fga-pg --timeout=300s
kubectl --context hub -n access get secret fga-pg-app -o jsonpath='{.data}' | jq 'keys'

Expected: the cluster Ready with a primary and a replica, and a Secret fga-pg-app that has a uri key (and others). Backups of this database and the restore test are in Part 5.

5.1 The server

The chart runs the schema migration as a Job before the server starts, reads the database address from fga-pg-app, and requires the pre-shared key from the Secret that external-secrets filled in section 2:

# platform/openfga-values.yaml
replicaCount: 2
datastore:
  engine: postgres
  uriSecret: fga-pg-app                 # CloudNativePG's application secret has a key called uri
  migrations:
    enabled: true                       # a Job runs the schema migration before the server starts
authn:
  method: preshared
  preshared:
    keysSecret: fga-keys                # key: keys
resources: { requests: { cpu: 100m, memory: 128Mi }, limits: { memory: 256Mi } }
helm upgrade --install openfga openfga/openfga -n access -f platform/openfga-values.yaml --kube-context hub --wait
kubectl --context hub -n access get pods,jobs

Expected: the migration Job Complete and two OpenFGA pods Running.

5.2 The model

The model has four types. A person is a user, identified by verified email. A carrier and a customer each have members. A shipment has a carrier and a customer, and can_view is true for any member of either. The text form is for reading, the JSON form is what the HTTP API takes:

# config/model.fga
model
  schema 1.1

type user

type carrier
  relations
    define member: [user]

type customer
  relations
    define member: [user]

type shipment
  relations
    define carrier: [carrier]
    define customer: [customer]
    define can_view: member from carrier or member from customer
{
  "schema_version": "1.1",
  "type_definitions": [
    { "type": "user" },
    { "type": "carrier",
      "relations": { "member": { "this": {} } },
      "metadata": { "relations": { "member": { "directly_related_user_types": [ { "type": "user" } ] } } } },
    { "type": "customer",
      "relations": { "member": { "this": {} } },
      "metadata": { "relations": { "member": { "directly_related_user_types": [ { "type": "user" } ] } } } },
    { "type": "shipment",
      "relations": {
        "carrier":  { "this": {} },
        "customer": { "this": {} },
        "can_view": { "union": { "child": [
          { "tupleToUserset": { "tupleset": { "relation": "carrier" },  "computedUserset": { "relation": "member" } } },
          { "tupleToUserset": { "tupleset": { "relation": "customer" }, "computedUserset": { "relation": "member" } } } ] } } },
      "metadata": { "relations": {
        "carrier":  { "directly_related_user_types": [ { "type": "carrier" } ] },
        "customer": { "directly_related_user_types": [ { "type": "customer" } ] },
        "can_view": { "directly_related_user_types": [] } } } }
  ]
}

The membership data (who belongs to which carrier or customer) is a small file in Git. The shipment-to-organisation links come from RDS: the carrier from shipments.carrier, the customer from the destination through a mapping file.

{
  "carrier:carrier-a": ["anita@carrier-a.example"],
  "carrier:carrier-b": ["bala@carrier-b.example"],
  "customer:customer-pune": ["chitra@pune.example"]
}
{ "Pune": "customer-pune" }

The loader creates the store once and writes the model. A new model version never changes the old one, so the printed FGA_MODEL_ID pins exactly the rules the API was tested with:

# ops/fga_load.py
#!/usr/bin/env python3
"""Create the OpenFGA store (once) and write the model. Prints FGA_STORE_ID and FGA_MODEL_ID.

  kubectl -n access port-forward svc/openfga 8080:8080 &
  FGA_URL=http://localhost:8080 FGA_TOKEN=$KEY python3 ops/fga_load.py config/model.json"""
import json, os, sys, urllib.request

url, token = os.environ.get("FGA_URL", "http://localhost:8080").rstrip("/"), os.environ.get("FGA_TOKEN", "")


def call(method, path, body=None):
    req = urllib.request.Request(url + path, method=method, data=json.dumps(body).encode() if body is not None else None,
                                 headers={"content-type": "application/json", **({"Authorization": "Bearer " + token} if token else {})})
    with urllib.request.urlopen(req, timeout=10) as r:
        return json.loads(r.read() or b"{}")


stores = call("GET", "/stores").get("stores", [])
store = next((s for s in stores if s["name"] == "coldchain"), None) or call("POST", "/stores", {"name": "coldchain"})
sid = store["id"]
model = json.load(open(sys.argv[1] if len(sys.argv) > 1 else "config/model.json"))
mid = call("POST", f"/stores/{sid}/authorization-models", model)["authorization_model_id"]
print(f"FGA_STORE_ID={sid}")
print(f"FGA_MODEL_ID={mid}")
export FGA_TOKEN=$(aws secretsmanager get-secret-value --secret-id coldchain/fga-key --query SecretString --output text)
kubectl --context hub -n access port-forward svc/openfga 8080:8080 &
sleep 3
FGA_URL=http://localhost:8080 python3 ops/fga_load.py config/model.json | tee /tmp/fga-ids.txt
export FGA_STORE_ID=$(grep STORE /tmp/fga-ids.txt | cut -d= -f2); export FGA_MODEL_ID=$(grep MODEL /tmp/fga-ids.txt | cut -d= -f2)

Expected: two lines, FGA_STORE_ID=... and FGA_MODEL_ID=.... If the server answers 401, the key in FGA_TOKEN does not match the one in the fga-keys Secret.

5.3 The tuples, from RDS

The partner API’s own image also contains the tuple sync (it is a small program that makes OpenFGA’s tuples equal to what RDS and the membership files say, adding what is missing and removing what is gone). It runs every five minutes as a CronJob, so a new shipment or a membership change reaches OpenFGA within that time. First the code, all of it:

# lib/fgalib.py
"""Small OpenFGA HTTP client (standard library only)."""
import json, os, urllib.request, urllib.error


class FgaError(Exception):
    pass


class Fga:
    def __init__(self, url=None, store_id=None, model_id=None, token=None, timeout=3):
        self.url = (url or os.environ.get("FGA_URL", "http://openfga.access.svc.cluster.local:8080")).rstrip("/")
        self.store_id = store_id or os.environ.get("FGA_STORE_ID", "")
        self.model_id = model_id or os.environ.get("FGA_MODEL_ID", "")
        self.token = token or os.environ.get("FGA_TOKEN", "")
        self.timeout = timeout

    def _post(self, path, body):
        req = urllib.request.Request(self.url + path, data=json.dumps(body).encode(), method="POST",
                                     headers={"content-type": "application/json"})
        if self.token:
            req.add_header("Authorization", "Bearer " + self.token)
        try:
            with urllib.request.urlopen(req, timeout=self.timeout) as r:
                raw = r.read()
                return json.loads(raw) if raw else {}
        except urllib.error.HTTPError as e:
            raise FgaError(f"{path}: {e.code} {e.read()[:200]!r}")
        except OSError as e:
            raise FgaError(f"{path}: {e}")

    def check(self, user, relation, obj):
        body = {"tuple_key": {"user": user, "relation": relation, "object": obj}}
        if self.model_id:
            body["authorization_model_id"] = self.model_id
        return bool(self._post(f"/stores/{self.store_id}/check", body).get("allowed"))

    def list_objects(self, user, relation, typ):
        body = {"user": user, "relation": relation, "type": typ}
        if self.model_id:
            body["authorization_model_id"] = self.model_id
        return self._post(f"/stores/{self.store_id}/list-objects", body).get("objects", [])

    def read_all(self, typ):
        """All tuples whose object is of this type, as a set of (user, relation, object)."""
        out, tok = set(), ""
        while True:
            body = {"tuple_key": {"object": typ + ":"}, "page_size": 100}
            if tok:
                body["continuation_token"] = tok
            r = self._post(f"/stores/{self.store_id}/read", body)
            for t in r.get("tuples", []):
                k = t["key"]
                out.add((k["user"], k["relation"], k["object"]))
            tok = r.get("continuation_token", "")
            if not tok:
                return out

    def write(self, writes=(), deletes=()):
        writes, deletes = list(writes), list(deletes)
        for i in range(0, max(len(writes), len(deletes)), 50):
            body = {}
            w, d = writes[i:i + 50], deletes[i:i + 50]
            if w:
                body["writes"] = {"tuple_keys": [{"user": u, "relation": r, "object": o} for u, r, o in w]}
            if d:
                body["deletes"] = {"tuple_keys": [{"user": u, "relation": r, "object": o} for u, r, o in d]}
            if self.model_id:
                body["authorization_model_id"] = self.model_id
            self._post(f"/stores/{self.store_id}/write", body)
# lib/pg.py
"""Thread-local PostgreSQL connection from the PG* environment variables."""
import threading
import psycopg

_local = threading.local()


def db():
    c = getattr(_local, "conn", None)
    if c is None or c.closed:
        c = psycopg.connect(autocommit=True, connect_timeout=5)
        _local.conn = c
    return c
# ops/sync_tuples.py
#!/usr/bin/env python3
"""Make OpenFGA's tuples match RDS and the membership files. Safe to run repeatedly; prints what it changed.

Tuples written:  user:<email> member carrier:<id> | customer:<id>      (from config/members.json)
                 carrier:<id> carrier shipment:<id>                     (from shipments.carrier)
                 customer:<id> customer shipment:<id>                   (from shipments.destination and config/customers.json)"""
import json, sys


def desired(shipments, members, customers):
    want = set()
    for obj, emails in members.items():
        kind = obj.split(":")[0]
        want |= {("user:" + e.lower(), "member", obj) for e in emails}
    for sid, carrier, destination in shipments:
        if carrier:
            want.add((f"carrier:{carrier}", "carrier", f"shipment:{sid}"))
        if destination in customers:
            want.add((f"customer:{customers[destination]}", "customer", f"shipment:{sid}"))
    return want


def sync(fga, shipments, members, customers, out=print):
    want = desired(shipments, members, customers)
    have = set()
    for typ in ("carrier", "customer", "shipment"):
        have |= fga.read_all(typ)
    add, drop = sorted(want - have), sorted(have - want)
    if add or drop:
        fga.write(add, drop)
    out(f"tuples: {len(want)} wanted, {len(add)} added, {len(drop)} removed")
    return len(add), len(drop)


if __name__ == "__main__":
    import psycopg
    from lib.fgalib import Fga
    members, customers = json.load(open("config/members.json")), json.load(open("config/customers.json"))
    with psycopg.connect() as c, c.cursor() as cur:
        cur.execute("SELECT shipment_id, carrier, destination FROM shipments")
        rows = cur.fetchall()
    sync(Fga(), rows, members, customers)

RDS is private, so the sync cannot run from your workstation; it runs as a Job from the image built in section 7, and section 7.3 runs it once and checks the result against OpenFGA itself. The function that decides what to add and what to remove has an offline test, shown in section 7.1.

Checkpoint 5. OpenFGA is up behind its key, the model is loaded, and FGA_STORE_ID and FGA_MODEL_ID are set.

6. Dex: partner sign-in

Dex is the one sign-in front for partners. Partners keep their own identity providers; Dex is an OpenID Connect provider that, behind the scenes, signs the user in elsewhere and issues its own token. The partner API trusts only Dex’s keys, so adding a carrier means adding a connector, not changing the API.

6.1 Lab users and the server

For the build you need users that exist without a partner’s help. Dex has a local password database; the three lab users below have no groups, because permission is OpenFGA’s job. Make three bcrypt hashes of one lab password and put them in the values file:

export LAB_PW=$(openssl rand -base64 12 | tr -d '/+=')
H() { htpasswd -bnBC 10 "" "$LAB_PW" | tr -d ':\n' | sed 's/$2y/$2a/'; }
sed -e "s#DOMAIN#$DOMAIN#g" -e "s#HASH_ANITA#$(H)#" -e "s#HASH_BALA#$(H)#" -e "s#HASH_CHITRA#$(H)#" \
  platform/dex-values.yaml > /tmp/dex-values.yaml
echo "lab password for the three users: $LAB_PW"
# platform/dex-values.yaml
# Dex: one sign-in front for partners. The lab users below prove the flow; real partners come in through connectors
# (platform/dex-federation.yaml). Replace DOMAIN and the three password hashes before applying.
replicaCount: 2
envFrom:
  - secretRef: { name: dex-secrets }     # from the SOPS file in Git: API_CLIENT_SECRET and the partners' client secrets
config:
  issuer: https://login.DOMAIN
  storage:
    type: kubernetes                     # Dex keeps its state in custom resources; no database
    config: { inCluster: true }
  web: { http: 0.0.0.0:5556 }
  oauth2:
    passwordConnector: local             # lets the test script ask for a token without a browser
    skipApprovalScreen: true
  enablePasswordDB: true
  staticPasswords:
    - { email: anita@carrier-a.example, username: anita, userID: "10000000-0000-4000-8000-00000000000a", hash: "HASH_ANITA" }
    - { email: bala@carrier-b.example,  username: bala,  userID: "10000000-0000-4000-8000-00000000000b", hash: "HASH_BALA" }
    - { email: chitra@pune.example,     username: chitra, userID: "10000000-0000-4000-8000-00000000000c", hash: "HASH_CHITRA" }
  staticClients:
    - id: coldchain-partner-api
      name: Cold chain partner API
      secretEnv: API_CLIENT_SECRET
      redirectURIs: ["https://api.DOMAIN/callback"]
  expiry:
    idTokens: 15m                        # short: the API checks the token on every call
helm upgrade --install dex dex/dex -n access -f /tmp/dex-values.yaml --kube-context hub --wait
kubectl --context hub -n access get pods -l app.kubernetes.io/name=dex
kubectl --context hub -n access logs deploy/dex --tail=5

Expected: two Dex pods Running and a log line that the server is listening on 0.0.0.0:5556. If the pods do not start, kubectl describe usually points at the missing dex-secrets Secret from section 3. Dex keeps its state in custom resources of the cluster (the kubernetes storage type), so there is no database to run.

6.2 A public name for Dex and for the API

Partners reach Dex and the API from the internet over HTTPS, through the load balancer built in Part 3. That load balancer terminates TLS with the certificate for events.<domain>. Give it the second certificate too (a list of certificate ARNs on the annotation: the first is the default, the others are chosen by the name the client asks for), publish the two names in Route 53, and map them with Knative:

# platform/access-public.yaml
# Two more public names on the load balancer from Part 3: login.DOMAIN (Dex) and api.DOMAIN (partner API).
apiVersion: networking.internal.knative.dev/v1alpha1
kind: ClusterDomainClaim
metadata: { name: login.DOMAIN }
spec: { namespace: access }
---
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata: { name: login.DOMAIN, namespace: access }
spec:
  ref: { apiVersion: v1, kind: Service, name: dex }      # Dex is a plain Deployment, so the mapping points at its Service
---
apiVersion: networking.internal.knative.dev/v1alpha1
kind: ClusterDomainClaim
metadata: { name: api.DOMAIN }
spec: { namespace: react }
---
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata: { name: api.DOMAIN, namespace: react }
spec:
  ref: { apiVersion: serving.knative.dev/v1, kind: Service, name: partner-api }
export EVENTS_CERT=$(cd infra/foundation && tofu output -raw events_cert_arn)
kubectl --context hub -n kourier-system annotate svc kourier-public --overwrite \
  service.beta.kubernetes.io/aws-load-balancer-ssl-cert="$EVENTS_CERT,$ACCESS_CERT"
sed "s#DOMAIN#$DOMAIN#g" platform/access-public.yaml | kubectl --context hub apply -f -
dig +short login.$DOMAIN api.$DOMAIN
curl -s https://login.$DOMAIN/.well-known/openid-configuration | jq '{issuer, jwks_uri, token_endpoint}'

Expected: both names resolve to the load balancer, and the discovery document shows "issuer": "https://login.<domain>". The dig answer exists because section 2.1 ran with TF_VAR_events_lb_hostname set, which created the two CNAMEs. If the discovery call returns a Knative 404, the mapping to Dex’s Service has not become ready: kubectl -n access get domainmapping shows why.

6.3 Get a token

Ask for an ID token the way the test script will, with the password grant (this is a convenience of the lab; real partners use the browser flow):

export API_CLIENT_SECRET=$(kubectl --context hub -n access get secret dex-secrets -o jsonpath='{.data.API_CLIENT_SECRET}' | base64 -d)
curl -s -u "coldchain-partner-api:$API_CLIENT_SECRET" https://login.$DOMAIN/token \
  -d grant_type=password -d username=anita@carrier-a.example -d "password=$LAB_PW" -d 'scope=openid email' \
  | jq -r .id_token | cut -d. -f2 | base64 -d 2>/dev/null | jq '{iss, aud, email, email_verified, exp}'

Expected: iss is https://login.<domain>, aud is coldchain-partner-api, email is the user’s address, email_verified is true, and exp is 15 minutes ahead. (The base64 -d may complain about padding on some systems; the JSON still prints.)

6.4 Federating a partner

To let a carrier’s own staff sign in, add an OpenID Connect connector. The partner gives you an issuer URL, a client ID and a client secret, and you give the partner your redirect URL https://login.<domain>/callback. The values go into the SOPS file from section 3 (the four placeholders), and the connector reads them from the environment:

# platform/dex-federation.yaml
# Partner identity providers, added as connectors. Each partner gives you an issuer URL, a client ID and a client secret.
# Apply only when the issuers are real: Dex contacts each issuer when it starts.
#   helm upgrade dex dex/dex -n access -f platform/dex-values.yaml -f platform/dex-federation.yaml
envFrom:
  - secretRef: { name: dex-secrets }
config:
  connectors:
    - type: oidc
      id: carrier-a
      name: Carrier A
      config:
        issuer: https://login.microsoftonline.com/TENANT_ID/v2.0     # example: a carrier's Microsoft Entra ID tenant
        clientID: $CARRIER_A_CLIENT_ID
        clientSecret: $CARRIER_A_CLIENT_SECRET
        redirectURI: https://login.DOMAIN/callback
        scopes: [openid, email, profile]
    - type: oidc
      id: customer-pune
      name: Pune hospital group
      config:
        issuer: https://PARTNER.okta.example.com                      # example: a customer's Okta
        clientID: $CUSTOMER_PUNE_CLIENT_ID
        clientSecret: $CUSTOMER_PUNE_CLIENT_SECRET
        redirectURI: https://login.DOMAIN/callback
        scopes: [openid, email, profile]
sops secrets/dex-secrets.enc.yaml            # edit the four partner values in your editor; SOPS re-encrypts on save
git commit -am "Carrier A and Pune connectors" && git push
sed -e "s#DOMAIN#$DOMAIN#g" platform/dex-federation.yaml > /tmp/dex-federation.yaml
helm upgrade dex dex/dex -n access -f /tmp/dex-values.yaml -f /tmp/dex-federation.yaml --kube-context hub --wait

Apply it only when the issuers are real: Dex contacts each issuer when it starts and will not come up with a placeholder. Then open https://login.<domain>/auth?client_id=coldchain-partner-api&response_type=code&scope=openid%20email&redirect_uri=https://api.<domain>/callback in a browser: Dex shows a button for each connector. The user’s email must match an entry in config/members.json, or the user signs in and sees nothing, which is the intended result for a stranger. Dex’s SAML connector carries a warning in its own documentation about being unmaintained, so use OIDC with partners wherever they can.

Checkpoint 6. The discovery document shows the right issuer, a token for a lab user has the expected claims, and (once you have a real partner) its connector signs a user in.

7. The partner API

The API is a small program with four read-only routes. The route code, the token check, the OpenFGA calls and the database access are separate pieces, so the logic is one function that is tested without a network.

# partner/app.py
#!/usr/bin/env python3
"""Partner API. Every request carries a Dex ID token; OpenFGA decides which shipments the caller may see."""
import json, os, re, sys, uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse

SHIPMENT = re.compile(r"^SHP-[0-9]{4,10}$")


class Denied(Exception):
    pass


class Unauthenticated(Exception):
    pass


class JwtVerifier:
    """Checks signature, issuer, audience and expiry against Dex's published keys."""
    def __init__(self, issuer, audience, jwks_url):
        import jwt
        self.jwt, self.issuer, self.audience = jwt, issuer, audience
        self.jwks = jwt.PyJWKClient(jwks_url, cache_keys=True)

    def __call__(self, token):
        try:
            key = self.jwks.get_signing_key_from_jwt(token).key
            return self.jwt.decode(token, key, algorithms=["RS256", "ES256"], issuer=self.issuer,
                                   audience=self.audience, options={"require": ["exp", "iss", "aud"]})
        except Exception as e:
            raise Unauthenticated(type(e).__name__)


class Pg:
    def __init__(self):
        from lib.pg import db
        self.db = db

    def shipments(self, ids):
        with self.db().cursor() as c:
            c.execute("""SELECT shipment_id, device_id, carrier, product, origin, destination, status,
                                start_time, end_time FROM shipments WHERE shipment_id = ANY(%s) ORDER BY shipment_id""", (ids,))
            cols = [d.name for d in c.description]
            return [dict(zip(cols, r)) for r in c.fetchall()]

    def cases(self, shipment_id):
        with self.db().cursor() as c:
            c.execute("""SELECT case_id, started_at, ended_at, max_temp_c, limit_c FROM excursion_cases
                         WHERE shipment_id = %s ORDER BY started_at""", (shipment_id,))
            cols = [d.name for d in c.description]
            return [dict(zip(cols, r)) for r in c.fetchall()]

    def deny(self, subject, action, obj):
        with self.db().cursor() as c:
            c.execute("INSERT INTO access_events (event_id, subject, action, object, allowed) VALUES (%s,%s,%s,%s,false)",
                      (str(uuid.uuid4()), subject, action, obj))


def clean(row):
    return {k: (v.isoformat() if hasattr(v, "isoformat") else (float(v) if hasattr(v, "as_tuple") else v)) for k, v in row.items()}


class Api:
    def __init__(self, verify, fga, pg):
        self.verify, self.fga, self.pg = verify, fga, pg

    def subject(self, authorization):
        if not authorization or not authorization.lower().startswith("bearer "):
            raise Unauthenticated("no bearer token")
        claims = self.verify(authorization[7:].strip())
        if not claims.get("email") or claims.get("email_verified") is not True:
            raise Unauthenticated("no verified email")
        return "user:" + claims["email"].lower()

    def handle(self, method, path, authorization):
        """Returns (status, body). Pure with respect to HTTP, so it is easy to test."""
        p = urlparse(path).path.rstrip("/") or "/"
        if p == "/healthz":
            return 200, {"ok": True}
        if method != "GET":
            return 405, {"error": "method not allowed"}
        try:
            who = self.subject(authorization)
        except Unauthenticated as e:
            return 401, {"error": "unauthenticated", "detail": str(e)}
        if p == "/v1/shipments":
            ids = [o.split(":", 1)[1] for o in self.fga.list_objects(who, "can_view", "shipment")]
            return 200, {"shipments": [clean(r) for r in self.pg.shipments(ids)] if ids else []}
        m = re.fullmatch(r"/v1/shipments/([^/]+)(/excursions)?", p)
        if not m or not SHIPMENT.match(m.group(1)):
            return 404, {"error": "not found"}
        sid, tail = m.group(1), m.group(2)
        if not self.fga.check(who, "can_view", "shipment:" + sid):
            self.pg.deny(who, "GET " + p, "shipment:" + sid)
            return 403, {"error": "forbidden"}      # same answer whether or not the shipment exists
        rows = self.pg.shipments([sid])
        if not rows:
            return 404, {"error": "not found"}
        if tail:
            return 200, {"shipment_id": sid, "excursions": [clean(r) for r in self.pg.cases(sid)]}
        return 200, clean(rows[0])


def serve(api, port):
    class H(BaseHTTPRequestHandler):
        def do_GET(self):
            try:
                status, body = api.handle("GET", self.path, self.headers.get("authorization"))
            except Exception as e:                  # a dependency failed: say so, never allow by default
                print("error", type(e).__name__, e, file=sys.stderr)
                status, body = 503, {"error": "unavailable"}
            b = json.dumps(body).encode()
            self.send_response(status); self.send_header("content-type", "application/json")
            self.send_header("content-length", str(len(b))); self.end_headers(); self.wfile.write(b)

        def log_message(self, *a):
            pass
    ThreadingHTTPServer(("0.0.0.0", port), H).serve_forever()


if __name__ == "__main__":
    from lib.fgalib import Fga
    issuer = os.environ["OIDC_ISSUER"]
    verify = JwtVerifier(issuer, os.environ["OIDC_AUDIENCE"],
                         os.environ.get("OIDC_JWKS", "http://dex.access.svc.cluster.local:5556/keys"))
    serve(Api(verify, Fga(), Pg()), int(os.environ.get("PORT", "8080")))

Read Api.handle first. For a shipment it does three things in order: verify the token, ask OpenFGA, then read the database. The answer to “may this person see this shipment” is never taken from the token and never defaulted to yes. A denial is written to access_events, which is one of the three kinds of custody record in section 9.

7.1 The offline test

The test replaces OpenFGA with a function that applies the model’s rule to a set of tuples, and the database with a stub. It covers the token cases (none, garbage, unverified email), both organisations, the customer, a shipment that does not exist, a list per user, and the denial record:

# partner/test_api.py
import sys; sys.path.insert(0, "/tmp/p4")
from partner.app import Api, Unauthenticated

# Tuples as ops/sync_tuples.py would write them for this model: shipment -> carrier / customer -> members.
T = {("carrier:carrier-a", "carrier", "shipment:SHP-1001"), ("carrier:carrier-b", "carrier", "shipment:SHP-1002"),
     ("carrier:carrier-a", "carrier", "shipment:SHP-1003"), ("customer:customer-pune", "customer", "shipment:SHP-1001"),
     ("user:anita@carrier-a.example", "member", "carrier:carrier-a"), ("user:bala@carrier-b.example", "member", "carrier:carrier-b"),
     ("user:chitra@pune.example", "member", "customer:customer-pune")}


class FakeFga:
    """Evaluates can_view = member from carrier or member from customer, as the model says."""
    def check(self, user, rel, obj):
        assert rel == "can_view"
        for u, r, o in T:
            if o == obj and r in ("carrier", "customer") and (user, "member", u) in T:
                return True
        return False
    def list_objects(self, user, rel, typ):
        return sorted({o for _, _, o in T if o.startswith(typ + ":") and self.check(user, rel, o)})


class FakePg:
    def __init__(self): self.denied = []
    def shipments(self, ids): return [{"shipment_id": i, "carrier": "x"} for i in ids if i != "SHP-9999"]
    def cases(self, sid): return [{"case_id": 1}]
    def deny(self, s, a, o): self.denied.append((s, a, o))

TOK = {"a": {"email": "anita@carrier-a.example", "email_verified": True},
       "b": {"email": "bala@carrier-b.example", "email_verified": True},
       "c": {"email": "chitra@pune.example", "email_verified": True},
       "u": {"email": "anita@carrier-a.example", "email_verified": False}}

def verify(t):
    if t not in TOK: raise Unauthenticated("bad")
    return TOK[t]

pg = FakePg(); api = Api(verify, FakeFga(), pg)
g = lambda path, tok=None: api.handle("GET", path, ("Bearer " + tok) if tok else None)
assert g("/v1/shipments")[0] == 401
assert g("/v1/shipments", "zzz")[0] == 401
assert g("/v1/shipments", "u")[0] == 401                                  # unverified email
assert g("/v1/shipments/SHP-1001", "a")[0] == 200
assert g("/v1/shipments/SHP-1001", "c")[0] == 200                         # the customer sees its own delivery
assert g("/v1/shipments/SHP-1002", "c")[0] == 403
assert g("/v1/shipments/SHP-1001", "b")[0] == 403 and pg.denied[-1] == ("user:bala@carrier-b.example", "GET /v1/shipments/SHP-1001", "shipment:SHP-1001")
assert g("/v1/shipments/SHP-9999", "a")[0] == 403                         # no existence leak
assert g("/v1/shipments/SHP-1001/excursions", "a") == (200, {"shipment_id": "SHP-1001", "excursions": [{"case_id": 1}]})
assert [s["shipment_id"] for s in g("/v1/shipments", "a")[1]["shipments"]] == ["SHP-1001", "SHP-1003"]
assert [s["shipment_id"] for s in g("/v1/shipments", "b")[1]["shipments"]] == ["SHP-1002"]
assert g("/v1/shipments/not-an-id", "a")[0] == 404
assert api.handle("POST", "/v1/shipments", "Bearer a")[0] == 405
assert api.handle("GET", "/healthz", None)[0] == 200
print("partner api: 13 checks passed")
python3 partner/test_api.py
python3 evidence/test_chain.py
python3 test_sync.py

Expected:

partner api: 13 checks passed
evidence chain: 9 scenarios passed
sync_tuples: 3 checks passed; tuples: 6 wanted, 6 added, 1 removed
model.json: 4 types

These tests prove the program’s logic against a stand-in. They do not prove OpenFGA’s own evaluation of the model; sections 7.3 and 10 do that against the real server.

7.2 The database additions, the image and the deployment

The API writes denials, and the evidence job keeps a log. Two tables and their grants are added to RDS by a Job run as the administrator (if you deleted the pg-admin Secret after Part 3, create it again from the managed secret as in Part 3, section 3):

-- Part 4 additions. Run as the administrator; every statement can be repeated.
\set ON_ERROR_STOP on
CREATE TABLE IF NOT EXISTS access_events (
  event_id text PRIMARY KEY,
  at       timestamptz NOT NULL DEFAULT now(),
  subject  text NOT NULL,
  action   text NOT NULL,
  object   text NOT NULL,
  allowed  boolean NOT NULL);

CREATE TABLE IF NOT EXISTS evidence_log (
  seq         bigserial PRIMARY KEY,
  record_key  text UNIQUE NOT NULL,
  sha256      text NOT NULL,
  prev_sha256 text NOT NULL,
  version_id  text NOT NULL,
  written_at  timestamptz NOT NULL DEFAULT now());

GRANT SELECT, INSERT ON access_events, evidence_log TO coldchain_app;
GRANT USAGE ON SEQUENCE evidence_log_seq_seq TO coldchain_app;
# platform/db-part4.yaml
apiVersion: batch/v1
kind: Job
metadata: { name: db-part4, namespace: react }
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: psql
          image: postgres:17-alpine
          command: ["psql", "-v", "ON_ERROR_STOP=1", "-f", "/sql/part4.sql"]
          env:
            - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-admin, key: host } } }
            - { name: PGDATABASE, value: coldchain }
            - { name: PGUSER, valueFrom: { secretKeyRef: { name: pg-admin, key: username } } }
            - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-admin, key: password } } }
            - { name: PGSSLMODE, value: require }
          volumeMounts: [{ name: sql, mountPath: /sql }]
      volumes: [{ name: sql, configMap: { name: db-part4 } }]
kubectl --context hub -n react create configmap db-part4 --from-file=part4.sql=db/part4.sql
kubectl --context hub apply -f platform/db-part4.yaml
kubectl --context hub -n react wait --for=condition=complete job/db-part4 --timeout=120s
kubectl --context hub -n react logs job/db-part4 | tail -3

Expected: the Job completes and the log ends with GRANT lines; a second run completes too. The image holds the API, the evidence program and the tuple sync, so one repository (coldchain/partner-api, created in Part 1) serves all three:

# Dockerfile
FROM python:3.12-slim
RUN pip install --no-cache-dir "psycopg[binary]>=3.2,<4" "PyJWT[crypto]>=2.8,<3" "boto3>=1.35"
WORKDIR /app
COPY lib lib
COPY partner partner
COPY evidence evidence
COPY ops/sync_tuples.py ops/sync_tuples.py
ENV PYTHONUNBUFFERED=1
USER 10001
CMD ["python", "-m", "partner.app"]
# scripts/build-partner.sh
#!/usr/bin/env bash
# usage: scripts/build-partner.sh <tag>   builds the partner API image (it also holds the evidence and tuple-sync jobs)
set -euo pipefail
TAG=${1:?tag}; REGION=${AWS_REGION:-ap-south-1}
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
ECR=$ACCOUNT.dkr.ecr.$REGION.amazonaws.com; REPO=coldchain/partner-api
aws ecr get-login-password --region "$REGION" | docker login --username AWS --password-stdin "$ECR"
docker build --platform linux/amd64 -t "$ECR/$REPO:$TAG" .
docker push "$ECR/$REPO:$TAG"
DIGEST=$(aws ecr describe-images --repository-name "$REPO" --image-ids imageTag="$TAG" --query 'imageDetails[0].imageDigest' --output text)
echo "$ECR/$REPO@$DIGEST"
chmod +x scripts/build-partner.sh
export ECR_PARTNER=$(scripts/build-partner.sh 1 | tail -1)
echo "$ECR_PARTNER"

Expected: one line, <account>.dkr.ecr.ap-south-1.amazonaws.com/coldchain/partner-api@sha256:.... Now the configuration, the Knative service and the three jobs of section 9 (the evidence writer, the verifier and the tuple sync):

# platform/partner-api.yaml
apiVersion: v1
kind: ConfigMap
metadata: { name: access-config, namespace: react }
data:
  members.json: |
    {
      "carrier:carrier-a": ["anita@carrier-a.example"],
      "carrier:carrier-b": ["bala@carrier-b.example"],
      "customer:customer-pune": ["chitra@pune.example"]
    }
  customers.json: |
    { "Pune": "customer-pune" }
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: partner-api, namespace: react }
spec:
  template:
    metadata:
      annotations: { autoscaling.knative.dev/min-scale: "2" }     # no cold start for a partner; two so one can restart
    spec:
      containers:
        - image: ECR_PARTNER
          command: ["python", "-m", "partner.app"]
          ports: [{ containerPort: 8080 }]
          env:
            - { name: OIDC_ISSUER, value: "https://login.DOMAIN" }
            - { name: OIDC_AUDIENCE, value: coldchain-partner-api }
            - { name: OIDC_JWKS, value: "http://dex.access.svc.cluster.local:5556/keys" }
            - { name: FGA_URL, value: "http://openfga.access.svc.cluster.local:8080" }
            - { name: FGA_STORE_ID, value: FGA_STORE_ID }
            - { name: FGA_MODEL_ID, value: FGA_MODEL_ID }
            - { name: FGA_TOKEN, valueFrom: { secretKeyRef: { name: fga-token, key: token } } }
            - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
            - { name: PGDATABASE, value: coldchain }
            - { name: PGUSER, value: coldchain_app }
            - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
            - { name: PGSSLMODE, value: require }
          resources: { requests: { cpu: 100m, memory: 128Mi }, limits: { memory: 256Mi } }
          readinessProbe: { httpGet: { path: /healthz } }
# platform/evidence.yaml
apiVersion: v1
kind: ServiceAccount
metadata: { name: evidence-sync, namespace: react }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: evidence-verify, namespace: react }
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: evidence-sync, namespace: react }
spec:
  schedule: "* * * * *"                 # every minute; the chain needs one writer at a time
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          serviceAccountName: evidence-sync
          restartPolicy: Never
          containers:
            - name: sync
              image: ECR_PARTNER
              command: ["python", "-m", "evidence.chain", "sync"]
              env:
                - { name: EVIDENCE_BUCKET, value: EVIDENCE_BUCKET }
                - { name: AWS_REGION, value: ap-south-1 }
                - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
                - { name: PGDATABASE, value: coldchain }
                - { name: PGUSER, value: coldchain_app }
                - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
                - { name: PGSSLMODE, value: require }
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: evidence-verify, namespace: react }
spec:
  schedule: "17 */6 * * *"              # every six hours; a failure shows as a failed Job, which Prometheus can alert on
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 0
      template:
        spec:
          serviceAccountName: evidence-verify
          restartPolicy: Never
          containers:
            - name: verify
              image: ECR_PARTNER
              command: ["python", "-m", "evidence.chain", "verify"]
              env:
                - { name: EVIDENCE_BUCKET, value: EVIDENCE_BUCKET }
                - { name: AWS_REGION, value: ap-south-1 }
                - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
                - { name: PGDATABASE, value: coldchain }
                - { name: PGUSER, value: coldchain_app }
                - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
                - { name: PGSSLMODE, value: require }
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: tuple-sync, namespace: react }
spec:
  schedule: "*/5 * * * *"               # new shipments and membership changes reach OpenFGA within five minutes
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: sync
              image: ECR_PARTNER
              command: ["python", "-m", "ops.sync_tuples"]
              env:
                - { name: FGA_URL, value: "http://openfga.access.svc.cluster.local:8080" }
                - { name: FGA_STORE_ID, value: FGA_STORE_ID }
                - { name: FGA_MODEL_ID, value: FGA_MODEL_ID }
                - { name: FGA_TOKEN, valueFrom: { secretKeyRef: { name: fga-token, key: token } } }
                - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
                - { name: PGDATABASE, value: coldchain }
                - { name: PGUSER, value: coldchain_app }
                - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
                - { name: PGSSLMODE, value: require }
              volumeMounts: [{ name: config, mountPath: /app/config }]
          volumes: [{ name: config, configMap: { name: access-config } }]
for f in platform/partner-api.yaml platform/evidence.yaml; do
  sed -e "s#ECR_PARTNER#$ECR_PARTNER#g" -e "s#DOMAIN#$DOMAIN#g" -e "s#EVIDENCE_BUCKET#$EVIDENCE_BUCKET#g" \
      -e "s#FGA_STORE_ID#$FGA_STORE_ID#g" -e "s#FGA_MODEL_ID#$FGA_MODEL_ID#g" $f | kubectl --context hub apply -f -
done
kubectl --context hub -n react get ksvc partner-api
kubectl --context hub -n react get cronjob

Expected: partner-api ready with two pods and the three CronJobs listed. The Gatekeeper rule from section 4 admits the image because it comes from our ECR path.

7.3 Tuples and the first answers

Run the tuple sync once now instead of waiting five minutes, and look at what OpenFGA holds:

kubectl --context hub -n react create job --from=cronjob/tuple-sync tuple-sync-1
kubectl --context hub -n react wait --for=condition=complete job/tuple-sync-1 --timeout=120s
kubectl --context hub -n react logs job/tuple-sync-1
curl -s -H "Authorization: Bearer $FGA_TOKEN" -H 'content-type: application/json' \
  localhost:8080/stores/$FGA_STORE_ID/check -d '{"tuple_key":{"user":"user:anita@carrier-a.example","relation":"can_view","object":"shipment:SHP-1001"}}'
curl -s -H "Authorization: Bearer $FGA_TOKEN" -H 'content-type: application/json' \
  localhost:8080/stores/$FGA_STORE_ID/check -d '{"tuple_key":{"user":"user:bala@carrier-b.example","relation":"can_view","object":"shipment:SHP-1001"}}'

Expected: the sync reports tuples: 11 wanted, 11 added, 0 removed on the first run (with the Part 3 seed data: three membership tuples, four carrier links, and four customer links because all four shipments go to Pune; the number depends on your rows), then {"allowed":true} for carrier A’s user on SHP-1001 and {"allowed":false} for carrier B’s user. That is OpenFGA itself applying the model, not a stub. Call the API as carrier A:

TOKEN=$(curl -s -u "coldchain-partner-api:$API_CLIENT_SECRET" https://login.$DOMAIN/token -d grant_type=password \
  -d username=anita@carrier-a.example -d "password=$LAB_PW" -d 'scope=openid email' | jq -r .id_token)
curl -s -H "Authorization: Bearer $TOKEN" https://api.$DOMAIN/v1/shipments | jq '.shipments[].shipment_id'
curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $TOKEN" https://api.$DOMAIN/v1/shipments/SHP-1002

Expected: the shipment IDs of carrier A, and 403 for carrier B’s shipment.

Checkpoint 7. The offline tests pass, the Job finished the migration, the image is in ECR, OpenFGA’s own check answers true and false as above, and the API returns the list and the 403.

8. Microcks: mock and contract test

A partner builds against a contract before the real service is ready, and a change that breaks the contract should be caught before the partner finds it. The contract is an OpenAPI file in Git: the partner API’s, and a second one for a carrier’s handover API that we do not implement but whose shape our code and the carrier agree on. Microcks serves a mock from the examples in the contract, and runs the real API against the same file.

# contracts/partner-api.openapi.yaml
openapi: 3.0.3
info:
  title: Cold Chain Partner API
  version: 1.0.0
  description: Read-only access to the shipments the caller's organisation may see. Authentication is a Dex ID token.
servers: [{ url: https://api.example.com }]
security: [{ bearer: [] }]
components:
  securitySchemes:
    bearer: { type: http, scheme: bearer, bearerFormat: JWT }
  schemas:
    Shipment:
      type: object
      required: [shipment_id, device_id, status]
      properties:
        shipment_id: { type: string, pattern: '^SHP-[0-9]{4,10}$' }
        device_id: { type: string }
        carrier: { type: string, nullable: true }
        product: { type: string, nullable: true }
        origin: { type: string, nullable: true }
        destination: { type: string, nullable: true }
        status: { type: string }
        start_time: { type: string, format: date-time }
        end_time: { type: string, format: date-time, nullable: true }
    Error:
      type: object
      required: [error]
      properties: { error: { type: string }, detail: { type: string } }
paths:
  /v1/shipments:
    get:
      operationId: listShipments
      summary: Shipments the caller may see
      responses:
        '200':
          description: The list, possibly empty
          content:
            application/json:
              schema:
                type: object
                required: [shipments]
                properties: { shipments: { type: array, items: { $ref: '#/components/schemas/Shipment' } } }
              examples:
                carrierA:
                  value:
                    shipments:
                      - { shipment_id: SHP-1001, device_id: reefer-001, carrier: carrier-a, product: vaccines, origin: Mumbai, destination: Pune, status: in_transit, start_time: '2026-11-20T06:00:00Z', end_time: null }
        '401': { description: No or invalid token, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
  /v1/shipments/{shipment_id}:
    get:
      operationId: getShipment
      parameters:
        - { name: shipment_id, in: path, required: true, schema: { type: string }, examples: { ok: { value: SHP-1001 } } }
      responses:
        '200':
          description: The shipment
          content:
            application/json:
              schema: { $ref: '#/components/schemas/Shipment' }
              examples:
                ok:
                  value: { shipment_id: SHP-1001, device_id: reefer-001, carrier: carrier-a, product: vaccines, origin: Mumbai, destination: Pune, status: in_transit, start_time: '2026-11-20T06:00:00Z', end_time: null }
        '401': { description: No or invalid token, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
        '403': { description: Not your shipment, or no such shipment, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
        '404': { description: Allowed, but the record is gone, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
  /v1/shipments/{shipment_id}/excursions:
    get:
      operationId: listExcursions
      parameters:
        - { name: shipment_id, in: path, required: true, schema: { type: string }, examples: { ok: { value: SHP-1001 } } }
      responses:
        '200':
          description: Excursion cases for the shipment
          content:
            application/json:
              schema:
                type: object
                required: [shipment_id, excursions]
                properties:
                  shipment_id: { type: string }
                  excursions:
                    type: array
                    items:
                      type: object
                      properties:
                        case_id: { type: integer }
                        started_at: { type: string, format: date-time }
                        ended_at: { type: string, format: date-time, nullable: true }
                        max_temp_c: { type: number, nullable: true }
                        limit_c: { type: number }
              examples:
                none: { value: { shipment_id: SHP-1001, excursions: [] } }
        '401': { description: No or invalid token, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
        '403': { description: Not your shipment, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
# contracts/carrier-handover.openapi.yaml
openapi: 3.0.3
info:
  title: Carrier Handover API
  version: 1.0.0
  description: The API a carrier offers for handover confirmations. We do not implement it; Microcks mocks it so our code and the carrier can build against the same contract.
servers: [{ url: https://carrier.example.com }]
paths:
  /handovers/{shipment_id}:
    get:
      operationId: getHandover
      parameters:
        - { name: shipment_id, in: path, required: true, schema: { type: string }, examples: { ok: { value: SHP-1001 } } }
      responses:
        '200':
          description: The handover record
          content:
            application/json:
              schema:
                type: object
                required: [shipment_id, handed_over_at, receiver]
                properties:
                  shipment_id: { type: string }
                  handed_over_at: { type: string, format: date-time }
                  receiver: { type: string }
              examples:
                ok: { value: { shipment_id: SHP-1001, handed_over_at: '2026-11-20T11:30:00Z', receiver: Pune hospital group } }
helm upgrade --install microcks microcks/microcks -n microcks --create-namespace -f platform/microcks-values.yaml --kube-context hub --wait --timeout 10m
kubectl --context hub -n microcks get pods
kubectl --context hub -n microcks port-forward svc/microcks 8585:8080 &

Expected: the Microcks, its database and its sign-in pods Running. Gatekeeper only guards react, so these upstream images are not blocked. The script imports both files and starts a contract test of the running API, passing a real token so the test can get past the sign-in:

# ops/microcks_run.sh
#!/usr/bin/env bash
# Imports the two contracts into Microcks and runs the contract test against the partner API.
# Needs: kubectl port-forward svc/microcks 8585:8080 -n microcks, and a Dex ID token in $TOKEN.
# With Microcks' sign-in on, set KC_SECRET (the service account's client secret); with it off, leave it empty.
set -euo pipefail
MICROCKS=${MICROCKS:-http://host.docker.internal:8585/api/}
AUTH=()
if [ -n "${KC_SECRET:-}" ]; then AUTH=(--keycloakClientId="${KC_ID:-microcks-serviceaccount}" --keycloakClientSecret="$KC_SECRET"); fi
CLI="docker run --rm -v $PWD/contracts:/contracts --add-host host.docker.internal:host-gateway quay.io/microcks/microcks-cli:latest microcks-cli"
$CLI import /contracts/partner-api.openapi.yaml:true,/contracts/carrier-handover.openapi.yaml:true \
     --microcksURL="$MICROCKS" "${AUTH[@]}"
$CLI test 'Cold Chain Partner API:1.0.0' http://partner-api.react.svc.cluster.local OPEN_API_SCHEMA \
     --microcksURL="$MICROCKS" "${AUTH[@]}" --waitFor=120sec \
     --operationsHeaders="{\"globals\":[{\"name\":\"Authorization\",\"values\":\"Bearer $TOKEN\"}]}"
chmod +x ops/microcks_run.sh
export TOKEN=$(curl -s -u "coldchain-partner-api:$API_CLIENT_SECRET" https://login.$DOMAIN/token -d grant_type=password \
  -d username=anita@carrier-a.example -d "password=$LAB_PW" -d 'scope=openid email' | jq -r .id_token)
ops/microcks_run.sh

Expected: the import prints the two services, and the test ends with a success result: every operation of the partner contract answered with a status and a body that match the schema. The token lives 15 minutes, which is enough for one run.

Then the mock. The carrier’s handover API does not exist here, but a request to Microcks returns the example from the contract:

curl -s "http://localhost:8585/rest/Carrier+Handover+API/1.0.0/handovers/SHP-1001" | jq .

Expected: the example record from the contract (SHP-1001, a time, the receiver). Now break the contract on purpose: in partner/app.py rename status in the shipment answer, rebuild, deploy and run the test again. It fails and names the operation and the schema rule that was broken. That is the check for “no breaking change unnoticed”. Put the same command in your pipeline after each deploy.

Checkpoint 8. The mock answers with the example, the contract test passes against the real API, and a deliberate break makes it fail.

9. Evidence: a hash-chained custody record, and the validation checklist

9.1 What is recorded, and how

Three kinds of fact matter for custody, and Parts 3 and 4 already write each into RDS: an excursion case opening and closing, a geofence event (the vehicle entered or left a site), and a refused access. The evidence program turns each fact into one small JSON object in the evidence bucket, with the key built from the fact’s identity. Every object holds the SHA-256 of the record before it, in the order the program wrote them, and the log of keys, hashes and S3 version IDs sits in RDS.

Three properties come from three details:

  • A repeat cannot add a version. The write uses If-None-Match: *, so it fails if the key exists. A crash between writing the object and logging it is recovered on the next run, which accepts the existing object only if its bytes are identical.
  • Deletion is blocked by the lock. The bucket’s default retention (Part 1: governance mode, 30 days, for the lab) keeps every version, and the writer’s IAM role has no delete right.
  • A change is visible from two sides. The verifier reads the logged version of every object, recomputes its hash, checks the chain, checks that no newer version of the key exists, checks that the version is locked, and lists the bucket for objects the log does not know. Changing the log breaks the match with the bucket; adding to the bucket breaks the match with the log.
# evidence/chain.py
#!/usr/bin/env python3
"""Chain-of-custody evidence: records go to an S3 bucket with Object Lock, and each record names the hash of the one before.

  python -m evidence.chain sync      write records that are not yet in the bucket
  python -m evidence.chain verify    check the bucket and the log against each other (exit 1 on any finding)
"""
import hashlib, json, os, sys
from datetime import datetime, timezone

GENESIS = "0" * 64


class Exists(Exception):
    pass


def canon(body):
    return json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()


def sha(data):
    return hashlib.sha256(data).hexdigest()


def iso(v):
    return v.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ") if isinstance(v, datetime) else v


# --------------------------------------------------------------------------- stores
class S3Store:
    def __init__(self, bucket):
        import boto3
        self.s3, self.bucket = boto3.client("s3"), bucket

    def put(self, key, data):
        from botocore.exceptions import ClientError
        try:      # If-None-Match: * makes the write fail when the key exists, so a repeat cannot add a version
            r = self.s3.put_object(Bucket=self.bucket, Key=key, Body=data, ContentType="application/json", IfNoneMatch="*")
            return r["VersionId"]
        except ClientError as e:
            if e.response["Error"]["Code"] in ("PreconditionFailed", "ConditionalRequestConflict"):
                raise Exists(key)
            raise

    def get(self, key, version_id=None):
        kw = {"VersionId": version_id} if version_id else {}
        r = self.s3.get_object(Bucket=self.bucket, Key=key, **kw)
        return r["Body"].read(), r["VersionId"]

    def latest_version(self, key):
        return self.s3.head_object(Bucket=self.bucket, Key=key)["VersionId"]

    def locked(self, key, version_id):
        from botocore.exceptions import ClientError
        try:
            r = self.s3.get_object_retention(Bucket=self.bucket, Key=key, VersionId=version_id)
            return bool(r.get("Retention", {}).get("Mode"))
        except ClientError:
            return False

    def keys(self, prefix="custody/"):
        out = []
        for page in self.s3.get_paginator("list_objects_v2").paginate(Bucket=self.bucket, Prefix=prefix):
            out += [o["Key"] for o in page.get("Contents", [])]
        return out


# --------------------------------------------------------------------------- repository
class Repo:
    """Where the records come from, and the log of what has been written."""
    def __init__(self):
        from lib.pg import db
        self.db = db

    def _q(self, sql, args=()):
        with self.db().cursor() as c:
            c.execute(sql, args)
            cols = [d.name for d in c.description]
            return [dict(zip(cols, r)) for r in c.fetchall()]

    def pending(self, limit=500):
        """Records not yet logged, oldest first. The key is built in SQL so the anti-join stays in the database."""
        out = []
        for r in self._q("""SELECT * FROM excursion_cases c WHERE NOT EXISTS (SELECT 1 FROM evidence_log l
                            WHERE l.record_key = 'custody/case/' || c.case_id || '/opened.json') ORDER BY detected_at LIMIT %s""", (limit,)):
            out.append((f"custody/case/{r['case_id']}/opened.json", r["detected_at"], {
                "kind": "case-opened", "case_id": r["case_id"], "device": r["device_id"], "room": r["room"],
                "shipment": r["shipment_id"], "started_at": iso(r["started_at"]), "detected_at": iso(r["detected_at"]),
                "limit_c": float(r["limit_c"]), "hold_s": r["hold_s"]}))
        for r in self._q("""SELECT * FROM excursion_cases c WHERE ended_at IS NOT NULL AND NOT EXISTS (SELECT 1 FROM evidence_log l
                            WHERE l.record_key = 'custody/case/' || c.case_id || '/closed.json') ORDER BY ended_at LIMIT %s""", (limit,)):
            out.append((f"custody/case/{r['case_id']}/closed.json", r["ended_at"], {
                "kind": "case-closed", "case_id": r["case_id"], "device": r["device_id"], "shipment": r["shipment_id"],
                "ended_at": iso(r["ended_at"]), "max_temp_c": float(r["max_temp_c"]) if r["max_temp_c"] is not None else None}))
        for r in self._q("""SELECT * FROM geofence_events g WHERE NOT EXISTS (SELECT 1 FROM evidence_log l
                            WHERE l.record_key = 'custody/geofence/' || g.event_id || '.json') ORDER BY received_at LIMIT %s""", (limit,)):
            out.append((f"custody/geofence/{r['event_id']}.json", r["received_at"], {
                "kind": "geofence", "event_id": r["event_id"], "event_type": r["event_type"], "geofence": r["geofence_id"],
                "device": r["device_id"], "sample_time": iso(r["sample_time"]), "received_at": iso(r["received_at"])}))
        for r in self._q("""SELECT * FROM access_events a WHERE NOT EXISTS (SELECT 1 FROM evidence_log l
                            WHERE l.record_key = 'custody/access/' || a.event_id || '.json') ORDER BY at LIMIT %s""", (limit,)):
            out.append((f"custody/access/{r['event_id']}.json", r["at"], {
                "kind": "access", "event_id": r["event_id"], "at": iso(r["at"]), "subject": r["subject"],
                "action": r["action"], "object": r["object"], "allowed": r["allowed"]}))
        out.sort(key=lambda x: (x[1], x[0]))
        return [(k, b) for k, _, b in out][:limit]

    def last_sha(self):
        r = self._q("SELECT sha256 FROM evidence_log ORDER BY seq DESC LIMIT 1")
        return r[0]["sha256"] if r else GENESIS

    def log(self, key, digest, prev, version_id):
        with self.db().cursor() as c:
            c.execute("INSERT INTO evidence_log (record_key, sha256, prev_sha256, version_id) VALUES (%s,%s,%s,%s) ON CONFLICT DO NOTHING",
                      (key, digest, prev, version_id))

    def entries(self):
        return self._q("SELECT seq, record_key, sha256, prev_sha256, version_id FROM evidence_log ORDER BY seq")


# --------------------------------------------------------------------------- the two jobs
def sync(repo, store, out=print):
    n, prev = 0, repo.last_sha()
    for key, body in repo.pending():
        body = dict(body, prev_sha256=prev, schema=1)
        data = canon(body)
        digest = sha(data)
        try:
            version = store.put(key, data)
        except Exists:        # written by an earlier run that stopped before logging: accept it only if it is the same bytes
            existing, version = store.get(key)
            if sha(existing) != digest:
                out(f"CONFLICT {key}: the object exists with different content; not logged")
                return 1
        repo.log(key, digest, prev, version)
        prev, n = digest, n + 1
    out(f"synced {n} record(s)")
    return 0


def verify(repo, store, out=print):
    bad, prev, seen = 0, GENESIS, set()
    entries = repo.entries()
    for e in entries:
        key, seen = e["record_key"], seen | {e["record_key"]}
        problems = []
        try:
            data, _ = store.get(key, e["version_id"])
        except Exception as ex:
            out(f"FAIL  #{e['seq']} {key}: the logged version cannot be read ({type(ex).__name__})")
            bad += 1; prev = e["sha256"]; continue
        if sha(data) != e["sha256"]:
            problems.append("content does not match the logged hash")
        if e["prev_sha256"] != prev or json.loads(data).get("prev_sha256") != prev:
            problems.append("chain link broken")
        if store.latest_version(key) != e["version_id"]:
            problems.append("a newer version of the object exists")
        if not store.locked(key, e["version_id"]):
            problems.append("no retention lock on the version")
        if problems:
            bad += 1; out(f"FAIL  #{e['seq']} {key}: " + "; ".join(problems))
        prev = e["sha256"]
    for k in sorted(set(store.keys()) - seen):
        bad += 1; out(f"FAIL  {k}: object in the bucket that the log does not know")
    out(f"checked {len(entries)} record(s): " + ("PASS" if not bad else f"{bad} problem(s)"))
    return 1 if bad else 0


if __name__ == "__main__":
    cmd = sys.argv[1] if len(sys.argv) > 1 else ""
    if cmd not in ("sync", "verify"):
        sys.exit(__doc__)
    repo, store = Repo(), S3Store(os.environ["EVIDENCE_BUCKET"])
    sys.exit(sync(repo, store) if cmd == "sync" else verify(repo, store))

The offline test runs nine scenarios against a fake bucket and a fake log: the normal path, a repeated run, the crash-between-steps recovery, a newer version, an edited log row, an unknown object, a missing version, a missing lock, and an object that exists with different content. It is in the file list above and ran in section 7.1.

9.2 Run it

The CronJob from section 7.2 syncs every minute. Make some facts, then look:

kubectl --context hub -n react get cronjob evidence-sync
aws s3api list-objects-v2 --bucket $EVIDENCE_BUCKET --prefix custody/ --query 'Contents[].[Key,Size]' --output text | head
kubectl --context hub -n react create job --from=cronjob/evidence-verify verify-1
kubectl --context hub -n react wait --for=condition=complete job/verify-1 --timeout=120s
kubectl --context hub -n react logs job/verify-1

Expected: objects under custody/case/, custody/geofence/ and (after you have made a refused call, section 10) custody/access/, and a verifier log ending checked N record(s): PASS. To see one record:

aws s3 cp s3://$EVIDENCE_BUCKET/custody/case/1/opened.json - | jq .
aws s3api get-object-retention --bucket $EVIDENCE_BUCKET --key custody/case/1/opened.json --query Retention

Expected: a record with kind, the case details, prev_sha256 and schema, and a retention block with the mode and the retain-until date.

9.3 A validation evidence checklist

This is engineering guidance, not legal advice. These are the items a quality unit usually asks to see for a system like this. Each has a place in this build; whether they are enough is for the company to decide.

Evidence Where this build keeps it
Intended use and requirements The Overview’s requirements table
Design and configuration, under change control The Git repository, pull requests and Argo CD history; OpenFGA model IDs; image digests
Access control: who may see what config/model.json, config/members.json, the access-denial test (section 10)
Secrets management Secrets Manager and SOPS homes (sections 2 and 3); CloudTrail for access to them
Audit trail of access access_events and the custody records
Record integrity and retention Object Lock, the hash chain and the verifier (section 9); retention period set from the quality policy
Test evidence Offline tests, the contract test, the access-denial test, the tamper test, with dates, versions and operators written down
Alert performance Part 3’s excursion test, with the worst of ten runs recorded
Backup and restore Part 5
Periodic review A dated review of members, connectors, retention and the verifier results

Production differences to decide before relying on it: use compliance mode with a retention period set by the quality policy (governance mode lets an authorised administrator remove versions, which section 10 uses to clean up; compliance mode does not, and its retention cannot be shortened, so do not use it in a lab); add the electronic-signature and identity-binding steps your regulation asks for; and send the verifier’s failure to a page, not just a failed Job.

Checkpoint 9. Records appear for a case, a geofence event and a refusal, and the verifier prints PASS.

10. The access-denial and tamper tests

10.1 Access denial

The script signs in the three lab users and compares the API’s answers with what the model says. It includes the cases an attacker would try: no token, a garbage token, a token whose signature has been altered, another organisation’s shipment, and a shipment that does not exist:

# ops/access_test.py
#!/usr/bin/env python3
"""Access-denial test. Signs the lab users in through Dex (password grant), calls the partner API, and compares each answer
with what the model says it should be.

  python3 ops/access_test.py --issuer https://login.$DOMAIN --api https://api.$DOMAIN --secret "$API_CLIENT_SECRET" --password "$LAB_PW"
"""
import argparse, base64, json, sys, urllib.error, urllib.parse, urllib.request

CASES = [   # (who, path, expected status)  -- data: SHP-1001 carrier-a to Pune, SHP-1002 carrier-b, SHP-1003 carrier-a, SHP-1004 carrier-a (test truck); all four go to Pune
    ("anita@carrier-a.example", "/v1/shipments/SHP-1001", 200),
    ("anita@carrier-a.example", "/v1/shipments/SHP-1002", 403),      # another carrier's shipment
    ("anita@carrier-a.example", "/v1/shipments/SHP-1001/excursions", 200),
    ("bala@carrier-b.example", "/v1/shipments/SHP-1001", 403),
    ("bala@carrier-b.example", "/v1/shipments/SHP-1002", 200),
    ("chitra@pune.example", "/v1/shipments/SHP-1001", 200),          # the customer sees its own delivery
    ("chitra@pune.example", "/v1/shipments/SHP-9999", 403),          # no such shipment: same answer as "not yours"
]
LISTS = {"anita@carrier-a.example": ["SHP-1001", "SHP-1003", "SHP-1004"], "bala@carrier-b.example": ["SHP-1002"], "chitra@pune.example": ["SHP-1001", "SHP-1002", "SHP-1003", "SHP-1004"]}


def call(url, headers=None, data=None):
    req = urllib.request.Request(url, data=data, headers=headers or {})
    try:
        with urllib.request.urlopen(req, timeout=15) as r:
            return r.status, json.loads(r.read() or b"{}")
    except urllib.error.HTTPError as e:
        try:
            return e.code, json.loads(e.read() or b"{}")
        except ValueError:
            return e.code, {}


def token(issuer, secret, user, pw):
    basic = base64.b64encode(f"coldchain-partner-api:{secret}".encode()).decode()
    body = urllib.parse.urlencode({"grant_type": "password", "username": user, "password": pw, "scope": "openid email"}).encode()
    s, r = call(issuer + "/token", {"Authorization": "Basic " + basic, "content-type": "application/x-www-form-urlencoded"}, body)
    if s != 200:
        sys.exit(f"cannot sign {user} in: {s} {r}")
    return r["id_token"]


def main():
    a = argparse.ArgumentParser()
    for k in ("issuer", "api", "secret", "password"):
        a.add_argument("--" + k, required=True)
    a = a.parse_args()
    toks = {u: token(a.issuer.rstrip("/"), a.secret, u, a.password) for u in LISTS}
    bad = 0

    def check(label, got, want):
        nonlocal bad
        ok = got == want
        bad += not ok
        print(f"{'ok  ' if ok else 'FAIL'} {label}: got {got}, want {want}")

    check("no token", call(a.api + "/v1/shipments")[0], 401)
    check("garbage token", call(a.api + "/v1/shipments", {"Authorization": "Bearer abc.def.ghi"})[0], 401)
    t = toks["anita@carrier-a.example"]; head, body, sig = t.split(".")
    check("signature altered", call(a.api + "/v1/shipments", {"Authorization": f"Bearer {head}.{body}.{sig[:-4]}AAAA"})[0], 401)
    for who, path, want in CASES:
        check(f"{who.split('@')[0]:6} GET {path}", call(a.api + path, {"Authorization": "Bearer " + toks[who]})[0], want)
    for who, want in LISTS.items():
        s, r = call(a.api + "/v1/shipments", {"Authorization": "Bearer " + toks[who]})
        got = sorted(x["shipment_id"] for x in r.get("shipments", []))
        check(f"{who.split('@')[0]:6} list", got, want)
    print("RESULT:", "PASS" if not bad else f"FAIL ({bad})")
    sys.exit(1 if bad else 0)


if __name__ == "__main__":
    main()
python3 ops/access_test.py --issuer https://login.$DOMAIN --api https://api.$DOMAIN --secret "$API_CLIENT_SECRET" --password "$LAB_PW"

Expected, in this shape:

ok   no token: got 401, want 401
ok   garbage token: got 401, want 401
ok   signature altered: got 401, want 401
ok   anita  GET /v1/shipments/SHP-1001: got 200, want 200
ok   anita  GET /v1/shipments/SHP-1002: got 403, want 403
...
RESULT: PASS

The list checks assume the Part 3 seed data (SHP-1001 to SHP-1004, carriers A and B, destination Pune). Change the two tables at the top of the script if your data differs. Afterwards each refusal is a row in access_events and, within a minute, an object under custody/access/.

10.2 Tamper

The script does what a person with access, or a mistake, might do, runs the verifier after each step and undoes the change where the lab allows:

# ops/tamper_test.sh
#!/usr/bin/env bash
# Tamper test for the evidence chain. Run from a workstation with your own AWS identity (the lab administrator).
# Each step does something an attacker or a mistake might do, runs the verifier, and undoes it where the lab allows.
set -uo pipefail
: "${EVIDENCE_BUCKET:?}" "${PG_HOST:?}"; CTX=${CTX:-hub}
K="kubectl --context $CTX -n react"
verify() { $K delete job tamper-verify --ignore-not-found >/dev/null
           $K create job --from=cronjob/evidence-verify tamper-verify >/dev/null
           $K wait --for=condition=complete --timeout=120s job/tamper-verify >/dev/null 2>&1 || $K wait --for=condition=failed --timeout=5s job/tamper-verify >/dev/null 2>&1
           $K logs job/tamper-verify | tail -${1:-4}; }
psql() { $K run "psql-$RANDOM" --rm -i --restart=Never --image=postgres:17-alpine --env=PGHOST="$PG_HOST" --env=PGDATABASE=coldchain \
           --env=PGSSLMODE=require --env=PGUSER="$1" --env=PGPASSWORD="$2" --command -- psql -v ON_ERROR_STOP=1 -tA -c "$3" 2>&1; }

KEY=$(aws s3api list-objects-v2 --bucket "$EVIDENCE_BUCKET" --prefix custody/ --query 'Contents[0].Key' --output text)
VID=$(aws s3api list-object-versions --bucket "$EVIDENCE_BUCKET" --prefix "$KEY" --query 'Versions[0].VersionId' --output text)
echo "== 0. baseline";                                  verify 1
echo "== 1. delete a locked version (expect AccessDenied)"
aws s3api delete-object --bucket "$EVIDENCE_BUCKET" --key "$KEY" --version-id "$VID" 2>&1 | head -2
echo "== 2. add a newer version of a record (writes are allowed; the lock keeps the original)"
echo '{"kind":"forged"}' > /tmp/forged.json
NEW=$(aws s3api put-object --bucket "$EVIDENCE_BUCKET" --key "$KEY" --body /tmp/forged.json --query VersionId --output text)
verify 3
echo "   undo (lab only: GOVERNANCE mode lets an authorised administrator remove the forged version; COMPLIANCE would not)"
aws s3api delete-object --bucket "$EVIDENCE_BUCKET" --key "$KEY" --version-id "$NEW" --bypass-governance-retention >/dev/null; verify 1
echo "== 3. change a hash in the log, first as the application role, then as the administrator"
psql coldchain_app "$($K get secret pg-app -o jsonpath='{.data.password}' | base64 -d)" "UPDATE evidence_log SET sha256 = repeat('f',64) WHERE record_key = '$KEY'" | grep -i "permission denied"
ADMIN_USER=$($K get secret pg-admin -o jsonpath='{.data.username}' | base64 -d); ADMIN_PW=$($K get secret pg-admin -o jsonpath='{.data.password}' | base64 -d)
OLD=$(psql "$ADMIN_USER" "$ADMIN_PW" "SELECT sha256 FROM evidence_log WHERE record_key = '$KEY'" | tail -1)
psql "$ADMIN_USER" "$ADMIN_PW" "UPDATE evidence_log SET sha256 = repeat('f',64) WHERE record_key = '$KEY'" >/dev/null; verify 4
psql "$ADMIN_USER" "$ADMIN_PW" "UPDATE evidence_log SET sha256 = '$OLD' WHERE record_key = '$KEY'" >/dev/null; verify 1
echo "== 4. an object the log does not know"
UNK=$(aws s3api put-object --bucket "$EVIDENCE_BUCKET" --key custody/unlogged.json --body /tmp/forged.json --query VersionId --output text)
verify 3
aws s3api delete-object --bucket "$EVIDENCE_BUCKET" --key custody/unlogged.json --version-id "$UNK" --bypass-governance-retention >/dev/null; verify 1
chmod +x ops/tamper_test.sh
EVIDENCE_BUCKET=$EVIDENCE_BUCKET PG_HOST=$PG_HOST ops/tamper_test.sh

Expected, in this shape (the record names and numbers are yours):

Step What happens What the verifier says
0 Nothing checked N record(s): PASS
1 Delete a locked version AccessDenied from S3
2 Add a newer version of a record FAIL ... a newer version of the object exists, then PASS after the lab cleanup
3 Edit the log as the application role permission denied for table evidence_log
3 Edit the log as the administrator FAIL ... content does not match the logged hash, and the next record chain link broken, then PASS after the restore
4 Put an object the log does not know FAIL ... object in the bucket that the log does not know, then PASS after cleanup

The cleanup steps use --bypass-governance-retention. That works in governance mode for a principal that has the right, which is what you want in a lab and exactly what a compliance-mode bucket forbids. Note what the test does not cover: someone who controls both the database administrator account and the bucket’s retention permissions can rebuild a consistent forged history. Separate those duties, and consider periodically copying the latest hash to a system that person cannot reach (a ticket, a signed email, another account).

Checkpoint 10. The access test prints PASS, and every tamper step is either refused or reported by the verifier, then clean after the lab undo.

11. Troubleshooting, cost, cleanup, design notes and checklist

Troubleshooting

Symptom Likely cause What to check
ClusterSecretStore not Ready The controller has no role aws eks list-pod-identity-associations; restart the external-secrets pods after the association is created
ExternalSecret SecretSyncedError Wrong secret name, property, or a missing KMS right kubectl describe externalsecret; the role’s policy in access.tf names AmazonMSK_coldchain_* and coldchain/*
Secret does not change after a new value Refresh interval Annotate the ExternalSecret with force-sync; consumers need a restart
SOPS says it cannot decrypt The caller has no kms:Decrypt Your IAM identity locally; Argo CD’s repo-server association in the cluster
Argo CD secrets Application shows a kustomize error Build options or plugin path kubectl -n argocd logs deploy/argocd-repo-server; compare the init container with the README of your KSOPS tag
A pod in react is refused It is not on the allow-list The message names the image; add it on purpose or move it to ECR
OpenFGA migration Job fails fga-pg-app has no uri, or the database is not ready kubectl -n access describe job; the CloudNativePG cluster status
fga_load.py returns 401 The key does not match The keys value of fga-keys against Secrets Manager
Dex does not start A connector issuer is unreachable Dex logs; remove the connector overlay until the issuer is real
Token is rejected with 401 Issuer or audience differs Decode the token; compare iss with OIDC_ISSUER (including a trailing slash) and aud with OIDC_AUDIENCE
API returns 503 OpenFGA, the database or Dex’s keys are unreachable The API logs name the failing dependency
Everyone gets 403 The tuples are missing kubectl logs job/tuple-sync-1; the emails in members.json must be lower case and match the token
Contract test fails on every operation The token expired or the headers were not passed Get a new token; check the operations-headers flag for your CLI version
Evidence sync logs CONFLICT An object exists with other content Someone wrote that key; investigate before anything else
Verifier reports an unknown object Something wrote to custody/ outside the sync The bucket’s access log and CloudTrail data events

Cost note

No prices were read for this post, so none are quoted. The additions are small: an extra database pair and volumes for OpenFGA, a few more pods (Dex, OpenFGA, Microcks with its own MongoDB and Keycloak, external-secrets, Gatekeeper), Secrets Manager secrets (billed per secret and per call), KMS requests for every decrypt, S3 storage and requests for the evidence objects (small), CloudTrail if you enable data events on the bucket, and the same load balancer as Part 3. Microcks is the heaviest component and can be stopped between uses. Price the items in the AWS Pricing Calculator for ap-south-1.

Cleanup

helm --kube-context hub -n microcks uninstall microcks
kubectl --context hub -n react delete cronjob evidence-sync evidence-verify tuple-sync
kubectl --context hub -n react delete ksvc partner-api
helm --kube-context hub -n access uninstall dex openfga
kubectl --context hub -n access delete cluster.postgresql.cnpg.io fga-pg
kubectl --context hub delete coldallowedregistry react-images-from-ecr
kubectl --context hub -n argocd delete application secrets
(cd infra/foundation && tofu destroy -target=aws_eks_pod_identity_association.evidence_writer -target=aws_eks_pod_identity_association.evidence_verifier)

The evidence objects stay: governance retention keeps every version for 30 days, and with versioning on, a plain delete only adds a marker. To remove lab objects early, an administrator uses --bypass-governance-retention on each version, as the tamper test does. Never use compliance mode with a long period in a lab: you cannot undo it, and the bucket (and the account that holds it) stays until the period ends. Part 5 covers the final teardown.

Design notes and version checks

What this blueprint chose, and what to confirm on your versions before relying on it.

  • Dex. Chart dex/dex, with the configuration under config: and Secrets as envFrom. Confirm the key names for your chart version. The passwordConnector: local setting and secretEnv on a static client are used for the lab flow. A connector’s issuer is contacted when Dex starts. ID tokens last 15 minutes here. Dex’s SAML connector is documented as unmaintained.
  • Identity is the email. Partners with no verified email in their tokens cannot be admitted by this design; use the provider’s own claim mapping to supply one. Membership lives in config/members.json, so a leaver is removed by a pull request, and the change reaches OpenFGA within five minutes.
  • Public names. The NLB annotation takes a comma-separated list of certificate ARNs on current in-tree providers; with the AWS Load Balancer Controller the annotation set differs. A Knative DomainMapping pointing at a plain Kubernetes Service (for Dex) depends on your Knative version; if yours refuses it, put Dex behind a Knative Service wrapper or an ingress of your choice.
  • OpenFGA. Chart keys (datastore.uriSecret, authn.preshared.keysSecret, datastore.migrations) and the uri key of CloudNativePG’s application Secret are as documented at the time of writing; confirm with helm show values openfga/openfga. The HTTP request shapes (check, list-objects, read, write, authorization models) are those of the Check and relationship-query documentation. Pin the model ID in every client, as the best-practice page recommends, so a new model version cannot change answers silently.
  • external-secrets. external-secrets.io/v1 resources and the no-auth store using Pod Identity; creationPolicy: Merge needs the target Secret to exist. If your release serves only v1beta1, change the apiVersion.
  • SOPS and KSOPS. The init-container recipe, the plugin path and the kustomize.buildOptions follow the KSOPS README for the tag you pick; check it, because the recipe has changed between major versions. SOPS protects Git, not the cluster: the decrypted Secret sits in etcd like any other.
  • Gatekeeper. The template uses the classic Rego syntax of the Sprint 4 policies; use the syntax your Gatekeeper release documents if you move to Rego v1. The allow-list is a prefix match, so a tag or a digest from an allowed repository passes. Add ratify or a signature check (Sprint 4) if you want signed images only.
  • Microcks. The chart values microcks.url, keycloak.url, keycloak.privateUrl and ingresses, the default service account name, and the CLI flags (--operationsHeaders, --waitFor, :true for a primary artifact) are from the project’s documentation for the CLI image tag you pull. The CLI finds Microcks’ sign-in service through a URL the server reports; in a port-forward lab that URL may not be reachable, and the lab fallback is to install with keycloak.enabled=false and leave KC_SECRET empty. Microcks’ own managed database and Keycloak are for discovery, not production.
  • S3. Conditional writes (If-None-Match: *) on a versioned bucket, GetObjectRetention and Object Lock behaviour are as in the S3 documentation; confirm your boto3 version supports IfNoneMatch on put_object. The lab uses governance mode; production evidence uses compliance mode with the retention period from the quality policy.
  • Evidence chain. One writer, one chain, written in the order the program found the facts. Facts logged late are chained late; the record’s own timestamps say when they happened. The chain proves the order of recording, not the truth of a reading.
  • Checked offline. The partner API logic, the tuple sync and the evidence chain (nine scenarios, including the tamper cases) were run against stand-ins on a laptop. That shows the logic is as described, not that your cluster, OpenFGA, Dex or S3 behave the same.
  • Not measured. Throughput of the API, OpenFGA latency under load, the real federation flow with a partner, the Microcks contract run, and cost.

Setup checklist

  • 1. Prerequisites: done when the exports print, the nodes are Ready and the Part 3 workloads run in react
  • 2. external-secrets: done when the store is valid, every ExternalSecret is synced and pg-app matches Secrets Manager
  • 3. SOPS and Argo CD: done when Git holds only encrypted values, a changed value fails the MAC check and dex-secrets appears through Argo CD
  • 4. Gatekeeper: done when the dry-run list is empty, the constraint is deny and an outside image is refused
  • 5. OpenFGA: done when the migration Job is complete, the model is loaded and the store and model IDs are set
  • 6. Dex: done when the discovery document shows the right issuer and a lab token has the expected claims
  • 7. Partner API: done when the offline tests pass, OpenFGA’s own check answers true and false, and the API returns the list and a 403
  • 8. Microcks: done when the mock answers, the contract test passes and a deliberate break fails it
  • 9. Evidence: done when case, geofence and access records appear and the verifier prints PASS
  • 10. Access and tamper tests: done when the access test prints PASS and every tamper step is refused or reported, then clean
  • 11. Cleanup path: done when you know which of these resources you will delete and in what order

Tools used

Dex, OpenFGA, SOPS, Microcks, CloudNativePG, external-secrets, OPA Gatekeeper, Knative, Argo CD and OpenTofu, plus Amazon RDS for PostgreSQL, S3 Object Lock, AWS KMS, AWS Secrets Manager, Amazon ECR and EKS Pod Identity.

What comes next

Part 5 operates the platform: Kanister backs up openGemini and CloudNativePG, a restore test brings back an RDS snapshot, Kube-burner and a larger simulator scale it, Kuberhealthy watches for a silent vehicle, and the teardown removes what costs money.

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.