This is Part 4 of the five-part build in the Sprint 5 project overview. Part 3 stores readings, opens excursion cases in RDS and records geofence events. Now outsiders arrive. A carrier should see its own shipments and nobody else’s. A hospital group should see what is being delivered to it. Nobody should hold a password in a file in Git, and nobody, including an administrator in a hurry, should be able to quietly change the record of what happened to a vaccine consignment. This part builds those four things: sign-in (Dex), per-shipment permission (OpenFGA on CloudNativePG), secrets (external-secrets with AWS Secrets Manager, and SOPS with KMS) with Gatekeeper admitting only our images, a contract for the partner API (Microcks), and a hash-chained custody record in S3 Object Lock.
Where this part fits. This part builds access, secrets and evidence (marked Part 4 on the overall diagram). Open the zoomable Sprint 5 architecture or read the Overview.
What this is and is not. This is engineering guidance, not legal or regulatory advice, and nothing here is validated or compliant on its own. The validation checklist in section 9 lists the evidence a quality unit usually asks for; deciding what a regulated product needs, and signing it off, is the company’s job. A hash chain and a locked bucket make tampering detectable and deletion hard, which is useful evidence, but they are not an electronic-signature system. If you find a mismatch with your versions, use the “Suggest a correction” link on this page.
What you build, and in what order
One image, the layers of this part. Open the zoomable diagram, or download it as SVG or PNG.
- Prerequisites and versions
- Secrets have one home each: external-secrets and Secrets Manager
- SOPS with KMS for values that must live in Git, and Argo CD
- Gatekeeper: only our images in
react - OpenFGA on CloudNativePG: the model and the tuples
- Dex: partner sign-in
- The partner API
- Microcks: mock and contract test
- Evidence: a hash-chained custody record, and the validation checklist
- The access-denial and tamper tests
- Troubleshooting, cost, cleanup, design notes and checklist
The decisions behind it
Every secret has one home. Values a pod reads and that we can rotate (the Kafka user, the database password, OpenFGA’s key) live in AWS Secrets Manager and reach the cluster through external-secrets. Values a partner gives us and that have no AWS home (their identity provider’s client secret) live in Git, encrypted with SOPS under the platform KMS key. Nothing is in both places, and no plain value is in either Git or a shell history.
Dex authenticates; OpenFGA decides. A partner’s token says who the caller is (a verified email). Which shipments that person may see is data in OpenFGA, not a claim in a token, so a membership change takes effect without waiting for a token to expire and without asking the partner to change their identity provider. The Overview’s wiring table mentions a groups claim: this build does not use one, for that reason.
The API says no by default. If OpenFGA, the database or the token keys cannot be reached, the API returns 503, never 200. A shipment that does not exist and a shipment you may not see both give 403, so the API does not reveal which shipment numbers exist.
Evidence is a chain, not a pile. Every custody record (an excursion case opened or closed, a geofence event, an access denial) is one small JSON object. Each names the SHA-256 of the record before it. The log of hashes sits in RDS, the objects in a locked bucket; changing either one is detectable from the other, and a verifier job checks both.
1. Prerequisites and versions
| Item | Version or detail | Why |
|---|---|---|
| Parts 1 to 3 complete | Every checkpoint passes | The hub, RDS, the evidence bucket and the shipments exist |
| Dex | v2.46.1 (chart dex/dex) |
Dex brief |
| OpenFGA | v1.18.x; chart openfga/openfga |
OpenFGA brief |
| CloudNativePG | The current operator release from cloudnative-pg/charts |
CloudNativePG brief |
| SOPS | v3.13.3 or newer |
SOPS brief |
| KSOPS | A tag from viaductoss/ksops on Docker Hub (set KSOPS_TAG) |
Argo CD has no built-in SOPS support |
| external-secrets | The current minor, pinned (helm search repo external-secrets/external-secrets) |
external-secrets brief |
| Gatekeeper | v3.23.1 or the chart you used in Sprint 4 |
OPA brief, Sprint 4 policies |
| Microcks | 1.15.0 chart; microcks-cli image from quay.io/microcks/microcks-cli |
Microcks brief |
| Tools on your workstation | sops, kubectl, helm, htpasswd, jq, Docker with buildx, Python 3.12 |
The new OpenTofu file reads module.eks, aws_kms_key.platform, aws_route53_zone.main and the pod_trust policy document from Part 1, and var.events_lb_hostname from Part 3. Set the shell variables this part uses:
export AWS_REGION=ap-south-1
export DOMAIN=coldchain.example.com
export EVIDENCE_BUCKET=coldchain-evidence-abc123 # the Object Lock bucket from Part 1
cd cold-chain/infra/foundation
export KMS_ARN=$(tofu output -raw kms_key_arn)
export PG_HOST=$(tofu output -raw pg_endpoint)
cd ../..
export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
mkdir -p infra platform secrets contracts config lib partner evidence ops scripts db
kubectl --context hub get nodes
kubectl --context hub -n react get ksvc,cronjob
Expected: three Ready nodes and the Part 3 Knative services and the mkt CronJob in react. Add the repositories used below:
helm repo add external-secrets https://charts.external-secrets.io
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo add openfga https://openfga.github.io/helm-charts
helm repo add dex https://charts.dexidp.io
helm repo add microcks https://microcks.io/helm
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm repo update
Checkpoint 1. The exports print, the nodes are Ready, the Part 3 workloads are in react, and helm search repo openfga finds a chart.
2. Secrets have one home each: external-secrets and Secrets Manager
2.1 The AWS side
One OpenTofu file adds: the Secrets Manager entries (the database password’s home and OpenFGA’s key), four Pod Identity roles (external-secrets, Argo CD’s repo server for KMS decryption, an evidence writer and an evidence verifier), and one ACM certificate for the two new public names, login and api. The evidence writer can add and read objects under custody/ and nothing else: it has no delete right and no right to bypass a retention lock. The verifier can only read.
# infra/foundation/access.tf
variable "evidence_bucket" { type = string } # the Object Lock bucket from Part 1, for example coldchain-evidence-abc123
# --- Secrets Manager: values that pods read through external-secrets --------------------
resource "aws_secretsmanager_secret" "pg_app" {
name = "${var.name}/pg-app"
kms_key_id = aws_kms_key.platform.arn
}
resource "random_password" "fga_key" {
length = 40
special = false
}
resource "aws_secretsmanager_secret" "fga_key" {
name = "${var.name}/fga-key"
kms_key_id = aws_kms_key.platform.arn
}
resource "aws_secretsmanager_secret_version" "fga_key" {
secret_id = aws_secretsmanager_secret.fga_key.id
secret_string = random_password.fga_key.result
}
# --- Pod Identity roles ------------------------------------------------------------------
locals {
sm_arns = [
"arn:aws:secretsmanager:${var.region}:${data.aws_caller_identity.me.account_id}:secret:AmazonMSK_${var.name}_*",
"arn:aws:secretsmanager:${var.region}:${data.aws_caller_identity.me.account_id}:secret:${var.name}/*",
]
}
data "aws_caller_identity" "me" {}
resource "aws_iam_role" "eso" {
name = "${var.name}-external-secrets"
assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "eso" {
role = aws_iam_role.eso.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Effect = "Allow", Action = ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"], Resource = local.sm_arns },
{ Effect = "Allow", Action = ["kms:Decrypt"], Resource = aws_kms_key.platform.arn },
]
})
}
resource "aws_eks_pod_identity_association" "eso" {
cluster_name = module.eks.cluster_name
namespace = "external-secrets"
service_account = "external-secrets"
role_arn = aws_iam_role.eso.arn
}
# Argo CD's repo server decrypts SOPS files with the platform key. It can decrypt; it cannot encrypt or manage the key.
resource "aws_iam_role" "argocd_sops" {
name = "${var.name}-argocd-sops"
assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "argocd_sops" {
role = aws_iam_role.argocd_sops.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{ Effect = "Allow", Action = ["kms:Decrypt", "kms:DescribeKey"], Resource = aws_kms_key.platform.arn }]
})
}
resource "aws_eks_pod_identity_association" "argocd_sops" {
cluster_name = module.eks.cluster_name
namespace = "argocd"
service_account = "argocd-repo-server"
role_arn = aws_iam_role.argocd_sops.arn
}
# Evidence: the writer can add objects and read them; it has no delete and no bypass of retention. The verifier only reads.
resource "aws_iam_role" "evidence_writer" {
name = "${var.name}-evidence-writer"
assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "evidence_writer" {
role = aws_iam_role.evidence_writer.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Effect = "Allow", Action = ["s3:PutObject", "s3:GetObject", "s3:GetObjectVersion"], Resource = "arn:aws:s3:::${var.evidence_bucket}/custody/*" },
{ Effect = "Allow", Action = ["kms:GenerateDataKey", "kms:Decrypt"], Resource = aws_kms_key.platform.arn },
]
})
}
resource "aws_eks_pod_identity_association" "evidence_writer" {
cluster_name = module.eks.cluster_name
namespace = "react"
service_account = "evidence-sync"
role_arn = aws_iam_role.evidence_writer.arn
}
resource "aws_iam_role" "evidence_verifier" {
name = "${var.name}-evidence-verifier"
assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}
resource "aws_iam_role_policy" "evidence_verifier" {
role = aws_iam_role.evidence_verifier.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{ Effect = "Allow", Action = ["s3:GetObject", "s3:GetObjectVersion", "s3:GetObjectRetention"], Resource = "arn:aws:s3:::${var.evidence_bucket}/custody/*" },
{ Effect = "Allow", Action = ["s3:ListBucket"], Resource = "arn:aws:s3:::${var.evidence_bucket}" },
{ Effect = "Allow", Action = ["kms:Decrypt"], Resource = aws_kms_key.platform.arn },
]
})
}
resource "aws_eks_pod_identity_association" "evidence_verifier" {
cluster_name = module.eks.cluster_name
namespace = "react"
service_account = "evidence-verify"
role_arn = aws_iam_role.evidence_verifier.arn
}
# --- Public names: login (Dex) and api (partner API) share one certificate -------------------
resource "aws_acm_certificate" "access" {
domain_name = "login.${var.domain}"
subject_alternative_names = ["api.${var.domain}"]
validation_method = "DNS"
}
resource "aws_route53_record" "access_validation" {
for_each = { for o in aws_acm_certificate.access.domain_validation_options : o.domain_name => o }
zone_id = aws_route53_zone.main.zone_id
name = each.value.resource_record_name
type = each.value.resource_record_type
records = [each.value.resource_record_value]
ttl = 60
allow_overwrite = true
}
resource "aws_acm_certificate_validation" "access" {
certificate_arn = aws_acm_certificate.access.arn
validation_record_fqdns = [for r in aws_route53_record.access_validation : r.fqdn]
}
resource "aws_route53_record" "access" {
for_each = var.events_lb_hostname == "" ? toset([]) : toset(["login", "api"])
zone_id = aws_route53_zone.main.zone_id
name = "${each.key}.${var.domain}"
type = "CNAME"
ttl = 60
records = [var.events_lb_hostname]
}
output "access_cert_arn" { value = aws_acm_certificate_validation.access.certificate_arn }
output "pg_app_secret_id" { value = aws_secretsmanager_secret.pg_app.id }
cp infra/access.tf infra/foundation/access.tf
export TF_VAR_events_lb_hostname=$(kubectl --context hub -n kourier-system get svc kourier-public -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
(cd infra/foundation && tofu apply -var "evidence_bucket=$EVIDENCE_BUCKET")
export ACCESS_CERT=$(cd infra/foundation && tofu output -raw access_cert_arn)
aws eks list-pod-identity-associations --cluster-name coldchain-hub --query 'associations[].[namespace,serviceAccount]' --output text
Keep TF_VAR_events_lb_hostname set on every apply from now on: the Route 53 record for events is created only while the variable has a value, and an apply without it would remove that record. Expected: the apply succeeds (the certificate waits for its DNS validation record, a few minutes), and the association list now includes external-secrets, argocd-repo-server, evidence-sync and evidence-verify.
2.2 Move the database password into Secrets Manager
The password for the coldchain_app role was generated in Part 3 and sits in the Kubernetes Secret pg-app. Put the same value in Secrets Manager. Nothing in PostgreSQL changes, so nothing can break:
aws secretsmanager put-secret-value --secret-id coldchain/pg-app --secret-string \
"$(kubectl --context hub -n react get secret pg-app -o jsonpath='{.data.password}' | base64 -d)"
aws secretsmanager describe-secret --secret-id coldchain/pg-app --query '[Name,KmsKeyId]' --output text
Rotation later means changing it in two places in order: ALTER ROLE coldchain_app PASSWORD ... as the administrator, then put-secret-value. Within the refresh interval (15 minutes here) the Kubernetes Secret follows, and pods pick the new value up at their next start. Automating that is Secrets Manager’s rotation feature with a Lambda, which this build does not use.
2.3 Install external-secrets and the store
helm upgrade --install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace --wait
kubectl --context hub -n external-secrets get pods
kubectl --context hub create namespace access
The store has no auth block on purpose: the controller then uses the AWS credentials its pod receives, which is the role from the Pod Identity association, so no access key exists anywhere. The ExternalSecrets that follow use creationPolicy: Merge for Secrets that Parts 2 and 3 already created by hand: external-secrets fills the named keys of the existing Secret instead of replacing it, so there is no moment without a Secret, and keys like host and bootstrap stay.
# platform/external-secrets.yaml
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata: { name: aws-sm }
spec:
provider:
aws:
service: SecretsManager
region: ap-south-1 # no auth block: the controller uses the role from its EKS Pod Identity association
---
# Kafka credentials, one secret in Secrets Manager (AmazonMSK_coldchain_ingest, keys username and password), three shapes.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: msk-scram, namespace: ingest }
spec:
refreshInterval: 15m
secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
target:
name: msk-scram
creationPolicy: Merge # the hand-made Secret from Part 2 already exists; take it over without a gap
data:
- { secretKey: username, remoteRef: { key: AmazonMSK_coldchain_ingest, property: username } }
- { secretKey: password, remoteRef: { key: AmazonMSK_coldchain_ingest, property: password } }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: msk-kn, namespace: react }
spec:
refreshInterval: 15m
secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
target: { name: msk-kn, creationPolicy: Merge }
data:
- { secretKey: user, remoteRef: { key: AmazonMSK_coldchain_ingest, property: username } }
- { secretKey: password, remoteRef: { key: AmazonMSK_coldchain_ingest, property: password } }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: msk-broker-auth, namespace: knative-eventing }
spec:
refreshInterval: 15m
secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
target: { name: msk-broker-auth, creationPolicy: Merge }
data:
- { secretKey: user, remoteRef: { key: AmazonMSK_coldchain_ingest, property: username } }
- { secretKey: password, remoteRef: { key: AmazonMSK_coldchain_ingest, property: password } }
---
# The application's database password. The Secret in Secrets Manager holds the same value as the role in PostgreSQL.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: pg-app, namespace: react }
spec:
refreshInterval: 15m
secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
target: { name: pg-app, creationPolicy: Merge }
data:
- { secretKey: password, remoteRef: { key: coldchain/pg-app } }
---
# OpenFGA's pre-shared key, for the server and for its clients.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: fga-key, namespace: access }
spec:
refreshInterval: 15m
secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
target: { name: fga-keys, creationPolicy: Owner, template: { data: { keys: "{{ .key }}" } } }
data:
- { secretKey: key, remoteRef: { key: coldchain/fga-key } }
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: fga-key, namespace: react }
spec:
refreshInterval: 15m
secretStoreRef: { kind: ClusterSecretStore, name: aws-sm }
target: { name: fga-token, creationPolicy: Owner, template: { data: { token: "{{ .key }}" } } }
data:
- { secretKey: key, remoteRef: { key: coldchain/fga-key } }
kubectl --context hub apply -f platform/external-secrets.yaml
kubectl --context hub get clustersecretstore aws-sm
kubectl --context hub get externalsecret -A
Expected: the store Valid and Ready, and each ExternalSecret SecretSynced with status Ready. Prove the pipeline by changing a value in Secrets Manager and watching it arrive without touching the cluster:
kubectl --context hub -n react annotate externalsecret pg-app force-sync=$(date +%s) --overwrite
kubectl --context hub -n react get secret pg-app -o jsonpath='{.data}' | jq 'keys'
kubectl --context hub -n access get secret fga-keys -o jsonpath='{.data.keys}' | base64 -d | wc -c
Expected: pg-app still has host and password, and fga-keys holds a 40-character value. Restart the consumers once so they read the Secrets from the store (kubectl -n ingest rollout restart deploy/vector, and the Knative services in react get a new revision on their next deploy).
Checkpoint 2. The store is valid, every ExternalSecret is synced, and the same password is in Secrets Manager and in pg-app.
3. SOPS with KMS for values that must live in Git, and Argo CD
Some values arrive from outside and have no AWS home: the client ID and secret that a carrier’s identity provider gives us for the Dex connector, and the secret of Dex’s own API client. They belong in Git, because Git is how this platform is deployed, but encrypted. SOPS encrypts only the values of the file, so a pull request still shows which keys changed. The rule file says which key encrypts what:
# .sops.yaml
creation_rules:
- path_regex: secrets/.*\.enc\.yaml$
encrypted_regex: '^(data|stringData)$' # keys and metadata stay readable in review; only values are encrypted
kms: 'KMS_ARN'
sed -i "s#KMS_ARN#$KMS_ARN#" .sops.yaml
Write the plain file once, encrypt it, and delete the plain copy. Generate the one value that is ours (API_CLIENT_SECRET) now; the partner values stay as placeholders until a partner gives you real ones:
# secrets/dex-secrets.plain.yaml
# Encrypt this file, then delete the plain copy: sops --encrypt secrets/dex-secrets.plain.yaml > secrets/dex-secrets.enc.yaml
apiVersion: v1
kind: Secret
metadata: { name: dex-secrets, namespace: access }
type: Opaque
stringData:
API_CLIENT_SECRET: REPLACE_WITH_RANDOM_32_CHARS
CARRIER_A_CLIENT_ID: REPLACE_WITH_VALUE_FROM_THE_PARTNER
CARRIER_A_CLIENT_SECRET: REPLACE_WITH_VALUE_FROM_THE_PARTNER
CUSTOMER_PUNE_CLIENT_ID: REPLACE_WITH_VALUE_FROM_THE_PARTNER
CUSTOMER_PUNE_CLIENT_SECRET: REPLACE_WITH_VALUE_FROM_THE_PARTNER
export API_CLIENT_SECRET=$(openssl rand -hex 24)
sed -i "s#REPLACE_WITH_RANDOM_32_CHARS#$API_CLIENT_SECRET#" secrets/dex-secrets.plain.yaml
sops --encrypt secrets/dex-secrets.plain.yaml > secrets/dex-secrets.enc.yaml
shred -u secrets/dex-secrets.plain.yaml 2>/dev/null || rm -f secrets/dex-secrets.plain.yaml
head -12 secrets/dex-secrets.enc.yaml
sops --decrypt secrets/dex-secrets.enc.yaml | grep -c REPLACE_WITH
Expected: the encrypted file shows stringData keys with ENC[AES256_GCM,...] values and a sops: block listing the KMS ARN; the decrypt prints 4 (the four placeholders). Check that SOPS protects integrity as well as secrecy: change one character inside an ENC[...] value and decrypt again. SOPS refuses with a MAC error.
3.1 Argo CD decrypts it
Argo CD renders manifests with kustomize in its repo server. KSOPS is a kustomize plugin that calls SOPS during the render. The repo server needs the plugin binary, kustomize options that allow exec plugins, and rights on the KMS key (the Pod Identity association from section 2.1 gives kms:Decrypt only).
# platform/argocd-ksops-values.yaml
# Adds the KSOPS kustomize plugin to Argo CD's repo server. Pick KSOPS_TAG from the project's releases and read its README
# for the exact init-container recipe of that tag. Apply with:
# helm upgrade argocd argo/argo-cd -n argocd --reuse-values -f platform/argocd-ksops-values.yaml
configs:
cm:
kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"
repoServer:
env:
- { name: XDG_CONFIG_HOME, value: /.config }
- { name: KUSTOMIZE_PLUGIN_HOME, value: /.config/kustomize/plugin }
- { name: AWS_REGION, value: ap-south-1 } # the role comes from the Pod Identity association
volumes:
- { name: custom-tools, emptyDir: {} }
initContainers:
- name: install-ksops
image: viaductoss/ksops:KSOPS_TAG
command: ["/bin/sh", "-c"]
args: ["mv ksops /custom-tools/ && mv $GOPATH/bin/kustomize /custom-tools/"]
volumeMounts: [{ mountPath: /custom-tools, name: custom-tools }]
volumeMounts:
- { mountPath: /usr/local/bin/kustomize, name: custom-tools, subPath: kustomize }
- { mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops, name: custom-tools, subPath: ksops }
export KSOPS_TAG=v4.3.2 # replace with the tag you chose from the project's releases
sed "s#KSOPS_TAG#$KSOPS_TAG#" platform/argocd-ksops-values.yaml > /tmp/ksops-values.yaml
helm upgrade argocd argo/argo-cd -n argocd --reuse-values -f /tmp/ksops-values.yaml --kube-context hub --wait
kubectl --context hub -n argocd rollout status deploy/argocd-repo-server
The generator and kustomization tell kustomize which encrypted file to decrypt; the Application points Argo CD at the folder:
# secrets/ksops-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
name: coldchain-secrets
annotations: { config.kubernetes.io/function: "exec:\n path: ksops\n" }
files:
- ./dex-secrets.enc.yaml
# secrets/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
generators:
- ksops-generator.yaml
# gitops/apps/secrets.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: secrets, namespace: argocd }
spec:
project: default
source:
repoURL: https://github.com/YOUR-ORG/cold-chain.git
targetRevision: main
path: secrets
destination: { server: https://kubernetes.default.svc, namespace: access }
syncPolicy: { automated: { prune: true, selfHeal: true } }
cp platform/argocd-app-secrets.yaml gitops/apps/secrets.yaml
git add .sops.yaml secrets gitops/apps/secrets.yaml && git commit -m "Dex secrets, SOPS-encrypted" && git push
kubectl --context hub -n argocd get application secrets
kubectl --context hub -n access get secret dex-secrets -o jsonpath='{.data}' | jq 'keys'
Expected: the Application Synced and Healthy, and the Secret dex-secrets with its five keys. Argo CD wrote the plain Secret into the cluster, so keep Kubernetes encryption at rest and RBAC on Secrets in mind: SOPS protects Git, not etcd. If the Application shows a plugin or kustomize error instead, kubectl -n argocd logs deploy/argocd-repo-server says whether it is the plugin path, the build options or the KMS call.
Checkpoint 3. The file in Git has only encrypted values, a changed value fails the integrity check, and dex-secrets appears in the cluster through Argo CD.
4. Gatekeeper: only our images in react
The Sprint 4 project used Gatekeeper and a registry allow-list the same way. Install it if the hub does not have it, then add a template that allows an image only when its name starts with an allowed prefix.
helm upgrade --install gatekeeper gatekeeper/gatekeeper -n gatekeeper-system --create-namespace --kube-context hub --wait
kubectl --context hub -n gatekeeper-system get pods
The allow-list is short and explicit. Knative adds a sidecar from gcr.io/knative-releases/, and the one-off psql pods in this series use the pinned postgres:17-alpine. Everything else in react must come from our ECR repositories. A default memory limit for containers goes in the same file as a LimitRange, because the resource rule in the Overview is simpler as a namespace default than as a policy. Start in dry run:
# platform/gatekeeper.yaml
# Same style as the Sprint 4 policies (see Sprint 4 project, Part 4, section 6). This one allows images from our ECR only.
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata: { name: coldallowedregistry }
spec:
crd:
spec:
names: { kind: ColdAllowedRegistry }
validation:
openAPIV3Schema:
type: object
properties:
prefixes:
type: array
items: { type: string }
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package coldallowedregistry
containers[c] { c := input.review.object.spec.containers[_] }
containers[c] { c := input.review.object.spec.initContainers[_] }
allowed(img) { startswith(img, input.parameters.prefixes[_]) }
violation[{"msg": msg}] {
c := containers[_]
not allowed(c.image)
msg := sprintf("image %v is not on the allow-list for this namespace", [c.image])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: ColdAllowedRegistry
metadata: { name: react-images-from-ecr }
spec:
enforcementAction: dryrun # change to deny after the audit shows no violations
match:
kinds: [{ apiGroups: [""], kinds: ["Pod"] }]
namespaces: [react]
parameters:
prefixes:
- ACCOUNT.dkr.ecr.ap-south-1.amazonaws.com/coldchain/
- gcr.io/knative-releases/ # Knative's queue-proxy sidecar
- postgres:17-alpine # the one-off psql pods in this series
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: react }
spec:
limits:
- type: Container
default: { memory: 256Mi }
defaultRequest: { cpu: 50m, memory: 64Mi }
sed "s#ACCOUNT#$ACCOUNT#" platform/gatekeeper.yaml | kubectl --context hub apply -f -
sleep 60
kubectl --context hub get coldallowedregistry react-images-from-ecr -o jsonpath='{.status.totalViolations}{"\n"}'
kubectl --context hub get coldallowedregistry react-images-from-ecr -o jsonpath='{range .status.violations[*]}{.name}: {.message}{"\n"}{end}'
Expected: a violation count and, for each violation, the pod and the image. Fix every real violation before you enforce: an image you built moves to the ECR path; an upstream image you need is added to the list on purpose. When the list is empty, switch to deny and try a pod from outside:
kubectl --context hub patch coldallowedregistry react-images-from-ecr --type merge -p '{"spec":{"enforcementAction":"deny"}}'
sleep 10
kubectl --context hub -n react run bad --image=nginx:1.27 --restart=Never
Expected: the last command is refused by the admission webhook with the message that the image is not on the allow-list for this namespace. A pod using postgres:17-alpine or one of our digests is admitted.
Checkpoint 4. The dry-run list is empty, the constraint is deny, and an image from outside is refused.
5. OpenFGA on CloudNativePG: the model and the tuples
OpenFGA stores its tuples in PostgreSQL. The business data is in RDS (Part 1), but OpenFGA’s data is a small, separate, operator-managed database, so a CNCF project keeps a CNCF project’s data and the two failure domains stay apart. Install the operator and create the cluster:
helm upgrade --install cnpg cnpg/cloudnative-pg -n cnpg-system --create-namespace --kube-context hub --wait
# platform/cnpg.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata: { name: fga-pg, namespace: access }
spec:
instances: 2 # one primary and one replica, spread by the operator's defaults
storage: { size: 5Gi, storageClass: gp3 }
bootstrap:
initdb: { database: app, owner: app }
resources: { requests: { cpu: 100m, memory: 256Mi }, limits: { memory: 512Mi } }
kubectl --context hub apply -f platform/cnpg.yaml
kubectl --context hub -n access wait --for=condition=Ready cluster/fga-pg --timeout=300s
kubectl --context hub -n access get secret fga-pg-app -o jsonpath='{.data}' | jq 'keys'
Expected: the cluster Ready with a primary and a replica, and a Secret fga-pg-app that has a uri key (and others). Backups of this database and the restore test are in Part 5.
5.1 The server
The chart runs the schema migration as a Job before the server starts, reads the database address from fga-pg-app, and requires the pre-shared key from the Secret that external-secrets filled in section 2:
# platform/openfga-values.yaml
replicaCount: 2
datastore:
engine: postgres
uriSecret: fga-pg-app # CloudNativePG's application secret has a key called uri
migrations:
enabled: true # a Job runs the schema migration before the server starts
authn:
method: preshared
preshared:
keysSecret: fga-keys # key: keys
resources: { requests: { cpu: 100m, memory: 128Mi }, limits: { memory: 256Mi } }
helm upgrade --install openfga openfga/openfga -n access -f platform/openfga-values.yaml --kube-context hub --wait
kubectl --context hub -n access get pods,jobs
Expected: the migration Job Complete and two OpenFGA pods Running.
5.2 The model
The model has four types. A person is a user, identified by verified email. A carrier and a customer each have members. A shipment has a carrier and a customer, and can_view is true for any member of either. The text form is for reading, the JSON form is what the HTTP API takes:
# config/model.fga
model
schema 1.1
type user
type carrier
relations
define member: [user]
type customer
relations
define member: [user]
type shipment
relations
define carrier: [carrier]
define customer: [customer]
define can_view: member from carrier or member from customer
{
"schema_version": "1.1",
"type_definitions": [
{ "type": "user" },
{ "type": "carrier",
"relations": { "member": { "this": {} } },
"metadata": { "relations": { "member": { "directly_related_user_types": [ { "type": "user" } ] } } } },
{ "type": "customer",
"relations": { "member": { "this": {} } },
"metadata": { "relations": { "member": { "directly_related_user_types": [ { "type": "user" } ] } } } },
{ "type": "shipment",
"relations": {
"carrier": { "this": {} },
"customer": { "this": {} },
"can_view": { "union": { "child": [
{ "tupleToUserset": { "tupleset": { "relation": "carrier" }, "computedUserset": { "relation": "member" } } },
{ "tupleToUserset": { "tupleset": { "relation": "customer" }, "computedUserset": { "relation": "member" } } } ] } } },
"metadata": { "relations": {
"carrier": { "directly_related_user_types": [ { "type": "carrier" } ] },
"customer": { "directly_related_user_types": [ { "type": "customer" } ] },
"can_view": { "directly_related_user_types": [] } } } }
]
}
The membership data (who belongs to which carrier or customer) is a small file in Git. The shipment-to-organisation links come from RDS: the carrier from shipments.carrier, the customer from the destination through a mapping file.
{
"carrier:carrier-a": ["anita@carrier-a.example"],
"carrier:carrier-b": ["bala@carrier-b.example"],
"customer:customer-pune": ["chitra@pune.example"]
}
{ "Pune": "customer-pune" }
The loader creates the store once and writes the model. A new model version never changes the old one, so the printed FGA_MODEL_ID pins exactly the rules the API was tested with:
# ops/fga_load.py
#!/usr/bin/env python3
"""Create the OpenFGA store (once) and write the model. Prints FGA_STORE_ID and FGA_MODEL_ID.
kubectl -n access port-forward svc/openfga 8080:8080 &
FGA_URL=http://localhost:8080 FGA_TOKEN=$KEY python3 ops/fga_load.py config/model.json"""
import json, os, sys, urllib.request
url, token = os.environ.get("FGA_URL", "http://localhost:8080").rstrip("/"), os.environ.get("FGA_TOKEN", "")
def call(method, path, body=None):
req = urllib.request.Request(url + path, method=method, data=json.dumps(body).encode() if body is not None else None,
headers={"content-type": "application/json", **({"Authorization": "Bearer " + token} if token else {})})
with urllib.request.urlopen(req, timeout=10) as r:
return json.loads(r.read() or b"{}")
stores = call("GET", "/stores").get("stores", [])
store = next((s for s in stores if s["name"] == "coldchain"), None) or call("POST", "/stores", {"name": "coldchain"})
sid = store["id"]
model = json.load(open(sys.argv[1] if len(sys.argv) > 1 else "config/model.json"))
mid = call("POST", f"/stores/{sid}/authorization-models", model)["authorization_model_id"]
print(f"FGA_STORE_ID={sid}")
print(f"FGA_MODEL_ID={mid}")
export FGA_TOKEN=$(aws secretsmanager get-secret-value --secret-id coldchain/fga-key --query SecretString --output text)
kubectl --context hub -n access port-forward svc/openfga 8080:8080 &
sleep 3
FGA_URL=http://localhost:8080 python3 ops/fga_load.py config/model.json | tee /tmp/fga-ids.txt
export FGA_STORE_ID=$(grep STORE /tmp/fga-ids.txt | cut -d= -f2); export FGA_MODEL_ID=$(grep MODEL /tmp/fga-ids.txt | cut -d= -f2)
Expected: two lines, FGA_STORE_ID=... and FGA_MODEL_ID=.... If the server answers 401, the key in FGA_TOKEN does not match the one in the fga-keys Secret.
5.3 The tuples, from RDS
The partner API’s own image also contains the tuple sync (it is a small program that makes OpenFGA’s tuples equal to what RDS and the membership files say, adding what is missing and removing what is gone). It runs every five minutes as a CronJob, so a new shipment or a membership change reaches OpenFGA within that time. First the code, all of it:
# lib/fgalib.py
"""Small OpenFGA HTTP client (standard library only)."""
import json, os, urllib.request, urllib.error
class FgaError(Exception):
pass
class Fga:
def __init__(self, url=None, store_id=None, model_id=None, token=None, timeout=3):
self.url = (url or os.environ.get("FGA_URL", "http://openfga.access.svc.cluster.local:8080")).rstrip("/")
self.store_id = store_id or os.environ.get("FGA_STORE_ID", "")
self.model_id = model_id or os.environ.get("FGA_MODEL_ID", "")
self.token = token or os.environ.get("FGA_TOKEN", "")
self.timeout = timeout
def _post(self, path, body):
req = urllib.request.Request(self.url + path, data=json.dumps(body).encode(), method="POST",
headers={"content-type": "application/json"})
if self.token:
req.add_header("Authorization", "Bearer " + self.token)
try:
with urllib.request.urlopen(req, timeout=self.timeout) as r:
raw = r.read()
return json.loads(raw) if raw else {}
except urllib.error.HTTPError as e:
raise FgaError(f"{path}: {e.code} {e.read()[:200]!r}")
except OSError as e:
raise FgaError(f"{path}: {e}")
def check(self, user, relation, obj):
body = {"tuple_key": {"user": user, "relation": relation, "object": obj}}
if self.model_id:
body["authorization_model_id"] = self.model_id
return bool(self._post(f"/stores/{self.store_id}/check", body).get("allowed"))
def list_objects(self, user, relation, typ):
body = {"user": user, "relation": relation, "type": typ}
if self.model_id:
body["authorization_model_id"] = self.model_id
return self._post(f"/stores/{self.store_id}/list-objects", body).get("objects", [])
def read_all(self, typ):
"""All tuples whose object is of this type, as a set of (user, relation, object)."""
out, tok = set(), ""
while True:
body = {"tuple_key": {"object": typ + ":"}, "page_size": 100}
if tok:
body["continuation_token"] = tok
r = self._post(f"/stores/{self.store_id}/read", body)
for t in r.get("tuples", []):
k = t["key"]
out.add((k["user"], k["relation"], k["object"]))
tok = r.get("continuation_token", "")
if not tok:
return out
def write(self, writes=(), deletes=()):
writes, deletes = list(writes), list(deletes)
for i in range(0, max(len(writes), len(deletes)), 50):
body = {}
w, d = writes[i:i + 50], deletes[i:i + 50]
if w:
body["writes"] = {"tuple_keys": [{"user": u, "relation": r, "object": o} for u, r, o in w]}
if d:
body["deletes"] = {"tuple_keys": [{"user": u, "relation": r, "object": o} for u, r, o in d]}
if self.model_id:
body["authorization_model_id"] = self.model_id
self._post(f"/stores/{self.store_id}/write", body)
# lib/pg.py
"""Thread-local PostgreSQL connection from the PG* environment variables."""
import threading
import psycopg
_local = threading.local()
def db():
c = getattr(_local, "conn", None)
if c is None or c.closed:
c = psycopg.connect(autocommit=True, connect_timeout=5)
_local.conn = c
return c
# ops/sync_tuples.py
#!/usr/bin/env python3
"""Make OpenFGA's tuples match RDS and the membership files. Safe to run repeatedly; prints what it changed.
Tuples written: user:<email> member carrier:<id> | customer:<id> (from config/members.json)
carrier:<id> carrier shipment:<id> (from shipments.carrier)
customer:<id> customer shipment:<id> (from shipments.destination and config/customers.json)"""
import json, sys
def desired(shipments, members, customers):
want = set()
for obj, emails in members.items():
kind = obj.split(":")[0]
want |= {("user:" + e.lower(), "member", obj) for e in emails}
for sid, carrier, destination in shipments:
if carrier:
want.add((f"carrier:{carrier}", "carrier", f"shipment:{sid}"))
if destination in customers:
want.add((f"customer:{customers[destination]}", "customer", f"shipment:{sid}"))
return want
def sync(fga, shipments, members, customers, out=print):
want = desired(shipments, members, customers)
have = set()
for typ in ("carrier", "customer", "shipment"):
have |= fga.read_all(typ)
add, drop = sorted(want - have), sorted(have - want)
if add or drop:
fga.write(add, drop)
out(f"tuples: {len(want)} wanted, {len(add)} added, {len(drop)} removed")
return len(add), len(drop)
if __name__ == "__main__":
import psycopg
from lib.fgalib import Fga
members, customers = json.load(open("config/members.json")), json.load(open("config/customers.json"))
with psycopg.connect() as c, c.cursor() as cur:
cur.execute("SELECT shipment_id, carrier, destination FROM shipments")
rows = cur.fetchall()
sync(Fga(), rows, members, customers)
RDS is private, so the sync cannot run from your workstation; it runs as a Job from the image built in section 7, and section 7.3 runs it once and checks the result against OpenFGA itself. The function that decides what to add and what to remove has an offline test, shown in section 7.1.
Checkpoint 5. OpenFGA is up behind its key, the model is loaded, and FGA_STORE_ID and FGA_MODEL_ID are set.
6. Dex: partner sign-in
Dex is the one sign-in front for partners. Partners keep their own identity providers; Dex is an OpenID Connect provider that, behind the scenes, signs the user in elsewhere and issues its own token. The partner API trusts only Dex’s keys, so adding a carrier means adding a connector, not changing the API.
6.1 Lab users and the server
For the build you need users that exist without a partner’s help. Dex has a local password database; the three lab users below have no groups, because permission is OpenFGA’s job. Make three bcrypt hashes of one lab password and put them in the values file:
export LAB_PW=$(openssl rand -base64 12 | tr -d '/+=')
H() { htpasswd -bnBC 10 "" "$LAB_PW" | tr -d ':\n' | sed 's/$2y/$2a/'; }
sed -e "s#DOMAIN#$DOMAIN#g" -e "s#HASH_ANITA#$(H)#" -e "s#HASH_BALA#$(H)#" -e "s#HASH_CHITRA#$(H)#" \
platform/dex-values.yaml > /tmp/dex-values.yaml
echo "lab password for the three users: $LAB_PW"
# platform/dex-values.yaml
# Dex: one sign-in front for partners. The lab users below prove the flow; real partners come in through connectors
# (platform/dex-federation.yaml). Replace DOMAIN and the three password hashes before applying.
replicaCount: 2
envFrom:
- secretRef: { name: dex-secrets } # from the SOPS file in Git: API_CLIENT_SECRET and the partners' client secrets
config:
issuer: https://login.DOMAIN
storage:
type: kubernetes # Dex keeps its state in custom resources; no database
config: { inCluster: true }
web: { http: 0.0.0.0:5556 }
oauth2:
passwordConnector: local # lets the test script ask for a token without a browser
skipApprovalScreen: true
enablePasswordDB: true
staticPasswords:
- { email: anita@carrier-a.example, username: anita, userID: "10000000-0000-4000-8000-00000000000a", hash: "HASH_ANITA" }
- { email: bala@carrier-b.example, username: bala, userID: "10000000-0000-4000-8000-00000000000b", hash: "HASH_BALA" }
- { email: chitra@pune.example, username: chitra, userID: "10000000-0000-4000-8000-00000000000c", hash: "HASH_CHITRA" }
staticClients:
- id: coldchain-partner-api
name: Cold chain partner API
secretEnv: API_CLIENT_SECRET
redirectURIs: ["https://api.DOMAIN/callback"]
expiry:
idTokens: 15m # short: the API checks the token on every call
helm upgrade --install dex dex/dex -n access -f /tmp/dex-values.yaml --kube-context hub --wait
kubectl --context hub -n access get pods -l app.kubernetes.io/name=dex
kubectl --context hub -n access logs deploy/dex --tail=5
Expected: two Dex pods Running and a log line that the server is listening on 0.0.0.0:5556. If the pods do not start, kubectl describe usually points at the missing dex-secrets Secret from section 3. Dex keeps its state in custom resources of the cluster (the kubernetes storage type), so there is no database to run.
6.2 A public name for Dex and for the API
Partners reach Dex and the API from the internet over HTTPS, through the load balancer built in Part 3. That load balancer terminates TLS with the certificate for events.<domain>. Give it the second certificate too (a list of certificate ARNs on the annotation: the first is the default, the others are chosen by the name the client asks for), publish the two names in Route 53, and map them with Knative:
# platform/access-public.yaml
# Two more public names on the load balancer from Part 3: login.DOMAIN (Dex) and api.DOMAIN (partner API).
apiVersion: networking.internal.knative.dev/v1alpha1
kind: ClusterDomainClaim
metadata: { name: login.DOMAIN }
spec: { namespace: access }
---
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata: { name: login.DOMAIN, namespace: access }
spec:
ref: { apiVersion: v1, kind: Service, name: dex } # Dex is a plain Deployment, so the mapping points at its Service
---
apiVersion: networking.internal.knative.dev/v1alpha1
kind: ClusterDomainClaim
metadata: { name: api.DOMAIN }
spec: { namespace: react }
---
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata: { name: api.DOMAIN, namespace: react }
spec:
ref: { apiVersion: serving.knative.dev/v1, kind: Service, name: partner-api }
export EVENTS_CERT=$(cd infra/foundation && tofu output -raw events_cert_arn)
kubectl --context hub -n kourier-system annotate svc kourier-public --overwrite \
service.beta.kubernetes.io/aws-load-balancer-ssl-cert="$EVENTS_CERT,$ACCESS_CERT"
sed "s#DOMAIN#$DOMAIN#g" platform/access-public.yaml | kubectl --context hub apply -f -
dig +short login.$DOMAIN api.$DOMAIN
curl -s https://login.$DOMAIN/.well-known/openid-configuration | jq '{issuer, jwks_uri, token_endpoint}'
Expected: both names resolve to the load balancer, and the discovery document shows "issuer": "https://login.<domain>". The dig answer exists because section 2.1 ran with TF_VAR_events_lb_hostname set, which created the two CNAMEs. If the discovery call returns a Knative 404, the mapping to Dex’s Service has not become ready: kubectl -n access get domainmapping shows why.
6.3 Get a token
Ask for an ID token the way the test script will, with the password grant (this is a convenience of the lab; real partners use the browser flow):
export API_CLIENT_SECRET=$(kubectl --context hub -n access get secret dex-secrets -o jsonpath='{.data.API_CLIENT_SECRET}' | base64 -d)
curl -s -u "coldchain-partner-api:$API_CLIENT_SECRET" https://login.$DOMAIN/token \
-d grant_type=password -d username=anita@carrier-a.example -d "password=$LAB_PW" -d 'scope=openid email' \
| jq -r .id_token | cut -d. -f2 | base64 -d 2>/dev/null | jq '{iss, aud, email, email_verified, exp}'
Expected: iss is https://login.<domain>, aud is coldchain-partner-api, email is the user’s address, email_verified is true, and exp is 15 minutes ahead. (The base64 -d may complain about padding on some systems; the JSON still prints.)
6.4 Federating a partner
To let a carrier’s own staff sign in, add an OpenID Connect connector. The partner gives you an issuer URL, a client ID and a client secret, and you give the partner your redirect URL https://login.<domain>/callback. The values go into the SOPS file from section 3 (the four placeholders), and the connector reads them from the environment:
# platform/dex-federation.yaml
# Partner identity providers, added as connectors. Each partner gives you an issuer URL, a client ID and a client secret.
# Apply only when the issuers are real: Dex contacts each issuer when it starts.
# helm upgrade dex dex/dex -n access -f platform/dex-values.yaml -f platform/dex-federation.yaml
envFrom:
- secretRef: { name: dex-secrets }
config:
connectors:
- type: oidc
id: carrier-a
name: Carrier A
config:
issuer: https://login.microsoftonline.com/TENANT_ID/v2.0 # example: a carrier's Microsoft Entra ID tenant
clientID: $CARRIER_A_CLIENT_ID
clientSecret: $CARRIER_A_CLIENT_SECRET
redirectURI: https://login.DOMAIN/callback
scopes: [openid, email, profile]
- type: oidc
id: customer-pune
name: Pune hospital group
config:
issuer: https://PARTNER.okta.example.com # example: a customer's Okta
clientID: $CUSTOMER_PUNE_CLIENT_ID
clientSecret: $CUSTOMER_PUNE_CLIENT_SECRET
redirectURI: https://login.DOMAIN/callback
scopes: [openid, email, profile]
sops secrets/dex-secrets.enc.yaml # edit the four partner values in your editor; SOPS re-encrypts on save
git commit -am "Carrier A and Pune connectors" && git push
sed -e "s#DOMAIN#$DOMAIN#g" platform/dex-federation.yaml > /tmp/dex-federation.yaml
helm upgrade dex dex/dex -n access -f /tmp/dex-values.yaml -f /tmp/dex-federation.yaml --kube-context hub --wait
Apply it only when the issuers are real: Dex contacts each issuer when it starts and will not come up with a placeholder. Then open https://login.<domain>/auth?client_id=coldchain-partner-api&response_type=code&scope=openid%20email&redirect_uri=https://api.<domain>/callback in a browser: Dex shows a button for each connector. The user’s email must match an entry in config/members.json, or the user signs in and sees nothing, which is the intended result for a stranger. Dex’s SAML connector carries a warning in its own documentation about being unmaintained, so use OIDC with partners wherever they can.
Checkpoint 6. The discovery document shows the right issuer, a token for a lab user has the expected claims, and (once you have a real partner) its connector signs a user in.
7. The partner API
The API is a small program with four read-only routes. The route code, the token check, the OpenFGA calls and the database access are separate pieces, so the logic is one function that is tested without a network.
# partner/app.py
#!/usr/bin/env python3
"""Partner API. Every request carries a Dex ID token; OpenFGA decides which shipments the caller may see."""
import json, os, re, sys, uuid
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse
SHIPMENT = re.compile(r"^SHP-[0-9]{4,10}$")
class Denied(Exception):
pass
class Unauthenticated(Exception):
pass
class JwtVerifier:
"""Checks signature, issuer, audience and expiry against Dex's published keys."""
def __init__(self, issuer, audience, jwks_url):
import jwt
self.jwt, self.issuer, self.audience = jwt, issuer, audience
self.jwks = jwt.PyJWKClient(jwks_url, cache_keys=True)
def __call__(self, token):
try:
key = self.jwks.get_signing_key_from_jwt(token).key
return self.jwt.decode(token, key, algorithms=["RS256", "ES256"], issuer=self.issuer,
audience=self.audience, options={"require": ["exp", "iss", "aud"]})
except Exception as e:
raise Unauthenticated(type(e).__name__)
class Pg:
def __init__(self):
from lib.pg import db
self.db = db
def shipments(self, ids):
with self.db().cursor() as c:
c.execute("""SELECT shipment_id, device_id, carrier, product, origin, destination, status,
start_time, end_time FROM shipments WHERE shipment_id = ANY(%s) ORDER BY shipment_id""", (ids,))
cols = [d.name for d in c.description]
return [dict(zip(cols, r)) for r in c.fetchall()]
def cases(self, shipment_id):
with self.db().cursor() as c:
c.execute("""SELECT case_id, started_at, ended_at, max_temp_c, limit_c FROM excursion_cases
WHERE shipment_id = %s ORDER BY started_at""", (shipment_id,))
cols = [d.name for d in c.description]
return [dict(zip(cols, r)) for r in c.fetchall()]
def deny(self, subject, action, obj):
with self.db().cursor() as c:
c.execute("INSERT INTO access_events (event_id, subject, action, object, allowed) VALUES (%s,%s,%s,%s,false)",
(str(uuid.uuid4()), subject, action, obj))
def clean(row):
return {k: (v.isoformat() if hasattr(v, "isoformat") else (float(v) if hasattr(v, "as_tuple") else v)) for k, v in row.items()}
class Api:
def __init__(self, verify, fga, pg):
self.verify, self.fga, self.pg = verify, fga, pg
def subject(self, authorization):
if not authorization or not authorization.lower().startswith("bearer "):
raise Unauthenticated("no bearer token")
claims = self.verify(authorization[7:].strip())
if not claims.get("email") or claims.get("email_verified") is not True:
raise Unauthenticated("no verified email")
return "user:" + claims["email"].lower()
def handle(self, method, path, authorization):
"""Returns (status, body). Pure with respect to HTTP, so it is easy to test."""
p = urlparse(path).path.rstrip("/") or "/"
if p == "/healthz":
return 200, {"ok": True}
if method != "GET":
return 405, {"error": "method not allowed"}
try:
who = self.subject(authorization)
except Unauthenticated as e:
return 401, {"error": "unauthenticated", "detail": str(e)}
if p == "/v1/shipments":
ids = [o.split(":", 1)[1] for o in self.fga.list_objects(who, "can_view", "shipment")]
return 200, {"shipments": [clean(r) for r in self.pg.shipments(ids)] if ids else []}
m = re.fullmatch(r"/v1/shipments/([^/]+)(/excursions)?", p)
if not m or not SHIPMENT.match(m.group(1)):
return 404, {"error": "not found"}
sid, tail = m.group(1), m.group(2)
if not self.fga.check(who, "can_view", "shipment:" + sid):
self.pg.deny(who, "GET " + p, "shipment:" + sid)
return 403, {"error": "forbidden"} # same answer whether or not the shipment exists
rows = self.pg.shipments([sid])
if not rows:
return 404, {"error": "not found"}
if tail:
return 200, {"shipment_id": sid, "excursions": [clean(r) for r in self.pg.cases(sid)]}
return 200, clean(rows[0])
def serve(api, port):
class H(BaseHTTPRequestHandler):
def do_GET(self):
try:
status, body = api.handle("GET", self.path, self.headers.get("authorization"))
except Exception as e: # a dependency failed: say so, never allow by default
print("error", type(e).__name__, e, file=sys.stderr)
status, body = 503, {"error": "unavailable"}
b = json.dumps(body).encode()
self.send_response(status); self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(b))); self.end_headers(); self.wfile.write(b)
def log_message(self, *a):
pass
ThreadingHTTPServer(("0.0.0.0", port), H).serve_forever()
if __name__ == "__main__":
from lib.fgalib import Fga
issuer = os.environ["OIDC_ISSUER"]
verify = JwtVerifier(issuer, os.environ["OIDC_AUDIENCE"],
os.environ.get("OIDC_JWKS", "http://dex.access.svc.cluster.local:5556/keys"))
serve(Api(verify, Fga(), Pg()), int(os.environ.get("PORT", "8080")))
Read Api.handle first. For a shipment it does three things in order: verify the token, ask OpenFGA, then read the database. The answer to “may this person see this shipment” is never taken from the token and never defaulted to yes. A denial is written to access_events, which is one of the three kinds of custody record in section 9.
7.1 The offline test
The test replaces OpenFGA with a function that applies the model’s rule to a set of tuples, and the database with a stub. It covers the token cases (none, garbage, unverified email), both organisations, the customer, a shipment that does not exist, a list per user, and the denial record:
# partner/test_api.py
import sys; sys.path.insert(0, "/tmp/p4")
from partner.app import Api, Unauthenticated
# Tuples as ops/sync_tuples.py would write them for this model: shipment -> carrier / customer -> members.
T = {("carrier:carrier-a", "carrier", "shipment:SHP-1001"), ("carrier:carrier-b", "carrier", "shipment:SHP-1002"),
("carrier:carrier-a", "carrier", "shipment:SHP-1003"), ("customer:customer-pune", "customer", "shipment:SHP-1001"),
("user:anita@carrier-a.example", "member", "carrier:carrier-a"), ("user:bala@carrier-b.example", "member", "carrier:carrier-b"),
("user:chitra@pune.example", "member", "customer:customer-pune")}
class FakeFga:
"""Evaluates can_view = member from carrier or member from customer, as the model says."""
def check(self, user, rel, obj):
assert rel == "can_view"
for u, r, o in T:
if o == obj and r in ("carrier", "customer") and (user, "member", u) in T:
return True
return False
def list_objects(self, user, rel, typ):
return sorted({o for _, _, o in T if o.startswith(typ + ":") and self.check(user, rel, o)})
class FakePg:
def __init__(self): self.denied = []
def shipments(self, ids): return [{"shipment_id": i, "carrier": "x"} for i in ids if i != "SHP-9999"]
def cases(self, sid): return [{"case_id": 1}]
def deny(self, s, a, o): self.denied.append((s, a, o))
TOK = {"a": {"email": "anita@carrier-a.example", "email_verified": True},
"b": {"email": "bala@carrier-b.example", "email_verified": True},
"c": {"email": "chitra@pune.example", "email_verified": True},
"u": {"email": "anita@carrier-a.example", "email_verified": False}}
def verify(t):
if t not in TOK: raise Unauthenticated("bad")
return TOK[t]
pg = FakePg(); api = Api(verify, FakeFga(), pg)
g = lambda path, tok=None: api.handle("GET", path, ("Bearer " + tok) if tok else None)
assert g("/v1/shipments")[0] == 401
assert g("/v1/shipments", "zzz")[0] == 401
assert g("/v1/shipments", "u")[0] == 401 # unverified email
assert g("/v1/shipments/SHP-1001", "a")[0] == 200
assert g("/v1/shipments/SHP-1001", "c")[0] == 200 # the customer sees its own delivery
assert g("/v1/shipments/SHP-1002", "c")[0] == 403
assert g("/v1/shipments/SHP-1001", "b")[0] == 403 and pg.denied[-1] == ("user:bala@carrier-b.example", "GET /v1/shipments/SHP-1001", "shipment:SHP-1001")
assert g("/v1/shipments/SHP-9999", "a")[0] == 403 # no existence leak
assert g("/v1/shipments/SHP-1001/excursions", "a") == (200, {"shipment_id": "SHP-1001", "excursions": [{"case_id": 1}]})
assert [s["shipment_id"] for s in g("/v1/shipments", "a")[1]["shipments"]] == ["SHP-1001", "SHP-1003"]
assert [s["shipment_id"] for s in g("/v1/shipments", "b")[1]["shipments"]] == ["SHP-1002"]
assert g("/v1/shipments/not-an-id", "a")[0] == 404
assert api.handle("POST", "/v1/shipments", "Bearer a")[0] == 405
assert api.handle("GET", "/healthz", None)[0] == 200
print("partner api: 13 checks passed")
python3 partner/test_api.py
python3 evidence/test_chain.py
python3 test_sync.py
Expected:
partner api: 13 checks passed
evidence chain: 9 scenarios passed
sync_tuples: 3 checks passed; tuples: 6 wanted, 6 added, 1 removed
model.json: 4 types
These tests prove the program’s logic against a stand-in. They do not prove OpenFGA’s own evaluation of the model; sections 7.3 and 10 do that against the real server.
7.2 The database additions, the image and the deployment
The API writes denials, and the evidence job keeps a log. Two tables and their grants are added to RDS by a Job run as the administrator (if you deleted the pg-admin Secret after Part 3, create it again from the managed secret as in Part 3, section 3):
-- Part 4 additions. Run as the administrator; every statement can be repeated.
\set ON_ERROR_STOP on
CREATE TABLE IF NOT EXISTS access_events (
event_id text PRIMARY KEY,
at timestamptz NOT NULL DEFAULT now(),
subject text NOT NULL,
action text NOT NULL,
object text NOT NULL,
allowed boolean NOT NULL);
CREATE TABLE IF NOT EXISTS evidence_log (
seq bigserial PRIMARY KEY,
record_key text UNIQUE NOT NULL,
sha256 text NOT NULL,
prev_sha256 text NOT NULL,
version_id text NOT NULL,
written_at timestamptz NOT NULL DEFAULT now());
GRANT SELECT, INSERT ON access_events, evidence_log TO coldchain_app;
GRANT USAGE ON SEQUENCE evidence_log_seq_seq TO coldchain_app;
# platform/db-part4.yaml
apiVersion: batch/v1
kind: Job
metadata: { name: db-part4, namespace: react }
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: psql
image: postgres:17-alpine
command: ["psql", "-v", "ON_ERROR_STOP=1", "-f", "/sql/part4.sql"]
env:
- { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-admin, key: host } } }
- { name: PGDATABASE, value: coldchain }
- { name: PGUSER, valueFrom: { secretKeyRef: { name: pg-admin, key: username } } }
- { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-admin, key: password } } }
- { name: PGSSLMODE, value: require }
volumeMounts: [{ name: sql, mountPath: /sql }]
volumes: [{ name: sql, configMap: { name: db-part4 } }]
kubectl --context hub -n react create configmap db-part4 --from-file=part4.sql=db/part4.sql
kubectl --context hub apply -f platform/db-part4.yaml
kubectl --context hub -n react wait --for=condition=complete job/db-part4 --timeout=120s
kubectl --context hub -n react logs job/db-part4 | tail -3
Expected: the Job completes and the log ends with GRANT lines; a second run completes too. The image holds the API, the evidence program and the tuple sync, so one repository (coldchain/partner-api, created in Part 1) serves all three:
# Dockerfile
FROM python:3.12-slim
RUN pip install --no-cache-dir "psycopg[binary]>=3.2,<4" "PyJWT[crypto]>=2.8,<3" "boto3>=1.35"
WORKDIR /app
COPY lib lib
COPY partner partner
COPY evidence evidence
COPY ops/sync_tuples.py ops/sync_tuples.py
ENV PYTHONUNBUFFERED=1
USER 10001
CMD ["python", "-m", "partner.app"]
# scripts/build-partner.sh
#!/usr/bin/env bash
# usage: scripts/build-partner.sh <tag> builds the partner API image (it also holds the evidence and tuple-sync jobs)
set -euo pipefail
TAG=${1:?tag}; REGION=${AWS_REGION:-ap-south-1}
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
ECR=$ACCOUNT.dkr.ecr.$REGION.amazonaws.com; REPO=coldchain/partner-api
aws ecr get-login-password --region "$REGION" | docker login --username AWS --password-stdin "$ECR"
docker build --platform linux/amd64 -t "$ECR/$REPO:$TAG" .
docker push "$ECR/$REPO:$TAG"
DIGEST=$(aws ecr describe-images --repository-name "$REPO" --image-ids imageTag="$TAG" --query 'imageDetails[0].imageDigest' --output text)
echo "$ECR/$REPO@$DIGEST"
chmod +x scripts/build-partner.sh
export ECR_PARTNER=$(scripts/build-partner.sh 1 | tail -1)
echo "$ECR_PARTNER"
Expected: one line, <account>.dkr.ecr.ap-south-1.amazonaws.com/coldchain/partner-api@sha256:.... Now the configuration, the Knative service and the three jobs of section 9 (the evidence writer, the verifier and the tuple sync):
# platform/partner-api.yaml
apiVersion: v1
kind: ConfigMap
metadata: { name: access-config, namespace: react }
data:
members.json: |
{
"carrier:carrier-a": ["anita@carrier-a.example"],
"carrier:carrier-b": ["bala@carrier-b.example"],
"customer:customer-pune": ["chitra@pune.example"]
}
customers.json: |
{ "Pune": "customer-pune" }
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: partner-api, namespace: react }
spec:
template:
metadata:
annotations: { autoscaling.knative.dev/min-scale: "2" } # no cold start for a partner; two so one can restart
spec:
containers:
- image: ECR_PARTNER
command: ["python", "-m", "partner.app"]
ports: [{ containerPort: 8080 }]
env:
- { name: OIDC_ISSUER, value: "https://login.DOMAIN" }
- { name: OIDC_AUDIENCE, value: coldchain-partner-api }
- { name: OIDC_JWKS, value: "http://dex.access.svc.cluster.local:5556/keys" }
- { name: FGA_URL, value: "http://openfga.access.svc.cluster.local:8080" }
- { name: FGA_STORE_ID, value: FGA_STORE_ID }
- { name: FGA_MODEL_ID, value: FGA_MODEL_ID }
- { name: FGA_TOKEN, valueFrom: { secretKeyRef: { name: fga-token, key: token } } }
- { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
- { name: PGDATABASE, value: coldchain }
- { name: PGUSER, value: coldchain_app }
- { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
- { name: PGSSLMODE, value: require }
resources: { requests: { cpu: 100m, memory: 128Mi }, limits: { memory: 256Mi } }
readinessProbe: { httpGet: { path: /healthz } }
# platform/evidence.yaml
apiVersion: v1
kind: ServiceAccount
metadata: { name: evidence-sync, namespace: react }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: evidence-verify, namespace: react }
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: evidence-sync, namespace: react }
spec:
schedule: "* * * * *" # every minute; the chain needs one writer at a time
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 2
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 0
template:
spec:
serviceAccountName: evidence-sync
restartPolicy: Never
containers:
- name: sync
image: ECR_PARTNER
command: ["python", "-m", "evidence.chain", "sync"]
env:
- { name: EVIDENCE_BUCKET, value: EVIDENCE_BUCKET }
- { name: AWS_REGION, value: ap-south-1 }
- { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
- { name: PGDATABASE, value: coldchain }
- { name: PGUSER, value: coldchain_app }
- { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
- { name: PGSSLMODE, value: require }
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: evidence-verify, namespace: react }
spec:
schedule: "17 */6 * * *" # every six hours; a failure shows as a failed Job, which Prometheus can alert on
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 0
template:
spec:
serviceAccountName: evidence-verify
restartPolicy: Never
containers:
- name: verify
image: ECR_PARTNER
command: ["python", "-m", "evidence.chain", "verify"]
env:
- { name: EVIDENCE_BUCKET, value: EVIDENCE_BUCKET }
- { name: AWS_REGION, value: ap-south-1 }
- { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
- { name: PGDATABASE, value: coldchain }
- { name: PGUSER, value: coldchain_app }
- { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
- { name: PGSSLMODE, value: require }
---
apiVersion: batch/v1
kind: CronJob
metadata: { name: tuple-sync, namespace: react }
spec:
schedule: "*/5 * * * *" # new shipments and membership changes reach OpenFGA within five minutes
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 1
template:
spec:
restartPolicy: Never
containers:
- name: sync
image: ECR_PARTNER
command: ["python", "-m", "ops.sync_tuples"]
env:
- { name: FGA_URL, value: "http://openfga.access.svc.cluster.local:8080" }
- { name: FGA_STORE_ID, value: FGA_STORE_ID }
- { name: FGA_MODEL_ID, value: FGA_MODEL_ID }
- { name: FGA_TOKEN, valueFrom: { secretKeyRef: { name: fga-token, key: token } } }
- { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
- { name: PGDATABASE, value: coldchain }
- { name: PGUSER, value: coldchain_app }
- { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
- { name: PGSSLMODE, value: require }
volumeMounts: [{ name: config, mountPath: /app/config }]
volumes: [{ name: config, configMap: { name: access-config } }]
for f in platform/partner-api.yaml platform/evidence.yaml; do
sed -e "s#ECR_PARTNER#$ECR_PARTNER#g" -e "s#DOMAIN#$DOMAIN#g" -e "s#EVIDENCE_BUCKET#$EVIDENCE_BUCKET#g" \
-e "s#FGA_STORE_ID#$FGA_STORE_ID#g" -e "s#FGA_MODEL_ID#$FGA_MODEL_ID#g" $f | kubectl --context hub apply -f -
done
kubectl --context hub -n react get ksvc partner-api
kubectl --context hub -n react get cronjob
Expected: partner-api ready with two pods and the three CronJobs listed. The Gatekeeper rule from section 4 admits the image because it comes from our ECR path.
7.3 Tuples and the first answers
Run the tuple sync once now instead of waiting five minutes, and look at what OpenFGA holds:
kubectl --context hub -n react create job --from=cronjob/tuple-sync tuple-sync-1
kubectl --context hub -n react wait --for=condition=complete job/tuple-sync-1 --timeout=120s
kubectl --context hub -n react logs job/tuple-sync-1
curl -s -H "Authorization: Bearer $FGA_TOKEN" -H 'content-type: application/json' \
localhost:8080/stores/$FGA_STORE_ID/check -d '{"tuple_key":{"user":"user:anita@carrier-a.example","relation":"can_view","object":"shipment:SHP-1001"}}'
curl -s -H "Authorization: Bearer $FGA_TOKEN" -H 'content-type: application/json' \
localhost:8080/stores/$FGA_STORE_ID/check -d '{"tuple_key":{"user":"user:bala@carrier-b.example","relation":"can_view","object":"shipment:SHP-1001"}}'
Expected: the sync reports tuples: 11 wanted, 11 added, 0 removed on the first run (with the Part 3 seed data: three membership tuples, four carrier links, and four customer links because all four shipments go to Pune; the number depends on your rows), then {"allowed":true} for carrier A’s user on SHP-1001 and {"allowed":false} for carrier B’s user. That is OpenFGA itself applying the model, not a stub. Call the API as carrier A:
TOKEN=$(curl -s -u "coldchain-partner-api:$API_CLIENT_SECRET" https://login.$DOMAIN/token -d grant_type=password \
-d username=anita@carrier-a.example -d "password=$LAB_PW" -d 'scope=openid email' | jq -r .id_token)
curl -s -H "Authorization: Bearer $TOKEN" https://api.$DOMAIN/v1/shipments | jq '.shipments[].shipment_id'
curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $TOKEN" https://api.$DOMAIN/v1/shipments/SHP-1002
Expected: the shipment IDs of carrier A, and 403 for carrier B’s shipment.
Checkpoint 7. The offline tests pass, the Job finished the migration, the image is in ECR, OpenFGA’s own check answers true and false as above, and the API returns the list and the 403.
8. Microcks: mock and contract test
A partner builds against a contract before the real service is ready, and a change that breaks the contract should be caught before the partner finds it. The contract is an OpenAPI file in Git: the partner API’s, and a second one for a carrier’s handover API that we do not implement but whose shape our code and the carrier agree on. Microcks serves a mock from the examples in the contract, and runs the real API against the same file.
# contracts/partner-api.openapi.yaml
openapi: 3.0.3
info:
title: Cold Chain Partner API
version: 1.0.0
description: Read-only access to the shipments the caller's organisation may see. Authentication is a Dex ID token.
servers: [{ url: https://api.example.com }]
security: [{ bearer: [] }]
components:
securitySchemes:
bearer: { type: http, scheme: bearer, bearerFormat: JWT }
schemas:
Shipment:
type: object
required: [shipment_id, device_id, status]
properties:
shipment_id: { type: string, pattern: '^SHP-[0-9]{4,10}$' }
device_id: { type: string }
carrier: { type: string, nullable: true }
product: { type: string, nullable: true }
origin: { type: string, nullable: true }
destination: { type: string, nullable: true }
status: { type: string }
start_time: { type: string, format: date-time }
end_time: { type: string, format: date-time, nullable: true }
Error:
type: object
required: [error]
properties: { error: { type: string }, detail: { type: string } }
paths:
/v1/shipments:
get:
operationId: listShipments
summary: Shipments the caller may see
responses:
'200':
description: The list, possibly empty
content:
application/json:
schema:
type: object
required: [shipments]
properties: { shipments: { type: array, items: { $ref: '#/components/schemas/Shipment' } } }
examples:
carrierA:
value:
shipments:
- { shipment_id: SHP-1001, device_id: reefer-001, carrier: carrier-a, product: vaccines, origin: Mumbai, destination: Pune, status: in_transit, start_time: '2026-11-20T06:00:00Z', end_time: null }
'401': { description: No or invalid token, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
/v1/shipments/{shipment_id}:
get:
operationId: getShipment
parameters:
- { name: shipment_id, in: path, required: true, schema: { type: string }, examples: { ok: { value: SHP-1001 } } }
responses:
'200':
description: The shipment
content:
application/json:
schema: { $ref: '#/components/schemas/Shipment' }
examples:
ok:
value: { shipment_id: SHP-1001, device_id: reefer-001, carrier: carrier-a, product: vaccines, origin: Mumbai, destination: Pune, status: in_transit, start_time: '2026-11-20T06:00:00Z', end_time: null }
'401': { description: No or invalid token, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
'403': { description: Not your shipment, or no such shipment, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
'404': { description: Allowed, but the record is gone, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
/v1/shipments/{shipment_id}/excursions:
get:
operationId: listExcursions
parameters:
- { name: shipment_id, in: path, required: true, schema: { type: string }, examples: { ok: { value: SHP-1001 } } }
responses:
'200':
description: Excursion cases for the shipment
content:
application/json:
schema:
type: object
required: [shipment_id, excursions]
properties:
shipment_id: { type: string }
excursions:
type: array
items:
type: object
properties:
case_id: { type: integer }
started_at: { type: string, format: date-time }
ended_at: { type: string, format: date-time, nullable: true }
max_temp_c: { type: number, nullable: true }
limit_c: { type: number }
examples:
none: { value: { shipment_id: SHP-1001, excursions: [] } }
'401': { description: No or invalid token, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
'403': { description: Not your shipment, content: { application/json: { schema: { $ref: '#/components/schemas/Error' } } } }
# contracts/carrier-handover.openapi.yaml
openapi: 3.0.3
info:
title: Carrier Handover API
version: 1.0.0
description: The API a carrier offers for handover confirmations. We do not implement it; Microcks mocks it so our code and the carrier can build against the same contract.
servers: [{ url: https://carrier.example.com }]
paths:
/handovers/{shipment_id}:
get:
operationId: getHandover
parameters:
- { name: shipment_id, in: path, required: true, schema: { type: string }, examples: { ok: { value: SHP-1001 } } }
responses:
'200':
description: The handover record
content:
application/json:
schema:
type: object
required: [shipment_id, handed_over_at, receiver]
properties:
shipment_id: { type: string }
handed_over_at: { type: string, format: date-time }
receiver: { type: string }
examples:
ok: { value: { shipment_id: SHP-1001, handed_over_at: '2026-11-20T11:30:00Z', receiver: Pune hospital group } }
helm upgrade --install microcks microcks/microcks -n microcks --create-namespace -f platform/microcks-values.yaml --kube-context hub --wait --timeout 10m
kubectl --context hub -n microcks get pods
kubectl --context hub -n microcks port-forward svc/microcks 8585:8080 &
Expected: the Microcks, its database and its sign-in pods Running. Gatekeeper only guards react, so these upstream images are not blocked. The script imports both files and starts a contract test of the running API, passing a real token so the test can get past the sign-in:
# ops/microcks_run.sh
#!/usr/bin/env bash
# Imports the two contracts into Microcks and runs the contract test against the partner API.
# Needs: kubectl port-forward svc/microcks 8585:8080 -n microcks, and a Dex ID token in $TOKEN.
# With Microcks' sign-in on, set KC_SECRET (the service account's client secret); with it off, leave it empty.
set -euo pipefail
MICROCKS=${MICROCKS:-http://host.docker.internal:8585/api/}
AUTH=()
if [ -n "${KC_SECRET:-}" ]; then AUTH=(--keycloakClientId="${KC_ID:-microcks-serviceaccount}" --keycloakClientSecret="$KC_SECRET"); fi
CLI="docker run --rm -v $PWD/contracts:/contracts --add-host host.docker.internal:host-gateway quay.io/microcks/microcks-cli:latest microcks-cli"
$CLI import /contracts/partner-api.openapi.yaml:true,/contracts/carrier-handover.openapi.yaml:true \
--microcksURL="$MICROCKS" "${AUTH[@]}"
$CLI test 'Cold Chain Partner API:1.0.0' http://partner-api.react.svc.cluster.local OPEN_API_SCHEMA \
--microcksURL="$MICROCKS" "${AUTH[@]}" --waitFor=120sec \
--operationsHeaders="{\"globals\":[{\"name\":\"Authorization\",\"values\":\"Bearer $TOKEN\"}]}"
chmod +x ops/microcks_run.sh
export TOKEN=$(curl -s -u "coldchain-partner-api:$API_CLIENT_SECRET" https://login.$DOMAIN/token -d grant_type=password \
-d username=anita@carrier-a.example -d "password=$LAB_PW" -d 'scope=openid email' | jq -r .id_token)
ops/microcks_run.sh
Expected: the import prints the two services, and the test ends with a success result: every operation of the partner contract answered with a status and a body that match the schema. The token lives 15 minutes, which is enough for one run.
Then the mock. The carrier’s handover API does not exist here, but a request to Microcks returns the example from the contract:
curl -s "http://localhost:8585/rest/Carrier+Handover+API/1.0.0/handovers/SHP-1001" | jq .
Expected: the example record from the contract (SHP-1001, a time, the receiver). Now break the contract on purpose: in partner/app.py rename status in the shipment answer, rebuild, deploy and run the test again. It fails and names the operation and the schema rule that was broken. That is the check for “no breaking change unnoticed”. Put the same command in your pipeline after each deploy.
Checkpoint 8. The mock answers with the example, the contract test passes against the real API, and a deliberate break makes it fail.
9. Evidence: a hash-chained custody record, and the validation checklist
9.1 What is recorded, and how
Three kinds of fact matter for custody, and Parts 3 and 4 already write each into RDS: an excursion case opening and closing, a geofence event (the vehicle entered or left a site), and a refused access. The evidence program turns each fact into one small JSON object in the evidence bucket, with the key built from the fact’s identity. Every object holds the SHA-256 of the record before it, in the order the program wrote them, and the log of keys, hashes and S3 version IDs sits in RDS.
Three properties come from three details:
- A repeat cannot add a version. The write uses
If-None-Match: *, so it fails if the key exists. A crash between writing the object and logging it is recovered on the next run, which accepts the existing object only if its bytes are identical. - Deletion is blocked by the lock. The bucket’s default retention (Part 1: governance mode, 30 days, for the lab) keeps every version, and the writer’s IAM role has no delete right.
- A change is visible from two sides. The verifier reads the logged version of every object, recomputes its hash, checks the chain, checks that no newer version of the key exists, checks that the version is locked, and lists the bucket for objects the log does not know. Changing the log breaks the match with the bucket; adding to the bucket breaks the match with the log.
# evidence/chain.py
#!/usr/bin/env python3
"""Chain-of-custody evidence: records go to an S3 bucket with Object Lock, and each record names the hash of the one before.
python -m evidence.chain sync write records that are not yet in the bucket
python -m evidence.chain verify check the bucket and the log against each other (exit 1 on any finding)
"""
import hashlib, json, os, sys
from datetime import datetime, timezone
GENESIS = "0" * 64
class Exists(Exception):
pass
def canon(body):
return json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode()
def sha(data):
return hashlib.sha256(data).hexdigest()
def iso(v):
return v.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ") if isinstance(v, datetime) else v
# --------------------------------------------------------------------------- stores
class S3Store:
def __init__(self, bucket):
import boto3
self.s3, self.bucket = boto3.client("s3"), bucket
def put(self, key, data):
from botocore.exceptions import ClientError
try: # If-None-Match: * makes the write fail when the key exists, so a repeat cannot add a version
r = self.s3.put_object(Bucket=self.bucket, Key=key, Body=data, ContentType="application/json", IfNoneMatch="*")
return r["VersionId"]
except ClientError as e:
if e.response["Error"]["Code"] in ("PreconditionFailed", "ConditionalRequestConflict"):
raise Exists(key)
raise
def get(self, key, version_id=None):
kw = {"VersionId": version_id} if version_id else {}
r = self.s3.get_object(Bucket=self.bucket, Key=key, **kw)
return r["Body"].read(), r["VersionId"]
def latest_version(self, key):
return self.s3.head_object(Bucket=self.bucket, Key=key)["VersionId"]
def locked(self, key, version_id):
from botocore.exceptions import ClientError
try:
r = self.s3.get_object_retention(Bucket=self.bucket, Key=key, VersionId=version_id)
return bool(r.get("Retention", {}).get("Mode"))
except ClientError:
return False
def keys(self, prefix="custody/"):
out = []
for page in self.s3.get_paginator("list_objects_v2").paginate(Bucket=self.bucket, Prefix=prefix):
out += [o["Key"] for o in page.get("Contents", [])]
return out
# --------------------------------------------------------------------------- repository
class Repo:
"""Where the records come from, and the log of what has been written."""
def __init__(self):
from lib.pg import db
self.db = db
def _q(self, sql, args=()):
with self.db().cursor() as c:
c.execute(sql, args)
cols = [d.name for d in c.description]
return [dict(zip(cols, r)) for r in c.fetchall()]
def pending(self, limit=500):
"""Records not yet logged, oldest first. The key is built in SQL so the anti-join stays in the database."""
out = []
for r in self._q("""SELECT * FROM excursion_cases c WHERE NOT EXISTS (SELECT 1 FROM evidence_log l
WHERE l.record_key = 'custody/case/' || c.case_id || '/opened.json') ORDER BY detected_at LIMIT %s""", (limit,)):
out.append((f"custody/case/{r['case_id']}/opened.json", r["detected_at"], {
"kind": "case-opened", "case_id": r["case_id"], "device": r["device_id"], "room": r["room"],
"shipment": r["shipment_id"], "started_at": iso(r["started_at"]), "detected_at": iso(r["detected_at"]),
"limit_c": float(r["limit_c"]), "hold_s": r["hold_s"]}))
for r in self._q("""SELECT * FROM excursion_cases c WHERE ended_at IS NOT NULL AND NOT EXISTS (SELECT 1 FROM evidence_log l
WHERE l.record_key = 'custody/case/' || c.case_id || '/closed.json') ORDER BY ended_at LIMIT %s""", (limit,)):
out.append((f"custody/case/{r['case_id']}/closed.json", r["ended_at"], {
"kind": "case-closed", "case_id": r["case_id"], "device": r["device_id"], "shipment": r["shipment_id"],
"ended_at": iso(r["ended_at"]), "max_temp_c": float(r["max_temp_c"]) if r["max_temp_c"] is not None else None}))
for r in self._q("""SELECT * FROM geofence_events g WHERE NOT EXISTS (SELECT 1 FROM evidence_log l
WHERE l.record_key = 'custody/geofence/' || g.event_id || '.json') ORDER BY received_at LIMIT %s""", (limit,)):
out.append((f"custody/geofence/{r['event_id']}.json", r["received_at"], {
"kind": "geofence", "event_id": r["event_id"], "event_type": r["event_type"], "geofence": r["geofence_id"],
"device": r["device_id"], "sample_time": iso(r["sample_time"]), "received_at": iso(r["received_at"])}))
for r in self._q("""SELECT * FROM access_events a WHERE NOT EXISTS (SELECT 1 FROM evidence_log l
WHERE l.record_key = 'custody/access/' || a.event_id || '.json') ORDER BY at LIMIT %s""", (limit,)):
out.append((f"custody/access/{r['event_id']}.json", r["at"], {
"kind": "access", "event_id": r["event_id"], "at": iso(r["at"]), "subject": r["subject"],
"action": r["action"], "object": r["object"], "allowed": r["allowed"]}))
out.sort(key=lambda x: (x[1], x[0]))
return [(k, b) for k, _, b in out][:limit]
def last_sha(self):
r = self._q("SELECT sha256 FROM evidence_log ORDER BY seq DESC LIMIT 1")
return r[0]["sha256"] if r else GENESIS
def log(self, key, digest, prev, version_id):
with self.db().cursor() as c:
c.execute("INSERT INTO evidence_log (record_key, sha256, prev_sha256, version_id) VALUES (%s,%s,%s,%s) ON CONFLICT DO NOTHING",
(key, digest, prev, version_id))
def entries(self):
return self._q("SELECT seq, record_key, sha256, prev_sha256, version_id FROM evidence_log ORDER BY seq")
# --------------------------------------------------------------------------- the two jobs
def sync(repo, store, out=print):
n, prev = 0, repo.last_sha()
for key, body in repo.pending():
body = dict(body, prev_sha256=prev, schema=1)
data = canon(body)
digest = sha(data)
try:
version = store.put(key, data)
except Exists: # written by an earlier run that stopped before logging: accept it only if it is the same bytes
existing, version = store.get(key)
if sha(existing) != digest:
out(f"CONFLICT {key}: the object exists with different content; not logged")
return 1
repo.log(key, digest, prev, version)
prev, n = digest, n + 1
out(f"synced {n} record(s)")
return 0
def verify(repo, store, out=print):
bad, prev, seen = 0, GENESIS, set()
entries = repo.entries()
for e in entries:
key, seen = e["record_key"], seen | {e["record_key"]}
problems = []
try:
data, _ = store.get(key, e["version_id"])
except Exception as ex:
out(f"FAIL #{e['seq']} {key}: the logged version cannot be read ({type(ex).__name__})")
bad += 1; prev = e["sha256"]; continue
if sha(data) != e["sha256"]:
problems.append("content does not match the logged hash")
if e["prev_sha256"] != prev or json.loads(data).get("prev_sha256") != prev:
problems.append("chain link broken")
if store.latest_version(key) != e["version_id"]:
problems.append("a newer version of the object exists")
if not store.locked(key, e["version_id"]):
problems.append("no retention lock on the version")
if problems:
bad += 1; out(f"FAIL #{e['seq']} {key}: " + "; ".join(problems))
prev = e["sha256"]
for k in sorted(set(store.keys()) - seen):
bad += 1; out(f"FAIL {k}: object in the bucket that the log does not know")
out(f"checked {len(entries)} record(s): " + ("PASS" if not bad else f"{bad} problem(s)"))
return 1 if bad else 0
if __name__ == "__main__":
cmd = sys.argv[1] if len(sys.argv) > 1 else ""
if cmd not in ("sync", "verify"):
sys.exit(__doc__)
repo, store = Repo(), S3Store(os.environ["EVIDENCE_BUCKET"])
sys.exit(sync(repo, store) if cmd == "sync" else verify(repo, store))
The offline test runs nine scenarios against a fake bucket and a fake log: the normal path, a repeated run, the crash-between-steps recovery, a newer version, an edited log row, an unknown object, a missing version, a missing lock, and an object that exists with different content. It is in the file list above and ran in section 7.1.
9.2 Run it
The CronJob from section 7.2 syncs every minute. Make some facts, then look:
kubectl --context hub -n react get cronjob evidence-sync
aws s3api list-objects-v2 --bucket $EVIDENCE_BUCKET --prefix custody/ --query 'Contents[].[Key,Size]' --output text | head
kubectl --context hub -n react create job --from=cronjob/evidence-verify verify-1
kubectl --context hub -n react wait --for=condition=complete job/verify-1 --timeout=120s
kubectl --context hub -n react logs job/verify-1
Expected: objects under custody/case/, custody/geofence/ and (after you have made a refused call, section 10) custody/access/, and a verifier log ending checked N record(s): PASS. To see one record:
aws s3 cp s3://$EVIDENCE_BUCKET/custody/case/1/opened.json - | jq .
aws s3api get-object-retention --bucket $EVIDENCE_BUCKET --key custody/case/1/opened.json --query Retention
Expected: a record with kind, the case details, prev_sha256 and schema, and a retention block with the mode and the retain-until date.
9.3 A validation evidence checklist
This is engineering guidance, not legal advice. These are the items a quality unit usually asks to see for a system like this. Each has a place in this build; whether they are enough is for the company to decide.
| Evidence | Where this build keeps it |
|---|---|
| Intended use and requirements | The Overview’s requirements table |
| Design and configuration, under change control | The Git repository, pull requests and Argo CD history; OpenFGA model IDs; image digests |
| Access control: who may see what | config/model.json, config/members.json, the access-denial test (section 10) |
| Secrets management | Secrets Manager and SOPS homes (sections 2 and 3); CloudTrail for access to them |
| Audit trail of access | access_events and the custody records |
| Record integrity and retention | Object Lock, the hash chain and the verifier (section 9); retention period set from the quality policy |
| Test evidence | Offline tests, the contract test, the access-denial test, the tamper test, with dates, versions and operators written down |
| Alert performance | Part 3’s excursion test, with the worst of ten runs recorded |
| Backup and restore | Part 5 |
| Periodic review | A dated review of members, connectors, retention and the verifier results |
Production differences to decide before relying on it: use compliance mode with a retention period set by the quality policy (governance mode lets an authorised administrator remove versions, which section 10 uses to clean up; compliance mode does not, and its retention cannot be shortened, so do not use it in a lab); add the electronic-signature and identity-binding steps your regulation asks for; and send the verifier’s failure to a page, not just a failed Job.
Checkpoint 9. Records appear for a case, a geofence event and a refusal, and the verifier prints PASS.
10. The access-denial and tamper tests
10.1 Access denial
The script signs in the three lab users and compares the API’s answers with what the model says. It includes the cases an attacker would try: no token, a garbage token, a token whose signature has been altered, another organisation’s shipment, and a shipment that does not exist:
# ops/access_test.py
#!/usr/bin/env python3
"""Access-denial test. Signs the lab users in through Dex (password grant), calls the partner API, and compares each answer
with what the model says it should be.
python3 ops/access_test.py --issuer https://login.$DOMAIN --api https://api.$DOMAIN --secret "$API_CLIENT_SECRET" --password "$LAB_PW"
"""
import argparse, base64, json, sys, urllib.error, urllib.parse, urllib.request
CASES = [ # (who, path, expected status) -- data: SHP-1001 carrier-a to Pune, SHP-1002 carrier-b, SHP-1003 carrier-a, SHP-1004 carrier-a (test truck); all four go to Pune
("anita@carrier-a.example", "/v1/shipments/SHP-1001", 200),
("anita@carrier-a.example", "/v1/shipments/SHP-1002", 403), # another carrier's shipment
("anita@carrier-a.example", "/v1/shipments/SHP-1001/excursions", 200),
("bala@carrier-b.example", "/v1/shipments/SHP-1001", 403),
("bala@carrier-b.example", "/v1/shipments/SHP-1002", 200),
("chitra@pune.example", "/v1/shipments/SHP-1001", 200), # the customer sees its own delivery
("chitra@pune.example", "/v1/shipments/SHP-9999", 403), # no such shipment: same answer as "not yours"
]
LISTS = {"anita@carrier-a.example": ["SHP-1001", "SHP-1003", "SHP-1004"], "bala@carrier-b.example": ["SHP-1002"], "chitra@pune.example": ["SHP-1001", "SHP-1002", "SHP-1003", "SHP-1004"]}
def call(url, headers=None, data=None):
req = urllib.request.Request(url, data=data, headers=headers or {})
try:
with urllib.request.urlopen(req, timeout=15) as r:
return r.status, json.loads(r.read() or b"{}")
except urllib.error.HTTPError as e:
try:
return e.code, json.loads(e.read() or b"{}")
except ValueError:
return e.code, {}
def token(issuer, secret, user, pw):
basic = base64.b64encode(f"coldchain-partner-api:{secret}".encode()).decode()
body = urllib.parse.urlencode({"grant_type": "password", "username": user, "password": pw, "scope": "openid email"}).encode()
s, r = call(issuer + "/token", {"Authorization": "Basic " + basic, "content-type": "application/x-www-form-urlencoded"}, body)
if s != 200:
sys.exit(f"cannot sign {user} in: {s} {r}")
return r["id_token"]
def main():
a = argparse.ArgumentParser()
for k in ("issuer", "api", "secret", "password"):
a.add_argument("--" + k, required=True)
a = a.parse_args()
toks = {u: token(a.issuer.rstrip("/"), a.secret, u, a.password) for u in LISTS}
bad = 0
def check(label, got, want):
nonlocal bad
ok = got == want
bad += not ok
print(f"{'ok ' if ok else 'FAIL'} {label}: got {got}, want {want}")
check("no token", call(a.api + "/v1/shipments")[0], 401)
check("garbage token", call(a.api + "/v1/shipments", {"Authorization": "Bearer abc.def.ghi"})[0], 401)
t = toks["anita@carrier-a.example"]; head, body, sig = t.split(".")
check("signature altered", call(a.api + "/v1/shipments", {"Authorization": f"Bearer {head}.{body}.{sig[:-4]}AAAA"})[0], 401)
for who, path, want in CASES:
check(f"{who.split('@')[0]:6} GET {path}", call(a.api + path, {"Authorization": "Bearer " + toks[who]})[0], want)
for who, want in LISTS.items():
s, r = call(a.api + "/v1/shipments", {"Authorization": "Bearer " + toks[who]})
got = sorted(x["shipment_id"] for x in r.get("shipments", []))
check(f"{who.split('@')[0]:6} list", got, want)
print("RESULT:", "PASS" if not bad else f"FAIL ({bad})")
sys.exit(1 if bad else 0)
if __name__ == "__main__":
main()
python3 ops/access_test.py --issuer https://login.$DOMAIN --api https://api.$DOMAIN --secret "$API_CLIENT_SECRET" --password "$LAB_PW"
Expected, in this shape:
ok no token: got 401, want 401
ok garbage token: got 401, want 401
ok signature altered: got 401, want 401
ok anita GET /v1/shipments/SHP-1001: got 200, want 200
ok anita GET /v1/shipments/SHP-1002: got 403, want 403
...
RESULT: PASS
The list checks assume the Part 3 seed data (SHP-1001 to SHP-1004, carriers A and B, destination Pune). Change the two tables at the top of the script if your data differs. Afterwards each refusal is a row in access_events and, within a minute, an object under custody/access/.
10.2 Tamper
The script does what a person with access, or a mistake, might do, runs the verifier after each step and undoes the change where the lab allows:
# ops/tamper_test.sh
#!/usr/bin/env bash
# Tamper test for the evidence chain. Run from a workstation with your own AWS identity (the lab administrator).
# Each step does something an attacker or a mistake might do, runs the verifier, and undoes it where the lab allows.
set -uo pipefail
: "${EVIDENCE_BUCKET:?}" "${PG_HOST:?}"; CTX=${CTX:-hub}
K="kubectl --context $CTX -n react"
verify() { $K delete job tamper-verify --ignore-not-found >/dev/null
$K create job --from=cronjob/evidence-verify tamper-verify >/dev/null
$K wait --for=condition=complete --timeout=120s job/tamper-verify >/dev/null 2>&1 || $K wait --for=condition=failed --timeout=5s job/tamper-verify >/dev/null 2>&1
$K logs job/tamper-verify | tail -${1:-4}; }
psql() { $K run "psql-$RANDOM" --rm -i --restart=Never --image=postgres:17-alpine --env=PGHOST="$PG_HOST" --env=PGDATABASE=coldchain \
--env=PGSSLMODE=require --env=PGUSER="$1" --env=PGPASSWORD="$2" --command -- psql -v ON_ERROR_STOP=1 -tA -c "$3" 2>&1; }
KEY=$(aws s3api list-objects-v2 --bucket "$EVIDENCE_BUCKET" --prefix custody/ --query 'Contents[0].Key' --output text)
VID=$(aws s3api list-object-versions --bucket "$EVIDENCE_BUCKET" --prefix "$KEY" --query 'Versions[0].VersionId' --output text)
echo "== 0. baseline"; verify 1
echo "== 1. delete a locked version (expect AccessDenied)"
aws s3api delete-object --bucket "$EVIDENCE_BUCKET" --key "$KEY" --version-id "$VID" 2>&1 | head -2
echo "== 2. add a newer version of a record (writes are allowed; the lock keeps the original)"
echo '{"kind":"forged"}' > /tmp/forged.json
NEW=$(aws s3api put-object --bucket "$EVIDENCE_BUCKET" --key "$KEY" --body /tmp/forged.json --query VersionId --output text)
verify 3
echo " undo (lab only: GOVERNANCE mode lets an authorised administrator remove the forged version; COMPLIANCE would not)"
aws s3api delete-object --bucket "$EVIDENCE_BUCKET" --key "$KEY" --version-id "$NEW" --bypass-governance-retention >/dev/null; verify 1
echo "== 3. change a hash in the log, first as the application role, then as the administrator"
psql coldchain_app "$($K get secret pg-app -o jsonpath='{.data.password}' | base64 -d)" "UPDATE evidence_log SET sha256 = repeat('f',64) WHERE record_key = '$KEY'" | grep -i "permission denied"
ADMIN_USER=$($K get secret pg-admin -o jsonpath='{.data.username}' | base64 -d); ADMIN_PW=$($K get secret pg-admin -o jsonpath='{.data.password}' | base64 -d)
OLD=$(psql "$ADMIN_USER" "$ADMIN_PW" "SELECT sha256 FROM evidence_log WHERE record_key = '$KEY'" | tail -1)
psql "$ADMIN_USER" "$ADMIN_PW" "UPDATE evidence_log SET sha256 = repeat('f',64) WHERE record_key = '$KEY'" >/dev/null; verify 4
psql "$ADMIN_USER" "$ADMIN_PW" "UPDATE evidence_log SET sha256 = '$OLD' WHERE record_key = '$KEY'" >/dev/null; verify 1
echo "== 4. an object the log does not know"
UNK=$(aws s3api put-object --bucket "$EVIDENCE_BUCKET" --key custody/unlogged.json --body /tmp/forged.json --query VersionId --output text)
verify 3
aws s3api delete-object --bucket "$EVIDENCE_BUCKET" --key custody/unlogged.json --version-id "$UNK" --bypass-governance-retention >/dev/null; verify 1
chmod +x ops/tamper_test.sh
EVIDENCE_BUCKET=$EVIDENCE_BUCKET PG_HOST=$PG_HOST ops/tamper_test.sh
Expected, in this shape (the record names and numbers are yours):
| Step | What happens | What the verifier says |
|---|---|---|
| 0 | Nothing | checked N record(s): PASS |
| 1 | Delete a locked version | AccessDenied from S3 |
| 2 | Add a newer version of a record | FAIL ... a newer version of the object exists, then PASS after the lab cleanup |
| 3 | Edit the log as the application role | permission denied for table evidence_log |
| 3 | Edit the log as the administrator | FAIL ... content does not match the logged hash, and the next record chain link broken, then PASS after the restore |
| 4 | Put an object the log does not know | FAIL ... object in the bucket that the log does not know, then PASS after cleanup |
The cleanup steps use --bypass-governance-retention. That works in governance mode for a principal that has the right, which is what you want in a lab and exactly what a compliance-mode bucket forbids. Note what the test does not cover: someone who controls both the database administrator account and the bucket’s retention permissions can rebuild a consistent forged history. Separate those duties, and consider periodically copying the latest hash to a system that person cannot reach (a ticket, a signed email, another account).
Checkpoint 10. The access test prints PASS, and every tamper step is either refused or reported by the verifier, then clean after the lab undo.
11. Troubleshooting, cost, cleanup, design notes and checklist
Troubleshooting
| Symptom | Likely cause | What to check |
|---|---|---|
ClusterSecretStore not Ready |
The controller has no role | aws eks list-pod-identity-associations; restart the external-secrets pods after the association is created |
ExternalSecret SecretSyncedError |
Wrong secret name, property, or a missing KMS right | kubectl describe externalsecret; the role’s policy in access.tf names AmazonMSK_coldchain_* and coldchain/* |
| Secret does not change after a new value | Refresh interval | Annotate the ExternalSecret with force-sync; consumers need a restart |
| SOPS says it cannot decrypt | The caller has no kms:Decrypt |
Your IAM identity locally; Argo CD’s repo-server association in the cluster |
Argo CD secrets Application shows a kustomize error |
Build options or plugin path | kubectl -n argocd logs deploy/argocd-repo-server; compare the init container with the README of your KSOPS tag |
A pod in react is refused |
It is not on the allow-list | The message names the image; add it on purpose or move it to ECR |
| OpenFGA migration Job fails | fga-pg-app has no uri, or the database is not ready |
kubectl -n access describe job; the CloudNativePG cluster status |
fga_load.py returns 401 |
The key does not match | The keys value of fga-keys against Secrets Manager |
| Dex does not start | A connector issuer is unreachable | Dex logs; remove the connector overlay until the issuer is real |
| Token is rejected with 401 | Issuer or audience differs | Decode the token; compare iss with OIDC_ISSUER (including a trailing slash) and aud with OIDC_AUDIENCE |
| API returns 503 | OpenFGA, the database or Dex’s keys are unreachable | The API logs name the failing dependency |
| Everyone gets 403 | The tuples are missing | kubectl logs job/tuple-sync-1; the emails in members.json must be lower case and match the token |
| Contract test fails on every operation | The token expired or the headers were not passed | Get a new token; check the operations-headers flag for your CLI version |
Evidence sync logs CONFLICT |
An object exists with other content | Someone wrote that key; investigate before anything else |
| Verifier reports an unknown object | Something wrote to custody/ outside the sync |
The bucket’s access log and CloudTrail data events |
Cost note
No prices were read for this post, so none are quoted. The additions are small: an extra database pair and volumes for OpenFGA, a few more pods (Dex, OpenFGA, Microcks with its own MongoDB and Keycloak, external-secrets, Gatekeeper), Secrets Manager secrets (billed per secret and per call), KMS requests for every decrypt, S3 storage and requests for the evidence objects (small), CloudTrail if you enable data events on the bucket, and the same load balancer as Part 3. Microcks is the heaviest component and can be stopped between uses. Price the items in the AWS Pricing Calculator for ap-south-1.
Cleanup
helm --kube-context hub -n microcks uninstall microcks
kubectl --context hub -n react delete cronjob evidence-sync evidence-verify tuple-sync
kubectl --context hub -n react delete ksvc partner-api
helm --kube-context hub -n access uninstall dex openfga
kubectl --context hub -n access delete cluster.postgresql.cnpg.io fga-pg
kubectl --context hub delete coldallowedregistry react-images-from-ecr
kubectl --context hub -n argocd delete application secrets
(cd infra/foundation && tofu destroy -target=aws_eks_pod_identity_association.evidence_writer -target=aws_eks_pod_identity_association.evidence_verifier)
The evidence objects stay: governance retention keeps every version for 30 days, and with versioning on, a plain delete only adds a marker. To remove lab objects early, an administrator uses --bypass-governance-retention on each version, as the tamper test does. Never use compliance mode with a long period in a lab: you cannot undo it, and the bucket (and the account that holds it) stays until the period ends. Part 5 covers the final teardown.
Design notes and version checks
What this blueprint chose, and what to confirm on your versions before relying on it.
- Dex. Chart
dex/dex, with the configuration underconfig:and Secrets asenvFrom. Confirm the key names for your chart version. ThepasswordConnector: localsetting andsecretEnvon a static client are used for the lab flow. A connector’s issuer is contacted when Dex starts. ID tokens last 15 minutes here. Dex’s SAML connector is documented as unmaintained. - Identity is the email. Partners with no verified email in their tokens cannot be admitted by this design; use the provider’s own claim mapping to supply one. Membership lives in
config/members.json, so a leaver is removed by a pull request, and the change reaches OpenFGA within five minutes. - Public names. The NLB annotation takes a comma-separated list of certificate ARNs on current in-tree providers; with the AWS Load Balancer Controller the annotation set differs. A Knative
DomainMappingpointing at a plain Kubernetes Service (for Dex) depends on your Knative version; if yours refuses it, put Dex behind a Knative Service wrapper or an ingress of your choice. - OpenFGA. Chart keys (
datastore.uriSecret,authn.preshared.keysSecret,datastore.migrations) and theurikey of CloudNativePG’s application Secret are as documented at the time of writing; confirm withhelm show values openfga/openfga. The HTTP request shapes (check,list-objects,read,write, authorization models) are those of the Check and relationship-query documentation. Pin the model ID in every client, as the best-practice page recommends, so a new model version cannot change answers silently. - external-secrets.
external-secrets.io/v1resources and the no-authstore using Pod Identity;creationPolicy: Mergeneeds the target Secret to exist. If your release serves onlyv1beta1, change theapiVersion. - SOPS and KSOPS. The init-container recipe, the plugin path and the
kustomize.buildOptionsfollow the KSOPS README for the tag you pick; check it, because the recipe has changed between major versions. SOPS protects Git, not the cluster: the decrypted Secret sits in etcd like any other. - Gatekeeper. The template uses the classic Rego syntax of the Sprint 4 policies; use the syntax your Gatekeeper release documents if you move to Rego v1. The allow-list is a prefix match, so a tag or a digest from an allowed repository passes. Add
ratifyor a signature check (Sprint 4) if you want signed images only. - Microcks. The chart values
microcks.url,keycloak.url,keycloak.privateUrlandingresses, the default service account name, and the CLI flags (--operationsHeaders,--waitFor,:truefor a primary artifact) are from the project’s documentation for the CLI image tag you pull. The CLI finds Microcks’ sign-in service through a URL the server reports; in a port-forward lab that URL may not be reachable, and the lab fallback is to install withkeycloak.enabled=falseand leaveKC_SECRETempty. Microcks’ own managed database and Keycloak are for discovery, not production. - S3. Conditional writes (
If-None-Match: *) on a versioned bucket,GetObjectRetentionand Object Lock behaviour are as in the S3 documentation; confirm your boto3 version supportsIfNoneMatchonput_object. The lab uses governance mode; production evidence uses compliance mode with the retention period from the quality policy. - Evidence chain. One writer, one chain, written in the order the program found the facts. Facts logged late are chained late; the record’s own timestamps say when they happened. The chain proves the order of recording, not the truth of a reading.
- Checked offline. The partner API logic, the tuple sync and the evidence chain (nine scenarios, including the tamper cases) were run against stand-ins on a laptop. That shows the logic is as described, not that your cluster, OpenFGA, Dex or S3 behave the same.
- Not measured. Throughput of the API, OpenFGA latency under load, the real federation flow with a partner, the Microcks contract run, and cost.
Setup checklist
- 1. Prerequisites: done when the exports print, the nodes are
Readyand the Part 3 workloads run inreact - 2. external-secrets: done when the store is valid, every ExternalSecret is synced and
pg-appmatches Secrets Manager - 3. SOPS and Argo CD: done when Git holds only encrypted values, a changed value fails the MAC check and
dex-secretsappears through Argo CD - 4. Gatekeeper: done when the dry-run list is empty, the constraint is
denyand an outside image is refused - 5. OpenFGA: done when the migration Job is complete, the model is loaded and the store and model IDs are set
- 6. Dex: done when the discovery document shows the right issuer and a lab token has the expected claims
- 7. Partner API: done when the offline tests pass, OpenFGA’s own check answers true and false, and the API returns the list and a 403
- 8. Microcks: done when the mock answers, the contract test passes and a deliberate break fails it
- 9. Evidence: done when case, geofence and access records appear and the verifier prints
PASS - 10. Access and tamper tests: done when the access test prints
PASSand every tamper step is refused or reported, then clean - 11. Cleanup path: done when you know which of these resources you will delete and in what order
Tools used
Dex, OpenFGA, SOPS, Microcks, CloudNativePG, external-secrets, OPA Gatekeeper, Knative, Argo CD and OpenTofu, plus Amazon RDS for PostgreSQL, S3 Object Lock, AWS KMS, AWS Secrets Manager, Amazon ECR and EKS Pod Identity.
What comes next
Part 5 operates the platform: Kanister backs up openGemini and CloudNativePG, a restore test brings back an RDS snapshot, Kube-burner and a larger simulator scale it, Kuberhealthy watches for a silent vehicle, and the teardown removes what costs money.


