Happy Dussehra! May good always win. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps

Sprint 5 project, Part 3: Store and React

Store every reading once in openGemini through Vector, check each one as it arrives with Knative functions fed by a Kafka source, raise a 5-minute temperature excursion within 60 seconds through Alertmanager, record cases and geofence events once in RDS, compute mean kinetic temperature per shipment, and show it all in Grafana and Perses, with a method for measuring the alert delay.

67 min read

  • openGemini
  • Knative
  • Vector
  • Apache Kafka
  • Amazon RDS for PostgreSQL
  • Alertmanager
  • Prometheus
  • Grafana
  • Perses
  • Amazon Location Service
  • Amazon EventBridge
  • AWS IoT Core
  • Amazon ECR
  • OpenTofu
  • Python
  • Kubernetes

This is Part 3 of the five-part build in the Sprint 5 project overview. Part 2 left readings flowing from simulated vehicles and a depot, through AWS IoT Core, onto the telemetry.raw topic on Amazon MSK, where Vector normalises them. This part gives those readings somewhere to live and something to trigger. openGemini stores each reading once. Knative functions, fed by a Kafka source, check each reading as it arrives and open an excursion case in Amazon RDS when a source has been above 8 °C for 5 minutes. Alertmanager pages. A second path takes geofence events from Amazon Location through EventBridge into Knative. A small job computes mean kinetic temperature per shipment. Grafana and Perses show it, and the last section gives a method for measuring the alert delay.

Where this part fits. This part builds storage, the reactions and the dashboards (marked Part 3 on the overall diagram). Open the zoomable Sprint 5 architecture or read the Overview.

Overall Sprint 5 architecture

What this is and is not. This is engineering guidance, not legal or regulatory advice, and nothing here is validated or compliant on its own. It is a reference blueprint for a lab: the alert rule, the thresholds and the retention are examples, and whether any of it is enough for a regulated product is for the company’s quality unit to decide. A software alert never replaces the reefer unit’s own controller and alarm. If you find a mismatch with your versions, use the “Suggest a correction” link on this page.

What you build, and in what order

Architecture: Vector reads the telemetry topic on Amazon MSK as its own consumer group and writes line protocol to openGemini, moving its offset only after the write. A Knative KafkaSource with its own consumer group delivers each reading to the excursion handler, which reads the last five minutes from openGemini, opens one case in RDS for PostgreSQL and posts an alert to Alertmanager, which routes it to Teams and email. Prometheus platform rules also feed Alertmanager and Perses reads Prometheus. Grafana reads openGemini. An IoT Core location rule sends positions to an Amazon Location tracker; ENTER and EXIT events go through EventBridge and an API destination to a gateway on Knative, which passes them to the Kafka broker, and a geofence handler records each event once in RDS. A cron job writes mean kinetic temperature per delivered shipment to RDS.

One image, the layers of this part. Open the zoomable diagram, or download it as SVG or PNG.

Flow in eight steps: openGemini on a disk with a database and a write test; the RDS schema and application role; the Vector sink with a count check and a replay check; Knative Serving, Eventing and the Kafka source; the handler images and the first case in RDS; alert routing and dashboards; Amazon Location geofences with an ENTER event recorded once; and the measured excursion test.

  1. Prerequisites and versions
  2. openGemini: disk, database and a write test
  3. RDS: the schema and an application role
  4. Vector into openGemini, written once
  5. Knative: install, images and the excursion handler
  6. Alerting: Alertmanager, rules, Grafana and Perses
  7. Geofences: Amazon Location, EventBridge and the gateway
  8. Mean kinetic temperature per shipment
  9. The excursion test: how to measure the delay
  10. Troubleshooting, cost, cleanup, design notes and checklist

The rule, in one place

A source is in excursion when every reading, walking back from the newest one, is above 8.0 °C with no gap longer than 60 seconds, and that run lasts at least 300 seconds. A source is one device, or one cold room of a depot. There is at most one open case per source. A reading at or below the limit closes the case. The limit and the hold time are environment variables, and they are examples: set them from the product’s labelled storage conditions and the quality policy.

Two decisions worth knowing before you start

Readings are stored by the device’s time, so a repeat overwrites. In line protocol a point is identified by its measurement, its tags and its timestamp. Vector writes the device’s own timestamp, with device_id, room and kind as tags. If the same reading arrives twice (MQTT QoS 1 and Kafka producers can both repeat), the second write lands on the same point and changes nothing. That is what “idempotent” means here, and section 4 has a replay check for it.

The handler asks openGemini, not its own memory. Each reading above the limit triggers one query for the preceding six minutes. A reading at or below the limit costs no query. The handler keeps no state that matters: a restart, a second replica or a replayed message gives the same answer, and the open case lives in PostgreSQL where Part 4’s partner API can read it.

1. Prerequisites and versions

Item Version or detail Why
Parts 1 and 2 complete Every checkpoint passes; telemetry.raw has data Everything here reads it
Knative v1.24.0 (26 Oct 2026), which needs Kubernetes 1.35 or later Knative brief
Knative Kafka extension The knative-v1.24.0 release of eventing-kafka-broker, if that tag exists; otherwise the newest release that matches KafkaSource and Kafka Broker
openGemini v1.5.2 (v1.5.3 was a release candidate) openGemini brief
Vector The chart version Part 2 pinned Vector brief
kube-prometheus-stack Pin the version helm search repo prometheus-community/kube-prometheus-stack shows Prometheus, Alertmanager, Grafana
Perses Chart perses/perses; pin what helm search repo perses shows Perses brief
Docker with buildx Any current version Builds the handler images
psql is not needed locally A Kubernetes Job runs it The RDS instance is private

Set the shell variables this part uses (values from Parts 1 and 2):

export AWS_REGION=ap-south-1
export DOMAIN=coldchain.example.com
cd cold-chain/infra/foundation
export PG_HOST=$(tofu output -raw pg_endpoint)
export PG_SECRET_ARN=$(tofu output -raw pg_secret_arn)
export BOOT=$(aws kafka get-bootstrap-brokers --cluster-arn $(tofu output -raw msk_cluster_arn) \
  --query BootstrapBrokerStringSaslScram --output text)
cd ../..
kubectl --context hub get nodes
kubectl --context hub -n ingest logs deploy/vector --tail=3

Expected: three Ready nodes and Vector log lines for real readings. If the simulators from Part 2 are not running, start them now (python simulator/fleet.py ...), because this part needs live data. Create the working folders:

mkdir -p handlers/excursion handlers/geofence db platform/knative platform/monitoring platform/dashboards/perses ops scripts

Checkpoint 1. The exports print, the cluster answers, and Vector is processing readings.

2. openGemini: disk, database and a write test

2.1 Storage on the hub

The hub has no volume driver yet, and openGemini needs a disk that outlives the pod. Add the Amazon EBS CSI driver as an EKS add-on, with Pod Identity as in Part 1.

# infra/foundation/ebs-csi.tf
resource "aws_iam_role" "ebs_csi" {
  name               = "${var.name}-ebs-csi"
  assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}

resource "aws_iam_role_policy_attachment" "ebs_csi" {
  role       = aws_iam_role.ebs_csi.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
}

resource "aws_eks_pod_identity_association" "ebs_csi" {
  cluster_name    = module.eks.cluster_name
  namespace       = "kube-system"
  service_account = "ebs-csi-controller-sa"
  role_arn        = aws_iam_role.ebs_csi.arn
}

resource "aws_eks_addon" "ebs_csi" {
  cluster_name = module.eks.cluster_name
  addon_name   = "aws-ebs-csi-driver"
  depends_on   = [aws_eks_pod_identity_association.ebs_csi]
}
(cd infra/foundation && tofu apply)
kubectl --context hub -n kube-system get pods -l app=ebs-csi-controller

Expected: the controller pods Running. Then a default storage class. Volumes are encrypted with the account’s default EBS key here; use a customer-managed key (kmsKeyId in the parameters, and a grant for the driver’s role) when the quality policy asks for key control.

# platform/storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations: { storageclass.kubernetes.io/is-default-class: "true" }
provisioner: ebs.csi.aws.com
parameters: { type: gp3, encrypted: "true" }
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
kubectl --context hub apply -f platform/storageclass.yaml

2.2 The server

One ts-server (the single binary that bundles ts-sql, ts-meta and ts-store) is enough for a lab and for the example load, which is roughly 80 messages a second. The openGemini brief covers the cluster layout for more. Check the image tag before you apply the manifest:

docker manifest inspect opengeminidb/opengemini-server:v1.5.2 >/dev/null && echo "tag v1.5.2 exists" \
  || docker manifest inspect opengeminidb/opengemini-server:1.5.2 >/dev/null && echo "tag 1.5.2 exists"

Whichever form prints, put it in the manifest. The image keeps its data in /var/lib/openGemini, which is where the volume goes. A NetworkPolicy limits who may connect, because authentication is off in this build (see the design notes).

# platform/opengemini.yaml
apiVersion: v1
kind: Namespace
metadata: { name: data }
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: opengemini, namespace: data }
spec:
  serviceName: opengemini
  replicas: 1
  selector: { matchLabels: { app: opengemini } }
  template:
    metadata: { labels: { app: opengemini } }
    spec:
      containers:
        - name: opengemini
          image: opengeminidb/opengemini-server:v1.5.2     # confirm the tag, see the check below
          ports: [{ name: http, containerPort: 8086 }]
          readinessProbe: { tcpSocket: { port: 8086 }, periodSeconds: 10 }
          resources:
            requests: { cpu: 500m, memory: 1Gi }
            limits: { memory: 2Gi }
          volumeMounts:
            - { name: data, mountPath: /var/lib/openGemini }
            - { name: logs, mountPath: /var/log/openGemini }
      volumes:
        - { name: logs, emptyDir: {} }
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources: { requests: { storage: 20Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: opengemini, namespace: data }
spec:
  selector: { app: opengemini }
  ports: [{ name: http, port: 8086, targetPort: 8086 }]
---
# Authentication is off in this build (see the design notes), so only named namespaces may connect.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: opengemini-clients, namespace: data }
spec:
  podSelector: { matchLabels: { app: opengemini } }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchExpressions:
              - { key: kubernetes.io/metadata.name, operator: In, values: [ingest, react, monitoring, data] }
      ports: [{ port: 8086 }]
kubectl --context hub apply -f platform/opengemini.yaml
kubectl --context hub -n data rollout status statefulset/opengemini --timeout=300s
kubectl --context hub -n data get pvc

Expected: the pod Ready and a Bound 20 GiB volume. The network policy only takes effect if the cluster’s network plugin enforces policies: on EKS the VPC CNI does so when its network policy feature is enabled (aws eks update-addon ... --configuration-values '{"enableNetworkPolicy":"true"}' on the vpc-cni add-on).

2.3 The database and its retention

Create the database through the HTTP API from a one-off pod. The retention policy rp400d keeps readings for 400 days with one-day shards, which is an example; take the period from the quality policy.

kubectl --context hub -n data run og-admin --rm -i --restart=Never --image=curlimages/curl --command -- sh -c '
curl -s -XPOST http://opengemini:8086/query --data-urlencode "q=CREATE DATABASE telemetry WITH DURATION 400d REPLICATION 1 SHARD DURATION 1d NAME rp400d"
echo
curl -s -G http://opengemini:8086/query --data-urlencode "q=SHOW RETENTION POLICIES ON telemetry"'

Expected: an empty result for the first call and a table showing rp400d with 400d and default true for the second. If the first call is refused from another pod while it works inside the pod, the server listens on loopback only, see the troubleshooting table.

2.4 A write and a read

Write one point the way Vector will (a measurement, tags, fields and a timestamp) and read it back:

kubectl --context hub -n data run og-test --rm -i --restart=Never --image=curlimages/curl --command -- sh -c '
curl -i -XPOST "http://opengemini:8086/write?db=telemetry&rp=rp400d&precision=s" --data-binary \
  "readings,device_id=reefer-000,room=-,kind=vehicle temp_c=5.2,seq=1i 1790000000"
curl -s -G http://opengemini:8086/query --data-urlencode "db=telemetry" \
  --data-urlencode "q=SELECT * FROM readings WHERE device_id='"'"'reefer-000'"'"'"
echo
curl -s -XPOST "http://opengemini:8086/write?db=telemetry&rp=rp400d&precision=s" --data-binary \
  "readings,device_id=reefer-000,room=-,kind=vehicle temp_c=5.2,seq=1i 1790000000"
curl -s -G http://opengemini:8086/query --data-urlencode "db=telemetry" \
  --data-urlencode "q=SELECT count(temp_c) FROM readings WHERE device_id='"'"'reefer-000'"'"'"'

Expected: HTTP/1.1 204 No Content for the write, one row from the first query, and a count of 1 after writing the identical point a second time. That count is the idempotency property in miniature. Delete the test point afterwards with DROP SERIES FROM readings WHERE device_id='reefer-000'.

Checkpoint 2. The volume is bound, the database has the retention policy, and the same point written twice counts once.

3. RDS: the schema and an application role

Business records live in RDS (the owner rule on essential services: a managed relational database is not skipped because it is not a CNCF project). The schema has five tables. shipments links a device to a delivery, excursion_cases has a partial unique index that allows one open case per source, geofence_events uses the CloudEvent ID as its key so a repeat is ignored, and shipment_summaries holds the derived values apart from the raw readings.

-- db/schema.sql
CREATE TABLE IF NOT EXISTS devices (
  device_id text PRIMARY KEY, kind text NOT NULL, label text);

CREATE TABLE IF NOT EXISTS shipments (
  shipment_id text PRIMARY KEY,
  device_id   text NOT NULL REFERENCES devices,
  carrier     text,
  product     text,
  origin      text,
  destination text,
  limit_c     numeric NOT NULL DEFAULT 8,
  start_time  timestamptz NOT NULL,
  end_time    timestamptz,
  status      text NOT NULL DEFAULT 'in_transit');

CREATE TABLE IF NOT EXISTS excursion_cases (
  case_id         bigserial PRIMARY KEY,
  device_id       text NOT NULL,
  room            text NOT NULL DEFAULT '-',
  shipment_id     text,
  started_at      timestamptz NOT NULL,
  detected_at     timestamptz NOT NULL DEFAULT now(),
  ended_at        timestamptz,
  max_temp_c      numeric,
  limit_c         numeric NOT NULL,
  hold_s          int NOT NULL,
  last_reading_at timestamptz);
CREATE UNIQUE INDEX IF NOT EXISTS one_open_case ON excursion_cases (device_id, room) WHERE ended_at IS NULL;

CREATE TABLE IF NOT EXISTS geofence_events (
  event_id    text PRIMARY KEY,
  event_type  text NOT NULL,
  geofence_id text NOT NULL,
  device_id   text NOT NULL,
  sample_time timestamptz NOT NULL,
  received_at timestamptz NOT NULL DEFAULT now(),
  position    jsonb);

CREATE TABLE IF NOT EXISTS shipment_summaries (
  shipment_id         text PRIMARY KEY REFERENCES shipments,
  computed_at         timestamptz NOT NULL DEFAULT now(),
  readings            int NOT NULL,
  mean_c              numeric, min_c numeric, max_c numeric, mkt_c numeric,
  minutes_above_limit numeric);

INSERT INTO devices VALUES
  ('reefer-001','truck','Truck 1'), ('reefer-002','truck','Truck 2'),
  ('reefer-003','container','Container 1'), ('reefer-004','truck','Test truck')
ON CONFLICT DO NOTHING;
INSERT INTO shipments (shipment_id, device_id, carrier, product, origin, destination, start_time) VALUES
  ('SHP-1001','reefer-001','carrier-a','vaccines','Mumbai','Pune', now()),
  ('SHP-1002','reefer-002','carrier-b','insulin','Mumbai','Pune', now()),
  ('SHP-1003','reefer-003','carrier-a','frozen biologics','Mumbai','Pune', now()),
  ('SHP-1004','reefer-004','carrier-a','test load','Mumbai','Pune', now())
ON CONFLICT DO NOTHING;

-- The handlers' role: no schema changes, no deletes.
\set ON_ERROR_STOP on
DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'coldchain_app') THEN CREATE ROLE coldchain_app LOGIN; END IF; END $$;
ALTER ROLE coldchain_app PASSWORD :'app_password';
GRANT SELECT ON devices TO coldchain_app;
GRANT SELECT, UPDATE ON shipments TO coldchain_app;
GRANT SELECT, INSERT, UPDATE ON excursion_cases TO coldchain_app;
GRANT USAGE ON SEQUENCE excursion_cases_case_id_seq TO coldchain_app;
GRANT SELECT, INSERT ON geofence_events, shipment_summaries TO coldchain_app;

The script ends by creating a login role for the handlers. That role can read and write only what it needs and cannot change the schema or delete. Put the connection details in Kubernetes Secrets (Part 4 moves this to external-secrets):

kubectl --context hub create namespace react
PGJSON=$(aws secretsmanager get-secret-value --secret-id "$PG_SECRET_ARN" --query SecretString --output text)
kubectl --context hub -n react create secret generic pg-admin \
  --from-literal=host="$PG_HOST" \
  --from-literal=username="$(jq -r .username <<<"$PGJSON")" \
  --from-literal=password="$(jq -r .password <<<"$PGJSON")"
APP_PW=$(openssl rand -base64 24 | tr -d '/+=')
kubectl --context hub -n react create secret generic pg-app \
  --from-literal=host="$PG_HOST" --from-literal=password="$APP_PW"
unset PGJSON APP_PW

The Job below carries the script in a ConfigMap and runs psql as the administrator, once:

# platform/db-init.yaml
apiVersion: v1
kind: ConfigMap
metadata: { name: db-schema, namespace: react }
data:
  schema.sql: |
    CREATE TABLE IF NOT EXISTS devices (
      device_id text PRIMARY KEY, kind text NOT NULL, label text);
    
    CREATE TABLE IF NOT EXISTS shipments (
      shipment_id text PRIMARY KEY,
      device_id   text NOT NULL REFERENCES devices,
      carrier     text,
      product     text,
      origin      text,
      destination text,
      limit_c     numeric NOT NULL DEFAULT 8,
      start_time  timestamptz NOT NULL,
      end_time    timestamptz,
      status      text NOT NULL DEFAULT 'in_transit');
    
    CREATE TABLE IF NOT EXISTS excursion_cases (
      case_id         bigserial PRIMARY KEY,
      device_id       text NOT NULL,
      room            text NOT NULL DEFAULT '-',
      shipment_id     text,
      started_at      timestamptz NOT NULL,
      detected_at     timestamptz NOT NULL DEFAULT now(),
      ended_at        timestamptz,
      max_temp_c      numeric,
      limit_c         numeric NOT NULL,
      hold_s          int NOT NULL,
      last_reading_at timestamptz);
    CREATE UNIQUE INDEX IF NOT EXISTS one_open_case ON excursion_cases (device_id, room) WHERE ended_at IS NULL;
    
    CREATE TABLE IF NOT EXISTS geofence_events (
      event_id    text PRIMARY KEY,
      event_type  text NOT NULL,
      geofence_id text NOT NULL,
      device_id   text NOT NULL,
      sample_time timestamptz NOT NULL,
      received_at timestamptz NOT NULL DEFAULT now(),
      position    jsonb);
    
    CREATE TABLE IF NOT EXISTS shipment_summaries (
      shipment_id         text PRIMARY KEY REFERENCES shipments,
      computed_at         timestamptz NOT NULL DEFAULT now(),
      readings            int NOT NULL,
      mean_c              numeric, min_c numeric, max_c numeric, mkt_c numeric,
      minutes_above_limit numeric);
    
    INSERT INTO devices VALUES
      ('reefer-001','truck','Truck 1'), ('reefer-002','truck','Truck 2'),
      ('reefer-003','container','Container 1'), ('reefer-004','truck','Test truck')
    ON CONFLICT DO NOTHING;
    INSERT INTO shipments (shipment_id, device_id, carrier, product, origin, destination, start_time) VALUES
      ('SHP-1001','reefer-001','carrier-a','vaccines','Mumbai','Pune', now()),
      ('SHP-1002','reefer-002','carrier-b','insulin','Mumbai','Pune', now()),
      ('SHP-1003','reefer-003','carrier-a','frozen biologics','Mumbai','Pune', now()),
      ('SHP-1004','reefer-004','carrier-a','test load','Mumbai','Pune', now())
    ON CONFLICT DO NOTHING;
    
    -- The handlers' role: no schema changes, no deletes.
    \set ON_ERROR_STOP on
    DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'coldchain_app') THEN CREATE ROLE coldchain_app LOGIN; END IF; END $$;
    ALTER ROLE coldchain_app PASSWORD :'app_password';
    GRANT SELECT ON devices TO coldchain_app;
    GRANT SELECT, UPDATE ON shipments TO coldchain_app;
    GRANT SELECT, INSERT, UPDATE ON excursion_cases TO coldchain_app;
    GRANT USAGE ON SEQUENCE excursion_cases_case_id_seq TO coldchain_app;
    GRANT SELECT, INSERT ON geofence_events, shipment_summaries TO coldchain_app;
---
apiVersion: batch/v1
kind: Job
metadata: { name: db-init, namespace: react }
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: psql
          image: postgres:17-alpine
          command: ["sh", "-c", "psql -v ON_ERROR_STOP=1 -v app_password=\"$APP_PASSWORD\" -f /sql/schema.sql"]
          env:
            - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-admin, key: host } } }
            - { name: PGDATABASE, value: coldchain }
            - { name: PGUSER, valueFrom: { secretKeyRef: { name: pg-admin, key: username } } }
            - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-admin, key: password } } }
            - { name: PGSSLMODE, value: require }
            - { name: APP_PASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
          volumeMounts: [{ name: sql, mountPath: /sql }]
      volumes: [{ name: sql, configMap: { name: db-schema } }]
kubectl --context hub apply -f platform/db-init.yaml
kubectl --context hub -n react wait --for=condition=complete job/db-init --timeout=180s
kubectl --context hub -n react logs job/db-init | tail -5

Expected: the Job completes and the log ends with GRANT lines. Run it again and it still completes: every statement is repeatable. The administrator secret in react is only for this Job; delete it once the schema exists if you want it out of the cluster (kubectl -n react delete secret pg-admin).

Checkpoint 3. db-init is complete, and a second run finishes without errors.

4. Vector into openGemini, written once

Part 2’s Vector printed events to its log. Now it writes them. The new values file keeps the Kafka source, the normalise and filter steps and the dead-letter output, and adds a step that shapes each event for line protocol and a sink for openGemini. Three details carry the guarantees:

  • The point’s time is the device’s time. The sink uses an event’s timestamp field as the point time, so the step renames ts to timestamp. A replayed reading then lands on the same point.
  • Tags are chosen on purpose. device_id, room and kind identify the series. Depot rooms have a room name; vehicles get -, so the tag is never empty.
  • The offset moves after the write. acknowledgements.enabled on the sink makes Vector commit a Kafka offset only once openGemini has accepted the batch. Without it the source acknowledges as soon as Vector has the event, and a crash in between loses it. Part 2 described this behaviour; this setting is what delivers it.
# platform/vector-values.yaml
role: Stateless-Aggregator
replicas: 2
env:
  - { name: KAFKA_BOOTSTRAP, valueFrom: { secretKeyRef: { name: msk-scram, key: bootstrap } } }
  - { name: KAFKA_USERNAME,  valueFrom: { secretKeyRef: { name: msk-scram, key: username } } }
  - { name: KAFKA_PASSWORD,  valueFrom: { secretKeyRef: { name: msk-scram, key: password } } }
customConfig:
  data_dir: /vector-data-dir
  api: { enabled: true, address: 127.0.0.1:8686 }
  sources:
    from_kafka:
      type: kafka
      bootstrap_servers: "${KAFKA_BOOTSTRAP}"
      group_id: vector-normaliser
      topics: [telemetry.raw]
      auto_offset_reset: earliest
      sasl: { enabled: true, mechanism: SCRAM-SHA-512, username: "${KAFKA_USERNAME}", password: "${KAFKA_PASSWORD}" }
      tls: { enabled: true }
  transforms:
    normalise:
      type: remap
      inputs: [from_kafka]
      drop_on_error: true
      reroute_dropped: true
      source: |
        . = parse_json!(.message)
        .device_id = to_string!(.device_id)
        .seq = to_int!(.seq)
        .temp_c = to_float!(.temp_c)
        .ts = parse_timestamp!(.ts, format: "%+")
        .kind = if starts_with(.device_id, "reefer") { "vehicle" } else if starts_with(.device_id, "depot") { "depot" } else { "other" }
    keep_valid:
      type: filter
      inputs: [normalise]
      condition: .temp_c > -60.0 && .temp_c < 60.0 && .seq > 0
    to_influx:
      type: remap
      inputs: [keep_valid]
      source: |
        .timestamp = .ts          # the device's time becomes the point's time, so a replay overwrites the same point
        del(.ts)
        .room = string(.room) ?? "-"
  sinks:
    opengemini:
      type: influxdb_logs
      inputs: [to_influx]
      version: "1"
      endpoint: http://opengemini.data.svc.cluster.local:8086
      database: telemetry
      retention_policy_name: rp400d
      measurement: readings
      tags: [device_id, room, kind]
      acknowledgements: { enabled: true }   # the Kafka offset moves only after openGemini accepted the write
      batch: { max_events: 500, timeout_secs: 1 }
    show_dropped:
      type: console
      inputs: [normalise.dropped]
      encoding: { codec: json }
helm upgrade vector vector/vector -n ingest -f platform/vector-values.yaml --kube-context hub
kubectl --context hub -n ingest rollout status deploy/vector
kubectl --context hub -n ingest logs deploy/vector --tail=10

Expected: two Running pods and no error lines; the sink’s health check passes at start-up. Field types matter: openGemini, like InfluxDB, fixes a field’s type at first write, so a field that is a float one day and an integer the next is rejected. The normalise step casts temp_c to a float and seq to an integer; the other fields come from the simulators as the same JSON types every time.

4.1 The count check

The simulators number their readings 1, 2, 3 and so on. If every reading is stored once, then for each source the count equals the last number minus the first plus one. ops/check_store.py tests exactly that:

# ops/check_store.py
#!/usr/bin/env python3
"""Compare what openGemini holds with what the simulators sent. Run through: kubectl -n data port-forward svc/opengemini 8086:8086
usage: check_store.py <since-epoch-seconds>
Every simulator numbers its readings 1, 2, 3 ... so count == max - min + 1 means nothing is missing and nothing is stored twice."""
import json, sys, urllib.parse, urllib.request
from datetime import datetime, timezone

URL, DB = "http://localhost:8086", "telemetry"
since = datetime.fromtimestamp(float(sys.argv[1]), timezone.utc).isoformat().replace("+00:00", "Z")
q = (f"SELECT count(seq), min(seq), max(seq) FROM readings WHERE time >= '{since}' GROUP BY device_id, room")
with urllib.request.urlopen(f"{URL}/query?" + urllib.parse.urlencode({"db": DB, "q": q}), timeout=30) as r:
    res = json.load(r)["results"][0]
ok = True
for s in sorted(res.get("series", []), key=lambda s: (s["tags"]["device_id"], s["tags"]["room"])):
    _, n, lo, hi = s["values"][0]
    good = n == hi - lo + 1
    ok &= good
    print(f"{s['tags']['device_id']:16} {s['tags']['room']:8} stored={n:5} first={lo} last={hi} {'ok' if good else 'MISMATCH'}")
print("RESULT:", "PASS" if ok and res.get("series") else "FAIL")

Start the simulators fresh (delete their *.spool.db files first, as in Part 2), let them run 10 minutes, note the start time, and run the check through a port-forward:

START=$(date +%s)                       # before you start the simulators
kubectl --context hub -n data port-forward svc/opengemini 8086:8086 &
python3 ops/check_store.py $START

Expected, in this shape:

reefer-001       -        stored=   61 first=1 last=61 ok
reefer-002       -        stored=   61 first=1 last=61 ok
reefer-003       -        stored=   61 first=1 last=61 ok
RESULT: PASS

The numbers depend on how long you ran. PASS means no missing and no doubled readings between the simulators and the store. A MISMATCH with a count below the range means a loss; above it cannot happen, because the same time gives the same point.

4.2 The replay check

This proves the “written once” property under a real replay. Stop Vector, move its consumer group back to the start of the topic, start it again, and run the check again. The counts must not change.

kubectl --context hub -n ingest scale deploy/vector --replicas=0
kubectl --context hub -n ingest run reset --rm -i --restart=Never --image=apache/kafka:4.4.0 \
  --overrides='{"spec":{"containers":[{"name":"reset","image":"apache/kafka:4.4.0","stdin":true,
  "command":["/bin/sh","/cfg/run.sh","/opt/kafka/bin/kafka-consumer-groups.sh","--reset-offsets","--group","vector-normaliser","--topic","telemetry.raw","--to-earliest","--execute"],
  "env":[{"name":"KAFKA_USER","valueFrom":{"secretKeyRef":{"name":"msk-scram","key":"username"}}},
  {"name":"KAFKA_PASS","valueFrom":{"secretKeyRef":{"name":"msk-scram","key":"password"}}},
  {"name":"KAFKA_BOOT","valueFrom":{"secretKeyRef":{"name":"msk-scram","key":"bootstrap"}}}],
  "volumeMounts":[{"name":"cfg","mountPath":"/cfg"}]}],"volumes":[{"name":"cfg","configMap":{"name":"kafka-client"}}]}}'
kubectl --context hub -n ingest scale deploy/vector --replicas=2
sleep 120 && python3 ops/check_store.py $START

Expected: the same stored= numbers as before the reset (plus the new readings that arrived meanwhile), and RESULT: PASS. Topic retention is seven days, so a reset replays up to seven days: on a long-running lab, set --to-datetime to a recent time instead of --to-earliest.

Checkpoint 4. The count check passes, and it still passes after the replay.

5. Knative: install, images and the excursion handler

5.1 Install Serving, the Kourier gateway and Eventing with the Kafka extension

Knative has two halves. Serving runs the HTTP functions and scales them. Eventing moves events: a KafkaSource turns Kafka messages into CloudEvents and calls a function, and a Kafka-backed Broker with Triggers routes the geofence events. Kourier is the small ingress that Serving uses here. The install follows the Knative brief:

KN=knative-v1.24.0
kubectl --context hub apply -f https://github.com/knative/serving/releases/download/$KN/serving-crds.yaml
kubectl --context hub apply -f https://github.com/knative/serving/releases/download/$KN/serving-core.yaml
kubectl --context hub apply -f https://github.com/knative-extensions/net-kourier/releases/download/$KN/kourier.yaml
kubectl --context hub patch configmap/config-network -n knative-serving --type merge \
  -p '{"data":{"ingress-class":"kourier.ingress.networking.knative.dev"}}'
kubectl --context hub apply -f https://github.com/knative/eventing/releases/download/$KN/eventing-crds.yaml
kubectl --context hub apply -f https://github.com/knative/eventing/releases/download/$KN/eventing-core.yaml
kubectl --context hub apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/$KN/eventing-kafka-controller.yaml
kubectl --context hub apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/$KN/eventing-kafka-broker.yaml
kubectl --context hub apply -f https://github.com/knative-extensions/eventing-kafka-broker/releases/download/$KN/eventing-kafka-source.yaml
kubectl --context hub -n knative-serving get pods
kubectl --context hub -n knative-eventing get pods

Expected: every pod Running within a few minutes. Two things to confirm for your release: that the Kafka extension has a release with the same tag (if the download fails with 404, use the newest eventing-kafka-broker release listed for the Knative line you installed), and the key name in config-network (older guides use ingress.class; the current install page uses ingress-class).

Kourier creates a public load balancer called kourier by default. This design uses its own, with a certificate, in section 7, so make the default one internal-only now:

kubectl --context hub -n kourier-system patch svc kourier -p '{"spec":{"type":"ClusterIP"}}'

5.2 Build the handler images and push them to ECR

The handlers are two small Python programs. The excursion handler holds the rule from the start of this part. The geofence program has three roles chosen by an environment variable: a gateway (the public front for EventBridge), a handler (records each event once) and a dead-letter logger. First the shared helpers:

# handlers/common.py
"""Shared helpers: openGemini queries, Alertmanager posts, PostgreSQL connections."""
import json, os, re, threading, urllib.parse, urllib.request
from datetime import datetime, timezone

OPENGEMINI = os.environ.get("OPENGEMINI_URL", "http://opengemini.data.svc.cluster.local:8086")
OG_DB = os.environ.get("OPENGEMINI_DB", "telemetry")
ALERTMANAGER = os.environ.get(
    "ALERTMANAGER_URL", "http://monitoring-kube-prometheus-alertmanager.monitoring.svc.cluster.local:9093")
SAFE = re.compile(r"^[A-Za-z0-9_.-]{1,64}$")          # device and room names go into InfluxQL, so they are checked


def iso(t):
    return datetime.fromtimestamp(t, timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")


def epoch(s):
    """RFC 3339 text to epoch seconds. Fractions are cut to 6 digits, which fromisoformat accepts."""
    s = re.sub(r"(\.\d{6})\d+", r"\1", s.replace("Z", "+00:00"))
    return datetime.fromisoformat(s).timestamp()


def og_query(q, timeout=3.0):
    """Run one InfluxQL statement against openGemini and return the rows of the first series."""
    url = f"{OPENGEMINI}/query?" + urllib.parse.urlencode({"db": OG_DB, "q": q})
    with urllib.request.urlopen(url, timeout=timeout) as r:
        res = json.load(r)["results"][0]
    if "error" in res:
        raise RuntimeError(res["error"])
    series = res.get("series") or []
    return series[0]["values"] if series else []


def series_points(device, room, t0, t1):
    """(epoch, temp_c) pairs for one source between two epoch times, oldest first."""
    assert SAFE.match(device) and SAFE.match(room)
    q = (f"SELECT temp_c FROM readings WHERE device_id='{device}' AND room='{room}' "
         f"AND time >= '{iso(t0)}' AND time <= '{iso(t1)}' ORDER BY time ASC")
    return [(epoch(t), float(v)) for t, v in og_query(q) if v is not None]


def post_alert(labels, annotations, starts, ends=None):
    alert = {"labels": labels, "annotations": annotations, "startsAt": iso(starts)}
    if ends:
        alert["endsAt"] = iso(ends)
    req = urllib.request.Request(f"{ALERTMANAGER}/api/v2/alerts", data=json.dumps([alert]).encode(),
                                 headers={"content-type": "application/json"}, method="POST")
    urllib.request.urlopen(req, timeout=3).read()


_local = threading.local()


def db():
    """One PostgreSQL connection per thread. libpq reads PGHOST, PGDATABASE, PGUSER, PGPASSWORD, PGSSLMODE."""
    import psycopg
    c = getattr(_local, "conn", None)
    if c is None or c.closed:
        c = psycopg.connect(autocommit=True, connect_timeout=5)
        _local.conn = c
    return c

The excursion handler. Read evaluate first: it is the whole rule, as a pure function with no network in it, so it can be tested on a laptop.

# handlers/excursion/app.py
#!/usr/bin/env python3
"""Excursion handler. A KafkaSource delivers one CloudEvent per reading; the body is the reading as JSON.

Rule: a source is in excursion when every reading, walking back from this one, is above the limit
with no gap longer than MAX_GAP_S, and the run lasts at least HOLD_S. One open case per source."""
import json, os, sys, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from common import SAFE, db, epoch, iso, og_query, post_alert, series_points

LIMIT_C = float(os.environ.get("LIMIT_C", "8.0"))
HOLD_S = int(os.environ.get("HOLD_S", "300"))
MAX_GAP_S = int(os.environ.get("MAX_GAP_S", "60"))
REFRESH_S = int(os.environ.get("REFRESH_S", "60"))       # Alertmanager expires alerts that are not refreshed
LATE_S = int(os.environ.get("LATE_S", "120"))            # older than this on arrival: a backlog replay, not live
STATE = {}                                               # per-source cache, rebuilt from PostgreSQL when empty


def evaluate(points, limit, hold_s, max_gap_s):
    """points: (epoch, temp) oldest first, the current reading last. Returns (hit, run_start_epoch)."""
    if not points or points[-1][1] <= limit:
        return False, None
    start = points[-1][0]
    for (t, v), (t_next, _) in zip(reversed(points[:-1]), reversed(points[1:])):
        if v <= limit or t_next - t > max_gap_s:
            break
        start = t
    return points[-1][0] - start >= hold_s, start


def open_case(dev, room):
    with db().cursor() as cur:
        cur.execute("SELECT case_id, shipment_id, extract(epoch FROM started_at) FROM excursion_cases "
                    "WHERE device_id=%s AND room=%s AND ended_at IS NULL", (dev, room))
        return cur.fetchone()


def alert_labels(dev, room, shipment):
    return {"alertname": "ColdChainExcursion", "severity": "critical", "source": "coldchain",
            "device": dev, "room": room, "shipment": shipment or "none"}


def handle_reading(d, now=None):
    now = now or time.time()
    dev, room = str(d["device_id"]), str(d.get("room") or "-")
    temp, ts = float(d["temp_c"]), epoch(d["ts"])
    if not (SAFE.match(dev) and SAFE.match(room)) or not -60.0 < temp < 60.0:
        return "ignored: invalid reading"
    st = STATE.setdefault(f"{dev}|{room}", {"open": None, "last_post": 0.0})

    if temp <= LIMIT_C:                                   # back in range: close a case if one is open
        if st["open"] is None:
            st["open"] = open_case(dev, room) is not None
        if not st["open"]:
            return "ok"
        case = open_case(dev, room)
        if case:
            case_id, shipment, started = case
            top = og_query(f"SELECT max(temp_c) FROM readings WHERE device_id='{dev}' AND room='{room}' "
                           f"AND time >= '{iso(float(started))}' AND time <= '{iso(ts)}'")
            peak = top[0][1] if top else temp
            with db().cursor() as cur:
                cur.execute("UPDATE excursion_cases SET ended_at=to_timestamp(%s), max_temp_c=%s, "
                            "last_reading_at=to_timestamp(%s) WHERE case_id=%s AND ended_at IS NULL",
                            (ts, peak, ts, case_id))
            post_alert(alert_labels(dev, room, shipment),
                       {"summary": f"{dev} {room} back in range at {temp:.1f} C (peak {peak:.1f} C)"}, float(started), ends=ts)
        st["open"] = False
        return "closed"

    # Above the limit: ask openGemini for the run leading up to this reading.
    lookback = HOLD_S + MAX_GAP_S
    pts = [p for p in series_points(dev, room, ts - lookback, ts) if p[0] < ts] + [(ts, temp)]
    hit, started = evaluate(pts, LIMIT_C, HOLD_S, MAX_GAP_S)
    if not hit:
        return "above limit, not yet long enough"
    with db().cursor() as cur:                            # one open case per source, enforced by the partial index
        cur.execute(
            "INSERT INTO excursion_cases (device_id, room, shipment_id, started_at, limit_c, hold_s, last_reading_at) "
            "VALUES (%s, %s, (SELECT shipment_id FROM shipments WHERE device_id=%s AND status='in_transit' "
            "ORDER BY start_time DESC LIMIT 1), to_timestamp(%s), %s, %s, to_timestamp(%s)) "
            "ON CONFLICT (device_id, room) WHERE ended_at IS NULL DO NOTHING RETURNING case_id, shipment_id",
            (dev, room, dev, started, LIMIT_C, HOLD_S, ts))
        new = cur.fetchone()
    shipment = new[1] if new else (open_case(dev, room) or (None, None))[1]
    st["open"] = True
    if new or now - st["last_post"] >= REFRESH_S:        # new case, or keep the alert alive
        late = now - ts > LATE_S
        labels = alert_labels(dev, room, shipment)
        if late:
            labels["late"] = "true"
        post_alert(labels, {"summary": f"{dev} {room} above {LIMIT_C:.1f} C for {HOLD_S // 60} min (now {temp:.1f} C)",
                            "description": f"Run started {iso(started)}; reading at {iso(ts)}; shipment {shipment or 'none'}."},
                   started)
        st["last_post"] = now
    return "excursion new" if new else "excursion open"


class Handler(BaseHTTPRequestHandler):
    def _send(self, code, text):
        body = text.encode()
        self.send_response(code); self.send_header("content-length", str(len(body))); self.end_headers(); self.wfile.write(body)

    def do_GET(self):
        self._send(200, "ok")

    def do_POST(self):
        raw = self.rfile.read(int(self.headers.get("content-length", 0)))
        try:
            d = json.loads(raw)
            if isinstance(d, dict) and "specversion" in d and "data" in d:
                d = d["data"]                              # structured-mode CloudEvent
            msg = handle_reading(d)
        except (ValueError, KeyError, TypeError) as e:     # a bad message must not be retried for ever
            msg = f"ignored: {type(e).__name__}: {e}"
        except Exception as e:                             # a failed dependency should be retried by the source
            print(f"error: {type(e).__name__}: {e}", flush=True)
            return self._send(500, str(e))
        print(msg, flush=True)
        self._send(200, msg)

    def log_message(self, *a):
        pass


if __name__ == "__main__":
    ThreadingHTTPServer(("0.0.0.0", int(os.environ.get("PORT", "8080"))), Handler).serve_forever()

The geofence program:

# handlers/geofence/app.py
#!/usr/bin/env python3
"""ROLE=gateway: public HTTPS front for EventBridge. Checks the API key, forwards the CloudEvent to the Kafka Broker, answers fast.
ROLE=handler: Trigger target. Records the geofence event once (the CloudEvent id is the key).
ROLE=dlq: logs events the Trigger could not deliver."""
import hmac, json, os, sys, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

ROLE = os.environ.get("ROLE", "handler")
API_KEY = os.environ.get("API_KEY", "")
BROKER_URL = os.environ.get("BROKER_URL", "http://kafka-broker-ingress.knative-eventing.svc.cluster.local/react/default")


def record(event):
    from common import db
    d = event["data"]
    with db().cursor() as cur:
        cur.execute("INSERT INTO geofence_events (event_id, event_type, geofence_id, device_id, sample_time, position) "
                    "VALUES (%s,%s,%s,%s,%s,%s) ON CONFLICT (event_id) DO NOTHING",
                    (event["id"], d["eventType"], d["geofenceId"], d["deviceId"], d["sampleTime"], json.dumps(d.get("position"))))
        inserted = cur.rowcount == 1
        if inserted and d["eventType"] == "ENTER" and d["geofenceId"].startswith("customer-"):
            cur.execute("UPDATE shipments SET status='delivered', end_time=%s "           # arrival ends the shipment
                        "WHERE device_id=%s AND status='in_transit' AND end_time IS NULL", (d["sampleTime"], d["deviceId"]))
    return "recorded" if inserted else "duplicate"


class Handler(BaseHTTPRequestHandler):
    def _send(self, code, text):
        body = text.encode()
        self.send_response(code); self.send_header("content-length", str(len(body))); self.end_headers(); self.wfile.write(body)

    def do_GET(self):
        self._send(200, "ok")

    def do_POST(self):
        raw = self.rfile.read(int(self.headers.get("content-length", 0)))
        if ROLE == "gateway":
            if not hmac.compare_digest(self.headers.get("x-api-key", ""), API_KEY) or not API_KEY:
                return self._send(401, "unauthorized")
            req = urllib.request.Request(BROKER_URL, data=raw, method="POST",
                                         headers={"content-type": "application/cloudevents+json"})
            try:
                urllib.request.urlopen(req, timeout=3).read()
            except Exception as e:
                print(f"broker error: {e}", flush=True)
                return self._send(502, "broker unavailable")        # EventBridge retries
            return self._send(202, "accepted")
        try:
            if self.headers.get("content-type", "").startswith("application/cloudevents+json"):
                event = json.loads(raw)
            else:                                                   # binary mode: attributes are in ce-* headers
                event = {"id": self.headers["ce-id"], "data": json.loads(raw)}
            if ROLE == "dlq":
                print("DEAD LETTER", json.dumps(event), flush=True)
                return self._send(200, "logged")
            msg = record(event)
        except (ValueError, KeyError, TypeError) as e:
            msg = f"ignored: {type(e).__name__}: {e}"
        except Exception as e:
            print(f"error: {type(e).__name__}: {e}", flush=True)
            return self._send(500, str(e))
        print(msg, flush=True)
        self._send(200, msg)

    def log_message(self, *a):
        pass


if __name__ == "__main__":
    ThreadingHTTPServer(("0.0.0.0", int(os.environ.get("PORT", "8080"))), Handler).serve_forever()

One Dockerfile builds either, chosen by APP:

# handlers/Dockerfile
FROM python:3.12-slim
ARG APP
RUN pip install --no-cache-dir "psycopg[binary]>=3.2,<4"
WORKDIR /app
COPY common.py /app/common.py
COPY ${APP}/ /app/${APP}/
ENV APP=${APP} PYTHONUNBUFFERED=1
USER 10001
CMD ["sh", "-c", "exec python /app/${APP}/app.py"]

Before building, run the offline tests. They check the rule at its edges (exactly 300 seconds fires, 290 does not, a dip below the limit restarts the run, a long gap breaks it) and the mean kinetic temperature function:

# handlers/test_logic.py
"""Offline checks for the detection rule and mean kinetic temperature. Run: python3 handlers/test_logic.py"""
import math, os, sys, types
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "excursion"))
sys.modules.setdefault("psycopg", types.ModuleType("psycopg"))
from app import evaluate
from mkt import mkt_c, minutes_above

def run(temps, step=10):
    pts = [(1000 + i * step, t) for i, t in enumerate(temps)]
    return evaluate(pts, 8.0, 300, 60)

# 31 readings above the limit at 10 s: the run is exactly 300 s
assert run([9.0] * 31)[0] is True
assert run([9.0] * 30)[0] is False                       # 290 s
assert run([9.0] * 15 + [7.9] + [9.0] * 20)[0] is False  # a dip below the limit restarts the run
assert run([5.0] * 10 + [9.0] * 31) == (True, 1100)      # run starts at the first reading above the limit
assert evaluate([(0, 9.0), (200, 9.0), (301, 9.0)], 8.0, 300, 60)[0] is False   # a 200 s gap breaks the run
assert run([8.0] * 40)[0] is False                       # equal to the limit is not above it
assert run([9.0] * 30 + [7.0])[0] is False               # the current reading is back in range
print("evaluate: 7 checks passed")

assert abs(mkt_c([5.0] * 10) - 5.0) < 1e-9
m = mkt_c([4.0, 6.0]); assert 5.0 < m < 5.1, m
print("mkt_c([5]*10) =", round(mkt_c([5.0]*10), 4), " mkt_c([4,6]) =", round(m, 4), " mean = 5.0")
print("mkt_c([2,8]) =", round(mkt_c([2.0, 8.0]), 4), " mkt_c([2]*9+[30]) =", round(mkt_c([2.0]*9+[30.0]), 3), " mean =", round((18+30)/10, 2))
print("minutes_above:", minutes_above([(0, 9.0), (60, 9.0), (120, 5.0), (180, 5.0)], 8.0), "(expected 2.0)")
python3 handlers/test_logic.py

Expected:

evaluate: 7 checks passed
mkt_c([5]*10) = 5.0  mkt_c([4,6]) = 5.0609  mean = 5.0
mkt_c([2,8]) = 5.5364  mkt_c([2]*9+[30]) = 12.427  mean = 4.8
minutes_above: 2.0 (expected 2.0)

Then build and push both images and note the digests:

# scripts/build-handlers.sh
#!/usr/bin/env bash
# usage: scripts/build-handlers.sh <tag>   builds both images, pushes them to ECR, prints the digests
set -euo pipefail
TAG=${1:?tag}; REGION=${AWS_REGION:-ap-south-1}
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
ECR=$ACCOUNT.dkr.ecr.$REGION.amazonaws.com
aws ecr get-login-password --region "$REGION" | docker login --username AWS --password-stdin "$ECR"
for pair in excursion:excursion-handler geofence:geofence-handler; do
  APP=${pair%%:*}; REPO=coldchain/${pair##*:}
  docker build --platform linux/amd64 --build-arg APP="$APP" -t "$ECR/$REPO:$TAG" handlers/
  docker push "$ECR/$REPO:$TAG"
  DIGEST=$(aws ecr describe-images --repository-name "$REPO" --image-ids imageTag="$TAG" \
           --query 'imageDetails[0].imageDigest' --output text)
  echo "$ECR/$REPO@$DIGEST"
done
chmod +x scripts/build-handlers.sh
scripts/build-handlers.sh 1 | tee /tmp/digests.txt
export ECR_EXCURSION=$(grep excursion-handler /tmp/digests.txt)
export ECR_GEOFENCE=$(grep geofence-handler /tmp/digests.txt)
echo "$ECR_EXCURSION"; echo "$ECR_GEOFENCE"

Expected: two lines like 123456789012.dkr.ecr.ap-south-1.amazonaws.com/coldchain/excursion-handler@sha256:.... The manifests below refer to the image by digest. Knative normally resolves a tag to a digest by asking the registry, and a private ECR needs credentials for that; a digest needs no lookup. The nodes pull from ECR with their instance role, which the EKS module gives read access to ECR; confirm with aws iam list-attached-role-policies on the node role if a pull fails.

5.3 Credentials for the Kafka source and the Broker

The KafkaSource reads telemetry.raw with its own consumer group, and the Broker creates its own topic. Both authenticate with the SCRAM user from Part 1. The two components read their credentials in different formats, so there are two Secrets:

KUSER=$(kubectl --context hub -n ingest get secret msk-scram -o jsonpath='{.data.username}' | base64 -d)
KPASS=$(kubectl --context hub -n ingest get secret msk-scram -o jsonpath='{.data.password}' | base64 -d)
kubectl --context hub -n react create secret generic msk-kn \
  --from-literal=user="$KUSER" --from-literal=password="$KPASS" --from-literal=saslType=SCRAM-SHA-512
kubectl --context hub -n knative-eventing create secret generic msk-broker-auth \
  --from-literal=protocol=SASL_SSL --from-literal=sasl.mechanism=SCRAM-SHA-512 \
  --from-literal=user="$KUSER" --from-literal=password="$KPASS"
kubectl --context hub -n knative-eventing create configmap kafka-broker-config \
  --from-literal=bootstrap.servers="$BOOT" \
  --from-literal=default.topic.partitions=6 --from-literal=default.topic.replication.factor=3 \
  --from-literal=auth.secret.ref.name=msk-broker-auth
unset KUSER KPASS

The MSK cluster from Part 1 uses the default ACL behaviour (everyone may act when no ACL exists), so the user can create the Broker’s topic. If you add ACLs later, this user needs rights on telemetry.raw, the Broker topic and the consumer groups.

5.4 Deploy the excursion handler and the Kafka source

The handler runs as a Knative Service with a minimum scale of one. Scale to zero would be valid for a bursty function, but a cold start (schedule a pod, start it, open a database connection) could use the whole 60-second budget, so this one stays warm and scales up to six pods under load.

# platform/knative/handlers.yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: excursion-handler, namespace: react }
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"       # a cold start would cost more than the alert budget
        autoscaling.knative.dev/max-scale: "6"
    spec:
      containerConcurrency: 16
      containers:
        - image: ECR_EXCURSION                       # repository@sha256:digest from the build script
          env:
            - { name: LIMIT_C, value: "8.0" }
            - { name: HOLD_S, value: "300" }
            - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
            - { name: PGDATABASE, value: coldchain }
            - { name: PGUSER, value: coldchain_app }
            - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
            - { name: PGSSLMODE, value: require }
          resources: { requests: { cpu: 100m, memory: 128Mi }, limits: { memory: 256Mi } }
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: event-gateway, namespace: react }
spec:
  template:
    metadata:
      annotations: { autoscaling.knative.dev/min-scale: "1" }    # EventBridge waits 5 s, so no cold start here
    spec:
      containers:
        - image: ECR_GEOFENCE
          env:
            - { name: ROLE, value: gateway }
            - { name: API_KEY, valueFrom: { secretKeyRef: { name: events-api-key, key: key } } }
          resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { memory: 128Mi } }
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: geofence-handler, namespace: react }
spec:
  template:
    spec:
      containers:
        - image: ECR_GEOFENCE
          env:
            - { name: ROLE, value: handler }
            - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
            - { name: PGDATABASE, value: coldchain }
            - { name: PGUSER, value: coldchain_app }
            - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
            - { name: PGSSLMODE, value: require }
          resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { memory: 128Mi } }
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata: { name: dead-letter, namespace: react }
spec:
  template:
    spec:
      containers:
        - image: ECR_GEOFENCE
          env: [{ name: ROLE, value: dlq }]

Apply the Service (and the other three in the file, which section 7 uses) with the image digests filled in:

sed -e "0,/ECR_EXCURSION/s##$ECR_EXCURSION#" -e "s#ECR_GEOFENCE#$ECR_GEOFENCE#g" -e "s#ECR_EXCURSION#$ECR_EXCURSION#g" \
  platform/knative/handlers.yaml | kubectl --context hub apply -f -
kubectl --context hub -n react get ksvc

Expected: the four services listed and READY True for excursion-handler first (the others may wait on the Broker). The sed replaces the placeholders: the first expression is harmless and the others substitute the two digests.

The KafkaSource turns each Kafka message into a CloudEvent and posts it to the handler, one at a time per partition, so order is kept. The handler returns a 2xx for anything it decides to ignore (a malformed message must not be retried for ever) and a 5xx only when a dependency failed, which makes the source retry with backoff.

# platform/knative/kafka-source.yaml
apiVersion: v1
kind: Namespace
metadata: { name: react }
---
apiVersion: sources.knative.dev/v1
kind: KafkaSource
metadata: { name: telemetry-to-excursion, namespace: react }
spec:
  consumerGroup: knative-excursion
  bootstrapServers: ["placeholder:9096"]    # replaced by the patch command in section 5.3
  topics: [telemetry.raw]
  initialOffset: latest
  consumers: 3
  net:
    sasl:
      enable: true
      user: { secretKeyRef: { name: msk-kn, key: user } }
      password: { secretKeyRef: { name: msk-kn, key: password } }
      type: { secretKeyRef: { name: msk-kn, key: saslType } }
    tls:
      enable: true
  sink:
    ref: { apiVersion: serving.knative.dev/v1, kind: Service, name: excursion-handler }
  delivery:
    retry: 5
    backoffPolicy: exponential
    backoffDelay: PT1S
kubectl --context hub apply -f platform/knative/kafka-source.yaml
kubectl --context hub -n react patch kafkasource telemetry-to-excursion --type merge \
  -p "{\"spec\":{\"bootstrapServers\":$(echo "$BOOT" | jq -R 'split(",")')}}"
kubectl --context hub -n react get kafkasource
kubectl --context hub -n react logs -l serving.knative.dev/service=excursion-handler -c user-container --tail=5

Expected: the source READY True and handler log lines such as ok (readings at or below the limit). The initialOffset: latest applies only when the consumer group has no committed offsets; afterwards the group resumes where it stopped, which is what makes the backlog after an outage arrive in order. The group name knative-excursion is separate from Vector’s vector-normaliser, so each reads the whole topic.

5.5 The first case

Force an excursion on a throwaway source with a scripted publish through IoT Core (this uses your AWS identity, not a device certificate). Thirty-one readings, 10 seconds apart, all at 9.0 °C, make a 300-second run. Write them straight into openGemini too, so the handler’s query has something to read even if Vector is slow, then publish the last reading through the real path:

for i in $(seq 0 30); do
  T=$(date -u -d "-$((300 - i*10)) seconds" +%Y-%m-%dT%H:%M:%S.000Z)
  aws iot-data publish --topic fleet/reefer-004/telemetry --cli-binary-format raw-in-base64-out \
    --payload "{\"device_id\":\"reefer-004\",\"seq\":$((i+1)),\"ts\":\"$T\",\"temp_c\":9.0}"
done
sleep 5
kubectl --context hub -n react logs -l serving.knative.dev/service=excursion-handler -c user-container --tail=5

The readings carry past timestamps (the last is “now”), so the run in openGemini is 300 seconds long when the last one lands. Expected: a log line excursion new once the 31st reading arrives, marked late if your clock-to-publish gap exceeds two minutes. Read the case back with a one-off psql:

kubectl --context hub -n react run psql --rm -i --restart=Never --image=postgres:17-alpine \
  --env=PGHOST="$PG_HOST" --env=PGDATABASE=coldchain --env=PGUSER=coldchain_app --env=PGSSLMODE=require \
  --env=PGPASSWORD="$(kubectl --context hub -n react get secret pg-app -o jsonpath='{.data.password}' | base64 -d)" \
  --command -- psql -c "SELECT case_id, device_id, shipment_id, started_at, ended_at, limit_c FROM excursion_cases"

Expected: one open case for reefer-004 linked to SHP-1004. Send a reading at 5.0 °C afterwards and the case closes with ended_at and max_temp_c filled in. Publishing the same last reading twice leaves one case: the partial unique index and ON CONFLICT DO NOTHING see to that.

Checkpoint 5. Knative pods are Running, the KafkaSource is ready, the handler logs readings, and the scripted run opens exactly one case in RDS.

6. Alerting: Alertmanager, rules, Grafana and Perses

One chart installs Prometheus, Alertmanager and Grafana (the stack from Sprint 2, reused). The values do four things. They route alerts labelled source="coldchain" straight away (group_wait: 0s, because the default 30 seconds would use half of the 60-second budget), they also send each of them to a test receiver so the delay can be measured, they route platform alerts to the same on-call receiver, and they add openGemini to Grafana as an InfluxDB data source (openGemini answers InfluxQL).

# platform/monitoring/values.yaml
alertmanager:
  alertmanagerSpec:
    secrets: [alertmanager-secrets]
  config:
    global:
      resolve_timeout: 5m
      smtp_smarthost: smtp.example.com:587
      smtp_from: coldchain-alerts@example.com
      smtp_auth_username: coldchain-alerts@example.com
      smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-secrets/smtp-password
    route:
      receiver: "null"
      group_by: [alertname, device, room]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        # Every cold-chain alert also goes to the test sink, then continues to the real receiver.
        - matchers: ['source="coldchain"']
          receiver: alert-sink
          continue: true
          group_wait: 0s
          group_interval: 1m
        - matchers: ['source="coldchain"', 'severity="critical"']
          receiver: oncall
          group_wait: 0s              # the 60 s budget has no room for the default 30 s
          group_interval: 1m
          repeat_interval: 1h
        - matchers: ['source="platform"']
          receiver: oncall
    receivers:
      - name: "null"
      - name: alert-sink
        webhook_configs:
          - url: http://alert-sink.monitoring.svc:8080/alert
            send_resolved: true
      - name: oncall
        msteamsv2_configs:
          - webhook_url_file: /etc/alertmanager/secrets/alertmanager-secrets/teams-webhook
            send_resolved: true
            title: '{{ .Status | toUpper }}: {{ .CommonLabels.alertname }} {{ .CommonLabels.device }}'
            text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ "\n" }}{{ end }}'
        email_configs:
          - to: quality@example.com
            send_resolved: true
            headers:
              Subject: '[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} {{ .CommonLabels.device }}'
grafana:
  admin: { existingSecret: grafana-admin, userKey: admin-user, passwordKey: admin-password }
  additionalDataSources:
    - name: openGemini
      type: influxdb
      access: proxy
      url: http://opengemini.data.svc.cluster.local:8086
      jsonData: { dbName: telemetry, httpMode: GET }
  sidecar: { dashboards: { enabled: true, label: grafana_dashboard } }
prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false

Secrets stay out of the values file: the Teams webhook and the SMTP password are mounted as files. Create them (use real values), then install:

kubectl --context hub create namespace monitoring
kubectl --context hub -n monitoring create secret generic grafana-admin \
  --from-literal=admin-user=admin --from-literal=admin-password="$(openssl rand -base64 18)"
kubectl --context hub -n monitoring create secret generic alertmanager-secrets \
  --from-literal=teams-webhook='https://prod-00.example.logic.azure.com/workflows/REPLACE_ME' \
  --from-literal=smtp-password='REPLACE_ME'
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && helm repo update
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring -f platform/monitoring/values.yaml --kube-context hub
kubectl --context hub -n monitoring rollout status statefulset/alertmanager-monitoring-kube-prometheus-alertmanager
kubectl --context hub -n monitoring exec alertmanager-monitoring-kube-prometheus-alertmanager-0 -c alertmanager -- \
  amtool check-config /etc/alertmanager/config_out/alertmanager.env.yaml

Expected: SUCCESS from amtool, with the receivers listed. Teams: msteamsv2_configs posts to a Workflows webhook; the older connector style is retired. A phone page is a receiver in the same list (an on-call service’s webhook, or SMS through your provider); this build stops at Teams and email.

The handler posts to Alertmanager’s API with a startsAt time, and Alertmanager treats an alert that is not refreshed as resolved after resolve_timeout (5 minutes here). That is why the handler posts again about once a minute while a case is open, and posts a resolve with endsAt when it closes the case. Deploy the test receiver and the platform rules:

# platform/monitoring/alert-sink.yaml
apiVersion: v1
kind: ConfigMap
metadata: { name: alert-sink, namespace: monitoring }
data:
  alert_sink.py: |
    #!/usr/bin/env python3
    """Test receiver for Alertmanager's webhook. Records when each alert arrived, by this pod's clock."""
    import json, threading, time
    from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
    
    SEEN, LOCK = [], threading.Lock()
    
    class H(BaseHTTPRequestHandler):
        def _json(self, obj):
            b = json.dumps(obj).encode()
            self.send_response(200); self.send_header("content-type", "application/json")
            self.send_header("content-length", str(len(b))); self.end_headers(); self.wfile.write(b)
        def do_POST(self):
            body = json.loads(self.rfile.read(int(self.headers.get("content-length", 0))) or b"{}")
            with LOCK:
                for a in body.get("alerts", []):
                    SEEN.append({"recv": time.time(), "status": a.get("status"), "labels": a.get("labels", {}),
                                 "startsAt": a.get("startsAt")})
            self._json({"ok": True})
        def do_GET(self):
            if self.path.startswith("/time"):
                return self._json({"now": time.time()})
            with LOCK:
                self._json(SEEN)
        def log_message(self, *a): pass
    
    ThreadingHTTPServer(("0.0.0.0", 8080), H).serve_forever()
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: alert-sink, namespace: monitoring }
spec:
  replicas: 1
  selector: { matchLabels: { app: alert-sink } }
  template:
    metadata: { labels: { app: alert-sink } }
    spec:
      containers:
        - name: sink
          image: python:3.12-slim
          command: ["python", "/app/alert_sink.py"]
          ports: [{ containerPort: 8080 }]
          volumeMounts: [{ name: app, mountPath: /app }]
      volumes: [{ name: app, configMap: { name: alert-sink } }]
---
apiVersion: v1
kind: Service
metadata: { name: alert-sink, namespace: monitoring }
spec:
  selector: { app: alert-sink }
  ports: [{ port: 8080, targetPort: 8080 }]
# platform/monitoring/rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: coldchain-platform, namespace: monitoring }
spec:
  groups:
    - name: coldchain-platform
      rules:
        - alert: ExcursionHandlerDown
          expr: sum(kube_deployment_status_replicas_available{namespace="react", deployment=~"excursion-handler.*"}) < 1
          for: 2m
          labels: { severity: critical, source: platform }
          annotations: { summary: "No excursion handler pod is available; temperature alerts are not being raised." }
        - alert: VectorDown
          expr: kube_deployment_status_replicas_available{namespace="ingest", deployment="vector"} < 1
          for: 2m
          labels: { severity: critical, source: platform }
          annotations: { summary: "Vector has no available pod; readings are not reaching openGemini (Kafka keeps them)." }
        - alert: OpenGeminiNotReady
          expr: kube_statefulset_status_replicas_ready{namespace="data", statefulset="opengemini"} < 1
          for: 2m
          labels: { severity: critical, source: platform }
          annotations: { summary: "openGemini is not ready; writes wait in Kafka and excursion checks fail." }
kubectl --context hub apply -f platform/monitoring/alert-sink.yaml -f platform/monitoring/rules.yaml
kubectl --context hub -n monitoring get pods -l app=alert-sink

The three rules watch the pipeline itself: no excursion handler, no Vector, no openGemini. A silent failure of the pipeline is as bad as a missed alert, so these go to the same on-call receiver with source: platform.

6.1 Perses

Perses shows the platform’s own health from Prometheus. It was introduced in Sprint 4 and is reused as it is.

# platform/dashboards/perses/coldchain.yaml
kind: Project
metadata: { name: coldchain }
spec: {}
---
kind: GlobalDatasource
metadata: { name: coldchain-prometheus }
spec:
  default: true
  plugin:
    kind: PrometheusDatasource
    spec:
      directUrl: http://monitoring-kube-prometheus-prometheus.monitoring.svc:9090
---
kind: Dashboard
metadata: { name: platform-health, project: coldchain }
spec:
  display: { name: Platform health }
  duration: 6h
  refreshInterval: 30s
  panels:
    handler:
      kind: Panel
      spec:
        display: { name: Excursion handler pods available }
        plugin: { kind: TimeSeriesChart, spec: {} }
        queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec: { query: 'sum(kube_deployment_status_replicas_available{namespace="react", deployment=~"excursion-handler.*"})' }
    vector:
      kind: Panel
      spec:
        display: { name: Vector pods available }
        plugin: { kind: TimeSeriesChart, spec: {} }
        queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec: { query: 'kube_deployment_status_replicas_available{namespace="ingest", deployment="vector"}' }
    store:
      kind: Panel
      spec:
        display: { name: openGemini ready replicas }
        plugin: { kind: TimeSeriesChart, spec: {} }
        queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec: { query: 'kube_statefulset_status_replicas_ready{namespace="data", statefulset="opengemini"}' }
    firing:
      kind: Panel
      spec:
        display: { name: Cold-chain alerts firing }
        plugin: { kind: TimeSeriesChart, spec: {} }
        queries:
          - kind: TimeSeriesQuery
            spec:
              plugin:
                kind: PrometheusTimeSeriesQuery
                spec: { query: 'sum by (alertname) (ALERTS{alertstate="firing", source=~"coldchain|platform"})', seriesNameFormat: '{{alertname}}' }
  layouts:
    - kind: Grid
      spec:
        display: { title: Pipeline }
        items:
          - { x: 0,  y: 0, width: 8, height: 7, content: { $ref: '#/spec/panels/handler' } }
          - { x: 8,  y: 0, width: 8, height: 7, content: { $ref: '#/spec/panels/vector' } }
          - { x: 16, y: 0, width: 8, height: 7, content: { $ref: '#/spec/panels/store' } }
          - { x: 0,  y: 7, width: 24, height: 7, content: { $ref: '#/spec/panels/firing' } }
helm repo add perses https://perses.github.io/helm-charts && helm repo update
helm upgrade --install perses perses/perses -n dashboards --create-namespace --kube-context hub
kubectl --context hub -n dashboards port-forward svc/perses 8080:8080 &
percli login http://localhost:8080
percli lint -f platform/dashboards/perses/coldchain.yaml
percli apply -f platform/dashboards/perses/coldchain.yaml

Expected: percli lint passes and the project coldchain has a dashboard called Platform health, with four panels that draw lines. Check helm show values perses/perses for the persistence settings before you rely on it; the dashboards are in Git, so a lost volume is re-applied, not rebuilt.

6.2 Grafana

Grafana draws the temperatures themselves, from openGemini, because that is where the readings are. The dashboard has a temperature panel per source with a red line at 8 °C, and a count of stored readings per minute, which is a quick way to see a silent source.

# platform/dashboards/grafana.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coldchain-temperatures
  namespace: monitoring
  labels: { grafana_dashboard: "1" }
data:
  coldchain-temperatures.json: |
    {
      "title": "Cold chain temperatures",
      "uid": "coldchain-temps",
      "schemaVersion": 39,
      "time": { "from": "now-3h", "to": "now" },
      "refresh": "30s",
      "panels": [
        { "type": "timeseries", "title": "Temperature by source (limit 8 C)", "gridPos": { "x": 0, "y": 0, "w": 24, "h": 10 },
          "datasource": "openGemini",
          "fieldConfig": { "defaults": { "unit": "celsius", "thresholds": { "mode": "absolute", "steps": [
            { "color": "green", "value": null }, { "color": "red", "value": 8 } ] },
            "custom": { "thresholdsStyle": { "mode": "line" } } }, "overrides": [] },
          "targets": [ { "refId": "A", "rawQuery": true, "resultFormat": "time_series", "alias": "$tag_device_id $tag_room",
            "query": "SELECT mean(\"temp_c\") FROM \"readings\" WHERE $timeFilter GROUP BY time($__interval), \"device_id\", \"room\" fill(null)" } ] },
        { "type": "timeseries", "title": "Readings stored per minute", "gridPos": { "x": 0, "y": 10, "w": 24, "h": 7 },
          "datasource": "openGemini",
          "targets": [ { "refId": "A", "rawQuery": true, "resultFormat": "time_series", "alias": "$tag_device_id",
            "query": "SELECT count(\"temp_c\") FROM \"readings\" WHERE $timeFilter GROUP BY time(1m), \"device_id\" fill(0)" } ] }
      ]
    }
kubectl --context hub apply -f platform/dashboards/grafana.yaml
kubectl --context hub -n monitoring port-forward svc/monitoring-grafana 3000:80 &

Open http://localhost:3000, sign in as admin (the password is in the grafana-admin Secret), and open Cold chain temperatures. Part 4 puts Grafana behind Dex; until then it is reachable only through the port-forward. Confirm the data source with Connections, Data sources, openGemini, Save and test.

Checkpoint 6. amtool reports a valid configuration, the Perses dashboard draws, and Grafana shows temperatures for the running simulators.

7. Geofences: Amazon Location, EventBridge and the gateway

The path has seven hops, and each one has a reason:

  1. An IoT rule sends the latitude and longitude of each vehicle message to an Amazon Location tracker with the location action.
  2. The tracker, linked to a geofence collection, checks each position and emits an ENTER or EXIT event to EventBridge. (A tracker with the default time-based filtering evaluates updates but stores at most one every 30 seconds per device, which is fine for vehicles.)
  3. An EventBridge rule matches Location Geofence Event and sends it to an API destination over HTTPS. The destination has a client timeout of 5 seconds, retries on timeout, and cannot do mutual TLS, so the receiving end uses an API key.
  4. An input transformer reshapes the event into a CloudEvent, because EventBridge’s own envelope is not one.
  5. A Knative gateway checks the API key, hands the event to the Kafka Broker and answers at once, so the 5-second window is never at risk.
  6. A Trigger delivers events of type coldchain.geofence to the geofence handler, with retries and a dead-letter service behind it.
  7. The handler writes the event to RDS under its CloudEvent ID. Amazon Location can deliver the same event more than once; the primary key turns a repeat into a no-op.

7.1 AWS side, with OpenTofu

# infra/foundation/location-events.tf
variable "domain" {}                       # already set in Part 1
variable "events_lb_hostname" { default = "" }   # set after the load balancer exists (section 7.2)

# --- Amazon Location: tracker, geofence collection, link --------------------------------
resource "aws_location_tracker" "fleet" {
  tracker_name = "${var.name}-fleet"
}

resource "aws_location_geofence_collection" "sites" {
  collection_name = "${var.name}-sites"
}

resource "aws_location_tracker_association" "fleet_sites" {
  tracker_name = aws_location_tracker.fleet.tracker_name
  consumer_arn = aws_location_geofence_collection.sites.collection_arn
}

# Role the IoT location action assumes.
resource "aws_iam_role" "iot_location" {
  name               = "${var.name}-iot-location"
  assume_role_policy = data.aws_iam_policy_document.iot_trust.json     # from iot-kafka.tf in Part 2
}

resource "aws_iam_role_policy" "iot_location" {
  role = aws_iam_role.iot_location.id
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{ Effect = "Allow", Action = "geo:BatchUpdateDevicePosition", Resource = aws_location_tracker.fleet.tracker_arn }]
  })
}

output "location_rule_role_arn" { value = aws_iam_role.iot_location.arn }
output "tracker_name"           { value = aws_location_tracker.fleet.tracker_name }
output "collection_name"        { value = aws_location_geofence_collection.sites.collection_name }

# --- Public name and certificate for the event gateway ---------------------------------
resource "aws_acm_certificate" "events" {
  domain_name       = "events.${var.domain}"
  validation_method = "DNS"
}

resource "aws_route53_record" "events_validation" {
  for_each = { for o in aws_acm_certificate.events.domain_validation_options : o.domain_name => o }
  zone_id  = aws_route53_zone.main.zone_id
  name     = each.value.resource_record_name
  type     = each.value.resource_record_type
  records  = [each.value.resource_record_value]
  ttl      = 60
}

resource "aws_acm_certificate_validation" "events" {
  certificate_arn         = aws_acm_certificate.events.arn
  validation_record_fqdns = [for r in aws_route53_record.events_validation : r.fqdn]
}

resource "aws_route53_record" "events" {
  count   = var.events_lb_hostname == "" ? 0 : 1
  zone_id = aws_route53_zone.main.zone_id
  name    = "events.${var.domain}"
  type    = "CNAME"
  ttl     = 60
  records = [var.events_lb_hostname]
}

output "events_cert_arn" { value = aws_acm_certificate_validation.events.certificate_arn }

# --- EventBridge: geofence events to the gateway through an API destination -----------
resource "random_password" "events_api_key" {
  length  = 40
  special = false
}

output "events_api_key" {
  value     = random_password.events_api_key.result
  sensitive = true
}

resource "aws_cloudwatch_event_connection" "gateway" {
  name               = "${var.name}-event-gateway"
  authorization_type = "API_KEY"
  auth_parameters {
    api_key {
      key   = "X-Api-Key"
      value = random_password.events_api_key.result
    }
  }
}

resource "aws_cloudwatch_event_api_destination" "gateway" {
  name                             = "${var.name}-event-gateway"
  connection_arn                   = aws_cloudwatch_event_connection.gateway.arn
  invocation_endpoint              = "https://events.${var.domain}/events"
  http_method                      = "POST"
  invocation_rate_limit_per_second = 20
}

data "aws_iam_policy_document" "events_trust" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "events_invoke" {
  name               = "${var.name}-events-invoke"
  assume_role_policy = data.aws_iam_policy_document.events_trust.json
}

resource "aws_iam_role_policy" "events_invoke" {
  role = aws_iam_role.events_invoke.id
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{ Effect = "Allow", Action = "events:InvokeApiDestination", Resource = aws_cloudwatch_event_api_destination.gateway.arn }]
  })
}

resource "aws_sqs_queue" "events_dlq" {
  name                      = "${var.name}-events-dlq"
  sqs_managed_sse_enabled   = true
  message_retention_seconds = 1209600
}

data "aws_iam_policy_document" "events_dlq" {
  statement {
    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.events_dlq.arn]
    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_cloudwatch_event_rule.geofence.arn]
    }
  }
}

resource "aws_sqs_queue_policy" "events_dlq" {
  queue_url = aws_sqs_queue.events_dlq.id
  policy    = data.aws_iam_policy_document.events_dlq.json
}

resource "aws_cloudwatch_event_rule" "geofence" {
  name = "${var.name}-geofence-events"
  event_pattern = jsonencode({
    source        = ["aws.geo"]
    "detail-type" = ["Location Geofence Event"]
  })
}

resource "aws_cloudwatch_event_target" "geofence" {
  rule     = aws_cloudwatch_event_rule.geofence.name
  arn      = aws_cloudwatch_event_api_destination.gateway.arn
  role_arn = aws_iam_role.events_invoke.arn

  input_transformer {
    input_paths = {
      id   = "$.id"
      time = "$.time"
      et   = "$.detail.EventType"
      gid  = "$.detail.GeofenceId"
      dev  = "$.detail.DeviceId"
      st   = "$.detail.SampleTime"
      pos  = "$.detail.Position"
    }
    # A CloudEvent in structured mode. Strings are quoted here; the position array is inserted as it is.
    input_template = <<-EOT
      {"specversion":"1.0","id":"<id>","source":"aws.geo","type":"coldchain.geofence","time":"<time>","datacontenttype":"application/json","data":{"eventType":"<et>","geofenceId":"<gid>","deviceId":"<dev>","sampleTime":"<st>","position":<pos>}}
    EOT
  }

  retry_policy {
    maximum_retry_attempts       = 5
    maximum_event_age_in_seconds = 3600
  }
  dead_letter_config { arn = aws_sqs_queue.events_dlq.arn }
}
(cd infra/foundation && tofu apply)
export ACM_ARN=$(cd infra/foundation && tofu output -raw events_cert_arn)
export TRACKER=$(cd infra/foundation && tofu output -raw tracker_name)
export COLLECTION=$(cd infra/foundation && tofu output -raw collection_name)
export LOC_ROLE=$(cd infra/foundation && tofu output -raw location_rule_role_arn)

The certificate waits for its DNS validation record, which OpenTofu creates in the Part 1 zone, so the apply can take a few minutes. Expected: the outputs print, and aws location describe-tracker --tracker-name $TRACKER shows the tracker.

7.2 Geofences and the IoT location rule

Geofences are polygons, one for a depot and one for a customer, each about 280 metres square. Coordinates are longitude then latitude, the ring is closed, and the exterior ring runs counter-clockwise. The depot name starts with depot- and the customer’s with customer-: the handler marks a shipment delivered when its vehicle enters a customer- geofence.

aws location put-geofence --collection-name $COLLECTION --geofence-id depot-navi-mumbai \
  --geometry 'Polygon=[[[73.0270,19.0305],[73.0325,19.0305],[73.0325,19.0355],[73.0270,19.0355],[73.0270,19.0305]]]'
aws location put-geofence --collection-name $COLLECTION --geofence-id customer-pune \
  --geometry 'Polygon=[[[73.8530,18.5180],[73.8585,18.5180],[73.8585,18.5230],[73.8530,18.5230],[73.8530,18.5180]]]'
aws location list-geofences --collection-name $COLLECTION --query 'Entries[].GeofenceId'

Expected: the two names. The location rule is created from a JSON file, because the substitution templates in the tracker name and device fields are supported through the API and CLI. The SQL passes on only messages that carry a position, so depot sensors (which have none) do not produce errors:

{
  "sql": "SELECT topic(2) AS device_id, lat, lon FROM 'fleet/+/telemetry' WHERE isUndefined(lat) = false",
  "awsIotSqlVersion": "2016-03-23",
  "ruleDisabled": false,
  "actions": [
    { "location": {
        "roleArn": "ROLE_ARN",
        "trackerName": "TRACKER",
        "deviceId": "${device_id}",
        "latitude": "${lat}",
        "longitude": "${lon}",
        "timestamp": { "value": "${timestamp()}", "unit": "MILLISECONDS" } } }
  ]
}
sed -e "s#ROLE_ARN#$LOC_ROLE#" -e "s#TRACKER#$TRACKER#" ops/location-rule.json > /tmp/location-rule.json
aws iot create-topic-rule --rule-name coldchain_position_to_location --topic-rule-payload file:///tmp/location-rule.json
aws iot get-topic-rule --rule-name coldchain_position_to_location --query 'rule.actions'

The action stamps each position with the time IoT Core received it. That is right while a vehicle is online; after an outage, the backlog arrives in a burst and the positions carry the arrival time, not the drive time. Order is kept (the spool drains oldest first), so ENTER and EXIT still come in the right sequence, but their times are late. To use the device’s own time, compute it in the rule’s SQL from the ts field (the time_to_epoch function) and confirm the pattern on your side.

7.3 The public name and the gateway

The gateway must be reachable by EventBridge, which has no fixed addresses, so it sits behind a Network Load Balancer that terminates TLS with the ACM certificate. The default Kourier load balancer is already internal-only; this extra Service creates a public one that forwards port 443, as plain TCP, to Kourier’s port 8080, and a DomainMapping ties the name events.<domain> to the gateway Service.

# platform/knative/public.yaml
apiVersion: v1
kind: Service
metadata:
  name: kourier-public
  namespace: kourier-system
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ACM_CERT_ARN
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
spec:
  type: LoadBalancer
  selector: { app: 3scale-kourier-gateway }          # confirm with: kubectl -n kourier-system get pods --show-labels
  ports: [{ name: https, port: 443, targetPort: 8080 }]
---
apiVersion: networking.internal.knative.dev/v1alpha1
kind: ClusterDomainClaim
metadata: { name: events.DOMAIN }
spec: { namespace: react }
---
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata: { name: events.DOMAIN, namespace: react }
spec:
  ref: { apiVersion: serving.knative.dev/v1, kind: Service, name: event-gateway }
kubectl --context hub -n kourier-system get pods --show-labels | head -3      # confirm app=3scale-kourier-gateway
sed -e "s#ACM_CERT_ARN#$ACM_ARN#" -e "s#DOMAIN#$DOMAIN#g" platform/knative/public.yaml | kubectl --context hub apply -f -
kubectl --context hub -n kourier-system get svc kourier-public -w

Wait for the EXTERNAL-IP (a host name). Then the API key (from OpenTofu) goes into a Secret, the DNS record is created by a second apply, and the Broker, Trigger and handlers come up:

export LB=$(kubectl --context hub -n kourier-system get svc kourier-public -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
kubectl --context hub -n react create secret generic events-api-key \
  --from-literal=key="$(cd infra/foundation && tofu output -raw events_api_key)"
(cd infra/foundation && tofu apply -var "events_lb_hostname=$LB")
kubectl --context hub apply -f platform/knative/broker.yaml
kubectl --context hub -n react get broker,trigger,ksvc,domainmapping

Expected: the Broker READY True, the Trigger ready, and the DomainMapping ready. Check the front door from your workstation. A request without the key must be refused; one with it must be accepted:

curl -s -o /dev/null -w "%{http_code}\n" -X POST https://events.$DOMAIN/events -d '{}'
curl -s -o /dev/null -w "%{http_code}\n" -X POST https://events.$DOMAIN/events \
  -H "X-Api-Key: $(cd infra/foundation && tofu output -raw events_api_key)" -H 'content-type: application/json' \
  -d '{"specversion":"1.0","id":"test-1","source":"manual","type":"coldchain.geofence","datacontenttype":"application/json","data":{"eventType":"EXIT","geofenceId":"depot-navi-mumbai","deviceId":"reefer-004","sampleTime":"2026-11-20T00:00:00Z","position":[73.0297,19.033]}}'

Expected: 401, then 202. Send the second request twice and the geofence table still has one row for test-1.

7.4 An ENTER event, end to end

Publish a reading for the test truck with a position inside the depot geofence. The IoT rule from Part 2 stores it, the location rule updates the tracker, and the tracker’s check sends the event:

aws iot-data publish --topic fleet/reefer-004/telemetry --cli-binary-format raw-in-base64-out \
  --payload "{\"device_id\":\"reefer-004\",\"seq\":1000,\"ts\":\"$(date -u +%Y-%m-%dT%H:%M:%S.000Z)\",\"temp_c\":5.1,\"lat\":19.0330,\"lon\":73.0297}"
sleep 20
aws location get-device-position --tracker-name $TRACKER --device-id reefer-004
kubectl --context hub -n react logs -l serving.knative.dev/service=geofence-handler -c user-container --tail=5

Expected: the tracker returns the position and the handler logs recorded. A read of the table shows the row:

kubectl --context hub -n react run psql2 --rm -i --restart=Never --image=postgres:17-alpine \
  --env=PGHOST="$PG_HOST" --env=PGDATABASE=coldchain --env=PGUSER=coldchain_app --env=PGSSLMODE=require \
  --env=PGPASSWORD="$(kubectl --context hub -n react get secret pg-app -o jsonpath='{.data.password}' | base64 -d)" \
  --command -- psql -c "SELECT event_id, event_type, geofence_id, device_id, sample_time FROM geofence_events ORDER BY received_at DESC LIMIT 5"

Expected: an ENTER for depot-navi-mumbai. The geofence-handler Service scales to zero when idle, so the first event after a quiet period waits for a pod to start; the gateway has already answered EventBridge by then, and the Trigger retries if the pod is not ready. A position outside every geofence produces no event.

Checkpoint 7. The gateway answers 401 without the key and 202 with it, a position inside the depot produces one ENTER row, and a repeated event produces no second row.

8. Mean kinetic temperature per shipment

Mean kinetic temperature (MKT) is one number for a temperature history that weighs warm periods more than cool ones, because degradation speeds up as it gets warmer. It is MKT = (ΔH/R) / -ln( mean( e^(-ΔH/(R·T)) ) ), with T in kelvin and ΔH/R taken as 10,000 K, the value in common use when nothing product-specific is known. A shipment that sat at 2 °C for nine readings and 30 °C for one has a plain mean of 4.8 °C and an MKT of 12.4 °C, and the second number is the one that says something about the damage.

The job runs every 15 minutes. It finds shipments that are delivered (the geofence handler set delivered_at) and have no summary yet, reads the readings of each from openGemini in 6-hour chunks (so no query returns too much), computes MKT, the maximum and the minutes above the limit, and inserts one row into shipment_summaries. ON CONFLICT DO NOTHING makes a second run harmless.

# handlers/excursion/mkt.py
#!/usr/bin/env python3
"""Mean kinetic temperature per delivered shipment, written to shipment_summaries. Run by a CronJob."""
import math, os, sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from common import db, series_points

DH_OVER_R = 10000.0                                       # K; about 83.144 kJ/mol divided by the gas constant
MAX_GAP_S = 60
CHUNK_S = 6 * 3600


def mkt_c(temps_c):
    """MKT = (dH/R) / -ln( (sum exp(-dH/R / T)) / n ), T in kelvin, result in Celsius."""
    ks = [-DH_OVER_R / (t + 273.15) for t in temps_c]
    m = max(ks)                                           # log-sum-exp, so the sum cannot underflow
    log_mean = m + math.log(sum(math.exp(k - m) for k in ks)) - math.log(len(ks))
    return DH_OVER_R / -log_mean - 273.15


def minutes_above(points, limit):
    """Time spent above the limit; each reading counts until the next one, capped by MAX_GAP_S."""
    total = 0.0
    for (t, v), (t2, _) in zip(points, points[1:]):
        if v > limit:
            total += min(t2 - t, MAX_GAP_S)
    return total / 60.0


def main():
    with db().cursor() as cur:
        cur.execute("SELECT shipment_id, device_id, extract(epoch FROM start_time), extract(epoch FROM end_time), limit_c "
                    "FROM shipments s WHERE end_time IS NOT NULL AND NOT EXISTS "
                    "(SELECT 1 FROM shipment_summaries m WHERE m.shipment_id = s.shipment_id)")
        todo = cur.fetchall()
    for shipment, device, t0, t1, limit in todo:
        pts, t = [], float(t0)
        while t < float(t1):
            pts += series_points(device, "-", t, min(t + CHUNK_S, float(t1)))
            t += CHUNK_S
        pts = sorted(set(pts))
        if not pts:
            print(f"{shipment}: no readings, skipped", flush=True)
            continue
        temps = [v for _, v in pts]
        with db().cursor() as cur:
            cur.execute("INSERT INTO shipment_summaries (shipment_id, readings, mean_c, min_c, max_c, mkt_c, minutes_above_limit) "
                        "VALUES (%s,%s,%s,%s,%s,%s,%s) ON CONFLICT (shipment_id) DO NOTHING",
                        (shipment, len(temps), sum(temps) / len(temps), min(temps), max(temps), mkt_c(temps),
                         minutes_above(pts, float(limit))))
        print(f"{shipment}: n={len(temps)} mkt={mkt_c(temps):.2f} C", flush=True)


if __name__ == "__main__":
    main()
# platform/mkt-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata: { name: mkt, namespace: react }
spec:
  schedule: "*/15 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: mkt
              image: ECR_EXCURSION
              command: ["python", "/app/excursion/mkt.py"]
              env:
                - { name: PGHOST, valueFrom: { secretKeyRef: { name: pg-app, key: host } } }
                - { name: PGDATABASE, value: coldchain }
                - { name: PGUSER, value: coldchain_app }
                - { name: PGPASSWORD, valueFrom: { secretKeyRef: { name: pg-app, key: password } } }
                - { name: PGSSLMODE, value: require }
sed -e "s#ECR_EXCURSION#$ECR_EXCURSION#" platform/mkt-cronjob.yaml | kubectl --context hub apply -f -
kubectl --context hub -n react create job --from=cronjob/mkt mkt-manual-1
kubectl --context hub -n react wait --for=condition=complete job/mkt-manual-1 --timeout=120s
kubectl --context hub -n react logs job/mkt-manual-1

Expected: a log line per shipment that was summarised (none if no shipment is delivered yet). After the ENTER event in section 7.4, if you mark the shipment delivered the same way (the customer-pune geofence for reefer-004), the next run writes a row. Read it with the same psql pattern as before: SELECT shipment_id, mkt_c, max_temp_c, minutes_above_limit FROM shipment_summaries.

Checkpoint 8. The manual job completes, and a delivered shipment has one summary row; running the job again adds none.

9. The excursion test: how to measure the delay

The target is a page within 60 seconds of the moment the condition becomes true. That moment is the 31st reading above the limit (the first reading above it, plus 300 seconds). This section is a method. It has not been run for this post, so no result is shown, only the shape of one.

What the test does:

  1. It publishes a scripted profile as reefer-004 through the real path (IoT Core, MSK, Knative): 2 minutes at 5 °C, 6 minutes at 9 °C, 1 minute at 5 °C, one reading every 10 seconds.
  2. It records the local time at which the condition became true.
  3. The alert-sink pod (section 6) records, by its own clock, when Alertmanager delivered the firing alert.
  4. It subtracts, after correcting for the difference between the two clocks, measured with the midpoint method over seven exchanges. If the clocks differ by more than a second the test refuses to run, because the result would be meaningless.
# ops/excursion_test.py
#!/usr/bin/env python3
"""Measured excursion test. Publishes a scripted profile as reefer-004 through AWS IoT Core and times the alert.

  python ops/excursion_test.py --endpoint $IOT_ENDPOINT --certdir certs --sink http://localhost:8088

Needs: certs/reefer-004/{cert,key}.pem, certs/AmazonRootCA1.pem, and
kubectl -n monitoring port-forward svc/alert-sink 8088:8080 running in another terminal."""
import argparse, json, ssl, sys, time, urllib.request
import paho.mqtt.client as mqtt

BASE_S, HOT_S, COOL_S, STEP = 120, 360, 60, 10          # 2 min in range, 6 min at 9.0 C, 1 min back in range
LIMIT_S = 300                                            # the rule: above the limit for 5 minutes
BUDGET_S = 60                                            # the target: a page within 60 s of the condition


def clock_offset(sink):
    """Sink clock minus local clock, from the exchange with the smallest round trip (midpoint method)."""
    best = None
    for _ in range(7):
        a = time.time(); s = json.load(urllib.request.urlopen(sink + "/time", timeout=5))["now"]; b = time.time()
        if best is None or b - a < best[0]:
            best = (b - a, s - (a + b) / 2)
    return best[1], best[0]


def latency(t_cond_local, recv_sink, offset):
    """Seconds from the moment the condition became true (local clock) to the alert's arrival (sink clock)."""
    return (recv_sink - offset) - t_cond_local


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--endpoint", required=True); ap.add_argument("--certdir", default="certs")
    ap.add_argument("--sink", default="http://localhost:8088"); ap.add_argument("--device", default="reefer-004")
    a = ap.parse_args()

    offset, rtt = clock_offset(a.sink)
    print(f"clock offset sink-local: {offset*1000:+.0f} ms (round trip {rtt*1000:.0f} ms)")
    if abs(offset) > 1.0:
        sys.exit("clocks differ by more than 1 s: fix time sync before measuring")
    before = len(json.load(urllib.request.urlopen(a.sink + "/alerts", timeout=5)))

    c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=a.device, protocol=mqtt.MQTTv311)
    c.tls_set(ca_certs=f"{a.certdir}/AmazonRootCA1.pem", certfile=f"{a.certdir}/{a.device}/cert.pem",
              keyfile=f"{a.certdir}/{a.device}/key.pem", tls_version=ssl.PROTOCOL_TLS_CLIENT)
    c.connect(a.endpoint, 8883, keepalive=30); c.loop_start()

    n_base, n_hot, n_cool = BASE_S // STEP, HOT_S // STEP, COOL_S // STEP
    t_cond = None
    for i in range(n_base + n_hot + n_cool):
        hot = n_base <= i < n_base + n_hot
        now = time.time()
        body = {"device_id": a.device, "seq": i + 1, "ts": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(now)) + f".{int(now*1000)%1000:03d}Z",
                "temp_c": 9.0 if hot else 5.0, "humidity": 60.0, "door_open": False, "compressor_on": True}
        info = c.publish(f"fleet/{a.device}/telemetry", json.dumps(body, separators=(",", ":")), qos=1)
        info.wait_for_publish(timeout=5)
        if i == n_base + LIMIT_S // STEP:                 # the 31st reading above the limit: the run is exactly 300 s
            t_cond = now
            print(f"condition true at reading {i+1}, {time.strftime('%H:%M:%S', time.localtime(now))}")
        time.sleep(STEP)
    c.loop_stop(); c.disconnect()

    time.sleep(5)
    seen = json.load(urllib.request.urlopen(a.sink + "/alerts", timeout=5))[before:]
    firing = [x for x in seen if x["labels"].get("device") == a.device and x["status"] == "firing"]
    if not firing or t_cond is None:
        sys.exit("FAIL: no firing alert reached the sink")
    lat = latency(t_cond, firing[0]["recv"], offset)
    print(f"alert reached the sink {lat:.1f} s after the condition (budget {BUDGET_S} s, clock uncertainty about {rtt/2*1000:.0f} ms)")
    print("RESULT:", "PASS" if lat <= BUDGET_S else "FAIL")


if __name__ == "__main__":
    main()
python3 -m venv .venv && . .venv/bin/activate && pip install paho-mqtt
kubectl --context hub -n monitoring port-forward svc/alert-sink 8088:8080 &
export IOT_ENDPOINT=$(aws iot describe-endpoint --endpoint-type iot:Data-ATS --query endpointAddress --output text)
python ops/excursion_test.py --endpoint $IOT_ENDPOINT --certdir certs --sink http://localhost:8088

Expected output, in this shape (the numbers are placeholders for what you will measure):

clock offset sink-local: +<ms> ms (round trip <ms> ms)
condition true at reading 31, HH:MM:SS
alert reached the sink <s> s after the condition (budget 60 s, clock uncertainty about <ms> ms)
RESULT: PASS

Run it at least ten times, at different times of day, and keep every result, not the best one. Where the time goes, for reading the numbers: the IoT rule and MSK add a second or two; the KafkaSource polls and delivers within about a second; the handler’s query and insert take milliseconds; Alertmanager’s group_wait: 0s removes the grouping delay; the webhook is the last hop. A result over 60 seconds usually means a cold handler, a source that is not at minimum scale, or Alertmanager’s group_wait left at its default. Record the worst of the runs in the quality file, with the versions and the date.

A second test, for no alert while in range: publish the same profile at 7.9 °C and confirm no firing alert and no row in excursion_cases. A third, for reseeding after an outage: stop the simulator’s network for ten minutes at 9 °C so it spools, restore it, and confirm that exactly one case opens, marked late, with the start time of the first hot reading.

Checkpoint 9. Ten runs are recorded, the worst is under 60 seconds or the cause is written down, and the in-range test shows no alert.

10. Troubleshooting, cost, cleanup, design notes and checklist

Troubleshooting

Symptom Likely cause What to check
openGemini refuses connections from other pods It listens on loopback only The bind-address entries in its config; set them to 0.0.0.0 in a mounted config file and restart
check_store.py shows stored below the range A write failed, or Vector restarted before an acknowledgement kubectl -n ingest logs deploy/vector for sink errors; field-type conflicts show as 400 responses
Write rejected: field type conflict The field was first written as another type SHOW FIELD KEYS FROM readings; drop the measurement on a lab database and rewrite
KafkaSource is not ready Wrong bootstrap list, TLS or SCRAM settings kubectl -n react describe kafkasource; check the msk-kn Secret keys against the source’s net.sasl section
Handler logs nothing The source’s consumer group is behind, or the topic has no messages kafka-consumer-groups.sh --describe --group knative-excursion through the client pod
A case opens twice The partial unique index is missing \d excursion_cases should show one_open_case; rerun db-init
No alert, but a case exists Alertmanager is unreachable from the handler The ALERTMANAGER_URL value and the pod’s logs; the handler returns 5xx so the source retries
Alert arrives late group_wait default, cold handler, or a refresh gap Section 9 notes; minScale on the Service
EventBridge shows failed invocations Certificate not trusted or key wrong The API destination’s DLQ in SQS; curl the endpoint as in 7.3
401 with the right key The Secret has a trailing newline Recreate it with --from-literal and no echo
No ENTER event Position outside the polygon, wrong winding, or tracker not linked to the collection aws location list-tracker-consumers; swap the ring direction
Grafana panel empty Data source URL or database Data source test; the query SELECT * FROM readings LIMIT 1 through the port-forward
Pods stuck in ImagePullBackOff ECR permission or a tag used instead of a digest The node role’s ECR policy; use the digest

Cost note

No prices were read for this post, so none are quoted. The recurring items are: an EBS volume for openGemini, the NLB and its traffic, the Knative handlers (one warm pod each at minimum scale), the Prometheus and Grafana volumes, Amazon Location tracker updates and geofence evaluations (billed per request), EventBridge invocations, SQS for the dead letters, and the RDS instance from Part 1. The load here is small. Price it in the AWS Pricing Calculator for ap-south-1, and set a budget alarm before you leave the lab running.

Cleanup

kubectl --context hub delete -f platform/knative/broker.yaml --ignore-not-found
kubectl --context hub -n react delete ksvc --all; kubectl --context hub -n react delete kafkasource --all
kubectl --context hub -n kourier-system delete svc kourier-public
helm --kube-context hub -n monitoring uninstall monitoring; helm --kube-context hub -n dashboards uninstall perses
kubectl --context hub delete -f platform/opengemini.yaml
aws location delete-geofence-collection --collection-name $COLLECTION 2>/dev/null   # OpenTofu removes it too
(cd infra/foundation && tofu destroy -target=aws_cloudwatch_event_rule.geofence)    # then the rest of the file
aws iot delete-topic-rule --rule-name coldchain_position_to_location

Volumes marked Retain stay; delete them by hand if you want the data gone. The RDS data remains until Part 5’s teardown.

Design notes and version checks

Things this blueprint chose, and things to confirm on your versions before relying on them.

  • Store. One openGemini server with a 20 GiB volume, database telemetry, retention rp400d. Confirm the image tag, that the server’s HTTP bind address is reachable from other pods, and where its config file lives if you turn on authentication. Authentication is off in this build, so the network policy is the only guard: enable it before any shared use.
  • Vector. The influxdb_logs sink with version: "1" and acknowledgements.enabled. Confirm the chart’s key names and that temp_c stays a float.
  • Knative. v1.24.0; confirm the Kafka extension has the same tag, that KafkaSource accepts the TLS and SASL settings used for MSK’s public certificate, the Broker’s auth-secret key names and the kafka-broker-config ConfigMap keys, and the DomainMapping and ClusterDomainClaim API versions.
  • Ingress. The Kourier pod label and the NLB annotations for ACM termination are the in-tree ones; if you run the AWS Load Balancer Controller, use its annotations instead.
  • EventBridge. The input transformer’s quoting is easy to get wrong: if the gateway receives a body that is not a CloudEvent, look at the DLQ message first. API destinations time out at 5 seconds, which is why the gateway answers before it forwards.
  • Amazon Location. Resource names for the tracker and collection in OpenTofu, and the location action for IoT rules (done by CLI here), depend on your provider version. Events can repeat; the table’s primary key absorbs that.
  • Alerting. Alertmanager treats an alert that is not refreshed as resolved after resolve_timeout; the handler refreshes about once a minute. Mean kinetic temperature uses ΔH/R = 10000 K; use the product’s own value if it has one.
  • Dashboards. Grafana reads openGemini through its InfluxDB data source with InfluxQL; check the Perses chart’s values for persistence.
  • Checked offline. The detection rule at its edges, the MKT arithmetic and the case open, refresh and close flow were run against fakes on a laptop. They show the logic is as described, not that the cluster behaves.
  • Not measured. The alert delay (section 9 is a method), live throughput, and cost.

Setup checklist

  • 1. Prerequisites: done when the exports print, the cluster answers and Vector logs real readings
  • 2. openGemini: done when the volume is bound, the retention policy exists and the same point written twice counts once
  • 3. RDS schema: done when db-init completes, and completes again on a rerun
  • 4. Vector sink: done when check_store.py passes, before and after a replay
  • 5. Knative and handlers: done when the KafkaSource is ready and the scripted run opens exactly one case
  • 6. Alerting and dashboards: done when amtool accepts the config, Perses draws and Grafana shows temperatures
  • 7. Geofences: done when the gateway answers 401 and 202, and an ENTER produces one row
  • 8. Mean kinetic temperature: done when a delivered shipment has one summary row and a second run adds none
  • 9. Excursion test: done when ten runs are recorded and the worst is under 60 seconds or the cause is written down
  • 10. Cleanup path: done when you know which of these resources you will delete and in what order

Tools used

openGemini, Knative, Vector, Apache Kafka, Prometheus, Perses and OpenTofu, plus Amazon RDS for PostgreSQL, Amazon Location Service, Amazon EventBridge, AWS IoT Core, Amazon ECR, Alertmanager and Grafana.

What comes next

Part 4 adds the partner API that reads shipments and cases from RDS, Dex sign-in for Grafana, and the signed-image and policy layers. Part 5 tests an RDS snapshot restore and closes the project.

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.