Sprint 4 project: Machine Monitoring for a Pharmaceutical Plant on GCP, Overview

Sprint 4 ends with one project: live monitoring for the machines of an oral solid dosage plant, built on Google Cloud, that any DevOps engineer can reproduce without a factory. This overview fixes the scenario, the requirements, the architecture, the Google Cloud design, the wiring of all 15 tools introduced this sprint, the cost, and the repository layout. Five build parts follow, one every half day. Read this post once, then build from Part 1.

What this is and is not. The posts give engineering guidance for a regulated setting. They are not legal or regulatory advice, and nothing here is validated or compliant on its own. Whether a system meets a company’s quality policy, or any regulation that applies to it, is decided by the company’s quality unit through its own validation. I also built the examples from documentation and from the tool briefs, not from a plant, and the section “What I could not test” lists exactly what has not run end to end.

The scenario

An oral solid dosage plant makes tablets and capsules. It wants one place to see its machines live, to be told quickly when something drifts, and to keep evidence that the data is trustworthy. The machines are:

• Tablet presses: compression force, turret speed.
• Fluid-bed dryers: inlet temperature, humidity.
• Autoclaves and sterilisers: temperature, pressure.
• Blister packaging lines: seal temperature, reject count.
• Cleanroom HVAC: differential pressure, particle counts.
• A cold store: temperature, with excursions to detect.

The example targets, which each plant would adjust, are about 60 machines, 30,000 tags sampled every 1 to 5 seconds, an alarm within 10 seconds of an excursion, no data lost during a 4-hour outage of the link between plant and cloud, and audit evidence kept for the period the company’s quality policy sets (the examples use 7 years).

There is no factory in this build. An OPC UA simulator and a vibration data generator stand in for the machines, and a small virtual machine stands in for the plant, so every step can be reproduced.

Requirements

Requirement	Example target	How the design meets it	Built in
Alarm quickly	Alarm within 10 s of an excursion, link up	Short sampling interval, no batching delay on alarm tags, a continuous query on the stored data, direct routing to the on-call person	Parts 2 and 3
No data loss in a link outage	4 hours	EMQX queues data on disk at the edge while the link is down; the cloud side acknowledges only after the database write; a chaos test proves it	Parts 2 and 5
Attributable and traceable changes	Every change reviewed and recorded	Pull-request infrastructure changes through Atlantis, Git-driven delivery through Argo CD, signed images checked at admission, admission rules, audit logs	Parts 1 and 4
Retention	Set by the company’s quality policy (examples: 7 years)	A Cloud Storage retention policy with bucket lock on the audit evidence bucket	Parts 1 and 4
Original data preserved	Raw values kept as recorded	Raw readings land unchanged in TimescaleDB; derived values are separate; the evidence bucket is write-once	Parts 3 and 4
Secrets not in Git	None in repositories	Secret Manager, delivered by external-secrets	Parts 1 and 4
Recoverable	Restore after loss	Database backups and Velero, exercised in a game day	Part 5

Two limits are worth saying now. The 10-second alarm target applies while the link to the cloud is up, because the alarm path runs on Google Cloud. A plant that needs alarms during an outage needs a plant-local rule as well; Part 3 shows an EMQX rule that does this for one tag, and it is an extension, not a validated safety function. And no part of this replaces a machine’s own alarms and interlocks.

Architecture

The shape is machines on an operational technology (OT) network, an edge Kubernetes cluster in the plant, one outbound bridge to Google Cloud, and a GKE cluster that stores, detects, shows, audits and serves.

The second diagram follows one tag value through the system.

1. A tag changes. An OPC UA server in the OT zone exposes each machine’s tags.
2. Telegraf reads it. Akri found the server; Telegraf subscribes to the tags and publishes each value to MQTT.
3. EMQX queues it. The edge broker holds data on disk if the link is down.
4. The bridge sends it. EMQX publishes to a Pub/Sub topic over outbound HTTPS. Nothing in the cloud connects into the plant.
5. Ingest pulls it. A second Telegraf, on GKE, pulls the subscription and acknowledges a message only after the insert succeeds.
6. The database keeps it. TimescaleDB, run by CloudNativePG, stores the original reading.
7. Drasi notices. A continuous query over the stored data raises an alarm when a tag leaves its range.
8. A person is told. Alertmanager routes the alarm, and kagent drafts triage advice for the on-call engineer. Agents advise; a person approves anything that changes production.

The tools and how they are wired

Every one of the 15 tools from this sprint appears below, followed by the tools reused from earlier sprints. “Builds in” names the part whose steps and checkpoints install and prove the tool. Where an earlier brief on this site covers a tool, the link goes there. There is no separate brief on this site for Grafana or Velero, so those rows have no link.

Tool	Role in the plant	Talks to	Builds in	Brief
Akri	Discovers OPC UA servers from the edge cluster	Kubernetes API, the OPC UA servers	Part 2	Akri
Telegraf (edge)	Reads machine tags over OPC UA, publishes MQTT	OPC UA servers, EMQX	Part 2	Telegraf
EMQX	Edge MQTT broker; disk-backed queue; forwards to Pub/Sub	Telegraf, Pub/Sub	Part 2	EMQX
Telegraf (ingest)	Pulls the Pub/Sub subscription and inserts rows	Pub/Sub, TimescaleDB	Part 3	Telegraf
CloudNativePG	Runs and backs up the PostgreSQL cluster	Kubernetes API, Cloud Storage	Part 3	CloudNativePG
TimescaleDB	Time-series history for every tag	Ingest, Drasi, Perses, KServe	Part 3	TimescaleDB
Drasi	Continuous queries that raise alarms	TimescaleDB, Alertmanager	Part 3	Drasi
Perses	Operator and quality dashboards as code	TimescaleDB, Prometheus	Part 3	Perses
Grafana Loki	Audit and event log store	Fluent Bit, Perses or Grafana	Part 4	Loki
OPA and Gatekeeper	Admission rules for both clusters	Kubernetes API server	Part 4	OPA
Kubescape	Posture and framework scans, kept as evidence	Kubernetes API, Cloud Storage	Part 4	Kubescape
external-secrets	Secret Manager values into Kubernetes Secrets	Secret Manager, Kubernetes API	Parts 1 and 4	external-secrets
Notary Project and Ratify	Sign images; check signatures at admission	Artifact Registry, Gatekeeper	Part 4	Notary and Ratify
Atlantis	Pull-request plan and apply for the cloud foundation	GitHub, OpenTofu, Google Cloud APIs	Part 1	Atlantis
KServe	Serves the predictive-maintenance model	TimescaleDB, Gateway	Part 5	KServe
Krkn	Game-day tests: link outage, node loss, broker restart	Kubernetes API (runs outside the cluster)	Part 5	Krkn
kagent	Alarm-triage agent for the on-call engineer	Alertmanager, read-only Kubernetes tools, a model	Part 5	kagent
Kubernetes, k3s	GKE in the cloud; k3s at the plant edge	everything	Part 1	Kubernetes, k3s
KubeEdge	Not used; an alternative edge platform	none	none	KubeEdge
Cilium	Network policy between plant zones on the edge cluster (GKE uses its own Dataplane V2)	Pods	Part 1	Cilium
Argo CD	Git-driven delivery to both clusters	Git, Kubernetes API	Parts 1 and 4	Argo
OpenTofu	Declares the Google Cloud foundation	Google Cloud APIs, state bucket	Part 1	OpenTofu
cert-manager	Certificates for brokers and gateways	Kubernetes API	Part 2	cert-manager
Prometheus, Alertmanager	Platform metrics; alarm routing	Drasi, kagent, receivers	Part 3	Prometheus
Grafana	Optional second dashboard viewer	Prometheus, Loki	Part 3	none on this site
OpenTelemetry	Traces from kagent and the model server	collectors	Part 5	OpenTelemetry
Fluent Bit	Ships logs to Loki (the Fluentd brief covers the family)	Loki	Part 4	Fluentd
Kgateway	Gateway for dashboards and model endpoints	Perses, KServe	Part 5	Kgateway
Falco	Runtime detection on the clusters	Kernel events, Loki	Part 4	Falco
Velero	Backup of cluster objects	Cloud Storage	Part 5	none on this site
OpenCost	Cost per namespace	Prometheus	Part 5	OpenCost
Keycloak and Google identity	Sign-in to dashboards and Argo CD	Perses, Argo CD	Part 4	Keycloak

A few of these tools do not meet each other directly, and the wiring says how they connect anyway. Drasi has no first-party Pub/Sub or Alertmanager reaction in its documentation that I found, so Part 3 connects it through a documented reaction and says exactly which. Notary has no Google Cloud KMS plugin, and Ratify v1 has no Google Cloud authentication provider, so Part 4 signs with a key outside Cloud KMS and gives Ratify its registry credentials as a Kubernetes secret. kagent 1.0 does not support Vertex AI as a model provider, so Part 5 uses a Gemini API key. If any of these turns out not to wire when the part is built, the part says so in plain words and the brief is the place to look.

The Google Cloud design

Project and folder layout

Use one project for the platform and, ideally, a separate one for the plant stand-in, under one folder, so that the plant can be shown to have no network path inward. A lab can collapse this to a single throwaway project; the design reads the same.

Resource	Name used in the series	Purpose
Folder	plant-monitoring	Groups the projects; one place for policy
Project	plant-cloud	GKE, Pub/Sub, storage, secrets, registry, DNS
Project	plant-edge	The VM that stands in for the plant
Region	asia-south1 (Mumbai)	One region for everything, near the author; change it if you prefer

Network

• cloud-vpc with one subnet for GKE nodes and secondary ranges for Pods and Services. GKE uses a VPC-native cluster.
• plant-vpc in the plant project, with no inbound firewall rules. The VM has no public address; it reaches the internet through Cloud NAT and is administered through IAP TCP forwarding.
• No peering and no VPN between the two. The only path from plant to cloud is outbound HTTPS from EMQX to the Pub/Sub API. That is the “one-way by default” bridge: nothing in the cloud can open a connection to the plant.

GKE

A GKE Standard cluster with Dataplane V2, which is Cilium based and enforces Kubernetes network policy, in the Regular release channel. Standard mode is chosen over Autopilot because several tools in this build install webhooks, CRDs and privileged agents (Falco, Kubescape, Gatekeeper) and the series needs control of node pools.

GKE versions move quickly. On 14 October 2026 the Regular channel default was 1.36.4-gke.1495002, and the 27 October release notes list minor versions 1.34 to 1.37 as available in the Regular channel. kagent 1.0 needs Kubernetes 1.37 or newer, so a build that includes kagent 1.0 needs a 1.37 cluster, and a build on 1.36 uses the kagent 0.10 line. Part 1 reads the versions the channel offers on the day with gcloud container get-server-config and pins what it finds.

Why GKE is fine here although Sprint 3 ran on Compute Engine: Sprint 3 used Submariner to join clusters, and Submariner’s tested network plugins did not match GKE’s, so that sprint ran Kubernetes on Compute Engine with Calico. Sprint 4 has no Submariner. The edge and cloud clusters are joined only by an outbound message path, so a managed control plane is the better choice.

Messaging: Pub/Sub, not Managed Service for Apache Kafka

The plan left this open until the docs were checked. I checked them on 8 November 2026 and chose Pub/Sub.

Question	Pub/Sub	Managed Service for Apache Kafka
Does EMQX have a native sink?	Yes: the EMQX brief documents a Google PubSub Producer connector and sink	EMQX also has a Kafka sink
Does the cloud consumer have a native source?	Telegraf’s cloud_pubsub input (since Telegraf 1.10.0) reads a pull subscription	Needs a Kafka consumer
What do you operate?	A topic and a subscription	A cluster: at least 3 vCPU and 3 GiB of RAM, spread over three zones, reached through Private Service Connect endpoints
How is it billed?	By data volume: $40 per TiB after the first 10 GiB each month for the basic delivery SKU	By capacity: $0.09 per DCU-hour in us-central1 (1 vCPU is 0.6 DCU, 1 GiB of RAM is 0.1 DCU), local storage, long-term storage, and at least four endpoints at $0.01 an hour each
Idle cost	Close to zero	About $277 a month at the minimum size (my arithmetic, below)

The arithmetic for the Kafka floor, using the us-central1 prices on Google’s page: 3 vCPU with 12 GiB of RAM is 1.8 plus 1.2, so 3.0 DCU, which is $0.27 an hour, or about $197 for 730 hours; storage is billed at 100 GiB per vCPU, so 300 GiB at $0.000232877 per GiB-hour is about $51; and four endpoints at $0.04 an hour are about $29. The total is about $277 a month before data transfer, and before the separate Kafka Connect cluster if you add one.

The arithmetic for Pub/Sub at the full example scale, which is my own estimate from assumptions: 30,000 tags sampled on average every 3 seconds is 10,000 values a second; at 60 bytes a value that is about 0.6 MB/s, or about 1.4 TiB a month. Throughput is charged for publishing and again for each subscription’s delivery, so about 2.8 TiB at $40 is about $113 a month. Message attributes and a few bytes of metadata count toward the size, so measure your own payload.

So at the example scale and below, Pub/Sub is the cheaper and simpler choice, and the lab itself fits inside the free 10 GiB. Kafka becomes the better choice when you need what Kafka gives (offset replay, consumer groups, the Kafka ecosystem, or sustained high volume). Google’s own pricing pages list roughly $2,000 a month for Pub/Sub against roughly $1,100 for managed Kafka at 10 MiB/s of producer bandwidth, under assumptions that differ between the two pages, so treat that as a sign that the crossover exists, not as a quote. Part 1 creates the topic and subscription and keeps the choice reversible, because EMQX can write to either.

One caution applies to any choice: Pub/Sub delivers at least once, so the same message can arrive twice. Part 3 makes the insert idempotent with a unique key on tag and timestamp.

Registry, secrets and keys

• Artifact Registry holds the images and Helm charts that Part 4’s signing and admission checks protect.
• Secret Manager holds credentials. It bills per active secret version ($0.06 per version per location per month, with six free) and per access ($0.03 per 10,000), and external-secrets reads from it.
• Cloud KMS holds customer-managed keys (CMEK) for the evidence bucket and for Secret Manager. One rule matters for the evidence bucket: Cloud Storage does not let you destroy the Cloud KMS key versions that encrypt objects in a locked bucket until those objects have met their retention time. Plan key rotation with that in mind.
• Signing keys: Notation has no Google Cloud KMS plugin, so Part 4 signs with a key kept outside Cloud KMS and says so.

Evidence storage: retention policy and bucket lock

The evidence bucket (audit logs, Kubescape reports, validation records) gets a retention policy so that objects cannot be deleted or overwritten until they are old enough. Locking the policy makes it permanent: once locked, the policy cannot be removed or shortened, only lengthened, and the bucket cannot be deleted until every object in it has met its retention period. That is the point in production and a trap in a lab. Use a throwaway bucket with a one-day period when you practise, and lock the real one only after your quality unit has agreed the period. I did not verify what happens to a whole project that contains a locked bucket when you delete the project, so keep locked buckets out of any project you plan to delete.

DNS and names

Cloud DNS hosts a public or private zone for the dashboards and the Atlantis webhook endpoint. DNS propagation is one of the usual first-hour failures, so Part 1 has a checkpoint for it.

Identity: Workload Identity Federation for GKE

Workloads on GKE get Google Cloud access without key files. Enabling Workload Identity Federation for GKE gives the project a fixed pool named PROJECT_ID.svc.id.goog, the cluster is registered as an identity provider in it, and node pools use the GKE metadata server. You then grant a role to a Kubernetes service account’s principal, in the form principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME. Part 1 uses this for external-secrets, the ingest Telegraf, CloudNativePG backups and Atlantis. The edge VM is not on GKE, so EMQX’s credential for publishing to one Pub/Sub topic is the one place Part 2 has to choose between Workload Identity Federation for an external workload and a service account key; it will say which it used.

The plant edge

The edge is a single-node k3s cluster on a Compute Engine VM in plant-vpc, standing in for a server in the plant’s DMZ. On a laptop, any Linux machine or virtual machine with k3s does the same job, and the series states the one difference: the VM reaches Google Cloud through Cloud NAT, a laptop through your own network. The edge cluster runs the OPC UA simulator, Akri, Telegraf and EMQX, with Cilium replacing k3s’s default network plugin so that network policy can keep the simulator, the ingest tools and the broker in separate zones. Part 1 gives the install flags. KubeEdge is the usual alternative for fleets of edge nodes and is not used.

The zones inside the edge cluster are three namespaces with default-deny network policy:

• ot-sim for the machines and the vibration generator, which accepts connections only from the ingest namespace.
• ingest for Akri and Telegraf, which may reach ot-sim and the broker and nothing else.
• mqtt for EMQX, which may reach Pub/Sub’s HTTPS endpoint outward and accept MQTT only from ingest.

ALCOA+ and change control

ALCOA+ describes data that is attributable, legible, contemporaneous, original, accurate, and also complete, consistent, enduring and available. The table maps each property to a technical control in this build. It is a mapping of engineering controls, not an assessment: whether a control is enough is for the company’s quality unit to decide.

Property	What the platform does	Where
Attributable	Every value carries machine, tag and source; every change to the platform is tied to a pull request and a person; Keycloak or Google sign-in identifies dashboard users	Parts 2 and 4
Legible	Open, documented formats (SQL tables, JSON), dashboards as code	Part 3
Contemporaneous	The source timestamp is recorded at the edge; the ingest path adds a receive time separately; clocks are synchronised (clock drift is in the troubleshooting lists)	Parts 2 and 3
Original	The raw reading is stored unchanged and never updated; derived values go in separate tables	Part 3
Accurate	Range checks and quality codes from OPC UA are kept; Drasi queries read the stored data; the simulator has fixed seeds so tests are repeatable	Parts 2 and 3
Complete	Disk-backed queue at the edge, acknowledge after write, idempotent insert, and a game day that checks counts after an outage	Parts 2, 3 and 5
Consistent	One time base, one naming scheme for tags, schema in version control	Parts 2 and 3
Enduring	Retention policy and bucket lock on evidence, database backups	Parts 1, 4 and 5
Available	Backups, restore tests, a documented recovery step	Part 5

Change control follows one path. Infrastructure changes are pull requests that Atlantis plans and applies after review. Workload changes are Git commits that Argo CD applies. Images must be signed, and Ratify with Gatekeeper rejects unsigned ones. Gatekeeper rules and Kubescape scans catch drift, and the scan results are kept as evidence, which is a record that a scan ran and what it found, not a certificate. Part 4 closes with a validation evidence checklist for the quality unit. Two cautions from the briefs carry over: Atlantis runs apply before the pull request merges by default, and Terraform runs on the Atlantis host with that host’s credentials, so Part 1 scopes those credentials tightly.

Prerequisites

Accounts, APIs and roles

• A Google Cloud account with billing enabled and permission to create a folder and two projects. If you use one throwaway project, owner on it is enough for a lab, but the roles below are what a least-privilege setup needs.
• A GitHub account and a repository for the platform code (Atlantis and Argo CD read from it).
• A Gemini API key for kagent (Part 5), and a domain name you can delegate to Cloud DNS (optional; Part 1 has a no-domain path).

APIs to enable in plant-cloud, which Part 1 does with one command and then verifies: Compute Engine, Kubernetes Engine, IAM, IAM Service Account Credentials, Cloud Resource Manager, Service Usage, Secret Manager, Cloud Key Management Service, Artifact Registry, Cloud DNS, Cloud Pub/Sub, Cloud Storage, and Cloud Logging. In plant-edge: Compute Engine, IAM and Identity-Aware Proxy.

Roles for the person doing the build, as role IDs: roles/resourcemanager.folderCreator, roles/resourcemanager.projectCreator and roles/billing.user to create the projects; then in the projects roles/serviceusage.serviceUsageAdmin, roles/compute.networkAdmin, roles/container.admin, roles/iam.serviceAccountAdmin, roles/resourcemanager.projectIamAdmin, roles/secretmanager.admin, roles/cloudkms.admin, roles/artifactregistry.admin, roles/dns.admin, roles/pubsub.admin and roles/storage.admin. The workload identities get far narrower roles (for example roles/pubsub.publisher on one topic, roles/pubsub.subscriber on one subscription and roles/secretmanager.secretAccessor on named secrets), and each part lists them as it creates them.

Quotas

New projects can start with low compute quotas. Before Part 1, open IAM and Admin, then Quotas, and check CPUs in asia-south1 and all-regions CPUs against the sizing below (3 GKE nodes of 8 vCPU, plus the 4 vCPU edge VM, so 28 vCPU), and request an increase if you are short. I could not check the default quotas of a fresh account, so treat the figure as the thing to verify rather than a promise.

Tools on your workstation and versions

Tool	Version to use	Source of the version
gcloud CLI	current	gcloud version on the day
OpenTofu, kubectl, Helm	current stable	each part pins the output of tofu version, kubectl version --client and helm version
GKE	1.36 or 1.37 (see above)	GKE release notes, 27 October 2026
Akri	v0.14.0	the Akri brief
Telegraf	v1.40.2	the Telegraf brief
EMQX	6.3.1	the EMQX brief
TimescaleDB	2.31.0	the TimescaleDB brief
CloudNativePG	1.30.x (v1.30.0 was published on 29 June 2026 per its Go module page; Part 3 reads the current patch)	pkg.go.dev, 8 November 2026
Perses	0.54.0	the Perses brief
Loki	3.7.x	the Loki brief
KServe	v0.21.0	the KServe brief
OPA	v1.22.0	the OPA brief
Kubescape	v4.0.19	the Kubescape brief
Notation and Ratify	v1.3.2 and v1.4.8	the Notary and Ratify brief
Atlantis	v0.49.0	the Atlantis brief
Krkn	v5.2.14	the Krkn brief
kagent	v1.0.0-rc3, a release candidate	the kagent brief
cert-manager	1.21	the cert-manager brief
Kgateway	2.4.x	the Kgateway brief
asyncua	1.1.x (1.1.8 was the newest I could find, on conda-forge; I did not read the PyPI page)	Part 2 pins it
Python	3.11 or newer	for the simulator

Versions I did not re-check today (k3s, Cilium, Argo CD, OpenTofu, Prometheus, OpenTelemetry, Falco, OpenCost, Keycloak, external-secrets, Drasi) are pinned in the part that installs them, against the official release page on that day. A quick check that the workstation is ready:

gcloud version
gcloud auth list
tofu version
kubectl version --client
helm version
python3 --version

Expected: each command prints a version, gcloud auth list shows your account as active, and Python is 3.11 or newer. A missing tool is the only thing this block can prove wrong.

Cost estimate, read on 8 November 2026

Prices below are list prices read from Google’s pages on the day, except the virtual machine prices, which I could only find on third-party price lists (gcloud-compute.com and devzero.io, both showing the same figures) and not on a Google page I could open. Treat the machine lines as approximate and confirm them in Google’s pricing calculator.

Item	Price read	Source
GKE cluster management fee	$0.10 per cluster-hour; the free tier gives $74.40 a month of credit per billing account, which covers one zonal or Autopilot cluster but not a regional one	GKE pricing page
e2-standard-4, asia-south1	$0.161 an hour ($117.51 a month)	third-party list
e2-standard-4, us-central1	$0.134 an hour ($97.84 a month)	third-party list
Pub/Sub throughput	$40 per TiB after the first 10 GiB a month	Pub/Sub pricing
Secret Manager	$0.06 per active version per location per month (six free); $0.03 per 10,000 accesses	Secret Manager pricing
Managed Kafka (for comparison, not used)	$0.09 per DCU-hour in us-central1	Managed Kafka pricing
Cloud Storage Standard	about $0.02 per GB-month in US regions; I did not read the asia-south1 line	Cloud Storage pricing examples
Cloud KMS, Cloud DNS, Artifact Registry, persistent disks	not captured, so not itemised; each is small next to compute at lab scale

The sizing is my estimate, not a measurement: a zonal GKE Standard cluster of three e2-standard-8 nodes, plus one e2-standard-4 for the edge. An e2-standard-8 is eight vCPU and 32 GiB; I assumed it costs twice an e2-standard-4, which matches how the e2 family scales but I did not read it. In asia-south1 that is 3 × $0.322 plus $0.161, so about $1.13 an hour, with the cluster fee covered by the free credit. Left on for a month (730 hours) that is about $823, or about $27 a day. A three-day build running 10 hours a day costs about $34 in compute, plus disks, and the Pub/Sub lab traffic stays inside the free 10 GiB. A regional cluster adds the $0.10 an hour fee and more nodes; a plant would also want it.

The three-node size is a starting point. I have not run the whole stack on it, and the part that installs each tool gives its resource requests, so expect to adjust the node count after Part 3.

Cleanup summary

Each part ends with a cleanup block that removes only what it created. The order, from the last part to the first:

1. Part 5: Krkn scenarios, KServe service, kagent, Velero schedule.
2. Part 4: policies and constraints, Ratify, signed test images, Loki. Do not lock the lab evidence bucket.
3. Part 3: dashboards, Drasi queries, TimescaleDB cluster and its backups.
4. Part 2: EMQX bridge, Telegraf, Akri, the simulator.
5. Part 1: tofu destroy through Atlantis, then delete the edge VM, Cloud NAT and the Pub/Sub topic. Cloud KMS key versions are scheduled for destruction, not removed at once, and a locked bucket cannot be deleted early.

The fastest way to stop all charges is to delete the two projects, which is why the lab should not contain a locked bucket.

Repository and namespace layout

One repository, plant-monitoring, holds everything. Parts create the folders as they need them.

plant-monitoring/
  infra/              OpenTofu: folder, projects, network, GKE, Pub/Sub, storage, KMS, secrets, DNS, registry, edge VM
  atlantis.yaml       Atlantis project definitions
  repos.yaml          Atlantis server-side repository rules
  gitops/             Argo CD applications: platform/ and apps/
  edge/               k3s manifests: simulator, Akri, Telegraf, EMQX, network policies
  cloud/              GKE manifests: CloudNativePG, Telegraf ingest, Drasi, Perses, Loki, KServe, gateway
  policy/             Gatekeeper templates and constraints, Kubescape frameworks, Ratify configuration
  simulator/          asyncua server and vibration generator
  models/             training notebook and KServe InferenceService
  chaos/              Krkn scenario files
  agents/             kagent resources
  docs/               validation evidence checklist, runbooks

Cluster	Namespaces
Edge (k3s)	ot-sim, akri, ingest, mqtt, cert-manager
Cloud (GKE)	argocd, atlantis, cert-manager, cnpg-system, plant-db, ingest, drasi-system, monitoring, dashboards, logging, kserve, model-serving, gatekeeper-system, ratify, kubescape, external-secrets, kagent, gateway, falco, velero, opencost, keycloak

These names are the series’ convention, not required by the tools.

The simulated plant

The simulator is a Python program using asyncua that serves an OPC UA address space. Each machine is a folder of variables under Plant/<Line>/<Machine>, written to once a second by a seeded random model with a slow drift and, on command, a scheduled excursion. A second small generator produces vibration and bearing-temperature values with a slowly rising trend, which gives the KServe model something to predict.

The lab runs 12 machines (two of each type) and about 240 tags. A knob raises that towards the 60 machines and 30,000 tags of the example; I have not run that scale, so any capacity claim about it is a design estimate.

Machine type	Tags per machine (lab)	Example tags, with units	Normal range	Scheduled excursion
Tablet press	24	CompressionForce (kN), TurretSpeed (rpm), Vibration (mm/s)	8 to 14 kN; 30 to 60 rpm	Force drifts out of range for 30 s
Fluid-bed dryer	20	InletTemp (°C), ProductTemp (°C), ExhaustHumidity (% RH)	55 to 65 °C	Inlet temperature overshoots
Autoclave / steriliser	18	ChamberTemp (°C), ChamberPressure (kPa)	121 °C ± 1 in the hold phase	Temperature dips during the hold
Blister line	20	SealTemp (°C), RejectCount (count), LineSpeed (blisters/min)	150 to 170 °C	Reject count jumps
Cleanroom HVAC	20	DiffPressure (Pa), ParticleCount05 (per m³), RoomTemp (°C)	10 to 15 Pa	Differential pressure falls below the limit
Cold store	18	Temp (°C), DoorOpen (bool)	2 to 8 °C	Door left open, temperature climbs

These ranges are illustrative values for a demonstration and are not specification limits for any product or process. A real plant sets its own limits.

What I could not test

Nothing in this post has run end to end. Specifically:

• I did not create any Google Cloud resource, so every GCP statement comes from Google’s documentation and pricing pages as I read them today, and every price is a list price that may differ in your region, contract or currency.
• The default compute quotas of a new project, and the machine prices in asia-south1, are not confirmed from Google’s own pages.
• The 3-node sizing is an estimate. The full stack has not been run on it.
• Several wiring choices are unverified until the part that builds them: Telegraf’s PostgreSQL output with TimescaleDB for the ingest, Drasi’s route to Alertmanager, and EMQX’s credential method from outside Google Cloud.
• kagent 1.0 is a release candidate that needs Kubernetes 1.37 or newer, which may not match the GKE version your channel offers.
• Nothing here has been validated for use in a regulated plant.

Where a part finds that something above is wrong, it will say so at the top and this overview will be corrected.

The five parts

1. Part 1: Google Cloud foundation and the plant edge: OpenTofu and Atlantis, the projects, network, GKE, Pub/Sub, Cloud Storage retention, Secret Manager, Artifact Registry, the edge k3s cluster and the network zones.
2. Part 2: Machines to MQTT: the OPC UA simulator, Akri discovery, Telegraf, EMQX, and the store-and-forward bridge to Google Cloud.
3. Part 3: Store, detect, show: TimescaleDB on CloudNativePG, Drasi alarms, Prometheus and Alertmanager routing, Perses and Grafana dashboards.
4. Part 4: Data integrity and change control: the Loki audit trail, immutable storage, Gatekeeper, Kubescape, Ratify, external-secrets, the Argo CD flow, access control and the validation evidence checklist.
5. Part 5: Predict and operate: the KServe model, kagent triage, a Krkn game day, backup and recovery, cost and cleanup.

The Part links go live as each part is published.

Sources: Managed Service for Apache Kafka pricing · Managed Service for Apache Kafka overview · Plan the cluster size · Pub/Sub pricing · Cloud Storage Bucket Lock · GKE pricing · GKE release notes · Secret Manager pricing · About Workload Identity Federation for GKE · Telegraf Google Cloud PubSub input · CloudNativePG on pkg.go.dev · e2-standard-4 prices (third party)