Happy Dussehra! May good always win. Merry Christmas! Warm wishes from Vizag. Happy New Year! Here’s to a great year ahead.

DevOps

Sprint 5 project, Part 1: AWS Foundation, Crossplane and the Depot Cluster

Build the AWS foundation for the cold-chain project: OpenTofu state, VPC, EKS hub, Amazon MSK with SCRAM, KMS, Secrets Manager, ECR and Route 53, then Argo CD, Crossplane creating Object Lock buckets and IoT Core things, and a k0s depot cluster on a VM.

23 min read

  • OpenTofu
  • Crossplane
  • Argo CD
  • k0s
  • EKS
  • Amazon MSK
  • AWS IoT Core
  • S3 Object Lock
  • AWS KMS
  • Secrets Manager
  • Amazon ECR
  • Route 53
  • EKS Pod Identity
  • Kubernetes
  • Helm

This is Part 1 of the five-part build in the Sprint 5 project overview. By the end you have an AWS account laid out for the project, a private network, an EKS hub, a Kafka cluster, keys and secrets, a registry and a DNS zone, Argo CD syncing from Git, Crossplane creating the locked evidence buckets and the IoT Core things from Kubernetes manifests, and a k0s cluster on a VM standing in for a depot.

Where this part fits. This part builds the AWS foundation, the EKS hub, Crossplane and the depot cluster (marked Part 1 on the overall diagram). Open the zoomable Sprint 5 architecture or read the Overview.

Overall Sprint 5 architecture

What this is and is not. This is engineering guidance, not legal or regulatory advice, and nothing here is validated or compliant on its own. It is a reference blueprint: it sets out the intended flow and the wiring between the tools. Versions, chart keys, provider kinds and field names change between releases, so each section gives a check to confirm them on your version. If you hit a bug or a mismatch, use the “Suggest a correction” link on this page.

What you build, and in what order

Architecture: a workstation runs OpenTofu with state in a versioned private S3 bucket. OpenTofu creates the VPC, an EKS hub with Pod Identity, Amazon MSK with SCRAM, a KMS key, Secrets Manager entries, ECR and Route 53. Argo CD on the hub syncs the gitops folder and installs Crossplane, which uses Pod Identity to create the evidence and backup S3 buckets with Object Lock and the IoT Core things and policies. A k0s depot cluster on a VM stands alone in this part and dials the hub in Part 2.

One image, the layers of this part. Open the zoomable diagram, or download it as SVG or PNG.

The work splits in two, by who owns the resource:

  • OpenTofu owns the base: state bucket, VPC, EKS, MSK, KMS, Secrets Manager containers, ECR and Route 53. These are the things Crossplane needs in order to exist, or that rarely change.
  • Crossplane owns what applications ask for: the evidence and backup buckets, and the IoT things, certificates and policies. They are Kubernetes objects, reviewed in Git, reconciled continuously.

Flow in eight steps: prepare the account and tools; create the state bucket by hand; apply OpenTofu for the network and EKS; add KMS, secrets and MSK; install Argo CD; install Crossplane; let Crossplane create the buckets and IoT things; bring up the k0s depot cluster.

  1. Prerequisites
  2. Account, region and variables
  3. The repository and the state bucket
  4. The foundation stack: network zones, EKS, KMS, secrets, MSK, ECR, Route 53
  5. Argo CD
  6. Crossplane and its AWS providers
  7. Evidence and backup buckets with Object Lock
  8. IoT Core things, policies and certificates
  9. The depot: k0s on a VM
  10. Troubleshooting, cost, cleanup, design notes

1. Prerequisites

Item Version or detail Why
AWS account with billing An account you may spend in; MFA on the root user MSK and EKS are billed hourly
AWS CLI v2 Current release; keep aws --version Bootstrap and checks
OpenTofu Current stable (tofu version) See OpenTofu
AWS provider ~> 6.0 Resources below
kubectl, Helm Current releases Cluster work
A GitHub repository cold-chain Layout in the overview Argo CD syncs from it
A domain you control Ability to set NS records Route 53 delegation
An Ubuntu 24.04 VM 2 vCPU, 4 GB RAM, 30 GB disk, outbound internet The depot stand-in
Your public IP Allowed on the EKS public endpoint Restricts the API

2. Account, region and variables

Use a dedicated account or a sandbox you can throw away: Object Lock buckets in compliance mode cannot be emptied early (see section 7). Create an IAM Identity Center user or an IAM user with MFA for the build, with administrator rights for the lab only. Do not use the root user for daily work.

export AWS_REGION=ap-south-1
export AWS_DEFAULT_REGION=$AWS_REGION
export SUFFIX=abc123                     # short and unique; bucket names are global
export STATE_BUCKET=coldchain-tfstate-$SUFFIX
export DOMAIN=coldchain.example.com
export MY_IP=$(curl -s https://ifconfig.me)/32
export GH_ORG=YOUR-GITHUB-ORG-OR-USER
aws sts get-caller-identity

Expected: your account ID and the user ARN. Checkpoint: the account ID is the one you intend to build in.

3. The repository and the state bucket

mkdir -p cold-chain/{infra/foundation,gitops/apps,crossplane,depot} && cd cold-chain
git init -b main

The state bucket is the one thing created by hand, because OpenTofu needs it before it can store anything.

aws s3api create-bucket --bucket $STATE_BUCKET --region $AWS_REGION \
  --create-bucket-configuration LocationConstraint=$AWS_REGION
aws s3api put-bucket-versioning --bucket $STATE_BUCKET \
  --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket $STATE_BUCKET \
  --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
aws s3api put-bucket-encryption --bucket $STATE_BUCKET \
  --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"AES256"}}]}'

Verify:

aws s3api get-bucket-versioning --bucket $STATE_BUCKET
aws s3api get-public-access-block --bucket $STATE_BUCKET

Expected: "Status": "Enabled" and all four public-access flags true. Checkpoint: versioning on, public access blocked.

OpenTofu’s S3 backend locks state with a lock file in the same bucket (use_lockfile = true), so no DynamoDB table is needed.

4. The foundation stack

4.1 Providers, backend and variables

infra/foundation/versions.tf:

terraform {
  required_version = ">= 1.10"
  required_providers {
    aws    = { source = "hashicorp/aws", version = "~> 6.0" }
    random = { source = "hashicorp/random", version = "~> 3.6" }
  }
  backend "s3" {
    # bucket and region are passed at init time (see below)
    key          = "foundation/terraform.tfstate"
    use_lockfile = true
    encrypt      = true
  }
}

provider "aws" {
  region = var.region
  default_tags {
    tags = { project = "cold-chain", managed-by = "opentofu" }
  }
}

infra/foundation/variables.tf:

variable "region"           { default = "ap-south-1" }
variable "name"             { default = "coldchain" }
variable "domain"           { type = string }
variable "my_ip_cidr"       { type = string }
variable "cluster_version"  { type = string }   # see the check below
variable "kafka_version"    { type = string }   # see the check below
variable "associate_scram"  { default = false } # set true on the second apply

Find the versions the account offers right now and write them into terraform.tfvars:

aws eks describe-cluster-versions --query 'clusterVersions[].[clusterVersion,status]' --output table
aws kafka list-kafka-versions --query 'KafkaVersions[].Version' --output text

Choose the newest Kubernetes version that the add-ons below list as supported (aws eks describe-addon-versions --addon-name eks-pod-identity-agent --kubernetes-version <v>), and a current Kafka version offered for Standard brokers.

# infra/foundation/terraform.tfvars
domain          = "coldchain.example.com"
my_ip_cidr      = "203.0.113.10/32"
cluster_version = "1.37"      # example: confirm with the commands above
kafka_version   = "3.9.x"     # example: confirm with list-kafka-versions

4.2 Network zones

One VPC, three private subnets for nodes and brokers, three public subnets for load balancers and the NAT gateway. A single NAT gateway keeps the lab cheap; use one per zone in production.

infra/foundation/network.tf:

data "aws_availability_zones" "az" { state = "available" }

locals {
  azs = slice(data.aws_availability_zones.az.names, 0, 3)
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 6.0"

  name = var.name
  cidr = "10.40.0.0/16"
  azs  = local.azs

  private_subnets = ["10.40.0.0/20", "10.40.16.0/20", "10.40.32.0/20"]
  public_subnets  = ["10.40.240.0/24", "10.40.241.0/24", "10.40.242.0/24"]

  enable_nat_gateway = true
  single_nat_gateway = true

  public_subnet_tags  = { "kubernetes.io/role/elb" = 1 }
  private_subnet_tags = { "kubernetes.io/role/internal-elb" = 1 }
}

The zones, and what may talk to what:

Zone Where Inbound Outbound
Public Public subnets Load balancers only (later parts) Internet
Hub Private subnets, EKS nodes From load balancers; API only from my_ip_cidr NAT, MSK 9096, AWS APIs
Data Private subnets, MSK 9096 from the hub node group and the IoT VPC destination only None needed
Depot Your VM or site Nothing from the internet 443 to the hub API, 8883 to IoT Core

4.3 KMS, Secrets Manager and the MSK credentials

MSK reads SCRAM credentials from Secrets Manager. The secret name must start with AmazonMSK_ and the secret must be encrypted with a customer-managed key, not the default key. The password is never written in code: the secret is created empty here and filled with the CLI.

infra/foundation/security.tf:

data "aws_caller_identity" "me" {}

resource "aws_kms_key" "platform" {
  description             = "cold-chain platform key"
  enable_key_rotation     = true
  deletion_window_in_days = 7
}

resource "aws_kms_alias" "platform" {
  name          = "alias/${var.name}-platform"
  target_key_id = aws_kms_key.platform.key_id
}

# Container only. The value is added with the CLI in section 4.6.
resource "aws_secretsmanager_secret" "msk_ingest" {
  name       = "AmazonMSK_${var.name}_ingest"
  kms_key_id = aws_kms_key.platform.arn
}

4.4 EKS with Pod Identity

infra/foundation/eks.tf:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 21.0"

  name               = "${var.name}-hub"
  kubernetes_version = var.cluster_version

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  endpoint_public_access       = true
  endpoint_public_access_cidrs = [var.my_ip_cidr]
  endpoint_private_access      = true

  enable_cluster_creator_admin_permissions = true

  addons = {
    coredns                = {}
    kube-proxy             = {}
    vpc-cni                = { before_compute = true }
    eks-pod-identity-agent = { before_compute = true }
  }

  eks_managed_node_groups = {
    main = {
      instance_types = ["m6i.large"]
      min_size       = 3
      max_size       = 5
      desired_size   = 3
    }
  }
}

4.5 MSK, ECR and Route 53

infra/foundation/data.tf:

resource "aws_security_group" "msk" {
  name   = "${var.name}-msk"
  vpc_id = module.vpc.vpc_id
}

# SCRAM over TLS listens on 9096. Only the hub nodes may connect for now;
# Part 2 adds the IoT Core VPC destination.
resource "aws_vpc_security_group_ingress_rule" "msk_from_nodes" {
  security_group_id            = aws_security_group.msk.id
  referenced_security_group_id = module.eks.node_security_group_id
  from_port                    = 9096
  to_port                      = 9096
  ip_protocol                  = "tcp"
}

resource "aws_msk_cluster" "main" {
  cluster_name           = var.name
  kafka_version          = var.kafka_version
  number_of_broker_nodes = 3

  broker_node_group_info {
    instance_type   = "kafka.m7g.large"
    client_subnets  = module.vpc.private_subnets
    security_groups = [aws_security_group.msk.id]
    storage_info {
      ebs_storage_info { volume_size = 100 }
    }
  }

  client_authentication {
    sasl { scram = true }
  }

  encryption_info {
    encryption_in_transit {
      client_broker = "TLS"
      in_cluster    = true
    }
    encryption_at_rest_kms_key_arn = aws_kms_key.platform.arn
  }
}

resource "aws_msk_scram_secret_association" "ingest" {
  count           = var.associate_scram ? 1 : 0
  cluster_arn     = aws_msk_cluster.main.arn
  secret_arn_list = [aws_secretsmanager_secret.msk_ingest.arn]
}

resource "aws_ecr_repository" "platform" {
  for_each             = toset(["ingest-handler", "excursion-handler", "geofence-handler", "partner-api", "simulator"])
  name                 = "${var.name}/${each.key}"
  image_tag_mutability = "IMMUTABLE"
  image_scanning_configuration { scan_on_push = true }
  encryption_configuration {
    encryption_type = "KMS"
    kms_key         = aws_kms_key.platform.arn
  }
}

resource "aws_route53_zone" "main" { name = var.domain }

infra/foundation/outputs.tf:

output "cluster_name"       { value = module.eks.cluster_name }
output "oidc_provider_arn"  { value = module.eks.oidc_provider_arn }
output "kms_key_arn"        { value = aws_kms_key.platform.arn }
output "msk_cluster_arn"    { value = aws_msk_cluster.main.arn }
output "msk_secret_arn"     { value = aws_secretsmanager_secret.msk_ingest.arn }
output "name_servers"       { value = aws_route53_zone.main.name_servers }

4.6 Apply, then add the credentials

cd infra/foundation
tofu init -backend-config="bucket=$STATE_BUCKET" -backend-config="region=$AWS_REGION"
tofu validate
tofu plan -out=plan1
tofu apply plan1          # EKS and MSK each take 15 to 30 minutes

Now put the SCRAM credentials in, then associate them (the second apply):

SECRET_ARN=$(tofu output -raw msk_secret_arn)
KMS_ARN=$(tofu output -raw kms_key_arn)
PASS=$(openssl rand -base64 24 | tr -d '/+=')
aws secretsmanager put-secret-value --secret-id "$SECRET_ARN" \
  --secret-string "{\"username\":\"ingest\",\"password\":\"$PASS\"}"
unset PASS
tofu apply -var associate_scram=true

Do not write associate_scram=true into terraform.tfvars and forget it: keep the flag on the command line, or move it into the file once the secret has a value, so a fresh checkout does not try to associate an empty secret.

Point kubectl at the hub and check:

aws eks update-kubeconfig --name coldchain-hub --region $AWS_REGION
kubectl get nodes
aws kafka describe-cluster-v2 --cluster-arn $(tofu output -raw msk_cluster_arn) \
  --query 'ClusterInfo.State'
aws kafka list-scram-secrets --cluster-arn $(tofu output -raw msk_cluster_arn)
aws kafka get-bootstrap-brokers --cluster-arn $(tofu output -raw msk_cluster_arn) \
  --query 'BootstrapBrokerStringSaslScram'

Expected: three nodes Ready; the state "ACTIVE"; the secret ARN listed; and a bootstrap string whose ports are 9096. Keep that string: Parts 2 and 3 use it.

Delegate the domain: set the four NS records printed by tofu output name_servers at your registrar.

Checkpoint: nodes Ready, MSK ACTIVE with the secret associated, nothing in the state bucket marked public.

5. Argo CD

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argocd argo/argo-cd -n argocd --create-namespace --wait
kubectl -n argocd get pods

Expected: the server, repo-server, application-controller, redis and applicationset pods Running. Argo CD pulls from Git, so the repository must be reachable. For a private repository, add it under Settings, Repositories, with a read-only deploy key.

Create the root application that syncs the gitops/apps folder. gitops/root.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR-ORG/cold-chain.git
    targetRevision: main
    path: gitops/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }
git add -A && git commit -m "foundation" && git push -u origin main   # after adding the remote
kubectl apply -f gitops/root.yaml
kubectl -n argocd get applications

Checkpoint: root shows Synced and Healthy.

6. Crossplane and its AWS providers

6.1 The identities

Crossplane’s provider pods need AWS rights. They get them through EKS Pod Identity: no keys in the cluster. Each provider gets its own role and its own service account name, so the association is exact. infra/foundation/crossplane-iam.tf:

data "aws_iam_policy_document" "pod_trust" {
  statement {
    actions    = ["sts:AssumeRole", "sts:TagSession"]
    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "xp_s3" {
  name               = "${var.name}-crossplane-s3"
  assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}

resource "aws_iam_role_policy" "xp_s3" {
  role = aws_iam_role.xp_s3.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["s3:*"]
      Resource = ["arn:aws:s3:::${var.name}-*", "arn:aws:s3:::${var.name}-*/*"]
    }]
  })
}

resource "aws_iam_role" "xp_iot" {
  name               = "${var.name}-crossplane-iot"
  assume_role_policy = data.aws_iam_policy_document.pod_trust.json
}

resource "aws_iam_role_policy" "xp_iot" {
  role = aws_iam_role.xp_iot.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["iot:*"]
      Resource = "*"
    }]
  })
}

resource "aws_eks_pod_identity_association" "xp_s3" {
  cluster_name    = module.eks.cluster_name
  namespace       = "crossplane-system"
  service_account = "provider-aws-s3"
  role_arn        = aws_iam_role.xp_s3.arn
}

resource "aws_eks_pod_identity_association" "xp_iot" {
  cluster_name    = module.eks.cluster_name
  namespace       = "crossplane-system"
  service_account = "provider-aws-iot"
  role_arn        = aws_iam_role.xp_iot.arn
}

iot:* on * is a lab shortcut. Narrow it to the actions the resources below use once they work. The S3 role is limited to buckets named coldchain-*.

tofu apply
aws eks list-pod-identity-associations --cluster-name coldchain-hub

6.2 Install Crossplane through Argo CD

gitops/apps/crossplane.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: crossplane
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.crossplane.io/stable
    chart: crossplane
    targetRevision: 2.4.3        # the version this blueprint pins; confirm the newest stable
  destination:
    server: https://kubernetes.default.svc
    namespace: crossplane-system
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]

6.3 The providers

gitops/apps/crossplane-providers.yaml points Argo CD at the crossplane/ folder:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: crossplane-providers
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR-ORG/cold-chain.git
    targetRevision: main
    path: crossplane
  destination:
    server: https://kubernetes.default.svc
    namespace: crossplane-system
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [SkipDryRunOnMissingResource=true]

crossplane/providers.yaml: the runtime configs pin each provider’s service account name so that the Pod Identity associations match.

apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-aws-s3
spec:
  serviceAccountTemplate:
    metadata:
      name: provider-aws-s3
---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-aws-iot
spec:
  serviceAccountTemplate:
    metadata:
      name: provider-aws-iot
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/upbound/provider-aws-s3:v2.6.1   # pin the newest v2 you can see
  runtimeConfigRef: { name: provider-aws-s3 }
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-iot
spec:
  package: xpkg.upbound.io/upbound/provider-aws-iot:v2.6.1
  runtimeConfigRef: { name: provider-aws-iot }

The ProviderConfig tells both providers to use Pod Identity. In the v2 providers, the namespaced aws.m.upbound.io API is used; put the managed resources in the same namespace as this config. crossplane/providerconfig.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: aws-resources
---
apiVersion: aws.m.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
  namespace: aws-resources
spec:
  credentials:
    source: PodIdentity

Commit, push and watch:

git add -A && git commit -m "crossplane and providers" && git push
kubectl -n argocd get applications
kubectl get providers
kubectl -n crossplane-system get pods

Expected: both providers INSTALLED True and HEALTHY True, and the provider pods Running with the service accounts provider-aws-s3 and provider-aws-iot. Check the identity reached the pod:

kubectl -n crossplane-system exec deploy/$(kubectl -n crossplane-system get deploy -o name | grep provider-aws-s3 | cut -d/ -f2) -- env | grep AWS_CONTAINER

Expected: AWS_CONTAINER_CREDENTIALS_FULL_URI is set, which is how Pod Identity delivers credentials.

Checkpoint: both providers Healthy.

7. Evidence and backup buckets with Object Lock

Object Lock needs versioning, and a bucket must be created with Object Lock enabled. Compliance mode means no one, including the root user, can delete a locked version until its retention ends. Practise on the drill bucket first and keep locked buckets out of any account you intend to delete early.

crossplane/buckets.yaml:

apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket
metadata:
  name: coldchain-drill-abc123          # replace abc123 with your SUFFIX
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    objectLockEnabled: true
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: BucketVersioning
metadata:
  name: coldchain-drill-abc123
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    bucketRef: { name: coldchain-drill-abc123 }
    versioningConfiguration:
      status: Enabled
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: BucketObjectLockConfiguration
metadata:
  name: coldchain-drill-abc123
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    bucketRef: { name: coldchain-drill-abc123 }
    rule:
      defaultRetention:
        mode: COMPLIANCE
        days: 1
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket
metadata:
  name: coldchain-evidence-abc123
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    objectLockEnabled: true
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: BucketVersioning
metadata:
  name: coldchain-evidence-abc123
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    bucketRef: { name: coldchain-evidence-abc123 }
    versioningConfiguration:
      status: Enabled
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: BucketObjectLockConfiguration
metadata:
  name: coldchain-evidence-abc123
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    bucketRef: { name: coldchain-evidence-abc123 }
    rule:
      defaultRetention:
        mode: GOVERNANCE       # lab. Production evidence uses COMPLIANCE and your own retention period.
        days: 30
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket
metadata:
  name: coldchain-backup-abc123
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: s3.aws.m.upbound.io/v1beta1
kind: BucketVersioning
metadata:
  name: coldchain-backup-abc123
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    bucketRef: { name: coldchain-backup-abc123 }
    versioningConfiguration:
      status: Enabled
  providerConfigRef: { kind: ProviderConfig, name: default }

Encryption with the platform key and a public access block are added the same way, with BucketServerSideEncryptionConfiguration and BucketPublicAccessBlock; confirm the field names with kubectl explain below. Commit, push, then:

kubectl -n aws-resources get buckets.s3.aws.m.upbound.io
kubectl explain bucketobjectlockconfigurations.s3.aws.m.upbound.io.spec.forProvider.rule
aws s3api get-object-lock-configuration --bucket coldchain-drill-abc123

Expected: READY True and SYNCED True; the CLI shows "ObjectLockEnabled": "Enabled" with the default retention. Now try the lock on the drill bucket:

echo test > e.txt
aws s3api put-object --bucket coldchain-drill-abc123 --key e.txt --body e.txt
VID=$(aws s3api list-object-versions --bucket coldchain-drill-abc123 --query 'Versions[0].VersionId' --output text)
aws s3api delete-object --bucket coldchain-drill-abc123 --key e.txt --version-id $VID

Expected: the last command fails with AccessDenied because the version is locked. A delete without a version ID only adds a delete marker and the locked version stays.

Checkpoint: the drill version cannot be deleted; the evidence and backup buckets show versioning enabled.

8. IoT Core things, policies and certificates

Each device gets one thing, one certificate and one policy limited to its own topic prefix. crossplane/iot-reefer-001.yaml:

apiVersion: iot.aws.m.upbound.io/v1beta1
kind: Thing
metadata:
  name: reefer-001
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: iot.aws.m.upbound.io/v1beta1
kind: Policy
metadata:
  name: reefer-001
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    policy: |
      {
        "Version": "2012-10-17",
        "Statement": [
          { "Effect": "Allow", "Action": "iot:Connect",
            "Resource": "arn:aws:iot:ap-south-1:ACCOUNT_ID:client/reefer-001" },
          { "Effect": "Allow", "Action": "iot:Publish",
            "Resource": "arn:aws:iot:ap-south-1:ACCOUNT_ID:topic/fleet/reefer-001/telemetry" }
        ]
      }
  providerConfigRef: { kind: ProviderConfig, name: default }
---
apiVersion: iot.aws.m.upbound.io/v1beta1
kind: Certificate
metadata:
  name: reefer-001
  namespace: aws-resources
spec:
  forProvider:
    region: ap-south-1
    active: true
  writeConnectionSecretToRef:
    name: reefer-001-cert
  providerConfigRef: { kind: ProviderConfig, name: default }

Replace ACCOUNT_ID. Two attachment objects bind the pieces (PolicyAttachment policy to certificate, ThingPrincipalAttachment thing to certificate); write them with kubectl explain for the exact selector fields on your provider version.

kubectl -n aws-resources get things.iot.aws.m.upbound.io,policies.iot.aws.m.upbound.io,certificates.iot.aws.m.upbound.io
aws iot list-things --query 'things[].thingName'
aws iot describe-endpoint --endpoint-type iot:Data-ATS

Expected: reefer-001 listed, and an endpoint of the form xxxx-ats.iot.ap-south-1.amazonaws.com. Keep the endpoint: Part 2 uses it.

With AWS-generated keys, the certificate’s private key lands in the connection secret. That secret is a credential: Part 4 moves such secrets under SOPS, and Part 2 copies the certificate to the depot. A certificate signing request generated on the device keeps the private key off the cluster; the field for that on this provider was not confirmed here, so check kubectl explain certificates.iot.aws.m.upbound.io.spec.forProvider.

Checkpoint: thing, policy and certificate exist and are attached.

9. The depot: k0s on a VM

The depot is a small cluster with no AWS credentials. It only dials out. On the Ubuntu VM:

curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
sudo k0s install controller --single
sudo k0s start
sleep 30
sudo k0s status
sudo k0s kubectl get nodes

Expected: k0s status shows Role: controller and Workloads: true, and the node is Ready. To use plain kubectl from the VM:

mkdir -p ~/.kube
sudo k0s kubeconfig admin > ~/.kube/config
kubectl get pods -A

Limit the VM’s outbound traffic to what the depot needs, and nothing inbound:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh        # from your IP only, in a real site
sudo ufw enable

Check reachability to the two endpoints it will use in Part 2:

curl -sk --max-time 5 -o /dev/null -w "%{http_code}\n" https://$(aws eks describe-cluster --name coldchain-hub --query cluster.endpoint --output text | sed 's#https://##')/healthz
nc -vz <your-iot-endpoint> 8883

Run the first from a machine with the AWS CLI; the second from the VM. Expected: an HTTP code (401 or 403 is fine: the API answered) and a successful connect on 8883. Checkpoint: k0s status healthy, the node Ready, 8883 reachable.

10. Troubleshooting

Symptom Likely cause Fix
tofu init cannot reach the bucket Wrong region or name in -backend-config Re-run with the values from section 2
Backend lock error A previous run was killed tofu force-unlock <ID> after confirming nobody else is running
Nodes not Ready VPC CNI or Pod Identity add-on not up before nodes Keep before_compute = true; check kubectl -n kube-system get pods
kubectl times out Your IP changed Update my_ip_cidr and apply
MSK association rejected Secret uses the default key, wrong name prefix, or has no value Name must start with AmazonMSK_; use the customer key; put a value first
Provider pod AccessDenied Association service account name differs Names must be exactly provider-aws-s3 and provider-aws-iot
Provider never Healthy Package version not found Pin a version listed on the Upbound Marketplace
Bucket SYNCED False Name taken, or a field name differs kubectl describe; confirm with kubectl explain
get-object-lock-configuration fails Bucket created without Object Lock Object Lock must be set at creation; recreate the drill bucket
k0s node NotReady Not enough time or memory Wait a minute; check sudo k0s status and journalctl -u k0scontroller

Cost note

Read on 18 November 2026 (US East list prices; Mumbai, EC2, IoT and S3 prices were not read): EKS control plane $0.10 per hour, Amazon MSK kafka.m7g.large about $0.204 per broker hour (three brokers about $0.61 per hour, roughly $447 a month), MSK storage $0.10 per GB-month. Add three m6i.large nodes and one NAT gateway. Treat MSK and the nodes as the cost to watch: build, check, and destroy in the same sitting if you can.

Cleanup

Order matters: Crossplane-managed resources first, then the OpenTofu stack, with MSK going last of the paid items.

# 1. Remove the Crossplane objects (Argo CD would put them back, so delete the applications first).
kubectl -n argocd delete application root crossplane-providers
kubectl -n aws-resources delete things.iot.aws.m.upbound.io,policies.iot.aws.m.upbound.io,certificates.iot.aws.m.upbound.io --all
# Buckets with locked versions cannot be emptied until retention ends. Governance mode
# can be bypassed with s3:BypassGovernanceRetention; compliance mode cannot.

# 2. Destroy the foundation.
cd infra/foundation
tofu destroy
# The KMS key is scheduled for deletion, not deleted at once (7 days here).

# 3. The state bucket, last.
aws s3 rb s3://$STATE_BUCKET --force   # empty versions first if the bucket has them

# 4. The depot VM.
sudo k0s stop && sudo k0s reset

Design notes and version checks

  • Versions. Crossplane 2.4.3 is pinned; confirm the newest stable. The provider tag v2.6.1 was the latest I saw for the family; confirm per provider (provider-aws-s3, provider-aws-iot). The EKS Kubernetes version and the MSK Kafka version are chosen with the commands in section 4.1, not copied from this post.
  • Provider API. The v2 providers use namespaced kinds in aws.m.upbound.io groups. Kind and field names in sections 7 and 8 follow that API; confirm with kubectl explain on your version, especially the attachment kinds and the certificate CSR field.
  • Authentication. source: PodIdentity and Pod Identity associations are the recommended route over static keys; IRSA is the alternative if the cluster has an OIDC provider and you prefer annotations.
  • MSK. Standard brokers with SASL/SCRAM. SCRAM needs a customer-managed key and an AmazonMSK_ secret name. Whether Express brokers accept SCRAM was not confirmed, so this build uses Standard. MSK Serverless is not supported by the IoT Core Kafka action.
  • State locking. use_lockfile = true needs OpenTofu 1.10 or newer.
  • Object Lock. Needs versioning and cannot be turned off. The evidence bucket is in governance mode here so the lab can be cleaned up; choose compliance mode and a retention period from your quality policy for real records.
  • Not wired in this part. The Kafka rule and VPC destination for IoT Core are in Part 2. Whether the Crossplane provider covers them was not confirmed, so Part 2 plans to create them in OpenTofu.
  • Credentials in state. The MSK password is never in code or state. The IoT private key is in a cluster Secret until Part 4.

Setup checklist

  • 1. Account and tools: done when aws sts get-caller-identity shows the intended account and the CLIs print versions.
  • 2. State bucket: done when versioning is Enabled and the public access block is on.
  • 3. Network and EKS: done when kubectl get nodes shows three Ready nodes.
  • 4. KMS, secrets, MSK: done when the cluster state is ACTIVE and list-scram-secrets shows the secret.
  • 5. Argo CD: done when root is Synced and Healthy.
  • 6. Crossplane: done when both providers are Healthy.
  • 7. Buckets: done when the drill version cannot be deleted and evidence and backup show versioning enabled.
  • 8. IoT things: done when aws iot list-things shows reefer-001 and the endpoint is noted.
  • 9. Depot: done when k0s status is healthy and port 8883 is reachable.

Tools used

OpenTofu, Crossplane, Argo CD, k0s, Apache Kafka, Kubernetes, external-secrets (used from Part 2).

What comes next

Part 2 sends vehicle readings to the hub: the simulator, EMQX on the depot, IoT Core to Kafka, Vector and Open Cluster Management to bring the depot under the hub.

Spotted something wrong or out of date?Suggest a correction. I review every suggestion before changing the page.