mrvitej.in

Archive · DevOps

Terraform: Remote State, Imports, Testing and a Safe CI Pipeline

S3 remote state with native locking, bootstrapping the state bucket, import blocks, terraform test, static analysis and a GitHub Actions pipeline with OIDC and saved plans.

6 min read

  • Terraform 1.14
  • S3 backend use_lockfile
  • terraform test
  • tflint
  • Trivy
  • GitHub Actions

The previous post covered writing Terraform. This one covers running it safely as a team: where state lives, how changes are tested, and how a pipeline applies exactly what was reviewed.

Why state needs care

The state file maps each resource in your code to a real resource ID. It is also where Terraform keeps attribute values, which can include passwords and keys. That gives three rules:

  1. Store it remotely, so the whole team and the pipeline use the same one.
  2. Lock it, so two applies can never run at once and corrupt it.
  3. Protect it like a secret: encryption, versioning, tight access.

Architecture

flowchart LR
  DEV[Engineer opens PR] --> CI[CI pipeline]
  CI --> PLAN[fmt, validate, lint,<br/>scan, test, plan]
  PLAN --> REV[Plan posted to PR<br/>human review]
  REV --> MERGE[Merge to main]
  MERGE --> APPLY[Apply saved plan<br/>after approval]
  APPLY --> AWS[AWS account]
  CI <--> STATE[S3 state bucket<br/>versioned, encrypted]

S3 backend with native locking

Since Terraform 1.10 the S3 backend can lock using a lock file stored next to the state, with S3 conditional writes. The DynamoDB lock table that older setups needed is no longer required.

# envs/prod/backend.tf
terraform {
  backend "s3" {
    bucket       = "acme-tfstate-111122223333"
    key          = "shop/prod/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}

If you are migrating an existing backend that used dynamodb_table, you can set both during the transition, then remove the DynamoDB setting once everyone runs 1.10 or later.

Bootstrapping the state bucket

The bucket must exist before any backend can use it. Create it once, with its own tiny Terraform configuration that keeps local state (or with the CLI):

# bootstrap/main.tf
resource "aws_s3_bucket" "state" {
  bucket = "acme-tfstate-111122223333"
  lifecycle { prevent_destroy = true }
}

resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" }
  }
}

resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Versioning is your undo button: if state is ever corrupted, restore the previous object version.

One state per environment and layer

Split state so the blast radius of a mistake stays small:

State key Contains Changes
network/prod VPC, subnets, Transit Gateway Rarely
data/prod RDS, ElastiCache, S3 buckets Occasionally
app/prod ECS or EKS services, IAM roles Often

Read another layer’s outputs with terraform_remote_state, or better, look the resource up with a data source by tag, which avoids tight coupling between states.

Importing existing resources

Most teams start with things created by hand. Bring them under management with an import block, which is planned and reviewed like any other change:

import {
  to = aws_s3_bucket.legacy_assets
  id = "acme-legacy-assets"
}

Let Terraform write the starting configuration for you:

terraform plan -generate-config-out=generated.tf

Review generated.tf, tidy it into your normal files, and run plan again until it shows no changes. That confirms the code matches reality. Then delete the import block.

Testing

Static checks

terraform fmt -check -recursive
terraform validate
tflint --recursive                     # provider-aware lint, e.g. invalid instance types
trivy config --severity HIGH,CRITICAL . # misconfigurations such as public buckets

terraform test

terraform test runs .tftest.hcl files with assertions. A plan-mode test is fast and creates nothing:

# modules/network/tests/network.tftest.hcl
variables {
  name = "test"
  cidr = "10.99.0.0/16"
  azs  = ["ap-south-1a", "ap-south-1b", "ap-south-1c"]
}

run "creates_three_private_subnets" {
  command = plan

  assert {
    condition     = length(aws_subnet.private) == 3
    error_message = "Expected one private subnet per AZ."
  }
}

run "private_subnets_do_not_assign_public_ips" {
  command = plan

  assert {
    condition     = alltrue([for s in aws_subnet.private : s.map_public_ip_on_launch == false])
    error_message = "Private subnets must not map public IPs."
  }
}
cd modules/network && terraform init && terraform test

An apply-mode test (command = apply) creates real resources in a sandbox account and destroys them afterwards. Use it for modules where behaviour, not just the plan, matters.

The pipeline

Principles:

  • Plans run on every pull request and are posted for review.
  • Applies run only from main, only the saved plan, and only after approval.
  • AWS access uses OIDC to assume a role. No access keys are stored anywhere.
  • Plan and apply use different roles: read-only for plan, write for apply.
# .github/workflows/terraform.yml
name: terraform
on:
  pull_request:
    paths: ["infra/**"]
  push:
    branches: [main]
    paths: ["infra/**"]

permissions:
  id-token: write
  contents: read
  pull-requests: write

env:
  TF_IN_AUTOMATION: "true"
  WORKDIR: infra/envs/prod

jobs:
  plan:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: infra/envs/prod
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: "1.14.x" }
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-plan
          aws-region: ap-south-1
      - run: terraform fmt -check -recursive
      - run: terraform init -input=false
      - run: terraform validate
      - run: terraform plan -input=false -var-file=prod.tfvars -out=tfplan
      - run: terraform show -no-color tfplan > plan.txt
      - uses: actions/upload-artifact@v4
        with: { name: tfplan, path: "${{ env.WORKDIR }}/tfplan" }
      - name: Comment plan on PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const plan = fs.readFileSync('${{ env.WORKDIR }}/plan.txt', 'utf8').slice(0, 60000);
            github.rest.issues.createComment({
              ...context.repo, issue_number: context.issue.number,
              body: '```\n' + plan + '\n```'
            });

  apply:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    needs: plan
    runs-on: ubuntu-latest
    environment: production        # requires a reviewer's approval
    defaults:
      run:
        working-directory: infra/envs/prod
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: "1.14.x" }
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/tf-apply
          aws-region: ap-south-1
      - uses: actions/download-artifact@v4
        with: { name: tfplan, path: "${{ env.WORKDIR }}" }
      - run: terraform init -input=false
      - run: terraform apply -input=false tfplan

The IAM role’s trust policy restricts who may assume it:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub": "repo:acme/infra:environment:production"
      }
    }
  }]
}

Applying the saved plan matters: if anything changed between review and apply, Terraform refuses the stale plan instead of doing something nobody reviewed.

Detecting drift

Someone will change something in the console. A scheduled job catches it:

on:
  schedule: [{ cron: "30 3 * * 1-5" }]   # weekdays 09:00 IST
# ...same setup as the plan job, then:
      - run: terraform plan -input=false -detailed-exitcode -var-file=prod.tfvars

-detailed-exitcode returns 2 when there are changes, which fails the job and alerts the team. Fix drift by updating the code or reverting the manual change, never by ignoring it.

Day-to-day rules

Situation Do
Stuck lock after a cancelled run Confirm nothing is running, then terraform force-unlock <ID>
Need to remove a resource from Terraform without deleting it removed block with lifecycle { destroy = false }
Rename or move a resource moved block
Need to look at state terraform state list and terraform state show <addr>; avoid editing state by hand
Provider upgrade Separate pull request, read the changelog, plan every environment

Summary

  • S3 backend with use_lockfile, versioning and encryption; one state per environment and layer.
  • import blocks plus -generate-config-out to adopt existing resources.
  • fmt, validate, tflint, trivy config and terraform test on every pull request.
  • Plan on PR, apply the saved plan on main after approval, OIDC roles split by plan and apply.
  • A scheduled drift check.