How to use this guide
Each topic has the same shape: the question as an interviewer asks it, a diagram where one helps, a model answer you can say in about a minute, a go deeper section for follow-ups, and the common mistake to avoid. Study one chapter at a time, then close the page and answer the questions out loud.
The answer formula for any technical question
- Restate the problem in one sentence and ask one clarifying question if it matters.
- Give the short answer first (the model answer).
- Name the trade-off (“this costs X, so I would choose it when Y”).
- Give a real example or a command you have actually used.
- Say how you would verify it worked and what you would monitor.
Last-week plan: days 1 to 2 containers and Kubernetes, day 3 Terraform and CI/CD, day 4 AWS and databases, day 5 observability and networking, days 6 to 7 system design (practise two designs end to end, out loud, with the six steps). Prepare four behavioural stories from your own work.
Commands and numbers here are study aids, so check the official documentation for the exact versions you use.
1. Docker and containers
1.1 What really happens when you run docker run nginx
Question: “Walk me through what happens when I type docker run nginx.”
Why they ask: it shows whether you understand a container as a Linux process, not a tiny VM.
flowchart TD
A["docker CLI"] --> B["dockerd (Docker daemon)"]
B --> C["Image missing? Pull from registry"]
C --> D["containerd + runc create the container"]
D --> E["Read-only image layers + thin writable layer (overlayfs)"]
E --> F["Namespaces and cgroups set up"]
F --> G["nginx starts as PID 1 inside the container"]
G --> H["Network attached (bridge, veth, NAT)"]
Model answer (60 seconds): The CLI sends the request to the Docker daemon. If the image is not local, the daemon pulls it layer by layer from the registry. containerd and runc then create the container: Linux namespaces give it its own view of processes, network, mounts, hostname and users, and cgroups limit CPU and memory. The image layers are mounted read-only with a thin writable layer on top (overlay filesystem). The process in the image’s entrypoint (nginx) starts as PID 1 inside that isolated environment, and a virtual network interface is attached to the default bridge. A container is just a normal Linux process with isolation, sharing the host kernel.
Go deeper
- Namespaces control what a process can see (PID, mount, network, UTS, IPC, user). Cgroups control what it can use (CPU, memory, I/O, pids).
- Many containers from one image share the same read-only layers; only the writable layer is per container, and it is deleted with the container.
- PID 1 is special: it does not get default signal handlers and must reap zombie children. Use
--init(tini) or an app that handles SIGTERM. - Docker (dockerd) is one runtime front end. Kubernetes talks to containerd or CRI-O directly; the runc step is the same.
Follow-ups
- Container vs VM? A VM virtualises hardware and runs its own kernel; a container shares the host kernel, so it starts in milliseconds and has weaker isolation.
- Where does container data live? Writable layer under
/var/lib/docker; use volumes for anything that must survive. - How do you see the isolation?
docker top,docker inspect,ls -l /proc/<pid>/ns,lsns.
Common mistake: saying “a container is a lightweight VM”.
1.2 Why is the image 2 GB, and how do you shrink it?
Question: “Your image is 2 GB. What is the impact and how do you fix it?”
flowchart TD
A["Problem: 2 GB image"] --> B["Slow pulls on every node and scale-out<br/>Slow rollbacks and recovery<br/>Registry cost, bigger attack surface"]
B --> C["Fix: small base, multi-stage build,<br/>fewer layers, .dockerignore"]
Model answer: Large images slow every deploy and autoscaling event because each new node pulls the whole thing, they cost more to store and transfer, and they carry tools an attacker could use. I shrink them with a minimal base (alpine, slim, distroless), a multi-stage build so compilers and build tools never reach the final image, combining RUN steps and cleaning package caches in the same layer, a .dockerignore so .git, node_modules and test data are not copied, and pinned minimal dependencies. I verify with docker history and dive.
Go deeper
- Layers are additive: deleting a file in a later layer does not shrink the image. Clean up in the same
RUN. - Order Dockerfile steps from least to most frequently changing (copy dependency manifest, install, then copy source) to maximise build cache.
- Alpine uses musl libc, which can break some binaries and slow some workloads;
-slimDebian or distroless is the safer production choice. - Multi-stage example:
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
FROM node:20-alpine
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
USER node
CMD ["node", "dist/server.js"]
Follow-ups
- Image vs layer? An image is the stack of read-only layers; each Dockerfile instruction that changes the filesystem makes a layer.
- Why pin versions? Reproducible builds and no surprise breakage from
latest. - Why is a node:alpine image smaller than node? The base OS is about 5 MB versus hundreds of MB for Debian.
Common mistake: adding rm -rf in a later RUN and expecting the image to shrink.
1.3 CMD vs ENTRYPOINT, exec vs shell form
Question: “What is the difference between CMD and ENTRYPOINT?”
| ENTRYPOINT | CMD | |
|---|---|---|
| Purpose | The executable that always runs | Default arguments (or default command) |
Override at docker run |
Only with --entrypoint |
Anything after the image name replaces it |
| Typical use | Make the image behave like a tool | Provide a default that users can change |
Model answer: ENTRYPOINT defines what the container is and is hard to override; CMD supplies default arguments that are easy to override. Together, ENTRYPOINT ["python","app.py"] with CMD ["--port","8000"] runs python app.py --port 8000, and docker run img --port 9000 replaces only the CMD part.
Go deeper
- Exec form (
["nginx","-g","daemon off;"]) runs the program directly as PID 1, so it receives SIGTERM fromdocker stop. Shell form (nginx -g ...) wraps it in/bin/sh -c, the shell becomes PID 1 and often swallows signals, sodocker stopwaits 10 seconds and then SIGKILLs. - In shell-form ENTRYPOINT, CMD arguments are ignored.
- Use a wrapper script that ends with
exec "$@"when you need setup (migrations, config templating) before starting the app, and still want the app to be PID 1.
Follow-ups
- Why does my container take 10 seconds to stop? Shell form or an app ignoring SIGTERM.
- How do you run a shell in an image with an entrypoint?
docker run --entrypoint /bin/sh -it image.
Common mistake: using shell form and then wondering why graceful shutdown never runs.
1.4 Exit code 137 and other container deaths
Question: “A container keeps dying with exit code 137. What do you do?”
flowchart TD
A["Exit 137 = 128 + 9 (SIGKILL)"] --> B{"OOMKilled true?"}
B -->|"yes"| C["Raise limit or fix leak"]
B -->|"no"| D["Find who sent SIGKILL<br/>(stop timeout, orchestrator, manual)"]
Model answer: 137 means the process was killed with SIGKILL (128 + 9). First I check docker inspect --format '{{.State.OOMKilled}}' <c> and dmesg | grep -i oom. If it was OOM-killed, the cgroup memory limit was hit: I look at real usage with docker stats, then either raise the limit, fix a leak, or tune the runtime (JVM heap, Node --max-old-space-size). If not OOM, I check who sent the kill: docker stop timing out because the app ignores SIGTERM, an orchestrator health-check kill, or a manual kill.
Go deeper
- Other codes: 0 clean exit, 1 generic error, 126 not executable, 127 command not found, 139 segfault (128+11), 143 SIGTERM (128+15).
- In Kubernetes the same shows as
OOMKilledinkubectl describe pod(last state), andCrashLoopBackOffwhen it repeats. - JVM before Java 10 did not respect cgroup limits; set heap to roughly 60 to 75 percent of the container limit.
Follow-ups
- Limit vs request in Kubernetes? Request is for scheduling; limit is the hard cap. Memory over limit is killed, CPU over limit is throttled.
Common mistake: raising the limit without checking for a leak.
1.5 Bind mounts vs named volumes, logs and persistence
Question: “Bind mount or volume for a production database? What happens to container logs?”
| Bind mount | Named volume | |
|---|---|---|
| Path | You choose a host path | Docker manages it |
| Created | You create it | Auto-created |
| Best for | Local development, hot reload | Production data |
| Portability | Tied to host layout and permissions | Portable, works with volume drivers |
Model answer: Bind mounts are good for development because you edit on the host and the container sees it immediately. For production data use named volumes: Docker owns the location and permissions, they survive container removal, and volume drivers can back them with NFS, EBS or other storage for multi-host use. A local volume only exists on one host, so for multi-host persistence you need a network-backed driver or, in Kubernetes, PVCs.
Go deeper
- Container logs (
docker logs --tail 200 -f <c>) persist across stop and restart, but are deleted with the container. Ship logs off the host (Fluent Bit, Loki, CloudWatch) with a log driver. - Use log rotation (
--log-opt max-size=10m --log-opt max-file=3) or the disk fills; this is a classic cause of full disks. - Use
tmpfsmounts for secrets or scratch data that must not touch disk.
Follow-ups
- Backup a volume? Run a temp container mounting the volume and tar it, or snapshot the underlying disk.
Common mistake: keeping database files in the writable layer.
2. Kubernetes
2.1 What Kubernetes is and how it works
Question: “Explain Kubernetes architecture and what the scheduler and kubelet do.”
flowchart TD
U["kubectl / CI"] --> API["kube-apiserver"]
API --> E["etcd: cluster state"]
API --> CP["scheduler and controller-manager"]
API --> K["kubelet on each node"]
K --> P["container runtime runs the Pods"]
Model answer: Kubernetes is a declarative control loop. You tell the API server the desired state; controllers constantly compare it with actual state and fix the difference. The control plane has the API server (the only component that talks to etcd), etcd (the source of truth), the scheduler (assigns unscheduled pods to nodes), and the controller manager (runs controllers such as Deployment, ReplicaSet and Node). Each node runs a kubelet, which watches for pods assigned to it and tells the container runtime to run them, plus kube-proxy or an eBPF dataplane for Services.
Go deeper
- The scheduler has two phases. Filter: drop nodes that cannot run the pod (resources, taints, node selectors, affinity, volume zone). Score: rank the remaining nodes (spread, resource balance, affinity weight) and pick the best.
- A pod is the smallest deployable unit: one or more containers sharing network and volumes.
- Deployment manages ReplicaSets, which manage Pods; a bare ReplicationController only keeps a replica count and does not do rolling updates or autoscaling by itself.
Follow-ups
- What if etcd is lost? No cluster state; restore from a snapshot. Back up etcd regularly.
- What if the control plane goes down? Running pods keep running; you just cannot change anything.
Common mistake: claiming the scheduler starts the pod (it only picks the node).
2.2 A rollout is stuck: debugging
Question: “You deployed a new version and the rollout is stuck. What do you do?”
flowchart TD
A["kubectl rollout status"] --> B["kubectl get pods (state?)"]
B --> C{"Pending / ImagePullBackOff / CrashLoopBackOff / not Ready"}
C --> D["kubectl describe pod (events)"]
D --> E["kubectl logs --previous"]
E --> F["Fix, or kubectl rollout undo"]
Model answer: I run kubectl rollout status deploy/x, then kubectl get pods to see the state of the new pods. Pending means scheduling (resources, taints, PVC); ImagePullBackOff means a bad tag or registry credentials; CrashLoopBackOff means the app fails on start (check kubectl logs --previous); Running but not Ready means a failing readiness probe or a dependency. kubectl describe pod events usually name the cause. If production is hurting I roll back immediately with kubectl rollout undo deploy/x, then fix forward.
Go deeper
- Rolling update is controlled by
maxSurgeandmaxUnavailable; the rollout stalls if new pods never become Ready, which is the safety behaviour you want. - Set
progressDeadlineSecondsso the Deployment reports failure instead of waiting forever. - Good probes: readiness gates traffic, liveness restarts a hung process, startup protects slow starters.
Follow-ups
- Check history?
kubectl rollout history deploy/x,--to-revision=N. - Why Pending?
describeshowsInsufficient cpu, untolerated taints, or an unbound PVC.
Common mistake: deleting pods instead of reading the events.
2.3 Upgrading a cluster with no downtime
Question: “How do you upgrade a Kubernetes cluster without downtime?”
Model answer: Read the release notes and deprecated APIs, upgrade a staging cluster first, back up etcd, then upgrade the control plane before the nodes (a kubelet must never be newer than the API server, and minor versions are upgraded one at a time). For nodes I prefer replacement: create new nodes (or a new node group) on the new version, then cordon and drain old nodes one at a time, relying on replicas spread across nodes and PodDisruptionBudgets so the app keeps serving, and finally remove the old nodes. On managed services such as EKS this is a managed control plane update plus rolling node group update.
Go deeper
kubectl cordonstops scheduling,kubectl drain --ignore-daemonsets --delete-emptydir-dataevicts pods respecting PDBs.- PDB example:
minAvailable: 2stops a drain from taking out too many replicas at once. - Check that add-ons (CNI, CoreDNS, ingress controller, CSI) support the target version.
- Run
kubentorplutoto find removed APIs before upgrading.
Follow-ups
- What breaks upgrades? Single-replica apps without PDBs, local storage on nodes, removed API versions.
- In-place or replace? Replace is easier to roll back and cleaner for immutable infrastructure.
Common mistake: draining all nodes at once.
2.4 DNS inside the cluster
Question: “A pod cannot resolve a service name. How do you debug it?”
flowchart TD
A["Pod asks name"] --> B["/etc/resolv.conf: search domains, ndots:5"]
B --> C["CoreDNS Service (kube-dns)"]
C --> D["cluster.local: Kubernetes records<br/>external names: upstream DNS"]
Model answer: I exec into the pod and check /etc/resolv.conf and nslookup kubernetes.default. Then I check CoreDNS pods and logs in kube-system, the kube-dns Service and its endpoints, and whether a NetworkPolicy blocks UDP and TCP 53 from the pod to CoreDNS. A service name resolves as svc.namespace.svc.cluster.local; short names only work through the search path.
Go deeper
ndots:5means any name with fewer than five dots is tried with each search suffix first, so external lookups likeapi.example.comcause several wasted queries; use a trailing dot or lowerndotsfor latency-sensitive pods.dnsPolicy:ClusterFirst(default),Default(node’s resolv.conf),None(you supplydnsConfig).- A default-deny egress NetworkPolicy must explicitly allow DNS.
- NodeLocal DNSCache reduces conntrack pressure and latency in big clusters.
Follow-ups
- Headless service? DNS returns pod IPs directly instead of one virtual IP.
- Intermittent 5-second DNS delays? Classic conntrack race with parallel A/AAAA queries; NodeLocal DNSCache helps.
Common mistake: blaming CoreDNS before checking NetworkPolicy.
2.5 Managed or self-managed Kubernetes
Question: “EKS or run your own cluster?”
Model answer: For most teams use managed (EKS, GKE, AKS): the provider runs and patches the control plane and etcd with an SLA, so you spend time on workloads. Choose self-managed when you must run on-prem or air-gapped, need unusual control-plane flags, or have cost or compliance reasons. Even on EKS you still own node upgrades, add-ons, networking, IAM mapping, observability and cost. For on-prem or edge, lightweight distributions help: K3s (single binary), RKE2 (hardened, CIS), kubeadm (flexible), Talos (immutable OS), Kops (builds clusters on AWS).
Go deeper
- Managed trade-offs: less control, version lag, provider limits; self-managed: you run etcd backups, certificate rotation and upgrades.
- Minikube or kind are for local testing only, not production.
Follow-ups
- Staging vs prod? Same tooling and versions in both so upgrades are rehearsed.
Common mistake: assuming “managed” means no operational work.
2.6 Persistent storage: PV, PVC, StorageClass, access modes
Question: “Explain PV, PVC and access modes. Why did my pods fail to mount a volume?”
Model answer: A PVC is a pod’s request for storage (size, access mode, class). A PV is the actual storage resource. A StorageClass lets the cluster provision PVs dynamically when a PVC appears. Access modes: RWO (read-write by one node, typical for EBS), RWX (read-write by many nodes, NFS/EFS/CephFS), ROX (read-only by many), and RWOP (one pod). If several pods on different nodes try to mount an RWO volume, the extras fail; fix by using an RWX-capable storage class, or giving each replica its own volume (StatefulSet with volumeClaimTemplates).
Go deeper
- EBS volumes are zonal: a pod can only mount a volume in its own AZ; use
volumeBindingMode: WaitForFirstConsumerso the volume is created where the pod lands. - Reclaim policy:
Deleteremoves the disk with the PVC,Retainkeeps it. - Resize with
allowVolumeExpansion: true.
Follow-ups
- Analogy? PVC is the ticket, PV is the seat you get.
Common mistake: running a multi-replica Deployment on a single RWO PVC.
2.7 Scheduling control: taints, tolerations, affinity, eviction
Question: “How do you dedicate nodes to a workload? What happens when a node runs out of disk or memory?”
Model answer: A taint on a node repels pods; only pods with a matching toleration can schedule there (kubectl taint nodes node1 dedicated=gpu:NoSchedule). Tolerations only allow; to attract pods to those nodes you also use nodeSelector or node affinity. Pod anti-affinity or topology spread constraints spread replicas across nodes and zones. When a node is under resource pressure the kubelet evicts pods to protect itself: hard thresholds evict immediately (for example memory.available<100Mi, nodefs.available<10%, imagefs.available<15%), soft thresholds evict only after a grace period. Evicted pods are rescheduled on other nodes, lowest-priority and BestEffort pods first.
Go deeper
- Taint effects:
NoSchedule,PreferNoSchedule,NoExecute(also evicts running pods). - QoS classes decide eviction order: BestEffort, then Burstable, then Guaranteed.
- Sidecar pattern (a helper container in the same pod, for example a log shipper) shares the pod’s volumes and lifecycle.
Follow-ups
- DaemonSet on tainted nodes? Add tolerations.
Common mistake: adding a toleration and expecting pods to be pulled to that node.
2.8 Helm
Question: “What is Helm and why use it?”
Model answer: Helm is the package manager for Kubernetes. A chart bundles templated manifests plus default values.yaml; installing it creates a versioned release you can upgrade and roll back (helm upgrade --install, helm rollback). It removes copy-paste YAML across environments: one chart, different values files per environment. Charts live in repositories (OCI registries, Artifact Hub, ChartMuseum, S3, GCS).
Go deeper
- Commands:
helm repo add bitnami https://charts.bitnami.com/bitnami,helm template(render locally),helm diff upgrade(plugin),helm history. - Pair with GitOps: keep values in Git, let Argo CD or Flux apply them.
- Helm 3 has no Tiller; release state is stored as Secrets in the namespace.
Follow-ups
- Helm vs Kustomize? Helm templates and packages; Kustomize patches plain YAML with overlays.
Common mistake: putting secrets in values.yaml committed to Git.
2.9 Privileged pod with hostPath
Question: “A pod runs privileged and mounts a hostPath. What is the risk and what would you do?”
Model answer: A privileged container has almost all host capabilities, and a hostPath mount exposes host files; together they allow container escape, reading other pods’ secrets, tampering with the kubelet or the container runtime socket. I would remove privileged, drop capabilities, use readOnlyRootFilesystem, replace hostPath with the right volume type, and enforce this cluster-wide with Pod Security Admission (restricted profile) or a policy engine (Kyverno, Gatekeeper). Also isolate with RBAC, NetworkPolicy and runtime sandboxing (gVisor, Kata) when running untrusted code.
Go deeper
- Checks:
securityContext.runAsNonRoot,allowPrivilegeEscalation: false, seccompRuntimeDefault. - Legitimate uses (node agents) should be isolated to a dedicated namespace with explicit exceptions.
Common mistake: fixing one pod instead of adding an admission policy.
3. Infrastructure as code: Terraform, Packer, Ansible, Vault
3.1 How Terraform works
Question: “Explain how Terraform actually works.”
flowchart TD
A["Write .tf (desired state)"] --> B["terraform init (providers, backend)"]
B --> C["terraform plan: refresh + diff vs state"]
C --> D["terraform apply: provider API calls"]
D --> E["State updated (terraform.tfstate)"]
Model answer: Terraform is declarative. You describe resources in HCL; providers translate them into cloud API calls. init downloads providers and configures the backend; plan reads the state file, refreshes real resource attributes, builds a dependency graph and shows the diff between desired and current; apply executes the changes in dependency order and writes the result back to state. State is how Terraform maps your code to real resource IDs, so protecting it matters.
Go deeper
- Remote state (S3 + DynamoDB locking, or S3 native locking in newer versions; Terraform Cloud; GCS) gives team sharing, locking and encryption. Never commit state: it contains secrets in plain text.
terraform importbrings existing resources under management;movedblocks refactor without destroy.- Variables, locals, outputs,
for_eachovercountwhen items have identity, data sources to read existing resources, modules for reuse. - OpenTofu is the open-source fork with the same workflow.
Follow-ups
- Drift? Real infrastructure changed outside Terraform;
planshows it andapplyor-refresh-onlyreconciles. - Why
for_eachovercount? Removing an item from the middle of acountlist shifts indexes and recreates resources.
Common mistake: hand-editing state without a backup.
3.2 State locking, hung applies and validation
Question: “Two engineers run terraform apply together. What happens? What if an apply hangs?”
Model answer: With a backend that supports locking, the first apply takes a lock on the state, and the second waits or fails with a lock error, which prevents two writers corrupting state. Disabling it (-lock=false) is dangerous and only for recovery. If an apply hangs, first take a backup (terraform state pull > backup.tfstate), check the cloud console and provider logs for what is stuck, stop the process, and re-run apply; if a lock remains, terraform force-unlock <ID> only after confirming no run is active. To make hangs less costly, split big configurations into smaller state files (per environment or component).
Go deeper
- Input validation fails fast at plan time:
variable "ami_id" {
type = string
validation {
condition = can(regex("^ami-", var.ami_id))
error_message = "ami_id must start with ami-."
}
}
- Also use
preconditionandpostconditionblocks,tflint,terraform validate, and policy checks (OPA, Sentinel, Checkov) in CI. remote-execandlocal-execprovisioners are a last resort; prefer user data, cloud-init, Packer images or Ansible.
Follow-ups
- Sensitive values? Mark
sensitive = true, but remember they still sit in state. - Destroy protection?
lifecycle { prevent_destroy = true }.
Common mistake: a single giant state file for the whole company.
3.3 Secrets and golden images: Vault, Packer, Ansible
Question: “How do you keep secrets out of code, and how do you make sure every server has the required security packages?”
Model answer: Secrets come from a secrets manager at runtime, never from Git. Terraform can read from HashiCorp Vault or AWS Secrets Manager through a data source, but values read that way still end up in state, so protect the state backend, and prefer having the application fetch secrets itself (IAM role, Vault agent, External Secrets) and letting the database generate or rotate its own password where possible. For mandated packages, build a golden image with Packer (or EC2 Image Builder): one template installs the agents and patches, runs tests, and outputs a versioned AMI that every instance launches from, so the fleet is consistent and rebuilds replace patching in place.
Go deeper
- Ansible: inventory groups hosts (
[webservers], per-hostansible_user/ansible_port), playbooks are idempotent tasks, roles organise reuse.ignore_errors: yeslets a play continue after a failed task, use sparingly.ansible-vault create|encrypt|editencrypts variable files; pass--ask-vault-passor a vault-id. - Idempotent means running it twice gives the same result with no extra changes.
- Immutable infrastructure: replace instances instead of modifying them.
Follow-ups
- Packer vs custom AMI by hand? Packer is repeatable, versioned and reviewable.
- Rotate a leaked key? Revoke first, rotate, redeploy, audit usage, add scanning.
Common mistake: believing Vault stops secrets appearing in Terraform state.
4. CI/CD, Git and release strategy
4.1 Jenkins: pipelines, agents, notifications
Question: “How do you run builds in containers, scale Jenkins, and alert on failure?”
Model answer: I use declarative pipelines in a Jenkinsfile. A Docker agent gives each build a clean, reproducible environment:
pipeline {
agent { docker { image 'node:20-alpine' } }
stages {
stage('Test') { steps { sh 'npm ci && npm test' } }
}
post {
failure { slackSend color: 'danger', message: "Build failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}" }
}
}
To scale, keep the controller light and run builds on agents: add static agents (Manage Jenkins, Nodes) or, better, dynamic agents that appear per build (Kubernetes plugin pods, EC2 plugin). A multibranch pipeline scans the repository and creates a job for every branch or PR that has a Jenkinsfile, so each branch is built and tested automatically.
Go deeper
- Never run builds on the controller; put secrets in the Jenkins credentials store and bind them with
withCredentials. - Back up
JENKINS_HOME; keep pipeline code in Git; shared libraries for reuse. - Slack: install the Slack plugin, create a webhook or app token, store it as a credential.
Follow-ups
- Jenkins HA? Controller is mostly single; use fast restores, configuration as code (JCasC) and ephemeral agents.
- Parallel stages?
parallel { ... }inside a stage.
Common mistake: snowflake agents with hand-installed tools.
4.2 Git workflow and branch protection
Question: “How do you change a risky infrastructure file safely? How do you protect main?”
Model answer: I clone, branch, read the file history and git blame to learn why a line exists, make the smallest change, commit with a message that explains the why, push and open a pull request so CI runs (lint, plan, tests) and a reviewer approves. On the repository, branch protection on main requires pull request reviews, passing status checks, up-to-date branches, optionally signed commits and linear history, and blocks force-pushes and deletion. CODEOWNERS routes reviews to the right people.
Go deeper
- Descriptive commit messages (summary line, blank line, reason) speed reviews and make
git logandgit bisectuseful. - SonarQube fits into the PR flow: PR opened, Sonar analysis (bugs, vulnerabilities, code smells, coverage, security hotspots), quality gate pass or fail, peer review, merge, then the Jenkins build. SonarLint shows issues in the IDE.
- Merge vs rebase vs squash: squash keeps main tidy; rebase for linear history; never rewrite shared history.
Follow-ups
- Undo a pushed bad commit?
git revert, notreset, on shared branches. - Trunk-based vs GitFlow? Short-lived branches and feature flags suit continuous delivery.
Common mistake: pushing straight to main with no checks.
4.3 GitOps vs push-based CD
Question: “What is GitOps and why prefer it over a pipeline that runs kubectl apply?”
flowchart TD
A["Developer merges PR"] --> B["Git repo (desired state)"]
B --> C["Agent in cluster (Argo CD / Flux) pulls"]
C --> D["Cluster reconciled to Git"]
D --> E["Drift detected and corrected or reported"]
Model answer: In push CD the CI system holds cluster credentials and applies manifests, so those credentials sit outside the cluster and the real state can drift from what was deployed. In GitOps Git is the single source of truth and an agent inside the cluster pulls and continuously reconciles. Benefits: no cluster credentials in CI, full audit trail in Git history, easy rollback with git revert, automatic drift correction, and consistent multi-cluster rollout.
Go deeper
- CI builds and pushes an image and updates the manifest repo (image tag bump); the GitOps agent deploys it. Separate the app repo from the config repo.
- Argo CD: Applications, sync policy (auto, prune, self-heal), ApplicationSets for fleets, health checks.
- Secrets still need a solution: Sealed Secrets, SOPS, External Secrets Operator.
Follow-ups
- Downside? Extra component to run, secrets handling needs thought, slower feedback than a direct apply.
Common mistake: confusing GitOps with “CI pipeline stored in Git”.
4.4 Blue-green, canary and rolling deployments
Question: “Explain blue-green deployment and how it compares with canary.”
flowchart TD
A["Blue = live v1.0.8"] --> B["Deploy v1.0.9 to idle Green"]
B --> C["Test and warm up Green"]
C --> D["Switch router or DNS to Green"]
D --> E["Keep Blue for instant rollback"]
Model answer: Blue-green keeps two identical production environments. The live one serves users; you deploy and test the new version on the idle one, then switch traffic (load balancer target group, Route 53 weighted record, ingress). Rollback is just switching back, even hours later, so the old environment stays until you are confident. Cost: double capacity during the switch, and database changes must stay backward compatible. Canary sends a small percentage (1, 5, 25 percent) to the new version and watches metrics before widening; rolling replaces instances gradually in place.
Go deeper
- Database migrations use expand and contract: add new column (compatible with both versions), deploy, backfill, switch reads, later drop the old column.
- Argo Rollouts or Flagger automate canary analysis against Prometheus metrics.
- DNS switching is slowed by TTL caching; load balancer or service mesh switching is instant.
Follow-ups
- When blue-green over canary? When you need an instant, complete switch and have the capacity; canary when you want gradual risk exposure with real traffic.
Common mistake: a schema change that breaks the old version, making rollback impossible.
4.5 Secrets pushed to GitHub: incident response
Question: “You accidentally pushed an AWS key to a public repo. What do you do?”
Model answer: Treat the key as compromised the moment it was pushed. First revoke or deactivate it in IAM and issue a new one if needed. Then check CloudTrail for use of that key (new users, EC2 instances, S3 access, unusual regions), terminate anything unauthorised and rotate related credentials. Remove it from the code, and clean history with git filter-repo or BFG, but understand that deleting is not enough because forks, clones and bots have already copied it. Finally prevent a repeat: pre-commit secret scanning (gitleaks, trufflehog), GitHub secret scanning and push protection, short-lived credentials (IAM roles, OIDC for CI), and a secrets manager.
Go deeper
- AWS also scans public GitHub and may quarantine the key and open a case.
- Use least-privilege IAM and budget alarms so abuse is detected quickly.
Common mistake: force-pushing the history clean and not rotating the key.
4.6 Log parsing with awk (common live question)
Question: “From an NGINX access log, print the top 10 client IPs, or count 5xx responses.”
# Top 10 IPs
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Count of 5xx responses (status is field 9 in the combined format)
awk '$9 ~ /^5/ {c++} END {print c}' access.log
# Top requested URLs returning 404
awk '$9==404 {print $7}' access.log | sort | uniq -c | sort -rn | head
# Requests per minute
awk '{print substr($4,2,17)}' access.log | sort | uniq -c
Model answer: NGINX combined log fields: $1 IP, $4 time, $6-$8 request, $9 status, $10 bytes. Awk selects fields, sort | uniq -c | sort -rn counts and ranks. At scale use a log platform (Loki, Elasticsearch, CloudWatch Logs Insights) instead of shell tools.
Go deeper: grep -c, cut -d' ' -f1, sed, jq for JSON logs, zcat for rotated files.
Common mistake: forgetting the log format changes field positions.
5. AWS
5.1 Leaked or public S3 bucket: incident response
Question: “You find an S3 bucket with sensitive data exposed publicly. What do you do?”
flowchart TD
A["Contain: S3 Block Public Access ON"] --> B["Investigate: CloudTrail data events, S3 access logs"]
B --> C["Assess: what data, who accessed, how long"]
C --> D["Remediate: fix policy/ACL, rotate secrets, notify"]
D --> E["Prevent: account-level BPA, SCP, Config rules, Macie"]
Model answer: Contain first: turn on S3 Block Public Access for the bucket and the account, and remove public ACLs or policy statements. Then investigate who accessed what using CloudTrail data events and S3 server access logs, scope the exposure, rotate any credentials found in the data, and notify security, legal and affected parties as required. Prevent a repeat: account-wide Block Public Access, an SCP that denies disabling it, AWS Config rules and Security Hub checks, Macie for sensitive-data discovery, default encryption (SSE-KMS), bucket policies enforcing TLS, and least-privilege IAM.
Go deeper
- Use pre-signed URLs or CloudFront with Origin Access Control instead of public buckets.
- Versioning plus MFA delete and Object Lock protect against deletion and ransomware.
- Access Analyzer finds buckets shared outside the account.
Common mistake: deleting the bucket and destroying the evidence.
5.2 IAM policies: reading them and blocking a user
Question: “How do you stop one IAM user from reading a bucket? Explain this policy.”
Model answer: IAM evaluation: everything is denied by default; an explicit Allow grants; an explicit Deny always wins. To block one user from one bucket, add a statement to the bucket policy with Effect: Deny, Principal set to the user’s ARN, Action: s3:* and both the bucket ARN and bucket/* as resources.
{
"Effect": "Deny",
"Principal": {"AWS": "arn:aws:iam::111122223333:user/alice"},
"Action": "s3:*",
"Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]
}
Go deeper
- Bucket-level actions (
s3:ListBucket) use the bucket ARN; object actions (s3:GetObject) usebucket/*. Missing one of them is the most common policy bug. s3:ListAllMyBucketsonly works withResource: "*".- A Deny on
arn:aws:s3:::*log*blocks any bucket whose name contains “log”. - Policy types: AWS managed (maintained by AWS), customer managed (yours, reusable), inline (embedded in one identity). Also resource policies, permission boundaries, session policies and SCPs, which limit but do not grant.
- Cross-account access needs both an identity policy and a resource policy to allow it.
- Prefer roles with short-lived credentials over IAM users and access keys.
Follow-ups
- Debug “access denied”? IAM policy simulator, CloudTrail error details, check for an SCP, boundary, bucket policy or KMS key policy deny.
Common mistake: forgetting that an SCP or KMS key policy can deny even when IAM allows.
5.3 Security groups, load balancers and scaling
Question: “Explain security groups, the load balancer types, and scale up vs scale out.”
| Load balancer | Layer | Best for |
|---|---|---|
| ALB | 7 (HTTP/HTTPS) | Host and path routing, microservices, WebSockets, gRPC |
| NLB | 4 (TCP/UDP/TLS) | Extreme performance, static IPs, non-HTTP |
| GWLB | 3 | Inline virtual appliances (firewalls) |
| CLB | 4 and 7, legacy | Migrate away |
Model answer: Security groups are stateful virtual firewalls on the network interface: ingress rules control inbound traffic, egress rules control outbound, and return traffic is allowed automatically. They are allow-only; reference other security groups instead of IP ranges for tiered apps. NACLs are stateless, subnet-level, and allow or deny. For capacity, vertical scaling (scale up) makes one server bigger, simple but has a ceiling and usually downtime; horizontal scaling (scale out) adds servers behind a load balancer, gives fault tolerance and handles spikes, but requires stateless services. At 98 percent CPU I would look at the cause first (a runaway process, a missing index), then scale out with an Auto Scaling Group using target tracking on CPU or request count, and scale up only for stateful, hard-to-distribute components such as a single primary database.
Go deeper
- Mixed instances policies and instance fleets combine on-demand, reserved/savings-plan and spot capacity across instance types for cost and resilience (used by EMR and ASG).
- Spot not launching: price or capacity unavailable, or a service quota; diversify instance types and AZs, or request a quota increase through AWS Support.
- Health checks: ELB health checks can replace unhealthy ASG instances; use connection draining (deregistration delay).
Common mistake: scaling out a service that keeps session state in memory.
5.4 EC2 troubleshooting: full disk, terminating instances, bastion, log shipping
Question: “An EC2 disk is full. An Auto Scaling Group keeps terminating instances. How do you investigate?”
flowchart TD
A["Instance unhealthy"] --> B["df -h and df -i: disk, inodes<br/>top: CPU<br/>free -m: memory and swap"]
B --> C["Clean logs or resize EBS<br/>Fix runaway process or leak<br/>Right-size the instance"]
Model answer: For a full disk I find the culprit with df -h, then du -xh / --max-depth=1 | sort -h, and check df -i for inode exhaustion. On the root volume I clean old logs (/var/log, journal, package cache), rotate logs with logrotate, and avoid anything that could break the OS. On an application volume I find large logs, old deployments or obsolete JARs; to grow it I take an EBS snapshot first, modify the volume size, then extend the partition and filesystem (growpart, xfs_growfs or resize2fs) with no downtime. For an ASG that keeps replacing instances, the usual cause is that instances fail health checks because of resource exhaustion (disk, CPU, memory) or a bad launch template or boot script; check the ASG activity history, instance system logs and application logs, not just the ASG settings.
Go deeper
- Bastion host: a hardened jump server in a public subnet; you authenticate to it, then SSH to private servers it is allowed to reach. Benefits: one controlled entry point, central logging, easy to turn off. Modern alternative: AWS Systems Manager Session Manager (no open port 22, IAM-based, fully audited).
- Ship logs to S3 on a schedule: attach an IAM role to the instance (no access keys), write a bash script with a shebang that runs
aws s3 cporaws s3 sync, schedule with cron or a systemd timer. Better: CloudWatch Agent or Fluent Bit for streaming, with S3 lifecycle rules for retention.
#!/bin/bash
set -euo pipefail
aws s3 sync /var/log/myapp s3://my-log-bucket/$(hostname)/ --exclude "*" --include "*.log"
Common mistake: adding long-lived access keys to the instance instead of using an instance role.
5.5 Databases on AWS: choosing, replicas, backups
Question: “Which database for which job? What is replica lag? What is Aurora backtracking?”
| Type | Examples | Use when |
|---|---|---|
| Relational | MySQL, PostgreSQL, Aurora, Oracle, SQL Server (RDS) | Transactions, joins, strict schema |
| Document | MongoDB, DynamoDB | Flexible schema, key-based access at scale |
| Key-value / cache | Redis, DynamoDB | Sessions, cache, leaderboards |
| Wide-column | Cassandra | Huge write-heavy, time-series style |
Model answer: SQL gives structure, joins and ACID transactions; NoSQL trades some of that for flexible schema and horizontal scale. RDS gives managed relational databases with automated backups, Multi-AZ failover and read replicas. Read replicas are asynchronous: after a write to the primary there is a short delay before the replica has it, so reading your own write from a replica can return nothing (replica lag). Mitigate by reading critical data from the primary, monitoring ReplicaLag, and designing for eventual consistency. Multi-AZ is for high availability (synchronous standby, automatic failover) and is not for scaling reads. Aurora backtracking (Aurora MySQL) rewinds the cluster in place to an earlier time within a configured window (up to 72 hours) without restoring from backup, useful after an accidental delete.
Go deeper
- Backups: automated backups with point-in-time restore, manual snapshots, cross-region copies. Restore creates a new instance.
- Marketplace AMIs: you pay for the EC2 instance plus the vendor’s software charge.
Common mistake: treating a read replica as a backup or as an HA failover target.
6. Databases and reliability
6.1 A query takes 30 seconds
Question: “A query takes 30 seconds in production. What do you do?”
Model answer: Measure before changing. Get the exact query and run EXPLAIN (ANALYZE, BUFFERS) (Postgres) or EXPLAIN (MySQL) to see the plan: sequential scan on a large table, missing index, poor row estimates, big sorts or joins. Fix with the right index (matching the filter and sort columns), rewriting the query (select only needed columns, avoid functions on indexed columns, avoid SELECT *, paginate), refreshing statistics (ANALYZE), and only then considering caching, partitioning or a read replica. Check for locks and blocking sessions (pg_stat_activity, SHOW PROCESSLIST), table bloat and resource pressure (CPU, IO, memory). Use the slow query log or pg_stat_statements to find the worst offenders over time.
Go deeper
- Index trade-off: faster reads, slower writes and more storage. Use
CREATE INDEX CONCURRENTLYin Postgres to avoid blocking writes. - A leading wildcard
LIKE '%x'and implicit type casts defeat indexes.
Common mistake: adding indexes blindly without reading the plan.
6.2 Add a column with zero downtime
Question: “How do you change a schema on a huge table without downtime?”
flowchart TD
A["Expand: add nullable column"] --> B["Deploy code that writes both"]
B --> C["Backfill in small batches"]
C --> D["Switch reads to new column"]
D --> E["Contract: drop old column later"]
Model answer: Use the expand and contract pattern so old and new application versions both keep working. Add the column as nullable or with a cheap default (modern Postgres adds a constant default without rewriting the table; older versions and some MySQL operations lock), deploy code that handles both shapes, backfill existing rows in small batches to avoid long locks, switch reads, add constraints with NOT VALID then VALIDATE, and only later remove the old column. For big MySQL tables use online schema change tools (gh-ost, pt-online-schema-change).
Common mistake: running a long ALTER TABLE in a single transaction during peak traffic.
6.3 Replica promotion and failover
Question: “The primary database is down. How do you promote a replica?”
Model answer: Confirm the primary is truly down (avoid split brain), pick the most up-to-date replica by checking replication lag and last received position, stop writes to the old primary (fence it), promote the replica (pg_ctl promote or SELECT pg_promote(), or RDS promote-read-replica / Multi-AZ failover), repoint the application (DNS name, connection string, proxy), rebuild a new replica from the new primary, and review for lost transactions because asynchronous replication can lose the last few. Use automation (Patroni, RDS Multi-AZ, Orchestrator) with a consensus store so promotion is safe and fast.
Go deeper
- RPO (how much data you can lose) and RTO (how long you can be down) drive the design: synchronous replication reduces RPO to near zero at a latency cost.
- Test failovers regularly in game days.
Common mistake: leaving the old primary reachable, so two nodes accept writes.
6.4 “Too many connections”
Question: “Postgres says too many connections. What now?”
Model answer: Every Postgres connection is a process and costs memory, so raising max_connections blindly hurts. Look at who is connected (pg_stat_activity), find idle or leaked connections, and fix application pool sizes. Put a connection pooler (PgBouncer in transaction mode, RDS Proxy) between the apps and the database so thousands of client connections share a small number of server connections. Set sensible timeouts (idle_in_transaction_session_timeout), and size pools as roughly cores times a small factor, not hundreds per service.
Go deeper
- Many microservices times many replicas times pool size multiplies quickly; budget total connections.
- Transaction pooling breaks session features (prepared statements in some modes, advisory locks,
SET).
Common mistake: scaling the application out and watching connections explode.
7. Monitoring, logging and tracing
7.1 Monitoring and alerting fundamentals
Question: “How do you monitor and alert on a production service?”
flowchart TD
A["App and infra emit metrics"] --> B["Prometheus scrapes (pull)"]
B --> C["Alert rules (PromQL, thresholds, SLO burn)"]
C --> D["Alertmanager: group, dedupe, route"]
D --> E["Slack / PagerDuty / email to on-call"]
E --> F["Runbook, mitigate, postmortem"]
Model answer: I monitor the four golden signals (latency, traffic, errors, saturation) plus the RED method for services (rate, errors, duration) and USE for resources (utilisation, saturation, errors). Prometheus pulls metrics from targets over HTTP (service discovery finds them), stores time series, and evaluates alert rules; Alertmanager groups, deduplicates, silences and routes notifications to the on-call. Dashboards in Grafana. For AWS-native stacks CloudWatch metrics and alarms with SNS do the same job; on-prem teams may use Nagios, Zabbix or an ELK stack.
Go deeper
- Pull vs push: pull makes target health obvious (
upmetric) and centralises config; push (Pushgateway, OTLP) is for short-lived jobs. - Alert on symptoms users feel (error rate, latency, SLO burn rate) rather than every cause (CPU 80 percent). Every alert needs an owner, a runbook link and a clear action, otherwise it causes alert fatigue.
- HTTP check: a GET to a health endpoint and alert on 4xx/5xx or timeouts (Blackbox exporter, synthetic checks).
- Incident flow: detect, acknowledge, triage, mitigate (rollback first), communicate, resolve, blameless postmortem with action items.
- SLI, SLO and error budgets: for example a 99.9 percent monthly availability SLO leaves about 43 minutes of error budget.
Follow-ups
- Dashboards by platform? Grafana for Kubernetes, CloudWatch for AWS-native, Kibana for ELK on-prem.
Common mistake: paging on CPU thresholds nobody can act on.
7.2 Centralised logging for 100+ microservices
Question: “Design logging for many microservices. EFK or Loki?”
flowchart TD
A["Containers write JSON to stdout"] --> B["Node agent (Fluent Bit / Alloy)"]
B --> C["Aggregator or direct to backend"]
C --> D["Elasticsearch / Loki / CloudWatch"]
D --> E["Kibana / Grafana"]
D --> F["Object storage for retention"]
Model answer: Applications log structured JSON to stdout; a DaemonSet agent on each node (Fluent Bit, Fluentd, Grafana Alloy) tails container logs, enriches them with Kubernetes metadata (namespace, pod, labels), and ships them to a backend. EFK (Elasticsearch, Fluentd, Kibana) indexes the full text of every log line: powerful search and analytics but heavy on storage, memory and operations. Loki indexes only labels and keeps log content compressed in object storage (S3), so it is much cheaper and simpler, and integrates with Grafana, but full-text search is slower because it scans chunks. Pick Elasticsearch when you need rich search and aggregation across logs; pick Loki when cost and simplicity matter and you query by labels and time.
Go deeper
- Add a correlation or trace ID to every log line.
- Control cost: set retention, drop noisy debug logs at the agent, sample, and tier old data to cheap storage.
- Avoid high-cardinality labels in Loki (user ID, request ID) because they explode the index.
- Alloy to Loki to S3 to Grafana: Alloy collects, Loki ingests (distributor, ingester), chunks and index are flushed to S3, queriers read them, Grafana shows them with LogQL.
Common mistake: shipping unstructured text, which makes parsing and alerting painful.
7.3 Distributed tracing
Question: “How do you trace one request across 20 services?”
Model answer: Instrument services with OpenTelemetry. Each request gets a trace ID; each operation is a span with a parent span ID, timing and attributes. The trace context travels in headers (W3C traceparent) across HTTP and messaging. Spans go to an OpenTelemetry Collector, which batches, samples and forwards them to a backend (Tempo, Jaeger, Zipkin, vendor tools); Grafana or the backend UI shows the waterfall so you can see which service or database call is slow. Link traces with logs and metrics using the trace ID (exemplars).
Go deeper
- Sampling: head-based (decide at the start, cheap) or tail-based (decide after seeing the whole trace, keep errors and slow ones).
- Auto-instrumentation agents get you started; add manual spans for business operations.
Common mistake: not propagating context across async boundaries such as queues.
7.4 Metrics at scale: Prometheus plus Thanos
Question: “Prometheus does not scale for long retention or many clusters. What do you do?”
Model answer: A single Prometheus keeps data locally and is not horizontally scalable or highly available by itself. Run replicas for HA, and add Thanos (or Mimir/Cortex/VictoriaMetrics): a sidecar next to each Prometheus uploads blocks to object storage (S3) for cheap long retention, the Store Gateway serves historical blocks, the Compactor downsamples and compacts them, and Thanos Query gives one PromQL endpoint across clusters, deduplicating HA replicas. Grafana points at Thanos Query.
Go deeper
- Control cardinality (labels with unbounded values) because it is the main cause of memory blow-ups.
- Use recording rules to precompute expensive queries; federation or remote write for aggregation.
Common mistake: adding user IDs or URLs as labels.
8. Networking and distributed-systems basics
8.1 TCP vs UDP, gRPC vs REST, webhooks vs polling
Question: “TCP or UDP? gRPC or REST? Webhooks or polling?”
| Choose | Why | |
|---|---|---|
| TCP | Web, APIs, databases, file transfer | Reliable, ordered, congestion control, handshake |
| UDP | DNS, live video/voice, games, QUIC | Low latency, no handshake, app handles loss |
| REST/JSON | Public APIs, browsers | Simple, human-readable, cacheable |
| gRPC | Internal service to service | HTTP/2, binary protobuf, streaming, strong contracts, faster |
| Webhook | Event notifications | Provider pushes in near real time |
| Polling | No public endpoint, simple | You ask repeatedly: wasteful, higher latency |
Model answer: TCP gives reliable ordered delivery through a three-way handshake, acknowledgements and retransmission; UDP is connectionless and cheap, so latency-sensitive traffic that tolerates loss uses it. gRPC suits internal microservices (compact, typed, bidirectional streaming) but is awkward for browsers and harder to debug than REST. Webhooks flip the direction: the provider calls your HTTPS endpoint when something happens, so you must verify the signature, validate the payload, respond quickly with 2xx, process asynchronously, and handle retries and duplicates idempotently.
Go deeper
- Handshake: SYN, SYN-ACK, ACK. TLS adds more round trips (TLS 1.3 reduces them).
- Health checks and timeouts: always set client timeouts and retries with backoff and jitter.
- HTTP codes: 2xx success, 3xx redirect, 4xx client error, 5xx server error, 429 too many requests (add
Retry-After).
Common mistake: trusting webhook payloads without signature verification.
8.2 Kafka basics
Question: “Explain Kafka. How do you get ordering and scale consumers?”
flowchart TD
A["Producers"] --> B["Topic split into partitions"]
B --> C["Brokers store partitions (replicated)"]
C --> D["Consumer group: one consumer per partition"]
D --> E["Offsets committed per partition"]
Model answer: Kafka is a distributed, replicated commit log. Producers append messages to a topic that is split into partitions; ordering is guaranteed only within a partition, so messages with the same key (for example an order ID) go to the same partition. Partitions are replicated across brokers for durability. Consumers in a consumer group share the partitions, each partition read by one consumer at a time, so adding consumers up to the partition count scales throughput. Consumers track offsets; consumer lag (latest offset minus committed offset) shows who is behind. Messages are retained by time or size, so consumers can replay.
Go deeper
- Delivery: at-most-once, at-least-once (default, make consumers idempotent), exactly-once with idempotent producers and transactions inside Kafka.
- Producer
acks=allwithmin.insync.replicasfor durability. - KRaft mode replaces ZooKeeper for metadata.
Common mistake: adding more consumers than partitions and expecting more throughput.
8.3 Why 3 or 5 nodes: quorum and split brain
Question: “Why do etcd, ZooKeeper and Raft clusters use odd sizes?”
| Nodes | Quorum | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 (worse than 1 in availability) |
| 3 | 2 | 1 |
| 4 | 3 | 1 (no better than 3) |
| 5 | 3 | 2 |
Model answer: Consensus systems need a majority (floor(n/2) + 1) to agree before committing or electing a leader. In a network partition only the side with a majority can continue, so two sides can never both think they are the leader (split brain). An even size adds cost without tolerating more failures, so use 3 for most clusters and 5 when you must survive two failures; more than 7 slows writes because every commit needs more acknowledgements.
Go deeper: spread members across availability zones; etcdctl endpoint status --write-out=table and etcdctl member list to check health.
Common mistake: running two-node “HA”.
8.4 Process vs thread
Question: “What is the difference between a process and a thread?”
Model answer: A process has its own address space, so it is isolated and a crash stays contained, but memory is heavier and processes communicate through IPC (pipes, sockets, shared memory). Threads run inside one process and share its memory, so they are cheap and share data easily, but need locks or atomics to avoid race conditions, and one bad thread can crash the whole process. On Linux both are created with clone; threads share the address space (CLONE_VM). Containers are processes with isolation, not threads.
Go deeper: Python’s GIL limits CPU-bound threads (use processes or native code); Node uses an event loop plus worker threads; context switches cost more between processes.
Common mistake: assuming more threads always means more speed.
9. Scripting, tooling and behavioural
9.1 Python, scripting and build tools
Question: “How do you use Python in DevOps? How would you check an API or monitor a server?”
import requests, psutil
r = requests.get("https://api.example.com/health", timeout=5)
print(r.status_code) # alert if not 200
print(psutil.cpu_percent(interval=1))
print(psutil.virtual_memory().percent)
print(psutil.disk_usage("/").percent)
print(psutil.getloadavg())
Model answer: I use Python for glue and automation: calling APIs with requests and checking status codes and latency, boto3 for AWS tasks, psutil to collect CPU, memory, disk and load on servers without agents, subprocess for commands, and argparse plus logging for proper CLIs. Always set timeouts, handle exceptions, and make scripts idempotent. A package is a directory of modules, which contain functions and classes. Build tools turn source into artifacts: Maven or Gradle for Java, npm for Node, pip or poetry (plus build backends) for Python; they manage dependencies, tests and packaging and plug into CI/CD with caching.
Go deeper: web server (static content, TLS, reverse proxy, for example Nginx) versus application server (business logic, auth, database access, for example Tomcat, Gunicorn); often the web server fronts the app server.
Common mistake: scripts without timeouts that hang forever.
9.2 Behavioural answers that land
Question: “Describe your day, your growth plan, a failure, a conflict.”
Model answer (framework): use STAR: Situation, Task, Action, Result, and finish with what you learned. For a typical day, show a rhythm: check alerts and dashboards, stand-up, planned work (tickets, reviews, releases), collaboration with developers, and learning time. For growth, name concrete steps (certifications, reading incident postmortems, mentoring, owning a reliability goal). For failures, pick a real incident, own your part, explain the fix and the process change that stopped it recurring. For conflict, show you listened, used data and agreed on a decision criterion.
Go deeper: prepare four stories you can bend to many questions: a production incident, an automation win, a disagreement, and a time you learned something fast. Quantify results (time saved, cost reduced, incidents avoided).
Common mistake: blaming others or giving vague answers with no numbers.
10. System design interviews
10.1 The six-step framework and the first five minutes
Question: “Design X.” (Any system.)
flowchart TD
A["1 Clarify requirements"] --> B["2 Estimate scale"]
B --> C["3 Define API"]
C --> D["4 High-level design"]
D --> E["5 Deep dive 1-2 components"]
E --> F["6 Bottlenecks, failures, scaling"]
What interviewers grade: judgment under ambiguity (turning a vague prompt into a testable scope), technical depth (why a cache, and what it costs), communication, and collaboration (asking questions and checking assumptions).
Step by step
- Clarify (first 5 minutes). Ask only questions whose answers change the architecture: users (DAU/MAU) and growth, read-heavy or write-heavy, latency targets, data you cannot lose (durability), consistency needs, scope (which features, which clients). For minor details, state an assumption with a short reason and move on.
- Estimate. Requests per second = requests per day / 86,400 (about 100,000 seconds in a day for quick maths). Peak is often 2 to 5 times average. Storage = records times size times retention. Say the numbers out loud so the interviewer can correct them.
- API. Two to five endpoints with request, response and error cases; choose REST or gRPC; think about idempotency and pagination.
- High-level design. Clients, load balancer, stateless app servers, database, cache, queue, object store, CDN; draw the data flow for the main read and write.
- Deep dive. Pick the hardest part (data model, ID generation, cache strategy, fan-out, consistency) and go deep with trade-offs.
- Bottlenecks. What breaks first? Single points of failure, hot keys, failure modes and what you can afford to lose.
Rules of thumb: start simple and add components only when a number justifies them; always name the trade-off; state consistency choices (strong vs eventual); finish by summarising.
Common mistakes: jumping into boxes before clarifying, over-engineering (multi-region, sharding at 100 requests per second), never stating trade-offs.
10.2 Design a URL shortener
Question: “Design a URL shortening service like bit.ly.”
Clarify and assume: system-generated codes (no custom aliases), links do not expire, analytics optional.
Estimate: 1 million new links per day is about 12 writes per second. If each link is read about 1,000 times, that is 1 billion redirects per day, about 11,600 reads per second on average: read-heavy. Ten years of links is about 3.65 billion rows; a row of about 500 bytes is under 2 TB.
API
POST /api/v1/shortenwith{"long_url": "..."}returns{"short_url": "https://sho.rt/aX9dQ2z"}.GET /{code}returns a301or302withLocation.
flowchart TD
A["Client"] --> B["CDN / edge cache"]
B --> C["Load balancer"]
C --> D["Stateless app servers (local LRU)"]
D --> E["Redis (cache-aside)"]
E --> F["Database: code -> long_url"]
D --> G["ID allocator (counter blocks)"]
Generating the code
- Hash of the URL (take the first 7 characters of a hash, base62): stateless and the same URL gives the same code, but collisions appear as data grows, so every write needs a database check and sometimes a retry.
- Counter + base62: a globally unique number encoded in base62. Seven characters give 62^7, about 3.5 trillion codes. No collision checks, but sequential codes are guessable, so apply a reversible secret permutation (bijective shuffle) to the counter before encoding.
- Avoid a single counter bottleneck: give each app server a block of IDs (for example 10,000) from a central allocator (database sequence, Redis
INCR, or a Snowflake-style generator) and issue from memory. Large blocks mean fewer calls but more wasted IDs if a server dies; small blocks the opposite. Concurrent requests need atomic increments or a unique constraint with retry.
Storage: the access pattern is one key lookup (code -> long_url), no joins. Start with Postgres or MySQL with a unique index on the code; at billions of rows shard by code or use a key-value store (DynamoDB, Cassandra).
Redirect and cache
- Cache-aside in Redis: hit returns immediately; miss reads the database once and fills the cache. Mappings are immutable, so TTL and LRU eviction are enough.
- A celebrity spike hits one hot key: add layers: CDN edge, an in-process LRU in each app server, then Redis, then the database. Pre-warm known viral links; monitor miss rate.
- 301 vs 302: 301 is cacheable, so browsers and CDNs skip you next time: cheapest, but you lose analytics and the ability to disable or change a link. 302 keeps every click flowing through you (analytics, control) at higher load. Control caching with
Cache-Control.
Failure modes: Redis down means more database load and latency but correct answers (the cache is an optimisation); database loss means permanent broken links, so use replicas, backups and point-in-time recovery; the ID generator must never repeat after a crash (persist the position, or skip ahead).
Link expiry: store expires_at, check at redirect, purge with TTL or a background job. Abuse: rate limit creation, scan target URLs.
Follow-ups: custom aliases? uniqueness check, reserved words. Analytics without slowing redirects? write click events to a queue or log, process asynchronously.
10.3 Design a rate limiter
Question: “Design an API rate limiter.”
Clarify: who is limited (account ID when logged in, IP when anonymous, or API key), the rule (for example 100 requests per minute), and what happens on excess (reject with 429 Too Many Requests and Retry-After, or queue).
Estimate: 10 million users at 100 requests per day is 1 billion requests per day, about 11,600 per second average and about 35,000 per second at 3 times peak. State per user (counter plus timestamp) is about 50 bytes, so 10 million users is about 500 MB: it fits in memory.
Algorithms
| Algorithm | How | Pros | Cons |
|---|---|---|---|
| Fixed window | Counter per clock window | Simple, cheap | Boundary burst: 100 at the end of one minute and 100 at the start of the next allows 200 in 2 seconds |
| Sliding window log | Store timestamps, drop older than the window, count | Exact | High memory and CPU |
| Sliding window counter | Weighted blend of current and previous window | Cheap, close to exact | Approximate |
| Token bucket | Bucket of N tokens refilled at a steady rate, each request spends one | Allows controlled bursts, two fields per user | Needs shared state |
| Leaky bucket | Queue drained at a fixed rate | Smooth output | Bursts are delayed or dropped |
Token bucket math: capacity 100, refill 100 per minute (1.67 tokens per second). On each request: tokens = min(capacity, tokens + elapsed * refill_rate); if tokens >= 1 allow and subtract one; otherwise reject with Retry-After = ceil((1 - tokens) / refill_rate) seconds. Store current_tokens and last_refill_ts per user.
flowchart TD
A["Request"] --> B["API gateway / edge"]
B --> C["Redis bucket for user (Lua script)"]
C --> D["Allowed: upstream service<br/>Limited: 429 + Retry-After + X-RateLimit headers"]
Making it scale: with many gateways the counters must be shared, typically in Redis. A naive read-modify-write has a race (two gateways read the same token count), so run the whole refill-check-update as one Lua script (atomic in Redis) and call it with EVALSHA. One Redis handles on the order of 100,000 operations per second; shard keys by user ID with Redis Cluster or consistent hashing for more. Keep Redis in the same region and AZ as the gateways (about 1 ms round trip); avoid cross-region calls on the request path.
Decisions to state: fail-open (allow traffic if Redis is down, protects availability) vs fail-closed (protects the backend); put the limiter at the edge so you reject early; return headers X-RateLimit-Limit, -Remaining, -Reset.
Follow-ups: different limits per endpoint? configurable rules by route, method, tier and API key (reads high, writes tighter, login strictest to stop credential stuffing). Redis too slow or loaded? local in-memory counters synced periodically (low latency, slightly relaxed accuracy), local deny lists with cooldown, and edge/WAF filtering. Distributed across regions? per-region limits with eventual sync; strict global limits cost latency.
10.4 Design a notification system
Question: “Design a system that sends push, SMS, email and in-app notifications.”
Clarify: transactional (OTP, receipts: seconds, must not be lost) vs marketing (minutes are fine, huge bursts); channels; user preferences and opt-outs; can we lose one? can we send a duplicate?
Estimate: 10 million notifications per day is only about 115 per second average, but a marketing blast to 1 million users at 9 AM is a burst the design must absorb.
API: POST /notify with user, channels, body and metadata; persist a record with status pending and return 202 Accepted with an id immediately. Delivery is asynchronous.
flowchart TD
A["POST /notify"] --> B["DB: notification row (pending)"]
B --> C["Outbox processor scans pending"]
C --> D["Queue per channel (push, sms, email, in-app)"]
D --> E["Workers (idempotency check, prefs, templates)"]
E --> F["Provider (APNs/FCM, SMS gateway, email)"]
E -->|"retries exhausted"| G["Dead-letter queue"]
Why a queue: sending synchronously ties your API to slow third parties, and a crash mid-send leaves an unknown state. The database is the source of truth; the queue holds a small ticket (just the notification ID). If a worker dies before acknowledging, the queue’s visibility timeout redelivers the ticket.
One queue per channel so a slow email provider cannot block push. Separate transactional and marketing queues and dedicated worker pools so a campaign never delays a login code.
Retries: exponential backoff with jitter (1, 2, 4, 8, 16 seconds, capped, randomised), 3 to 5 attempts, then a dead-letter queue for inspection and replay; send malformed (poison) messages to the DLQ immediately. Alert on DLQ depth and annotate failures with reason, attempts and correlation ID. Drop expired messages (an OTP has an expiry).
Duplicates (the trap): a worker sends the SMS, crashes before acknowledging, and the ticket is redelivered. Delivery is at-least-once, so make sends idempotent: before calling the provider, claim the notification ID in Redis with SET key owner_token NX EX ttl; if the key exists, drop the ticket; release with compare-and-delete only if you own it. Honest caveat: exactly-once is impossible in general, because if the provider accepted the message but you crashed before recording it, a retry can still duplicate; mitigate with provider idempotency keys and audit logs.
The two-writes problem and the outbox: writing to the database and then enqueuing are two separate writes; a crash between them loses the notification. Fix with the transactional outbox: only write the row (pending) in the database; a background processor scans pending rows and enqueues them, so the enqueue is derived from database state. Enqueuing twice is harmless because workers are idempotent. Index by (status, created_at), batch enqueues.
Other points: respect preferences and opt-outs before calling providers; per-provider rate limits and circuit breakers; fail over to a secondary provider during outages; central gateway to deduplicate across teams (LinkedIn cut emails by about half with a central gateway); priority buckets (Uber separates urgent trip updates from promotions); in-app uses a seen flag.
10.5 Design a news feed (timeline)
Question: “Design a home feed like Twitter or Instagram.”
Clarify: posts only from accounts you follow, newest first (ranking is a separate, bigger problem); feed loads in about 1 second; a new post appears for followers within seconds; text, images, video; occasional duplicates after refresh are acceptable.
Estimate: 10 million DAU opening the app 5 times is 50 million feed reads per day, about 580 per second; 3 posts per user per month is about 1 million posts per day, about 12 writes per second. Read-to-write is roughly 48 to 1.
Tables: users (with a celebrity flag), follows (follower, followee; about 2 billion rows for 10 million users following 200 each), posts (indexed by (author_id, created_at DESC)). API: POST /posts, GET /feed?limit=50.
Why the simple design breaks (fan-out on read): per feed open, read the follow list (about 300 accounts), query posts for each, merge and sort. At 600 opens per second that is 180,000 post lookups per second (900,000 at 5 times peak).
Fan-out on write (push): precompute each user’s feed when someone posts.
flowchart TD
A["Create post: save post + outbox row (one transaction)"] --> B["Outbox scanner -> queue"]
B --> C["Fan-out workers (idempotent)"]
C --> D["Push post ID to each follower's Redis list (LPUSH + LTRIM)"]
E["GET /feed"] --> F["Read ID list from Redis"]
F --> G["Multi-get post content from post cache"]
G --> H["Miss: read DB and fill cache"]
- Cache 1, the feed index: a bounded list of post IDs per active user (500 IDs times 8 bytes is about 4 KB each; 10 million users is about 40 GB). Cache 2, the post cache: post content, about 1 KB per post, hot set with LRU (about 30 GB for a month of posts).
- Precompute only for active users (for example opened the app in the last 30 days); build an inactive user’s feed on demand when they return.
- Workers must be idempotent (use a set or sorted set keyed by post ID, or a unique constraint on
(follower_id, post_id)) because queues deliver at least once.
The celebrity problem: one post by an account with 50 million followers means 50 million writes and wasted work for people who never open the app. Use a hybrid: normal accounts push; accounts over a follower threshold (for example 100,000) are not pushed and are pulled at read time and merged with the precomputed list, sorted, and trimmed to the top 50. Keep a small shared cache of each celebrity’s recent posts so a reader following 800 celebrities makes 800 cache reads, not 800 database queries.
Storage: posts are append-only and time-ordered, a fit for wide-column stores (Cassandra) and sharding (about 360 million posts a year); follows and users are relational. Media: upload to object storage (S3) and serve through a CDN; the database and caches hold only URLs. Viral posts: replicate hot cache entries and let the CDN carry the bytes.
Follow-ups: posts take 8 minutes to reach followers? measure queue lag and worker throughput, add workers (watch cost), and prioritise fan-out to recently active followers first. Ranked feed? score candidates at read time with relevance signals (age, relationship, engagement) and measure the trade-off against latency.
10.6 Design an image upload pipeline
Question: “Design photo upload with thumbnails.”
flowchart TD
A["Client asks API for upload URL"] --> B["API returns pre-signed S3 URL"]
B --> C["Client uploads directly to S3"]
C --> D["S3 event -> queue"]
D --> E["Workers: validate, scan, resize, thumbnails"]
E --> F["Store variants + metadata in DB"]
F --> G["Serve via CDN"]
Answer: never stream large files through app servers. Issue a short-lived pre-signed URL so the client uploads straight to object storage. An object-created event goes to a queue; workers validate type and size, scan for malware, strip metadata (EXIF location), generate thumbnails and formats, and write metadata to the database. Serve images through a CDN with long cache lifetimes and versioned keys. Handle failures with retries and a DLQ; use multipart upload for big files and lifecycle rules for cleanup of abandoned uploads.
10.7 Design a payment system
Question: “Design payment processing. How do you avoid charging twice?”
Answer: correctness beats speed. Use a payment state machine (created, authorised, captured, failed, refunded) stored in a relational database with transactions. Every create or charge request carries an idempotency key; the server stores the key with the result so a retry returns the same outcome instead of charging again. Never trust the client’s amount; compute server-side. Call the payment provider with timeouts; on an unknown result (timeout) do not assume failure: query the provider or wait for its webhook, and reconcile. Keep an append-only ledger (double-entry: every debit has a credit) and run daily reconciliation against provider and bank reports. Use the outbox pattern so internal events (order paid) are not lost, and verify webhook signatures. Protect card data by never storing it (use the provider’s hosted fields or tokenisation, PCI scope stays small).
Follow-ups: retries and duplicates? idempotency keys and unique constraints. Partial failure between charge and order update? outbox plus reconciliation job.
10.8 Design a webhook delivery system
Question: “You must deliver events to customers’ HTTP endpoints reliably.”
Answer: write the event to a durable store and a queue; workers POST to the customer’s URL with a timeout (a few seconds), an HMAC signature header, a unique event ID and a timestamp (so receivers can verify and de-duplicate and reject replays). Treat any 2xx as success; retry other results with exponential backoff and jitter over hours or days, then mark the endpoint failing and notify the customer; offer a replay button and a delivery log. Isolate slow or dead endpoints so they cannot block others (per-endpoint queues or concurrency limits, circuit breaker), cap payload size and guard against SSRF (block private IP ranges). Delivery is at-least-once, so customers must be idempotent.
10.9 Design a caching layer
Question: “Where and how do you add caching?”
Answer: cache where reads dominate and data tolerates staleness: browser and CDN (static assets, public pages), application cache (Redis or Memcached), and database buffer cache. The default pattern is cache-aside: read the cache, on a miss read the database and populate. Alternatives: write-through (simpler reads, slower writes), write-back (fast writes, risk of data loss). Keep data fresh with TTLs plus explicit invalidation (commit the database write, then evict or update the cache). Protect against: cache stampede (many requests miss one expired hot key at once) with request coalescing or a per-key lock and TTL jitter; cache penetration (queries for keys that do not exist) with negative caching or a Bloom filter; hot keys with local caches or key replication. Choose eviction (LRU is the common default), size by hit rate, and monitor hit ratio, evictions and latency. The cache is an optimisation, not a source of truth: the system must still work when it is empty.
10.10 Design a background job queue
Question: “Design a system for running asynchronous jobs.”
Answer: producers enqueue a job (ID, type, payload, priority, attempts) into a durable queue (SQS, RabbitMQ, Kafka, or a database-backed queue); workers poll, process and acknowledge only on success; a visibility timeout makes unacknowledged jobs reappear, giving at-least-once processing, so jobs must be idempotent. Add retries with backoff and jitter, a max attempt count and a dead-letter queue, plus delayed or scheduled jobs and priority queues. Scale workers on queue depth and age of the oldest message; apply per-tenant concurrency limits to stop noisy neighbours; make long jobs report progress and heartbeat to extend the visibility timeout. Observe queue length, processing time and failure rate. Use the outbox pattern when a job must be enqueued atomically with a database change.
11. AWS CloudOps Engineer Associate: flows and questions
This chapter covers the six exam domains of the AWS Certified CloudOps Engineer Associate and the interview questions that come with the same material: monitoring, reliability, deployment, security, networking and cost. Every topic gives a flow where one helps, a model answer, deeper points and the usual mistake.
How the exam and interviews overlap: both test operations judgment. They ask “what do you check first?”, “which service fits?” and “what is the least risky fix?”. Practise answering in the order detect, diagnose, fix, prevent.
Domain 1: Monitoring, logging and remediation
11.1.1 CloudWatch vs CloudTrail
Question: “What is the difference between CloudWatch and CloudTrail?”
| CloudWatch | CloudTrail | |
|---|---|---|
| Answers | “How is it performing?” | “Who did what, when?” |
| Data | Metrics, logs, alarms, dashboards, events, traces | API calls and console actions |
| Typical use | Alarm on CPU, search app logs | Find who deleted a security group |
Model answer: CloudWatch is the performance and health side: metrics (with namespaces and dimensions), alarms, logs, dashboards and Logs Insights. CloudTrail is the audit side: it records management events (control-plane API calls) by default, can record data events (S3 object access, Lambda invokes) and insight events, and keeps 90 days of management history for free. For longer retention or an organisation-wide record, create a trail that delivers to S3 (encrypted, with log file validation) and optionally to CloudWatch Logs so you can build metric filters and alarms on API activity.
Go deeper
- Data events are off by default and billed per event; turn them on only for sensitive buckets.
- Use an organisation trail from the management account so member accounts cannot switch it off.
Common mistake: looking in CloudWatch for “who changed this” instead of CloudTrail.
11.1.2 Logs: groups, streams, Logs Insights, metric filters
Question: “Your application writes errors to a log. How do you alert on them, and how do you investigate a spike?”
flowchart TD
A["App or agent writes logs"] --> B["CloudWatch Logs: log group and stream"]
B --> C["Metric filter turns matches into a metric"]
C --> D["Alarm"]
D --> E["SNS topic -> email / chat / ticket"]
B --> F["Logs Insights query for investigation"]
Model answer: Logs sit in log streams (one source) inside log groups (an application or service, where retention and encryption are set). A metric filter matches a pattern such as ERROR or a JSON field ({ $.statusCode = 500 }) and increments a custom metric; an alarm on that metric notifies through SNS. For investigation I use Logs Insights, for example:
fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)
| sort @timestamp desc
Go deeper
- Retention defaults to never expire: set it per log group, and export or subscribe to S3, Firehose, OpenSearch or Lambda for archiving and processing.
- Subscription filters stream logs in near real time; metric filters only count matches going forward.
- Prefer structured JSON logs so filters can use field names.
Common mistake: leaving retention unlimited and paying for years of debug logs.
11.1.3 The CloudWatch Agent and the missing memory metric
Question: “Why is there no memory or disk metric for my EC2 instance, and how do you get it?”
Model answer: EC2 publishes hypervisor-level metrics (CPU, network, disk I/O, status checks) every 5 minutes by default, or every 1 minute with detailed monitoring. Memory use and disk space are inside the operating system, so AWS cannot see them. Install the CloudWatch Agent: give the instance an IAM role with CloudWatchAgentServerPolicy, install the agent (Systems Manager can distribute it), write its JSON config (metrics such as mem_used_percent, disk_used_percent, plus log files to collect), and start it. The metrics appear under the CWAgent namespace and logs in CloudWatch Logs.
Go deeper
- Collect only what you need, use least-privilege roles, and consider 1-second high-resolution metrics only for critical systems.
- It works for on-premises servers too (via credentials or hybrid activation).
- For containers use Container Insights.
Common mistake: granting administrator permissions to the instance just to make the agent work.
11.1.4 Alarms: states, missing data, composite alarms
Question: “How do you design CloudWatch alarms that do not cause alert fatigue?”
Model answer: An alarm has three states: OK, ALARM and INSUFFICIENT_DATA. Pick a metric and statistic, set a threshold, and require it to hold across several periods (for example CPU above 80 percent for 3 of 5 datapoints) instead of reacting to one spike. Decide how to treat missing data (notBreaching, breaching, ignore, missing), because a stopped instance sends none. Attach actions: SNS notification, Auto Scaling policy, EC2 action (stop, terminate, recover), or an EventBridge event for automation. Use composite alarms to combine conditions with AND/OR so you page only when, say, high latency and high error rate happen together.
Go deeper
- Anomaly detection alarms learn a band instead of a fixed threshold.
- Alarm on symptoms that users feel (error rate, latency, queue age) before causes (CPU).
- Use SNS with email, SMS, HTTPS or Lambda subscribers; confirm subscriptions and test them.
Common mistake: alarming on 40 percent CPU spikes that are normal.
11.1.5 EventBridge, SNS and Systems Manager: automated remediation
Question: “Someone opens a security group to the world. How do you detect and fix it automatically?”
flowchart TD
A["API call or Config change"] --> B["CloudTrail / AWS Config detects"]
B --> C["EventBridge rule matches the pattern"]
C --> D["Target: SSM Automation runbook or Lambda"]
D --> E["Rule removed, SNS notifies the owner"]
Model answer: CloudTrail records the AuthorizeSecurityGroupIngress call and an AWS Config rule (such as restricted-ssh) marks the resource non-compliant. Either signal reaches EventBridge, whose rule filters on an event pattern and targets a Systems Manager Automation runbook or a Lambda function that revokes the rule. SNS then tells the owner. Config can also attach the runbook directly as a remediation action, with manual or automatic execution.
Go deeper
- EventBridge pieces: event buses (default, custom, partner), rules with patterns, targets with retry and dead-letter queue, Scheduler for cron-style jobs, Pipes for point-to-point integrations, Schema Registry.
- Systems Manager family: Run Command, Automation (runbooks as YAML/JSON documents with steps like
aws:createSnapshot), Patch Manager, State Manager, Inventory, Session Manager, Parameter Store, Change Manager and maintenance windows. - Config records configuration history, evaluates rules, supports conformance packs and aggregators across accounts.
Common mistake: forgetting the IAM role the automation assumes.
11.1.6 Picking and sizing compute and storage for performance
Question: “How do you choose an EC2 instance type, and how do you monitor EBS performance?”
| Family | Ratio (vCPU:RAM) | Use |
|---|---|---|
| M | about 1:4 | General purpose |
| C | about 1:2 | CPU-bound: batch, encoding, game servers |
| R, X, U | about 1:8 and up | In-memory databases, caches, analytics |
| I, D | local NVMe or dense disk | High IOPS or throughput |
| P, G, Inf, Trn | accelerators | ML training, inference, graphics |
| T | burstable | Low steady load with bursts |
Model answer: Match the workload profile to the family, then right-size using real metrics. Suffixes help: g means Graviton (Arm, better price-performance), a AMD, n enhanced networking. T-series instances use CPU credits: watch CPUCreditBalance and CPUCreditUsage; sustained load drains credits and throttles, so move to M or C or enable unlimited mode. For EBS watch VolumeReadOps/WriteOps (divide by period for IOPS), VolumeReadBytes/WriteBytes (throughput), VolumeQueueLength (saturation), VolumeThroughputPercentage and BurstBalance (gp2 only). gp3 gives a baseline of 3,000 IOPS and 125 MB/s independent of size and lets you buy more; io2 and io2 Block Express provide guaranteed high IOPS for databases. Also check the instance’s own EBS bandwidth limit.
Go deeper
- Placement groups: cluster (one AZ, lowest latency, HPC), partition (rack isolation for Hadoop, Cassandra, Kafka), spread (each instance on separate hardware, small critical sets).
- Networking: ENA enhanced networking, EFA for HPC and ML, ENI for extra interfaces.
- Compute choice: EC2 for predictable sustained performance and large memory, containers (ECS/EKS/Fargate) for packaging and density, Lambda for short event-driven work (up to 15 minutes, 128 MB to 10,240 MB).
Common mistake: buying a bigger instance when the bottleneck is EBS throughput or a T-series credit limit.
11.1.7 S3 performance and storage classes
Question: “Uploads of large files from another continent are slow. What do you do? Which storage class for which data?”
Model answer: Use multipart upload (parts upload in parallel and retry individually; recommended above about 100 MB and required above 5 GB; object limit 5 TB) and S3 Transfer Acceleration, which sends data to the nearest edge location and over the AWS backbone through the bucket.s3-accelerate.amazonaws.com endpoint. Clean up incomplete multipart uploads with a lifecycle rule.
| Class | Use | Notes |
|---|---|---|
| Standard | Hot data | Multi-AZ |
| Intelligent-Tiering | Unknown or changing access | Moves objects automatically |
| Standard-IA | Infrequent, needs rapid access | 30-day minimum, retrieval fee |
| One Zone-IA | Re-creatable infrequent data | Single AZ |
| Glacier Instant Retrieval | Archive with millisecond access | 90-day minimum |
| Glacier Flexible Retrieval | Archive, minutes to hours | 90-day minimum |
| Glacier Deep Archive | Cheapest, 12 to 48 hours | 180-day minimum |
Go deeper: request-rate scaling is per prefix (thousands of requests per second); use byte-range fetches and CloudFront for read-heavy content.
Common mistake: moving small objects to IA or Glacier and paying minimum-size and minimum-duration charges.
11.1.8 RDS and Lambda performance
Question: “The database is slow and connection counts spike. And Lambda has cold starts. What do you do?”
Model answer (RDS): Watch CPUUtilization, DatabaseConnections, FreeableMemory, ReadIOPS/WriteIOPS, latency and DiskQueueDepth. Use Performance Insights to see DB load by wait event and the top SQL, then fix queries and indexes before resizing. Scale up for write-heavy load, add read replicas for read-heavy load, and put RDS Proxy in front to pool connections, absorb spikes from many Lambdas or microservices and speed up failover. Tune parameter groups carefully.
Model answer (Lambda): Memory (128 MB to 10,240 MB) also sets CPU and network (about one vCPU at roughly 1,769 MB), so more memory often reduces duration and can cost less overall. Cold starts come from creating a new execution environment; reduce them with smaller packages, lighter runtimes, and provisioned concurrency for latency-critical paths. Reserved concurrency caps a function to protect downstream systems (it is also a guaranteed slice of account concurrency, default 1,000 per region). Buffer bursts with SQS, SNS or EventBridge, and use dead-letter queues or on-failure destinations. Put Lambda in a VPC only when needed.
Common mistake: confusing provisioned concurrency (pre-warmed) with reserved concurrency (a limit).
Domain 2: Reliability and business continuity
11.2.1 Fault tolerance vs high availability
Question: “What is the difference between high availability and fault tolerance, and how do you build each on AWS?”
Model answer: High availability means the service recovers quickly from a failure with little downtime. Fault tolerance means users see no interruption at all, which needs full redundancy and costs more. Design for failure at three levels: instance (Auto Scaling group replaces bad instances), Availability Zone (spread across at least two AZs behind a load balancer, Multi-AZ databases) and Region (replication plus DNS failover for disaster recovery).
flowchart TD
U["Users"] --> R["Route 53"]
R --> L["Load balancer across AZs"]
L --> A["AZ-a: Auto Scaling group"]
L --> B["AZ-b: Auto Scaling group"]
A --> D["Database: primary + standby"]
B --> D
Common mistake: putting all instances in one AZ and calling it highly available.
11.2.2 Auto Scaling in depth
Question: “Explain how an Auto Scaling group decides to add and remove instances, and how you stop it from flapping.”
Model answer: An ASG has a launch template (AMI, type, security groups, role, user data), min, desired and max capacity, subnets in several AZs, and health checks (EC2 status, or ELB health to also replace instances failing the app check). Scaling policies:
| Policy | How it works | When |
|---|---|---|
| Target tracking | Keep a metric near a target (CPU 50 percent, request count per target) | Default choice |
| Step scaling | Different adjustments for different alarm breach sizes | Spiky load |
| Simple scaling | One adjustment, then wait for cooldown | Legacy |
| Scheduled | Change capacity at set times | Predictable peaks |
| Predictive | Forecasts from history | Regular daily patterns |
To stop flapping use cooldowns or instance warm-up, scale out fast and scale in slowly, and use lifecycle hooks to finish work (drain connections, upload logs) before termination. Instance refresh rolls out a new launch template gradually. Warm pools keep pre-initialised instances for slow-booting apps. Enable connection draining (deregistration delay) on the target group.
Common mistake: scaling on CPU for an app that is memory- or queue-bound. Scale workers on queue depth per instance.
11.2.3 Load balancers
Question: “ALB, NLB or Gateway Load Balancer?”
| ALB | NLB | GWLB | |
|---|---|---|---|
| Layer | 7 (HTTP/HTTPS, gRPC) | 4 (TCP, UDP, TLS) | 3 (IP) |
| Strengths | Path and host routing, redirects, auth, WAF | Very high throughput, static IP per AZ, preserves source IP | Insert third-party firewalls and inspection appliances |
Model answer: Pick ALB for web apps and microservices routing by path or host; NLB for extreme performance, non-HTTP protocols or fixed IPs (and PrivateLink services); GWLB for traffic inspection. Health checks decide which targets receive traffic; cross-zone load balancing spreads evenly across AZs (on by default for ALB). Sticky sessions pin a client to a target; connection draining protects in-flight requests.
Go deeper: a 502 means the target returned an invalid response, 503 no healthy targets, 504 target timeout (check idle timeout and keep-alive). Use access logs (S3) and target response time metrics.
Common mistake: security group on targets not allowing the load balancer’s security group.
11.2.4 Caching and loose coupling
Question: “How do you make a read-heavy, spiky system more reliable and cheaper?”
Model answer: Cache at every layer: CloudFront at the edge, ElastiCache (Redis or Memcached) in front of the database, DAX for DynamoDB, and application-level caches. Redis adds persistence, replication, Multi-AZ failover and data structures; Memcached is simple and multi-threaded. Decouple producers and consumers with SQS (queue, buffering, retries, dead-letter queue), SNS (fan-out), EventBridge (event routing) and Step Functions (workflows). A queue lets the consumer scale on backlog and absorbs bursts instead of failing.
Go deeper
- SQS standard (at-least-once, best-effort order) vs FIFO (exactly-once processing, ordered per message group, lower throughput). Visibility timeout must exceed processing time; long polling reduces cost and empty receives.
- Cache strategies: lazy loading (cache-aside), write-through, TTL; invalidate carefully.
- Use abstraction: DNS names, load balancers and service discovery instead of hard-coded IPs.
Common mistake: a retry storm with no dead-letter queue and no backoff.
11.2.5 Data stores and replication
Question: “Multi-AZ or read replica? And how do DynamoDB and Aurora differ for resilience?”
| RDS Multi-AZ | Read replica | |
|---|---|---|
| Purpose | Availability, automatic failover | Read scaling, cross-Region DR |
| Replication | Synchronous to a standby (not readable in classic Multi-AZ) | Asynchronous |
| Failover | Automatic, DNS endpoint flips | Manual promotion |
Model answer: Use Multi-AZ for production availability and replicas to offload reads or place data in another Region. Aurora stores six copies across three AZs, fails over in tens of seconds, supports up to 15 replicas and Global Database (typical replication under a second) for cross-Region DR; Aurora Serverless v2 scales capacity automatically. DynamoDB replicates across three AZs automatically; add global tables for multi-Region active-active, point-in-time recovery (35 days) and on-demand backups, and pick on-demand or provisioned capacity with auto scaling.
Common mistake: assuming Multi-AZ is a backup. It copies mistakes (a dropped table) instantly.
11.2.6 Networking for availability
Question: “Design a VPC that survives an AZ outage.”
Model answer: Use at least two AZs with a public and a private subnet in each; put load balancers in public subnets, application and database tiers in private ones. Deploy one NAT gateway per AZ and route each private subnet to its own, so an AZ failure does not remove egress for the others. Use Route 53 health checks and failover for external endpoints. Interface endpoints and gateway endpoints (S3, DynamoDB) reduce dependence on NAT.
Common mistake: a single NAT gateway in one AZ serving every private subnet.
11.2.7 Disaster recovery strategies
Question: “Explain RPO, RTO and the four DR strategies.”
Model answer: RPO is how much data you can lose (time since the last good copy). RTO is how long you can be down. The strategies trade cost for speed:
flowchart TD
A["Backup and restore: hours, cheapest"] --> B["Pilot light: core data live, compute off"]
B --> C["Warm standby: scaled-down full copy running"]
C --> D["Multi-site active-active: near-zero RTO, highest cost"]
| Strategy | RTO | What runs in the DR Region |
|---|---|---|
| Backup and restore | Hours | Only backups |
| Pilot light | Tens of minutes | Replicated data stores, servers off |
| Warm standby | Minutes | Everything, at reduced size |
| Active-active | Near zero | Everything serving traffic |
Tools: AWS Backup (policies, vaults, cross-Region and cross-account copies, Vault Lock for immutability), S3 Cross-Region Replication, RDS and Aurora cross-Region replicas, DynamoDB global tables, AWS Elastic Disaster Recovery for servers, AMI and EBS snapshot copies, CloudFormation to rebuild infrastructure, Route 53 failover.
Go deeper
- Test restores regularly and document runbooks; an untested backup is only a hope.
- EBS snapshots are incremental and stored in S3 (managed by AWS); use Data Lifecycle Manager to schedule them.
Common mistake: forgetting to replicate encryption keys (KMS keys are Regional) and secrets for the DR Region.
11.2.8 S3 resilience: versioning, lifecycle, replication
Question: “How do you protect S3 data from deletion and Region loss?”
Model answer: Turn on versioning (overwrites and deletes create versions or delete markers), add MFA Delete for the strictest protection, and use Object Lock (governance or compliance mode) for WORM retention. Add lifecycle rules to transition older versions to cheaper classes and expire them. For another Region or account use Cross-Region Replication (or same-Region replication); both buckets need versioning, S3 needs an IAM role, and replication is not retroactive by default (use S3 Batch Replication for existing objects). Delete markers are not replicated by default.
Common mistake: assuming turning on replication copies existing objects.
Domain 3: Deployment, provisioning and automation
11.3.1 Infrastructure as code with CloudFormation
Question: “How does CloudFormation work, and how do you update production safely?”
flowchart TD
T["Template (YAML/JSON)"] --> S["Create stack"]
S --> R["Resources created in dependency order"]
U["Change set"] --> V["Review changes"]
V --> W["Execute update"]
W --> X{"Failure?"}
X -- yes --> Y["Automatic rollback"]
Model answer: A template declares resources; a stack is the deployed instance. For updates, create a change set first to preview adds, modifications and replacements. Replacements (new physical ID) are the risk for stateful resources, so protect them with DeletionPolicy: Retain or Snapshot, and stack policies that block updates to critical resources. Failed creates roll back automatically; UPDATE_ROLLBACK_FAILED needs the cause fixed and a continue-rollback.
Go deeper
- Template sections: Parameters, Mappings, Conditions, Resources, Outputs, plus
Fn::Ref,GetAtt,Sub,ImportValueand exports. DependsOn, wait conditions and creation policies withcfn-signalensure instances finish bootstrapping.- Drift detection finds resources changed outside CloudFormation. Nested stacks and cross-stack references keep templates modular. Helper scripts (
cfn-init,cfn-hup) configure instances. - Rollback triggers use CloudWatch alarms during an update.
Common mistake: editing resources by hand in the console and then wondering why the next update fails.
11.3.2 StackSets and Resource Access Manager
Question: “You must deploy the same baseline to 40 accounts in three Regions. How?”
Model answer: CloudFormation StackSets: one administrator account (or a delegated administrator) deploys a template to many accounts and Regions. With service-managed permissions and AWS Organizations, new accounts in an organisational unit receive stacks automatically. Control rollout with maximum concurrent accounts and failure tolerance so one bad Region stops the rest; use drift detection to verify consistency. Resource Access Manager (RAM) shares resources (subnets, Transit Gateway, Route 53 resolver rules, License Manager configs) across accounts without copying them, centralising ownership and cost tracking.
Common mistake: not checking that the execution roles exist when using self-managed permissions.
11.3.3 Golden images and configuration management
Question: “How do you keep instances consistent and patched?”
Model answer: Bake AMIs with EC2 Image Builder (pipeline: source image, components, tests, distribution to Regions and accounts, scheduled rebuilds) and launch from them for faster starts and fewer surprises; keep user data for light, last-mile config. For fleets, Systems Manager provides Patch Manager (baselines, maintenance windows), State Manager (desired configuration), Run Command, Inventory and Session Manager. Immutable infrastructure beats patching in place: replace instances with a new AMI via Auto Scaling instance refresh.
Common mistake: snapshotting a hand-configured server and calling that a pipeline.
11.3.4 Deployment strategies
Question: “Compare recreate, rolling, blue-green and canary. Which would you choose?”
| Strategy | How | Downtime | Rollback | Cost |
|---|---|---|---|---|
| Recreate | Stop old, start new | Yes | Redeploy | Lowest |
| Rolling | Replace in batches | None, reduced capacity | Slow | Low |
| Blue-green | Full new environment, switch traffic | None | Instant switch back | Highest (two environments) |
| Canary | Small percentage first, then all | None | Fast | Medium |
| A/B test | Split users to compare features | None | Flag off | Medium |
Model answer: Blue-green when instant rollback matters and you can afford duplicates; canary when you want real-traffic evidence with limited blast radius; rolling for simple stateless services; recreate only for non-production or when two versions cannot coexist (schema changes). Always pair with health checks, alarms and automatic rollback.
Common mistake: a database migration that breaks the old version, which makes blue-green rollback impossible. Use backward-compatible (expand and contract) migrations.
11.3.5 CodeDeploy on EC2, Lambda and ECS
Question: “How does CodeDeploy deploy to each compute type?”
| Target | Options |
|---|---|
| EC2 or on-premises | In-place (take instances out in batches, run lifecycle hooks) or blue-green (new ASG, shift load balancer) |
| Lambda | All-at-once, linear (10 percent every N minutes) or canary (10 percent, then the rest after N minutes), using alias weights |
| ECS | Blue-green with two target groups, canary or linear shifts, test listener, rollback on alarms |
Model answer: An appspec file defines the files and lifecycle hooks (BeforeInstall, AfterInstall, ApplicationStart, ValidateService). For Lambda, hooks are BeforeAllowTraffic and AfterAllowTraffic functions that run tests. Attach CloudWatch alarms so a bad metric triggers automatic rollback. For pipelines, CodePipeline orchestrates source, build (CodeBuild) and deploy stages, with manual approval gates where needed.
Common mistake: CodeDeploy agent not installed or the instance role lacking S3 access to the artifact.
11.3.6 Common deployment failures and how to diagnose
Question: “A deployment fails or the new version is unhealthy. Walk me through diagnosis.”
flowchart TD
A["Deployment fails"] --> B["Read events: CodeDeploy / CloudFormation"]
B --> C["Check logs: CloudWatch, agent logs"]
C --> D{"Cause?"}
D --> E["Drift or config difference"]
D --> F["Dependency or version conflict"]
D --> G["Health check, network or IAM"]
D --> H["Capacity or quota"]
E --> I["Roll back, fix, redeploy"]
F --> I
G --> I
H --> I
Model answer: Start with the failure events, then logs, then metrics and traces (X-Ray for distributed tracing, Container Insights for containers). Typical causes: configuration drift between environments, dependency conflicts (pin versions, use lock files and build once, deploy the same artifact everywhere), health checks pointing at the wrong port or path, security groups, NACLs or routes blocking traffic, missing IAM permissions, exhausted service quotas or capacity, and traffic spikes arriving mid-deploy. Roll back first, then fix forward. Prevent with automated tests, canary stages, SLO alarms and infrastructure as code.
Common mistake: debugging forward under pressure instead of rolling back to the last known good version.
Domain 4: Security and compliance
11.4.1 Shared responsibility
Question: “Who is responsible for what in AWS?”
Model answer: AWS is responsible for security of the cloud: facilities, hardware, hypervisor, managed-service infrastructure. You are responsible for security in the cloud: IAM, data, encryption choices, network configuration, guest OS patching on EC2, application code. The split moves with the service: on EC2 you patch the OS; on RDS AWS patches the engine and OS but you control parameters, access and encryption; on Lambda you own only code and permissions. AWS Artifact gives you AWS compliance reports and agreements.
Common mistake: assuming a managed service means AWS handles your access policies.
11.4.2 IAM fundamentals
Question: “Explain users, groups, roles and how you apply least privilege.”
Model answer: Users are long-lived identities, groups collect users for shared permissions, and roles are assumed for temporary credentials (STS). Prefer roles for workloads (instance profiles, task roles, Lambda execution roles, IRSA for Kubernetes pods) and for cross-account access, so no access keys sit in code. Protect the root user: MFA, no access keys, use only for the few root-only tasks. Enforce MFA, a password policy (length, complexity, rotation, reuse), and prefer IAM Identity Center for workforce SSO across accounts. MFA types: virtual authenticator apps, passkeys, FIDO security keys, hardware tokens.
Start from the minimum, then widen: use Access Advisor (last-accessed data) to remove unused permissions and IAM Access Analyzer to generate policies from CloudTrail activity.
Common mistake: attaching AdministratorAccess to get something working and never removing it.
11.4.3 Policy evaluation: identity, boundaries, SCPs
Question: “A user has an allow policy but still gets access denied. Why?”
flowchart TD
A["Request"] --> B{"Explicit deny anywhere?"}
B -- yes --> X["DENIED"]
B -- no --> C{"SCP allows?"}
C -- no --> X
C -- yes --> D{"Permissions boundary allows?"}
D -- no --> X
D -- yes --> E{"Identity or resource policy allows?"}
E -- no --> X
E -- yes --> OK["ALLOWED"]
Model answer: The default is deny; an explicit deny always wins. Effective permission is the intersection of: SCPs (organisation guardrails, never grant anything, also bind the member account root user), permissions boundaries (maximum set for a user or role), session policies, and the identity-based policies, plus resource-based policies where they apply. Within one account a resource policy can grant access on its own; across accounts both sides must allow. Policy anatomy: Version, Statement with Effect, Action, Resource, optional Principal and Condition (for example aws:SourceIp, aws:MultiFactorAuthPresent, aws:RequestedRegion, tag-based aws:ResourceTag). Troubleshoot with the IAM Policy Simulator, CloudTrail’s error message and Access Analyzer.
Go deeper
- Permissions boundaries let you delegate role creation safely: developers create roles but cannot exceed the boundary.
- Organisation policy types: SCPs, tag policies, backup policies, AI services opt-out policies.
- Managed policies: AWS managed (maintained by AWS) vs customer managed (versioned, reusable, yours).
Common mistake: believing an SCP can grant permissions. It only limits.
11.4.4 Multi-account strategy
Question: “How do you structure and secure many AWS accounts?”
Model answer: Use AWS Organizations with a simple, flat OU structure (security, infrastructure, workloads by environment, sandbox), consolidated billing, SCPs and organisation-wide CloudTrail. Control Tower sets up a landing zone with an automated multi-account baseline, an Account Factory for vending new accounts, and guardrails: preventive (SCP-based, block the action, such as public S3 or leaving a Region) and detective (Config rules, report non-compliance). Separate log archive and security tooling accounts, use IAM Identity Center for access, and keep a break-glass procedure for emergencies.
Common mistake: deep OU nesting that makes SCP inheritance impossible to reason about.
11.4.5 Encryption: KMS, in transit and at rest
Question: “How does KMS envelope encryption work, and how do you encrypt data at rest and in transit?”
flowchart TD
A["App asks KMS: GenerateDataKey"] --> B["KMS returns plaintext key + encrypted key"]
B --> C["App encrypts data with plaintext key"]
C --> D["App discards plaintext key"]
D --> E["Store ciphertext + encrypted key"]
Model answer: KMS keys never leave KMS; direct encryption is limited to 4 KB, so large data uses envelope encryption: KMS issues a data key, you encrypt locally and store the encrypted data key beside the data. Key types: AWS owned, AWS managed (aws/s3, rotated yearly automatically), and customer managed (your policy, optional automatic rotation, aliases, can be disabled or scheduled for deletion with a 7 to 30 day wait). A key policy is the primary control; grants and IAM policies layer on top. Keys are Regional (multi-Region keys exist). CloudHSM gives dedicated, single-tenant hardware modules when you must control the HSM.
At rest: S3 SSE-S3, SSE-KMS or SSE-C, EBS, RDS, EFS and DynamoDB encryption with KMS. In transit: TLS everywhere, certificates from ACM on load balancers and CloudFront (ACM certificates auto-renew), SSL for RDS connections, and aws:SecureTransport bucket policy conditions to refuse plain HTTP.
Go deeper: SSE-KMS adds KMS request costs and quotas; S3 Bucket Keys cut that. Encrypted snapshots can only be shared if the key is shared too.
Common mistake: deleting a KMS key. Data encrypted with it becomes unrecoverable, so disable first.
11.4.6 Secrets: Secrets Manager vs Parameter Store
Question: “Where do you keep database passwords and how do you rotate them?”
| Secrets Manager | Parameter Store | |
|---|---|---|
| Purpose | Secrets, with managed rotation | Configuration and simple secrets |
| Rotation | Built-in via Lambda (RDS has ready-made) | Not built in |
| Cost | Per secret per month plus API calls | Standard tier free |
| Versioning | AWSCURRENT, AWSPREVIOUS stages | Parameter history |
| Types | Key-value JSON | String, StringList, SecureString (KMS) |
Model answer: Use Secrets Manager when you need automatic rotation, cross-account sharing or replication across Regions. Use Parameter Store SecureString for cheap hierarchical config (/app/env/key). Applications fetch at runtime with an IAM role, never from source control or AMIs.
Common mistake: putting secrets in environment variables baked into images or CloudFormation parameters in plain text.
11.4.7 Detection and compliance services
Question: “Which service answers which security question?”
| Service | Question it answers |
|---|---|
| GuardDuty | Is there malicious or anomalous activity? (analyses CloudTrail, VPC Flow Logs, DNS logs, EKS, S3 events) |
| Inspector | Which workloads have known vulnerabilities? (EC2, ECR images, Lambda; continuous) |
| Macie | Is there sensitive data (PII) in S3? |
| Security Hub | One place to see and prioritise findings across accounts and Regions (aggregates the services above, standards checks) |
| Config | What changed, and is it compliant? (history, rules, remediation, conformance packs) |
| Audit Manager | Can I produce audit evidence continuously? |
| Trusted Advisor | Are there best-practice gaps across cost, performance, security, fault tolerance, service limits? |
| Detective | What is the root cause of this finding? |
| CloudTrail | Who made which API call? |
Model answer: A finding flows from a detector (GuardDuty, Inspector, Macie, Config) to Security Hub, then through EventBridge to a remediation Lambda or Systems Manager runbook and a notification. Config rules can evaluate in detective mode (after change) or proactive mode (before provisioning, in CloudFormation hooks).
Common mistake: enabling these only in one Region or one account. Use a delegated administrator and organisation-wide enablement.
11.4.8 Network defence in depth
Question: “Security groups vs NACLs, and what else protects a web app?”
| Security group | NACL | |
|---|---|---|
| Level | Instance or ENI | Subnet |
| State | Stateful (return traffic automatic) | Stateless (allow both directions, including ephemeral ports) |
| Rules | Allow only | Allow and deny, numbered, evaluated in order |
Model answer: Layer defences: edge (CloudFront, WAF, Shield), network (NACLs, security groups, Network Firewall, private subnets), identity (IAM), data (encryption), and monitoring (VPC Flow Logs, GuardDuty). WAF attaches to CloudFront, ALB, API Gateway, AppSync and Cognito; it uses web ACLs with managed rule groups, rate-based rules, geo and IP match. Shield Standard is automatic and free for common DDoS; Shield Advanced adds response team support, cost protection and extra detection. Network Firewall offers stateless and stateful rule engines, domain filtering and Suricata-compatible rules, deployed in dedicated subnets with route-table steering; Firewall Manager applies WAF, Shield and security group policy across an organisation.
Common mistake: referencing CIDR blocks in security groups where referencing another security group would be tighter and self-maintaining.
Domain 5: Networking and content delivery
11.5.1 VPC building blocks and internet access
Question: “What makes a subnet public, and how do private instances reach the internet?”
flowchart TD
I["Internet"] --> G["Internet Gateway"]
G --> P["Public subnet: route 0.0.0.0/0 to IGW"]
P --> N["NAT gateway (with Elastic IP)"]
N --> Q["Private subnet: route 0.0.0.0/0 to NAT"]
Model answer: A subnet is public when its route table sends 0.0.0.0/0 to an Internet Gateway; instances there also need a public or Elastic IP, and security groups and NACLs must allow the traffic. Private instances use a NAT gateway that sits in a public subnet (private subnets route 0.0.0.0/0 to it); it allows outbound-initiated traffic only. For IPv6, an egress-only internet gateway does the same job. The main route table has a local route for the VPC CIDR; the most specific route wins. Reserved addresses: five per subnet.
Go deeper: NAT gateways are AZ-scoped, auto-scale to 100 Gbps and bill per hour plus per GB; use VPC endpoints to keep AWS traffic off them.
Common mistake: putting the NAT gateway in a private subnet.
11.5.2 Troubleshooting connectivity
Question: “An EC2 instance cannot reach the internet or a service. How do you troubleshoot?”
Model answer: Work outward in order: (1) public IP or Elastic IP present, (2) subnet route table has the right default route and target, (3) IGW exists and is attached, or NAT gateway is in a public subnet with its own route to the IGW, (4) security group outbound rules, (5) NACL inbound and outbound including ephemeral ports 1024 to 65535 (stateless), (6) OS firewall, DNS resolution (VPC DNS settings), and the destination’s own rules. Use VPC Reachability Analyzer to check a path without sending traffic, and VPC Flow Logs to see ACCEPT or REJECT. A REJECT with matching inbound ACCEPT often points at a NACL return-path problem.
| Misconfiguration | Result |
|---|---|
| No public IP | No direct internet |
| Missing 0.0.0.0/0 route | No egress |
| NACL missing return rule | Return packets dropped |
| NAT gateway in private subnet | No egress for anyone |
| Overlapping CIDRs between VPCs | Peering impossible |
Common mistake: editing security groups when the NACL is the blocker.
11.5.3 Private access and managing instances without SSH
Question: “How do you reach AWS services privately and administer servers without opening port 22?”
Model answer: Gateway endpoints (S3 and DynamoDB; free; added as a route table target) and interface endpoints (powered by PrivateLink; an ENI with a private IP in your subnet, billed hourly and per GB; works for most services, partner services and your own NLB-fronted services). Endpoint policies restrict what can be done; enable private DNS so default service names resolve to the endpoint. For administration use Systems Manager Session Manager: install the SSM Agent, attach a role with AmazonSSMManagedInstanceCore, ensure outbound HTTPS (443) to the SSM endpoints (via NAT or VPC endpoints ssm, ssmmessages, ec2messages), and grant users IAM permission to ssm:StartSession. No inbound ports, no key pairs, and sessions can be logged to S3 or CloudWatch Logs.
Common mistake: instances in a private subnet with no route and no endpoints for Systems Manager, so they never register.
11.5.4 Connecting networks: peering, Transit Gateway, VPN, Direct Connect
Question: “How do you connect ten VPCs and an office or data centre?”
flowchart TD
A["VPC A"] --> T["Transit Gateway"]
B["VPC B"] --> T
C["VPC C"] --> T
T --> V["Site-to-Site VPN"]
T --> D["Direct Connect"]
V --> O["On-premises"]
D --> O
| Option | Use | Limits |
|---|---|---|
| VPC peering | A few VPCs, same or different account or Region | One-to-one, not transitive, no overlapping CIDRs, update route tables both sides |
| Transit Gateway | Hub and spoke for many VPCs and on-premises, route tables per attachment | Hourly and per GB cost |
| Site-to-Site VPN | Quick encrypted link over the internet | Two tunnels for redundancy, about 1.25 Gbps per tunnel, internet latency |
| Client VPN | Individual remote users | Managed OpenVPN endpoint |
| Direct Connect | Dedicated private link, consistent latency, high throughput | Weeks to provision; not encrypted by default |
Model answer: Transit Gateway for scale and on-premises integration; peering for simple pairs. Direct Connect uses virtual interfaces: private (to a VPC), transit (to a Transit Gateway), public (to AWS public services). For resilience use two connections at different locations, or Direct Connect with a VPN as backup, and BGP for routing. Common issues: overlapping CIDRs, missing return routes, asymmetric routing, tunnel down because of mismatched settings, and bandwidth limits.
Common mistake: expecting peering to forward traffic through one VPC to another.
11.5.5 DNS with Route 53
Question: “Which Route 53 routing policy fits which requirement?”
| Policy | Use |
|---|---|
| Simple | One resource, or several values returned randomly |
| Weighted | Gradual migration, canary, A/B (percentage split) |
| Latency | Send users to the Region with the lowest latency |
| Failover | Active-passive with health checks |
| Geolocation | Route by user location (compliance, language) |
| Geoproximity | Route by distance with adjustable bias (Traffic Flow) |
| IP-based | Route by client CIDR ranges |
| Multivalue answer | Up to eight healthy records returned, basic client-side balancing |
Model answer: A hosted zone holds records: public for the internet, private associated with VPCs for internal names. Records: A, AAAA, CNAME (not allowed at the zone apex), MX, TXT, NS, and Alias (free for queries to AWS targets, works at the apex, follows target IP changes). Health checks monitor endpoints, other health checks or CloudWatch alarms; they drive failover and can raise alarms. Route 53 Resolver handles VPC DNS: inbound endpoints let on-premises resolvers query AWS private zones; outbound endpoints with forwarding rules send chosen domains (conditional forwarding) to on-premises DNS. Endpoints sit in at least two subnets and security groups must allow port 53.
Go deeper: TTL controls caching: lower it before a planned migration; resolver rules can be shared with RAM.
Common mistake: using a CNAME for the root domain instead of an Alias.
11.5.6 CloudFront and S3 static sites
Question: “How does CloudFront speed up and protect content, and why might users see stale content?”
Model answer: CloudFront caches content at edge locations; a request that misses goes to the origin (S3, ALB, custom). The cache key (path plus the headers, cookies and query strings you include) decides whether a request counts as the same object; keep it minimal for a better hit ratio. Cache policies and origin request policies control what is cached and what is forwarded. Default TTL is 24 hours. For stale content: shorten TTL, use versioned file names (cache busting), or create an invalidation (path-based, billed beyond the free allowance). Security: force HTTPS, Origin Access Control so only CloudFront can read a private S3 bucket, signed URLs (one file) and signed cookies (many files), geographic restrictions, field-level encryption and WAF. Also consider Lambda@Edge or CloudFront Functions for request rewriting.
S3 static hosting: create a bucket, upload index.html and error document, enable website hosting, allow public read (or better, keep it private behind CloudFront), and point a Route 53 Alias at the distribution. The website endpoint serves HTTP only, so use CloudFront for HTTPS.
Common mistake: forwarding all cookies and headers to the origin, which destroys the cache hit ratio.
11.5.7 Network logs: which log answers what
Question: “Where do you look for network and access problems?”
| Log | Use |
|---|---|
| VPC Flow Logs | Accepted and rejected traffic by ENI, subnet or VPC (metadata only, not payload) |
| CloudTrail | API calls that changed networking |
| Route 53 query logs | DNS troubleshooting and security |
| Resolver query logs | VPC DNS activity |
| ELB access logs | Client IPs, latency, status codes |
| CloudFront logs | Viewer requests and cache results |
| WAF logs | Which rule blocked which request |
| Transit Gateway flow logs | Traffic between attachments |
| Global Accelerator flow logs | Traffic through accelerator |
Model answer: Send them to S3 or CloudWatch Logs and query with Athena or Logs Insights. A flow log record shows source, destination, ports, protocol, bytes, action and log status; use it to prove a NACL or security group is dropping traffic.
Common mistake: expecting flow logs to show packet contents.
Domain 6: Cost and performance optimization
11.6.1 Cost visibility and governance
Question: “Your AWS bill jumped 40 percent. What do you do?”
flowchart TD
A["Alert: Budget or anomaly"] --> B["Cost Explorer: group by service, account, tag"]
B --> C["Find the driver"]
C --> D["Fix: right-size, delete, schedule, commit"]
D --> E["Prevent: budgets, tag policy, anomaly detection"]
Model answer: Open Cost Explorer and group by service, linked account, Region, usage type and cost allocation tag to find what moved. Use the Cost and Usage Report (to S3, queried with Athena) for line-item detail, Cost Anomaly Detection for automatic alerts, and Budgets for thresholds on cost, usage, Reserved Instance or Savings Plan coverage and utilisation; budgets can notify through SNS and run actions (apply an IAM or SCP policy, stop instances). Billing alarms in CloudWatch (billing metric in us-east-1, after enabling alerts) are the simpler option. Typical culprits: NAT gateway data, cross-AZ and internet data transfer, idle or oversized instances, unattached EBS volumes and old snapshots, forgotten load balancers, log retention, and noisy data events.
Go deeper: a tagging strategy (lowercase, consistent keys such as environment, owner, cost-center, application) enforced with tag policies and SCP or IAM conditions; tags must be activated as cost allocation tags in the billing console before they appear in reports. FinOps loop: inform, optimise, operate.
Common mistake: tagging after the bill arrives. Tags only apply going forward.
11.6.2 Right-sizing and finding waste
Question: “How do you find idle or oversized resources?”
Model answer: Compute Optimizer analyses historical utilisation (needs the CloudWatch Agent for memory data) and recommends sizes for EC2, Auto Scaling groups, EBS, Lambda and Fargate. Trusted Advisor (full checks on Business and Enterprise support) flags idle load balancers, low-utilisation instances, unassociated Elastic IPs and unattached volumes. Then act: downsize, schedule non-production off-hours (Instance Scheduler, scheduled scaling), delete orphaned resources, move to Graviton, switch gp2 to gp3, and add S3 lifecycle and Intelligent-Tiering.
Common mistake: right-sizing on average CPU only. Check peaks, memory, network and storage too.
11.6.3 Purchasing options
Question: “On-Demand, Reserved, Savings Plans or Spot?”
| Option | Discount | Commitment | Use |
|---|---|---|---|
| On-Demand | None | None | Unpredictable or short-term |
| Compute Savings Plan | Up to about 66 percent | 1 or 3 years of spend per hour | Flexible across EC2, Fargate, Lambda |
| EC2 Instance Savings Plan | Up to about 72 percent | Family in a Region | Stable fleets |
| Reserved Instances | Up to about 72 percent | 1 or 3 years, standard or convertible | Steady usage, also RDS and others |
| Spot | Up to about 90 percent | None; can be reclaimed with a two-minute warning | Fault-tolerant work |
| Dedicated Hosts | n/a | Optional | Licensing or compliance |
Model answer: Cover the steady baseline with Savings Plans or Reserved Instances, scale the variable part with On-Demand, and run interruptible work on Spot. Good Spot workloads: stateless web tiers behind load balancers, batch and data processing, CI runners, containers workers, big-data jobs. Poor fits: databases, long non-checkpointed jobs, single-instance critical services. Make Spot safe with diversified instance types and AZs, the capacity-optimised allocation strategy, Capacity Rebalancing, handling the interruption notice (metadata or EventBridge), checkpointing, and a mixed-instances ASG with an On-Demand base.
Common mistake: committing to a three-year plan before right-sizing.
11.6.4 Managed vs self-managed services
Question: “Should we run our own database on EC2 or use RDS?”
| Managed (RDS, DynamoDB, Aurora, ECS Fargate, Lambda) | Self-managed (on EC2) | |
|---|---|---|
| Operations | AWS handles patching, backups, failover | You do |
| Control | Limited (no OS access, parameter limits) | Full |
| Setup time | Minutes | Longer |
| Unit price | Higher per hour | Lower compute price, higher people cost |
| Security duty | Shared, less for you | Yours entirely |
Model answer: Compare total cost of ownership, not instance price: staff time, on-call burden, backup and failover engineering, compliance, and speed of delivery. Choose managed by default; choose self-managed for features the service lacks, unusual versions, deep tuning, licensing constraints or very large steady scale where the economics flip. Monitoring differs: with managed services you rely on CloudWatch, Performance Insights and enhanced monitoring, without host access.
Common mistake: counting only the instance bill and ignoring engineer time.
11.6.5 Performance quick reference
Question: “Give a quick checklist for improving performance.”
Model answer: Match instance family to workload; use placement groups for HPC; enable enhanced networking; choose the right EBS type (gp3 default, io2 for guaranteed IOPS, st1 and sc1 for throughput and cold data); cache with CloudFront, ElastiCache and DAX; scale reads with replicas and Aurora; use RDS Proxy; use S3 multipart and Transfer Acceleration; put Global Accelerator in front of global TCP and UDP apps for static anycast IPs and faster failover; and measure with CloudWatch, X-Ray and Performance Insights before changing anything.
Common mistake: optimising before measuring.
One-minute recap
| Domain | Remember |
|---|---|
| Monitoring | CloudWatch = performance, CloudTrail = audit, Config = configuration. Alarm, then EventBridge, then automate. |
| Reliability | Multi-AZ always, one NAT per AZ, decouple with queues, pick DR by RTO and RPO, test restores. |
| Deployment | IaC with change sets, bake AMIs, blue-green or canary with alarms and auto rollback. |
| Security | Deny wins, SCP limits, roles over keys, envelope encryption, defence in depth, organisation-wide detection. |
| Networking | Routes decide public or private, NACLs are stateless, Transit Gateway scales, Alias at the apex, cache keys matter. |
| Cost | Tag, measure, right-size, commit to the baseline, Spot for the flexible, managed by default. |
How to answer scenario questions
- Clarify the requirement (availability, latency, cost, compliance).
- Detect: which metric, log or alarm shows it?
- Diagnose from the outside in: DNS, network, security, compute, data.
- Fix with the least risky, reversible change.
- Prevent with automation, alarms and infrastructure as code.
12. AWS DevOps Engineer Professional: interview questions and flows
This chapter goes a level above chapter 11. The questions are about designing and operating delivery and resilience at scale: multi-account pipelines, safe rollouts, infrastructure as code with guardrails, automated incident response and security automation. Answer them in the order requirement, design, safeguards, how you detect failure, how you roll back.
Domain 1: SDLC automation
12.1.1 Designing a CI/CD pipeline
Question: “Design a CI/CD pipeline for a containerised service on AWS. What are the stages and the quality gates?”
flowchart TD
A["Git push or PR"] --> B["Build: CodeBuild"]
B --> C["Unit tests + static analysis"]
C --> D["Publish: ECR image / S3 artifact"]
D --> E["Deploy to test, integration tests"]
E --> F["Manual approval"]
F --> G["Deploy to production: canary"]
G --> H{"Alarms OK?"}
H -- no --> I["Automatic rollback"]
Model answer: Small, frequent releases with checks at every stage, not only at the end. CodePipeline orchestrates source (GitHub, Bitbucket, GitLab through a connection; CodeCommit is closed to new customers, so use an external Git provider), CodeBuild builds and tests (a fully managed build fleet that scales per build), artifacts go to S3, ECR or CodeArtifact, and deployment uses CodeDeploy, CloudFormation or ECS. Put cheap, fast tests first (lint, unit, SAST), slower ones later (integration, DAST, load), then production with a canary and alarms wired to automatic rollback. Build once and promote the same artifact through environments; never rebuild per environment.
Go deeper
- Test types: static analysis (SAST) and unit tests run on every commit; integration and acceptance tests on a deployed test environment; performance tests before release; DAST against a running app.
- Run pipelines on pull requests for fast feedback; protect the main branch; use trunk-based or GitHub Flow for short-lived branches.
Common mistake: a pipeline that rebuilds the artifact for production, so what was tested is not what is deployed.
12.1.2 CodeBuild, buildspec and artifacts
Question: “How does buildspec.yml work and how do you make builds faster and safer?”
Model answer: A buildspec defines phases (install, pre_build, build, post_build), environment variables, artifacts (files to output) and cache paths. Make builds faster with a local or S3 cache for dependencies, a custom build image in ECR with tools pre-installed, and smaller compute types after measuring. Make them safer with a least-privilege service role, secrets from Secrets Manager or Parameter Store through env.secrets-manager and env.parameter-store (never plain environment variables), builds in a VPC only when they need private resources, and image scanning in ECR (basic or enhanced with Inspector). Logs and metrics go to CloudWatch; build reports capture test results.
Go deeper: a failing build with no artifacts usually means a wrong artifact path, a missing dependency, or a service role without access to S3, ECR or the secret. Lifecycle rules on S3 and ECR keep storage costs bounded.
Common mistake: putting a password in the buildspec file or in a CodeBuild environment variable of type PLAINTEXT.
12.1.3 Multi-account pipelines
Question: “How do you deploy from one pipeline into dev, test and production accounts?”
flowchart TD
T["Tools account: CodePipeline + artifact bucket + KMS key"] --> D["Dev account: deploy role"]
T --> Q["Test account: deploy role"]
T --> P["Prod account: deploy role"]
Model answer: Keep the pipeline in a central tools (CI/CD) account and deploy into workload accounts by assuming cross-account IAM roles that trust the pipeline’s role. The artifact bucket must allow the target accounts to read it, and artifacts must be encrypted with a customer managed KMS key whose key policy allows those accounts (the default aws/s3 key cannot be shared). Separate accounts isolate blast radius, give per-environment quotas and billing, and let SCPs enforce different guardrails. A single account is simpler and fine for small teams; multi-account is the right default for production.
Common mistake: using the AWS managed S3 key for artifacts, then wondering why the other account gets AccessDenied.
12.1.4 Deployment strategies per compute target
Question: “Compare in-place, blue-green, canary and linear for EC2, ECS and Lambda. How do you choose?”
| Target | Options | Rollback |
|---|---|---|
| EC2 and on-premises (CodeDeploy) | In-place: one-at-a-time, half, all-at-once. Blue-green: new Auto Scaling group, shift load balancer | In-place: redeploy previous. Blue-green: switch back |
| ECS | Rolling (circuit breaker), CodeDeploy blue-green with two target groups, canary or linear | Alarms or circuit breaker |
| Lambda | Alias weights: all-at-once, canary, linear | Point alias back |
| EKS | Rolling updates, or GitOps with Argo CD or Flux, progressive delivery with Argo Rollouts or Flagger | Revert Git |
Model answer: One-at-a-time has the smallest blast radius but is slow; batched is faster with partial validation; blue-green needs duplicate capacity but gives instant rollback and zero downtime; canary exposes a small percentage first; linear shifts evenly over time. For a multi-service application, deploy green fully, run smoke and integration tests on a test listener, then shift traffic (Route 53 weighted records, ALB target group weights, or CodeDeploy) while watching alarms; keep blue until confident. Use immutable deployments (new AMI or image, replace instances) for production because mutable patching and SSH fixes cause drift; reserve mutable changes for emergencies.
Go deeper
- Database changes must be backward compatible (expand and contract) or rollback fails.
- Enable the ECS deployment circuit breaker with rollback, and CodeDeploy alarm-based rollback.
- Lambda hooks (
BeforeAllowTraffic,AfterAllowTraffic) run validation.
Common mistake: a blue-green design whose database schema change makes the old version fail.
12.1.5 Secrets and artifacts
Question: “Where do you store secrets and build artifacts in a pipeline?”
Model answer: Secrets in Secrets Manager when you need automatic rotation, up to 64 KB and cross-Region replication; in Parameter Store for configuration and cheaper simple secrets (standard 4 KB, advanced 8 KB). Both encrypt with KMS and are audited by CloudTrail. Artifacts: S3 for packages and templates (versioning, lifecycle), ECR for images (scan on push, lifecycle policies, immutable tags, cross-Region replication), CodeArtifact for language packages (npm, pip, Maven) with upstream proxying, and AMIs for golden images. Retain only what you need and sign or scan before promotion.
Common mistake: using latest as an image tag, which makes rollbacks and audits impossible.
12.1.6 Troubleshooting a failed pipeline
Question: “A pipeline fails or does not start. How do you diagnose it stage by stage?”
| Stage | Typical cause |
|---|---|
| Source does not trigger | Webhook or connection broken, branch name mismatch (main vs master), repo permissions |
| Build fails | Buildspec syntax, wrong artifact path, missing dependency, role cannot read secret or S3 |
| Deploy fails | Wrong ECR tag, AppSpec or task definition mismatch, deployment role lacks permission, health check fails |
| Provisioning fails | CloudFormation error, quota exhausted, missing AMI or image |
Model answer: Open the failing action’s details, read the CodeBuild or CodeDeploy logs in CloudWatch, then CloudFormation events or ECS service events. Most failures are IAM, configuration or missing resource problems. Reproduce locally (CodeBuild local agent), fix, re-run, and add a test or lint rule so it cannot recur.
Common mistake: granting broad admin permissions to the pipeline role to make the error go away.
Domain 2: Configuration management and infrastructure as code
12.2.1 CloudFormation drift, stack policies and safe updates
Question: “Someone changed a resource by hand. How do you detect it and bring the stack back, and how do you protect critical resources?”
Model answer: Run drift detection on the stack. Each resource shows IN_SYNC, MODIFIED, DELETED or NOT_CHECKED. Decide per case: revert the change by updating the stack (a change set shows the effect first), or, if the manual change must stay, update the template to match. Prevent recurrence by removing console write access (SCPs, least privilege), running drift detection on a schedule, and using AWS Config to flag it. Stack policies deny updates to named resources (for example the production database) unless explicitly overridden; add DeletionPolicy: Retain or Snapshot and termination protection on the stack.
Common mistake: assuming drift detection covers every property of every resource.
12.2.2 Modular templates and cross-account rollout
Question: “How do you structure CloudFormation for many teams, accounts and Regions?”
| Technique | Use | Trade-off |
|---|---|---|
| Nested stacks | One application with several components, one lifecycle | Large parent, cascading updates |
Cross-stack references (export and ImportValue) |
Share outputs such as VPC IDs | Tight coupling, exports cannot change while imported |
| StackSets | Same stack in many accounts and Regions | Needs administrator and execution roles or service-managed permissions |
| Modules and registry | Reusable resource patterns | Governance of versions |
Model answer: Separate templates by lifecycle and ownership (network, shared services, application), pass values through parameters or SSM parameters to limit coupling, and use StackSets (service-managed with Organizations) for baselines such as logging, IAM roles and Config rules.
Common mistake: exporting everything and creating a dependency web that blocks updates.
12.2.3 Policy as code: cfn-lint, CloudFormation Guard and hooks
Question: “How do you stop insecure infrastructure from being deployed?”
flowchart TD
A["Pull request"] --> B["cfn-lint: syntax and resource rules"]
B --> C["CFN Guard: policy rules"]
C --> D["Change set review"]
D --> E["Deploy"]
E --> F["AWS Config detects drift and violations"]
Model answer: Shift left. cfn-lint catches syntax, property and type errors; CloudFormation Guard evaluates declarative rules (no public S3 buckets, required tags, encryption on) and fails the build; CloudFormation hooks and Config proactive rules evaluate before provisioning. Then detect after the fact with Config rules and Security Hub. Layer SCPs as the hard limit.
Common mistake: relying only on after-the-fact detection.
12.2.4 AWS CDK
Question: “What is the CDK and when would you choose it over plain CloudFormation?”
Model answer: The CDK defines infrastructure in a programming language (TypeScript, Python, Java, C#, Go) and synthesises CloudFormation templates. Constructs come in three levels: L1 (direct CloudFormation resources), L2 (sensible defaults and helper methods), L3 (patterns combining resources). Commands: cdk init, cdk bootstrap (creates the staging bucket and roles per account and Region), cdk synth, cdk diff, cdk deploy, cdk destroy. Choose CDK for loops, conditions, reuse and unit testing; choose plain templates for simplicity and when teams are not developers. CDK Pipelines gives a self-mutating pipeline.
Common mistake: forgetting to bootstrap each target account and Region.
12.2.5 Systems Manager, AppConfig and Service Catalog
Question: “Which tool for patching, which for feature flags, which for governed self-service?”
| Need | Service |
|---|---|
| Patch fleets on a schedule, baselines, compliance | Patch Manager with maintenance windows |
| Keep a desired state (agents, files, settings) | State Manager associations |
| Run commands or scripts at scale, no SSH | Run Command |
| Runbooks and remediation | Automation |
| Shell access without inbound ports | Session Manager |
| Track what is installed | Inventory |
| Configuration and secrets | Parameter Store |
| Dynamic app configuration, feature flags, gradual rollout with validators and automatic rollback | AppConfig |
| Approved, versioned products developers can launch themselves | Service Catalog (portfolios, products, launch constraints, tag options) |
Model answer: Use Parameter Store for static values read at start-up and AppConfig when behaviour must change at runtime without redeploying: it validates (JSON Schema or Lambda), deploys gradually with a strategy and rolls back on alarms. Service Catalog gives governance: users launch pre-approved CloudFormation or CDK products through a launch role, so they need no broad permissions.
Common mistake: pushing feature toggles through full deployments.
12.2.6 Landing zones and governance
Question: “How do Organizations, Control Tower and Config fit together?”
Model answer: Organizations gives OUs, SCPs and consolidated billing; Control Tower automates a landing zone (management, log archive and audit accounts), IAM Identity Center, an Account Factory for new accounts, and preventive (SCP) and detective (Config) guardrails; AWS Config records configuration, evaluates rules in detective or proactive mode, applies remediation through SSM Automation, packages rules as conformance packs and aggregates results across accounts with an aggregator. CloudTrail tells you who changed it; Config tells you what changed and whether it is compliant.
Common mistake: treating Config and CloudTrail as alternatives.
Domain 3: Resilient cloud solutions
12.3.1 Finding and removing single points of failure
Question: “Review this architecture for single points of failure.”
| Component | Weakness | Fix |
|---|---|---|
| One EC2 instance | Instance or OS failure | Auto Scaling group behind a load balancer, multiple AZs |
| RDS single instance | AZ or host failure | Multi-AZ, backups, replicas |
| One NAT gateway | AZ failure removes egress | One per AZ |
| S3 in one Region | Region-level loss | Cross-Region replication |
| Single Region | Regional outage | Multi-Region with DNS or Global Accelerator failover |
| Hard-coded IPs, one DNS provider, one certificate expiry | Hidden single points | Service discovery, monitoring, certificate renewal alarms |
Model answer: Ask of every component “what happens if this fails, and how large is the failure scope (instance, AZ, Region, account)?” Fix the biggest scope you can afford, measure with health checks, and test with failure injection. Match the investment to the RTO and RPO.
Common mistake: adding a second Region while the one NAT gateway in a single AZ remains.
12.3.2 RDS and DynamoDB availability
Question: “What are the HA and DR options for relational and NoSQL data?”
Model answer: RDS Multi-AZ uses synchronous replication and automatic failover (a Multi-AZ DB cluster adds two readable standbys); read replicas use asynchronous replication for read scaling and cross-Region copies, with possible lag and data loss if promoted; automated backups with point-in-time recovery and snapshots are the cold option. Aurora keeps six copies over three AZs and offers Global Database. DynamoDB is replicated over three AZs; global tables provide multi-Region active-active with last-writer-wins conflict resolution, PITR restores within the last 35 days, and transactions (ACID) cover multi-item changes. Design access patterns first, since DynamoDB schemas follow queries.
Common mistake: using a read replica as the high-availability mechanism.
12.3.3 Containers and serverless resilience
Question: “How do you make ECS, EKS and Lambda workloads resilient?”
Model answer: ECS: use Fargate to remove node management or EC2 with an Auto Scaling capacity provider; place tasks across AZs with spread placement; run as a service so failed tasks are replaced; use the deployment circuit breaker; health checks on the target group; autoscale on CPU, memory or requests per target. EKS: multi-AZ node groups, replicas with pod anti-affinity or topology spread, PodDisruptionBudgets, probes, Karpenter or Cluster Autoscaler, and sometimes multiple clusters. Lambda: versions and aliases with weighted shifting, retries and dead-letter queues or on-failure destinations (asynchronous), SQS decoupling, reserved concurrency to protect dependencies, provisioned concurrency for latency, and VPC configuration across multiple subnets. Idempotent handlers matter because delivery is at least once.
Common mistake: a multi-AZ cluster with all replicas of a service scheduled on one node.
12.3.4 Edge and global traffic management
Question: “Global Accelerator or Route 53 for regional failover? CloudFront Functions or Lambda@Edge?”
| Route 53 | Global Accelerator | |
|---|---|---|
| Works at | DNS (cached by clients, TTL delays) | Network layer with two static anycast IPs |
| Failover speed | Depends on TTL and health check interval | Fast, no DNS caching |
| Policies | Weighted, latency, geolocation, failover and more | Traffic dials, endpoint weights |
| Best for | Flexible DNS routing, any endpoint | TCP and UDP apps needing static IPs and quick failover |
| CloudFront Functions | Lambda@Edge | |
|---|---|---|
| Runtime | Lightweight JavaScript at the edge | Node.js or Python at regional edge caches |
| Triggers | Viewer request and response | Viewer and origin request and response |
| Use | Header changes, redirects, URL rewrites, simple auth | Heavier logic, network calls, body access |
Model answer: Use both together when needed: Global Accelerator for fast network-layer failover and static IPs, Route 53 for DNS policies and health-based failover. Choose CloudFront Functions for the cheapest, fastest simple manipulations and Lambda@Edge when you need more runtime, memory or network access.
Common mistake: relying on a long DNS TTL for failover.
12.3.5 Disaster recovery model selection
Question: “The business needs RTO of 30 minutes and RPO of 5 minutes. Which DR model?”
Model answer: Backup and restore (hours) and pilot light (tens of minutes) are too slow or marginal; warm standby (a scaled-down full environment with live replication) fits, scaling up on failover; active-active if the RTO must be near zero and the cost is justified. An RPO of 5 minutes needs continuous or near-continuous replication (Aurora Global Database, DynamoDB global tables, S3 replication with replication time control, database log shipping), not nightly backups. Automate failover with Route 53 or Global Accelerator, rehearse with game days, and keep infrastructure as code so the standby is reproducible.
Common mistake: choosing a model by cost alone and never testing the failover.
Domain 4: Monitoring and logging
12.4.1 Metrics, logs and traces
Question: “What are the three pillars of observability and how do they work together during an incident?”
flowchart TD
A["Metric alarm: what is wrong"] --> B["Trace: where is the latency or error"]
B --> C["Logs: why did it happen"]
C --> D["Fix, then add a better alarm"]
Model answer: Metrics say what is happening (numeric time series), traces say where in a distributed request path, logs say why (event detail). Alarm on metrics, find the slow or failing service with the X-Ray service map, then read that service’s logs for the cause. Correlate with a request ID or trace ID written into every log line. Mind metric cardinality: custom metrics with unbounded dimensions (user ID, request ID) are expensive and unhelpful.
Common mistake: alarming on logs when a metric would be cheaper and faster.
12.4.2 Logs at scale: ingestion, retention, subscription filters
Question: “Ship logs from many accounts to a central place for real-time analysis. How?”
flowchart TD
A["App logs in CloudWatch Logs"] --> B["Subscription filter"]
B --> C["Kinesis Data Streams: fan-out, replay"]
B --> D["Firehose: delivery to S3 or OpenSearch"]
B --> E["Lambda: transform, route"]
Model answer: Collect with the CloudWatch Agent, Fluent Bit, the Lambda or ECS log drivers, or the SDK. Set retention per log group. Use subscription filters to stream matching events in near real time: Kinesis Data Streams for high throughput, ordering, multiple consumers and replay; Firehose for managed delivery to S3, Redshift, OpenSearch or third parties; Lambda for transformation and enrichment. A log group has a limited number of subscription filters (two), so fan out downstream from one stream. For cross-account aggregation, create a CloudWatch Logs destination in the central account that the member accounts subscribe to. Encrypt log groups with KMS, restrict with IAM and resource policies, and export to S3 for cheap long-term storage, queried with Athena.
Common mistake: keeping verbose debug logs forever in CloudWatch.
12.4.3 Logs Insights, metric filters and cost
Question: “How do you query logs efficiently and keep the cost down?”
Model answer: Logs Insights runs interactive SQL-like queries (fields, filter, stats, parse, sort, limit) on data already in CloudWatch Logs; narrow the time range and select log groups carefully because you pay per GB scanned. It suits recent troubleshooting; use Athena for archived logs in S3. Metric filters turn log patterns into custom metrics for alarms. Control cost with retention, filtering noisy logs at the source, sampling, log classes for infrequent access, and avoiding high-cardinality dimensions.
Common mistake: querying 30 days across dozens of log groups for a one-hour incident.
12.4.4 Alarms that do not cry wolf
Question: “How do you reduce false positives and automate response?”
Model answer: Tune period, evaluation periods and datapoints to alarm (M of N), and set the missing data treatment. Use anomaly detection (a learned band that adapts to daily and weekly patterns), metric math to alarm on ratios such as errors divided by requests, and composite alarms (AND, OR, NOT) to page only when several conditions agree. Alarm actions: SNS, Auto Scaling, EC2 actions, Systems Manager OpsItems and incidents, and EventBridge for automation. Prefer user-facing signals (latency, error rate, queue age) to raw resource use.
Common mistake: static thresholds on metrics with strong daily patterns.
12.4.5 X-Ray, CloudTrail and Config for investigation
Question: “Which tool do you open first for latency, for an unexpected change, for an access event?”
| Symptom | Tool |
|---|---|
| Slow request across services | X-Ray: traces, segments, subsegments, service map, annotations (indexed) and metadata (not indexed), sampling rules to control cost |
| “Who deleted this?” | CloudTrail: management events by default, data events opt-in, Insights for unusual call rates, log file validation, organisation trail, delivery to S3 and CloudWatch Logs |
| “What changed and is it compliant?” | AWS Config: configuration history, rules, aggregators |
| Slow SQL | RDS Performance Insights |
Model answer: Combine them: CloudTrail shows who and when, Config shows what and the before and after state, X-Ray shows request paths, CloudWatch shows trends. Instrument the app with the X-Ray SDK or OpenTelemetry (ADOT), run the daemon or collector, and lower the sampling rate in production.
Common mistake: tracing every request at high volume and paying for it.
12.4.6 EventBridge vs SNS vs SQS
Question: “When do you use each for event-driven operations?”
| EventBridge | SNS | SQS | |
|---|---|---|---|
| Model | Content-based routing on JSON patterns | Pub/sub fan-out | Queue, pull-based |
| Strength | AWS and SaaS sources, cross-account buses, archive and replay, schemas, input transformers | Multi-protocol delivery (email, SMS, HTTPS, Lambda, SQS), filter policies, FIFO topics | Buffering, retry, back-pressure, DLQ |
| Typical use | Route CloudTrail, Config, Health and alarm events to remediation | Notify many subscribers | Decouple and absorb bursts |
Model answer: Detect with an event source, route with EventBridge (rule, target, retry policy, dead-letter queue), notify through SNS, buffer through SQS where consumers may be slow. Know the limits (for example a handful of targets per rule and a maximum event size, so send large payloads by reference to S3) and monitor failed invocations. Pipes connect a source to a target with filtering and enrichment; Scheduler runs timed jobs.
Common mistake: no dead-letter queue, so failed events vanish.
Domain 5: Incident and event response
12.5.1 Events vs incidents and the response loop
Question: “What is the difference between an event and an incident, and how do you automate response?”
flowchart TD
A["Detect: alarm, Config, GuardDuty, Health"] --> B["Route: EventBridge"]
B --> C["Notify: SNS to chat or pager"]
B --> D["Remediate: SSM Automation or Lambda"]
D --> E["Record: OpsCenter OpsItem"]
E --> F["Review and improve"]
Model answer: An event is a state change (instance stopped, object created, Health notice); an incident is a service-impacting condition that needs action. The three pillars of response are detection and notification, automated remediation and investigation. Automate the safe, repeatable fixes (restart a service, revoke an open security group, replace an unhealthy instance) and keep humans in the loop for risky ones with approval steps. Always notify, record and measure time to detect and recover.
Common mistake: automating a remediation with no rate limit or guard, which causes a loop.
12.5.2 AWS Health
Question: “AWS has a problem affecting your resources. How do you find out and act?”
Model answer: AWS Health (the Personal Health Dashboard and the Health API and events) reports service issues, scheduled maintenance (for example instance retirement) and account notifications specific to your resources. Route Health events through EventBridge to SNS, chat or runbooks, for example to replace instances scheduled for retirement. In an organisation, enable organisational view to aggregate Health across accounts. Use the Health API with a Business or higher support plan, and check the public service status for wide issues.
Common mistake: checking the public status page only and missing account-specific events.
12.5.3 Systems Manager Automation and OpsCenter
Question: “How do you build an auditable, safe automated remediation?”
Model answer: Write a Systems Manager Automation runbook (schemaVersion, parameters, mainSteps) using actions such as aws:executeAwsApi, aws:runCommand, aws:executeScript, aws:branch, aws:approve, aws:waitForAwsResourceProperty. It runs under an assume role with least privilege, can run across accounts and Regions, and can be triggered by EventBridge, Config remediation, alarms or on demand; approval steps provide human-in-the-loop control. OpsCenter collects OpsItems (structured incident records with related resources and operational data) from alarms and events, links runbooks, aggregates across accounts and lowers mean time to resolve. Incident Manager adds response plans, escalation and on-call.
Common mistake: a runbook with an over-broad role and no logging.
12.5.4 Systematic troubleshooting
Question: “Production is failing and you do not know why. Walk me through your approach.”
Model answer: Be systematic and evidence-driven: (1) define the impact and when it started; (2) ask what changed (deployments, config, scaling, certificates, quotas); (3) form a hypothesis and check it against metrics, logs, traces and CloudTrail; (4) mitigate first (roll back, fail over, scale) to restore service; (5) fix the root cause; (6) write a blameless post-incident review with actions. Work layer by layer: DNS, edge, load balancer, network and security groups, compute, application, data, IAM. Specific kits: deployments (events and logs), Auto Scaling (activity history, launch template, quotas, AZ capacity), containers (describe, events, probes), performance (X-Ray, Performance Insights), network (Flow Logs, Reachability Analyzer), IAM (CloudTrail AccessDenied, Policy Simulator).
Common mistake: changing several things at once, so nobody knows what fixed it.
12.5.5 Chaos engineering
Question: “How do you prove that your resilience design actually works?”
Model answer: Use AWS Fault Injection Service to run controlled experiments: stop instances, throttle or fail APIs, inject network latency or disrupt an AZ, with stop conditions tied to CloudWatch alarms so the experiment aborts if the blast radius grows. Start in a test environment with a hypothesis (the service stays within its SLO), observe, learn and fix, then run game days in production with approvals. Pair with the DR rehearsal and runbook reviews.
Common mistake: running experiments with no stop condition.
Domain 6: Security and compliance
12.6.1 Access at scale: federation, Identity Center, ABAC
Question: “How do 500 employees get access to 30 accounts without IAM users?”
flowchart TD
A["User signs in at the IdP"] --> B["SAML or OIDC assertion"]
B --> C["IAM Identity Center / STS"]
C --> D["Temporary credentials for a permission set role"]
Model answer: Federate the corporate identity provider (Okta, Entra ID, ADFS) with IAM Identity Center: users and groups are assigned permission sets per account, which create roles; STS issues temporary credentials (AssumeRoleWithSAML, AssumeRoleWithWebIdentity). No long-lived keys and one place to disable a leaver. Use ABAC (tags on principals and resources, conditions such as aws:PrincipalTag/team equals aws:ResourceTag/team) to scale permissions without writing a role per team; use RBAC when job functions are stable. For workloads use roles (instance profiles, task roles, IRSA, OIDC for CI systems such as GitHub Actions).
Common mistake: sharing one IAM user’s access keys among CI jobs.
12.6.2 Finding over-permissioned and externally shared resources
Question: “How do you find resources shared outside the organisation and remove unused permissions?”
Model answer: IAM Access Analyzer has external access findings (S3 buckets, roles, KMS keys, secrets and more shared outside your zone of trust), unused access findings, policy validation (also in CI), custom policy checks and policy generation from CloudTrail. Access Advisor shows last-accessed data per service. For debugging an AccessDenied: identify the principal, action and resource; read the CloudTrail event; check the evaluation chain (explicit deny, SCPs, boundaries, session policies, identity and resource policies); test with the Policy Simulator. Cross-account access needs permission on both sides.
Common mistake: debugging the identity policy while an SCP or resource policy is denying.
12.6.3 KMS and secrets rotation
Question: “How do you manage encryption keys and rotate database credentials?”
Model answer: Use customer managed KMS keys for control: key policy first, then IAM and grants, separation of administrators and users, one key per environment or data class, aliases, CloudTrail logging of every use. Envelope encryption: GenerateDataKey, encrypt locally, store the encrypted data key with the data. Automatic rotation changes the backing key material yearly (configurable) without changing the key ID, and old material stays to decrypt older data. For RDS credentials use Secrets Manager rotation (a Lambda rotates the password; single-user rotation briefly invalidates the old credential, alternating-user keeps one valid at all times for zero downtime). Applications fetch the secret at runtime with caching; replicate secrets to the DR Region; share across accounts with a resource policy and key access.
Common mistake: rotating a password but leaving applications with the cached old one.
12.6.4 Detection and findings consolidation
Question: “Design organisation-wide threat detection and automated response.”
flowchart TD
A["GuardDuty, Inspector, Macie, Config, IAM Access Analyzer"] --> B["Security Hub: ASFF findings, aggregated"]
B --> C["EventBridge rule on severity"]
C --> D["Lambda or SSM runbook: isolate, revoke, tag"]
C --> E["Ticket and notification"]
Model answer: Enable the detectors in every account and Region through Organizations with a delegated administrator account. GuardDuty analyses CloudTrail, VPC Flow Logs, DNS logs and optionally S3, EKS, Lambda and malware signals; Inspector scans EC2, ECR and Lambda for vulnerabilities continuously; Macie finds PII in S3; Config and Access Analyzer cover configuration and access. Security Hub normalises everything into the AWS Security Finding Format (ASFF), scores compliance against standards (CIS, AWS Foundational Security Best Practices) and aggregates across accounts and Regions; custom products send findings with BatchImportFindings. Route high-severity findings through EventBridge to automated remediation: isolate the instance with a quarantine security group, revoke keys, snapshot for forensics, notify. Tune with suppression rules and trusted IP lists to cut noise.
Common mistake: enabling GuardDuty only in the main Region.
12.6.5 Network defence and auditing
Question: “How do you defend a public web application end to end, and prove what happened afterwards?”
Model answer: Edge: CloudFront with WAF (managed rules, rate limits) and Shield. Network: private subnets, security groups referencing other groups, NACLs as a coarse layer, Network Firewall for inspection, VPC endpoints, Flow Logs. Identity and data: least privilege, TLS, KMS encryption. Audit: CloudTrail organisation trail with log file validation (digest files prove logs were not altered), delivery to a locked-down S3 bucket in the log archive account with MFA delete or Object Lock, CloudWatch Logs integration with metric filters and alarms for root login, IAM changes, console sign-in without MFA and security group changes, Insights for anomalous activity. Remember global service events are recorded in us-east-1 unless the trail is multi-Region.
Common mistake: leaving the CloudTrail bucket writable by the same account that could be compromised.
12.7 Quick recap
| Domain | One-line memory |
|---|---|
| SDLC automation | Build once, promote the artifact, test early, deploy safely with alarms and automatic rollback; pipeline in a tools account assuming roles. |
| Configuration and IaC | Everything as code, detect drift, validate with cfn-lint and Guard, protect with stack policies, govern with Organizations, Control Tower and Config. |
| Resilience | Remove single points of failure by scope, pick DR by RTO and RPO, use Global Accelerator for fast failover, rehearse failures. |
| Monitoring | Metrics tell what, traces tell where, logs tell why; subscription filters fan out; anomaly and composite alarms cut noise. |
| Incident response | Detect, route with EventBridge, remediate with SSM, record in OpsCenter, review, inject faults to test. |
| Security | Federate, use ABAC, Access Analyzer, customer managed keys, rotate secrets, aggregate findings in Security Hub, audit with CloudTrail. |
Final checklist
- Can I explain a container as a process with namespaces and cgroups, and debug exit code 137?
- Can I debug a stuck rollout, DNS failure and Pending pod using only
kubectl describe, logs and events? - Can I explain Terraform state, locking and drift, and why secrets in state are a risk?
- Can I describe GitOps, blue-green and canary, and a safe database migration (expand and contract)?
- Can I read an IAM policy and say why an explicit deny wins?
- Can I walk through an incident: contain, investigate, remediate, prevent?
- Can I design a URL shortener, rate limiter, notification system and news feed with numbers and trade-offs?
- Can I pick the right DR strategy from an RPO and RTO, and explain why one NAT gateway per AZ matters?
- Can I design a multi-account pipeline with cross-account roles, a shared KMS key and automatic rollback?
- Can I choose between EventBridge, SNS and SQS, and wire detection to automated remediation?
- Do I have four STAR stories ready, with measurable results?