Handbook 3 of 3Cloud Delivery & DevOps

The path a change takes

The figure below is the path the reference service actually uses, end to end. A developer's only actions are opening a pull request, merging it once it is reviewed and green, and pushing a version tag. Everything after the tag is automated, gated and reversible. Each stage is a chapter of this handbook.

Three properties make this path trustworthy, and the rest of the handbook explains how each is achieved.

• Everything is reviewed. Code, infrastructure, pipeline definitions and even accepted security risks change only through pull requests.
• Everything is reproducible. The release builds from locked dependencies and pinned base images. The infrastructure is rebuilt from code, and a drift check reports any change made by hand.
• Everything is reversible. Releases roll forward and back through the same deploy, and database changes are written so the previous release keeps working.

What the reference build proved

A path is only as good as its evidence. These properties were each tested on the reference build, on AWS, with traffic flowing.

One artifact, promoted through environments

The single most important rule on the path is build once, promote the same artifact. A release builds each component exactly once. Staging, production and any other environment then run those same bytes, identified by a version tag and, underneath it, a content digest that cannot change. Only the configuration differs: addresses, limits and secrets arrive from the environment at runtime.

Two consequences follow. A version must name exact bytes, so nothing is ever deployed as latest: two pulls of latest can be different code, and a rollback has no name to return to. And a host never builds from source: building on a server needs compilers and internet access there, and two hosts building the same commit can still differ, because a package was republished or a base image patched in between.

name: Release

# A version tag on main publishes the production images (api, web and the
# TLS edge) to GitHub's container registry, for linux/amd64 and linux/arm64
# (Apple Silicon, AWS Graviton) under one tag:
#   git tag -a v1.4.0 -m "v1.4.0" && git push origin v1.4.0
#   -> ghcr.io/<owner>/triage-assistant-api:1.4.0, :1.4, :sha-<commit>, :latest
#      ghcr.io/<owner>/triage-assistant-web:  the same tags
#      ghcr.io/<owner>/triage-assistant-edge: the same tags
# :latest (docker/metadata-action's default) moves with every release, before
# the smoke test below: deploy by version, never by latest.
# Then it pulls what it published, on both architectures, and runs the
# stack from those images before writing the release notes. A host deploys
# it with `make deploy tag=1.4.0` (docs/runbooks/demo-vm.md).
#
# Pull requests that change the images' inputs run the same builds, on both

The environments

Each environment has one job. The table is the reference service's set, and it is a reasonable default for any team.

Production data never flows down to laptops. When a bug genuinely needs production-shaped data, the answer is an anonymized snapshot in staging, or synthetic data with the same shape, not a copy of production on someone's machine.

The same service at three sizes

A path like this does not require a large platform. The same service, with the same images, runs at three sizes, and each step up adds pieces only when a named need appears.

The reference build ran all three. The cloud column ran on AWS without a domain name, a real identity provider or a paid model, which the service's production checks allow only when each exception is named. Chapter 12 tells that story.

When the laptop is Windows

Everything in this handbook assumes a Linux shell, a container engine and a coding agent that can run the project's checks. A corporate Windows laptop can provide all three, but the choice of where the work lives decides whether the stack runs at full speed or breaks in ways that look like bugs in your code. The short version: the repository, the toolchain and the agent live inside a WSL2 distribution's own file system, the container engine runs there too, and the project ships a dev container so the setup is the same on every machine.

Before the first install, one ticket. Almost everything below installs per user without administrator rights: the editor, Git, the container desktop, the agent. Five things do not, and they are better asked for together than discovered one at a time: enabling the Linux subsystem and the virtual-machine platform (a reboot), the device-management policy for it (it can silently disable custom networking and ignore the limits you set), either a container-desktop seat or approval for the free engine, an allowlist entry for the agent's executable and the shells it spawns in the endpoint-security tool, and the proxy's root certificate as a file with the proxy's address. A locked-down laptop with those five settled is a normal workstation; without them it is a week of puzzles.

Where the files live. Keep every repository in the distribution's home directory, never under /mnt/c. Two mechanisms, not one: file operations across the boundary go through a file-server protocol with a round trip per metadata call, and file-change events raised on the Windows side never reach Linux, so a development server's reload and a test watcher silently stop working. The coding agent's own troubleshooting page lists the same symptom (fewer search matches than expected) with the same fix. Open the Linux side from Windows with explorer.exe . or the \\wsl$ share when you need the file manager.

Which container engine. Docker Desktop installs per user without administrator rights and uses the distribution as its backend, with the limits set in .wslconfig rather than in its own settings; it needs a paid subscription for commercial use in a company with more than 250 employees or more than ten million dollars in revenue, at roughly nine to twenty-four dollars per user per month depending on the tier, so the question goes to procurement before the first install. Docker Engine installed inside the distribution is the same open-source engine without the desktop product, free, and runs with systemd enabled in wsl.conf; you manage its updates, certificates and DNS, and the two must not coexist in one distribution. Rancher Desktop and Podman Desktop give a Docker-compatible socket without the license, and need WSL or Hyper-V enabled once by an administrator. Microsoft now ships its own container runtime inside WSL, which needs no engine at all, but it has no Compose support yet, so it cannot run this stack today.

The dev container. The portable form of all this is a container definition in the repository: a base image with the toolchain, pinned versions of the package managers, the agent installed through its published dev-container feature, and the host's Docker socket passed in so Compose works from inside. The editor reopens the folder in that container, and a new laptop goes from clone to a running stack without a hand-built environment; the agent's own reference container adds an outbound firewall with an allowlist, which is the setting its documentation requires for unattended runs. The same definition runs in a hosted codespace when a laptop cannot run Docker at all: at the list price of a four-core machine, six hours a day for a month is roughly fifty dollars per developer, and the source then lives on the vendor's cloud, which is a policy question before it is a cost question.

The agent on Windows. Install it inside the distribution with the Linux installer, not from PowerShell; a native Windows install works too, with PowerShell or Git Bash as its shell, but the command sandbox runs only on Linux, macOS and WSL2, so unattended runs on native Windows have no isolation. In WSL2 the sandbox needs two small packages and excludes Docker commands by configuration, because Docker is incompatible with it; an excluded command runs with full access, and the agent can edit the Compose file before running it, so keep Compose commands behind a permission prompt. Two things an administrator must know: policy deployed through the Windows registry is not read inside the distribution unless a key says so, so a laptop can be locked down on one side and open on the other; and Windows-side endpoint sensors do not see processes inside the WSL virtual machine unless the vendor's WSL plug-in is installed, which Microsoft's own product has.

Proxies and inspected TLS. A corporate proxy that re-signs TLS needs its root certificate in five places, not one: the Windows store (the installers validate against it), the distribution's certificate bundle, every container image that fetches packages during a build, and the tools that carry their own bundles rather than the system's, which includes the Python package manager (a system-certificates switch) and npm (a certificate-file setting). The agent trusts the operating-system store by default and takes HTTPS_PROXY and NO_PROXY from its settings file's environment block, which matters because its background processes inherit the first shell that started them, not yours; it speaks plain HTTP proxies only, so a SOCKS or NTLM proxy needs a gateway in front. A VPN that changes DNS usually breaks the distribution's generated resolver file first; turning generation off and writing the resolver by hand is the documented fix. If a repository must stay on the Windows drive, the development servers' file watchers can fall back to polling through their environment variables, at a cost in battery and latency.

Git across the boundary. Commit a .gitattributes with * text=auto eol=lf so line endings are a repository decision rather than each developer's core.autocrlf, and set that option to false on the Windows side; a shell script with carriage returns fails inside a container with a message about a file that plainly exists. On a Windows checkout, enable long paths in Git and in Windows (the 260-character limit is reached by any node_modules), turn on Developer Mode for symbolic links, and expect the executable bit to be lost on NTFS: set it in the index with git update-index --chmod=+x. Inside the distribution's file system none of this applies, which is one more reason to work there.

Traps

Questions

1. Why should staging and production run the same image rather than each building from the same commit?

   Answer

   Because a build is not reproducible bit for bit: packages get republished and base images patched between builds. Running the same image, identified by its digest, means what was tested is exactly what runs.

2. What is the difference between a release and a deploy in this path?

   Answer

   A release is a version tag that builds, scans and publishes images once. A deploy tells an environment to run a particular released version. One release can be deployed many times, to many environments, and an older one can be deployed again as a rollback.

3. Why is latest never deployed?

   Answer

   It is a movable label: two pulls can return different code, and there is no name to roll back to. A deploy must name exact bytes.

4. What makes a rollback safe when the release being undone changed the database?

   Answer

   Writing migrations so they only add (expand) before anything is removed (contract), so the previous release still runs on the newer schema. A rollback then deploys the older code and leaves the schema alone. Chapter 13 covers it.

The model: snapshots and labels

Most confusion with Git disappears once the model is clear. A commit is a snapshot of the whole project plus a pointer to its parent commit. Its identifier is a hash of the snapshot, the parent, the author, the time and the message, so a commit can never change: change anything, even the message, and you have a different commit. A branch is a label pointing at one commit, which moves forward when you commit. A tag is a label that stays put: committing does not move it, and by convention nobody does, though git tag -f and a force-push can unless a tag ruleset forbids it. HEAD is where you are.

A remote is simply another copy of the same database. fetch downloads its new commits and moves your origin/main label. pull is a fetch followed by a merge. push uploads your commits and asks the remote to move its branch label, which it refuses unless your history contains its history.

One practical consequence matters for operations: a running service reads the disk, not Git. On restart it loads whatever the working tree holds, including uncommitted edits or half-resolved conflict markers. That is why a production host is never a developer's checkout: a deploy puts a specific, built release on it (Chapter 3).

Three places a file lives

In git status --short the left column compares the staging area with the last commit, and the right column compares the working tree with the staging area. MM means "staged, then edited again": add it again. Untracked files appear in neither diff, which is how a new file gets forgotten.

Merge, rebase and squash

Three ways to bring a branch's work into main produce three different histories. Teams pick one per repository; the reference service squash-merges and requires a linear history.

A conflict looks the same in all three. Git writes both versions into the file between <<<<<<<, ======= and >>>>>>> markers; you edit the file to its final form, stage it and continue, or abort and start again. If a commit you rewrote disappears, git reflog still knows where every label pointed recently.

Trunk-based development

For a service that deploys continuously, the default is trunk-based development: one permanent branch, main, always deployable, and short branches that live for hours or days, each merged through a pull request and then deleted. GitHub flow is the same idea. GitFlow, with its develop, release/* and hotfix/* branches, exists for software shipped in several supported versions at once, such as installed products. Recognize it; do not pick it for a service.

The pull request as a gate

The pull request is where a change is explained, verified and reviewed. Its value depends on the rules around it, and on GitHub those rules are branch protection. On the reference repository, main requires:

• six green checks: lint, test-api, build, security, web-build and e2e, on a branch that is up to date with main;
• one approving review, including one from the code owners of every path touched, with approvals dismissed when new commits arrive;
• a linear history; with squash as the only merge method the repository allows, every change lands as one commit.

Code owners say whose approval a path needs. The pull request template asks the author for the evidence a reviewer needs, so verification is written down rather than assumed.

Closes #

## What and why

<!-- One paragraph: the change, and the reason for it. -->

## How I verified it

<!-- Commands and numbers, not "tested locally". E.g. `make check`,
     `make e2e`, `make drills d="db-freeze"`, a before/after latency. -->

## Checklist

- [ ] `make check` passes locally (lint, types, tests; CI adds the scans, image checks and e2e)
- [ ] Tests cover the failure paths, not only the happy path (dependency down / hung, invalid input, forbidden)
- [ ] Browser- or nginx-visible change: `make prod-up && make e2e`
- [ ] Database path, pools or timeouts touched: the database drills (`make drills d="db-freeze db-hang pgbouncer-freeze"`)
- [ ] Migration: backward compatible (expand now, contract in a later release), `lock_timeout`, concurrent indexes
- [ ] New metric labels are bounded; new alerts have a promtool test
- [ ] New dependency: one line on why, and why not the standard library or an existing one
- [ ] Docs: the handbook chapter, an ADR for a decision, a runbook for a procedure, the gotcha list for a trap
- [ ] No secrets, personal data or real customer data in code, tests, fixtures or logs

# Decisions are reviewed by whoever owns the architecture
/docs/adr/             @Parzon

# Accepting a vulnerability or exempting a secret is a decision too
/.trivyignore.yaml     @Parzon
/.gitleaks.toml        @Parzon
/.gitleaksignore       @Parzon

Habits that keep history useful

• Branch names as type/issue-short-description, such as fix/31-pool-leak. Create the branch before the first edit.
• Commit messages in the Conventional Commits style (feat:, fix:, docs:), with an imperative subject and the reason in the body.
• Small pull requests with one concern each. This matters more with AI coding tools, which produce large diffs quickly; a large diff gets skimmed, not reviewed.
• Never commit secrets, generated output or environments (.venv, node_modules, build directories). A secret that reaches a commit is rotated, not just deleted: on a public repository it was copied the moment it was pushed.

When an agent wrote the commits

Nothing in the pull-request gate changes when a coding agent authored the change; what changes is how much weight each part carries. The person who opens the pull request owns it: the kernel and the Apache projects both settled on that rule, with an attribution trailer on the commit (Assisted-by: or an equivalent) so the history says what was generated. A commit-message hook enforces the trailer. Small pull requests matter more: one vendor's review service reports an average of 7.5 findings on pull requests over a thousand lines and half a finding under fifty, which is as much about what reviewers can read as about what agents write. The files that define the gates, the pipeline, the lint configuration, the hooks and the lockfiles, get a code-owners entry, so a change to them needs a named reviewer and an agent cannot loosen a rule inside the same pull request that needs it loosened. Required status checks stay the merge gate; a review bot comments and never blocks, runs in comment-only mode for a few weeks first, reads its rules from a file the repository owns, and is capped on nits. Review effort follows the risk tier of the change: strict for authentication, personal data, money and migrations, sampled elsewhere. The four layers that this sits on top of are drawn in chapter 13 of the software handbook.

Traps

Questions

1. Why can a commit never change, while a branch changes all the time?

   Answer

   A commit's ID is a hash of its snapshot, its parent, its author and time, and its message, so any change makes a different commit. A branch is only a label pointing at a commit, and it moves forward with each new commit.

2. When is rebasing safe, and when is it not?

   Answer

   It is safe on your own commits that nobody else has. It is unsafe on commits others already have, because rebasing gives them new IDs and their copies no longer match.

3. What does a squash merge give a team, and what does it cost?

   Answer

   One commit per change on main, so one revert undoes a whole feature and the history reads as a list of changes. The small commits are no longer on main, but they remain visible in the pull request.

4. Which branch protection settings turn a pull request into a real gate?

   Answer

   Required status checks on an up-to-date branch, required reviews including code owners, stale approvals dismissed on new commits, and a linear history. Without them, a pull request is only a suggestion.

5. Why should a production server never run from a Git checkout?

   Answer

   A service reads its files from disk, including uncommitted edits and conflict markers, and a checkout can drift from any recorded version. A deploy should place a specific built release on the host.

Continuous integration: the merge gate

On every pull request, the reference service runs six jobs on fresh machines: five in parallel, and the image build once lint has passed. All six are required by branch protection (Chapter 2), so a pull request cannot merge until they pass. Each job is the same command a developer can run locally, which keeps "it passed on my machine" and "it passed in CI" the same statement.

Two kinds of test sit in this gate, and they catch different bugs. A unit test checks one function in isolation, in milliseconds. An integration test runs real parts together, such as the api against a real Postgres, connection pooler and cache, and catches wiring bugs a unit test cannot see. The test-api job builds a throwaway stack for the integration tests and deletes it afterwards. The e2e job goes one step further and drives a real browser through the production-shaped stack, which once caught a bug no unit test could reproduce. Handbook 2 covers testing in depth.

name: CI

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

# A newer push to the same branch cancels the run still in progress.
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

# Least privilege for the GITHUB_TOKEN: these jobs only read the repo.
permissions:
  contents: read

# Actions are pinned to full commit SHAs (tags can be moved to malicious
# code, as in the 2025 tj-actions/changed-files compromise). Dependabot
# keeps the SHAs and the version comments current.

test-api:
  runs-on: ubuntu-24.04
  steps:
    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
    # The exact command developers run: a throwaway compose stack with real
    # Postgres, PgBouncer and Valkey, migrations + `alembic check`, then
    # pytest with the coverage gate from pyproject.toml.
    - run: make .env
    - run: make test-api

Where CI runs, and what it costs

GitHub Actions runs each job on a fresh, GitHub-hosted virtual machine. Standard runners are free for public repositories, such as the reference one; a private repository gets a monthly quota of minutes and is billed by the minute beyond it, and larger runners are always billed. Three habits keep the bill and the wait down: cache dependency downloads keyed on the lockfile, run expensive workflows only when their inputs change (path filters), and cancel superseded runs. A self-hosted runner executes whatever a workflow tells it to. Never attach one to a public repository that accepts pull requests from outside: a stranger's pull request would run code on your machine.

Continuous delivery: a tag becomes a release

A release starts when someone pushes a version tag, such as v0.8.2, to a commit on main. Versions follow semantic versioning: MAJOR.MINOR.PATCH, where the major number changes for a breaking change, the minor for a compatible feature, and the patch for a fix. Teams choose between deploying every merge to main (continuous deployment) and deploying on a tag (continuous delivery). The reference service deploys on a tag.

Each step guards against a specific failure. Checking that the tag points at a commit on main stops anyone releasing code that never passed review. Scanning again at release catches vulnerabilities published after the merge. Testing the published images, rather than the build outputs, proves the registry holds what the team thinks it holds. Two gaps remain. The release-time scan and image check run on images the verify job builds for them on its amd64 machine, from the same commit: neither the arm64 images nor the exact published bytes are scanned. Scanning the pushed index by digest, once per architecture, would close that gap. And the copy to ECR does not wait for the smoke test, so a release that fails it still lands in ECR under a tag that can never be re-pushed. Only the deploy waits. Making the copy depend on the smoke test would close the gap.

# Tags can be pushed from any commit; only code that went through a
# reviewed, green PR into main gets released.
- name: The tag points at a commit on main
  run: git merge-base --is-ancestor "$GITHUB_SHA" origin/main
- name: Production api image - non-root, no dev tools, every module imports
  run: make image-check
# Again at release, not only at merge: vulnerabilities published since
# the PR merged block the release too (ADR-0022).
- name: No fixable HIGH or CRITICAL vulnerability in what ships (Trivy)
  run: make scan

# A tag push publishes by digest; a PR only builds.
outputs: ${{ github.event_name == 'push' && format('type=image,name={0},push-by-digest=true,name-canonical=true,push=true', env.IMAGE) || 'type=cacheonly' }}
# What went into the image, stored next to it in the registry:
# the build's provenance and a software bill of materials.
provenance: ${{ github.event_name == 'push' && 'mode=max' || 'false' }}
sbom: ${{ github.event_name == 'push' }}

Tags, digests and multi-architecture images

An image is the immutable build output, identified by a digest: a hash of its content, such as sha256:21115358…. A tag is a movable label, like a Git branch. The release workflow moves 0.8 on every 0.8.x release on purpose; a re-push can move any tag unless the registry forbids it. So deploy an exact version from a registry with immutable tags, or deploy by digest, and never deploy latest or a moving tag such as 0.8.

The reference release builds each architecture on a native runner (an Arm machine for arm64) rather than emulating one on the other. Emulated builds are slow and occasionally behave differently. Each build pushes its image by digest, and a final job joins the digests under the version tags.

When the release was copied from GitHub's registry to Amazon ECR, the index digest in ECR was identical to the one on GitHub: the same bytes in a second place. Pushing a different image over an existing tag in ECR was then refused, because the repositories have immutable tags. A tag in that registry always names the same bytes.

Traps

Questions

1. What is the difference between a unit test and an integration test, and why does CI run both?

   Answer

   A unit test checks one function in isolation and is fast. An integration test runs real components together and catches wiring bugs, such as a wrong connection setting or a missing migration. Each catches failures the other cannot.

2. Why does the release workflow test the images it published rather than the ones it built?

   Answer

   Because what matters is what hosts will pull. Testing the published images, on each architecture, proves the registry holds working images under the release tags.

3. What is the difference between a tag and a digest, and which should a deploy use?

   Answer

   A tag is a movable label; a digest is a hash of the content and never changes. Deploy by digest, or by an exact version tag in a registry that makes tags immutable. Never by latest or a tag that is moved on purpose, such as a minor line.

4. Why might a team build arm64 images at all?

   Answer

   Arm servers such as AWS Graviton cost less for the same work, and many developer laptops are Arm. One tag with both images lets every host pull the right one.

5. What does build provenance answer that a version number does not?

   Answer

   Which commit and which workflow run produced this exact image. A version number is only a label someone chose; provenance is a record the build wrote and stored with the image. It proves the answer only when it is signed and the signature is checked; the reference release's is not signed.

What you ship but did not write

A small service's own code is a thin layer on top of a much larger stack. Each layer has its own way of changing under you, and its own control.

The bottom layer is easy to forget. A scanner runs with access to your CI environment, so a compromised scanner is a compromised pipeline. This has happened: attackers published a Trivy release, binaries and container image alike, and moved almost every tag of its GitHub Action, all to code that stole CI credentials (CVE-2026-33634). Teams that ran a scanner image pinned to a digest from before the attack, and did not take the new release, were not affected. A digest pin stops a tag from being re-pointed; a reviewed bump is the chance to catch a bad new release. The reference service runs its scanners that way, from the same commands locally and in CI, with no access to the Docker socket.

Pinning, and taking updates

Pinning alone would freeze a service with every vulnerability it ever had. The other half is a bot, Dependabot here, that opens pull requests to move the pins: grouped monthly for packages and actions, weekly for base images, and immediately for a known vulnerability. Each is an ordinary pull request, so CI and review decide whether it merges. A major version gets read like any other change.

FROM ghcr.io/astral-sh/uv:0.12.21@sha256:a7aed3216253ee804de3e2d8afa5073baa1a177335345d43845cd4165e43b711 AS uv

# ---- deps: virtualenv with runtime dependencies only ----------------------
FROM python:3.13-slim@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS deps

# Dependency update PRs, monthly and grouped (one PR per ecosystem, not one
# per package). Monthly, not weekly: a vulnerable dependency still gets its
# own security PR at once, and everything else can wait for the batch. The
# Dockerfiles' base images are the exception, weekly: security updates do
# not cover OS packages, and a rebuilt base is how a .trivyignore.yaml
# entry gets removed before it expires (ADR-0026).
# Each is an ordinary PR: CI must pass before merging, and a
# major version gets read like any other change. GitHub Actions stay pinned
# to commit SHAs; Dependabot moves the SHA and its version comment together.

Scanning, and what to do with a finding

Two kinds of scan run in the reference pipeline. Vulnerability scanning (Trivy) compares the packages inside each production image, and the web app's runtime dependencies, with databases of known vulnerabilities. Secret scanning (gitleaks) reads every commit in the history for things that look like keys and passwords. Both run on every pull request; the vulnerability scan runs again at release and weekly, because new advisories are published for images nobody changed. Each of those runs builds the images itself on an amd64 machine, so it checks the same source, not the exact published bytes, and not the arm64 images.

The note at the bottom left of the figure is a lesson learned. The weekly scan of the third-party images the stack merely runs, such as Keycloak, Grafana and pgvector, first found 336 findings in 12 of 14 images, nearly all waiting on releases from those projects. A job that fails every Monday for things the team cannot change is an email nobody reads, so that scan now reports instead of failing, and is read when those images are upgraded. The service's own images still fail on any fixable finding.

The first scan also changed the images themselves. Two vulnerabilities lived inside pip, which the production image never used, so pip was removed and the image check now refuses an image that contains it. Another lived in nginx's optional modules, so the image moved to nginx's slim variant. Removing what you do not need is often a better fix than patching it.

# Accepted vulnerabilities (make scan, make scan-compose; ADR-0022). A
# finding goes here only when it cannot be fixed now: no fixed version is
# used by the image yet, or the vulnerable code is shown to be unreachable.
# Each entry needs:
#   statement:  why it is safe for now, and who accepted it
#   expired_at: at most 90 days out; Trivy reports it again after that
# scripts/check_trivyignore.py refuses an entry without both, and the code
# owners must approve every change to this file (.github/CODEOWNERS).

#!/usr/bin/env python3
"""Every accepted vulnerability in .trivyignore.yaml has a reason and an
expiry date no more than 90 days out.

Trivy honours `statement` and `expired_at` but requires neither: without
this check a risk accepted once stays accepted forever, for no stated
reason. Run by `make scan` and `make scan-compose`.
"""

A secret found in a commit is handled in a fixed order: rotate it first, because once pushed it may already be copied, and only then remove it. A fake credential in a test is allowed only by its exact fingerprint, with the value shown to be fake. GitHub's push protection adds a second layer by refusing the push itself.

Traps

Questions

1. Why pin a base image by digest when the tag already names a version?

   Answer

   A tag can be re-pointed to different content, including a rebuilt image with other packages. A digest is a hash of the content, so it always means the same bytes. Updates then arrive as a reviewed change to the digest.

2. Why does the policy block only findings that have a fixed version?

   Answer

   A finding with no fix leaves nothing to do but wait. Blocking on it keeps the check permanently red, and people learn to ignore it. With --ignore-unfixed the blocking scan does not even report them, so the reference service's weekly run adds a report-only scan without that flag, which keeps them in sight. The accepted-risks file, with a reason and an expiry that forces a fresh decision, is for the other case: a fix exists upstream but cannot be taken yet.

3. Why scan again at release and weekly if every pull request is scanned?

   Answer

   Vulnerabilities are published every day for packages that have not changed. An image that was clean when its pull request merged can have a known critical vulnerability a week later.

4. A real API key is found in a commit. What happens first, and why?

   Answer

   Rotate the key, so the leaked value stops working. Removing it from history comes second: on a public repository it may already have been copied.

Three callers, three routes

The old model gave every caller a long-lived secret: a password, an access key, an SSH key. Each one had to be stored somewhere, rotated by someone, and revoked when a person left. The modern model replaces all of them with the same pattern. A caller proves who it is to a system that already knows it, and receives credentials with a short, fixed lifetime.

The rest of this chapter takes each lane in turn. Pipelines get the most space, because that is where the change is newest and where the configuration is easiest to get subtly wrong.

People: single sign-on, and no standing access to production

An engineer signs in once a day through the company's identity provider (Okta, Microsoft Entra ID and others), with MFA. AWS IAM Identity Center maps the groups that provider asserts to roles in each account: development, staging, production. The command-line tools then fetch short-lived role credentials as needed. When the sign-in session ends, usually after a working day, everything stops working until the person signs in again. A stolen laptop's credentials die on their own.

Production adds one more rule: people do not deploy by hand. The pipeline deploys through its own role. People are read-only by default and get elevated access only on request, for a fixed time, and every use of it is recorded. This is often called break-glass or just-in-time access.

The reference build used a single throwaway account, so it signed in with the simpler aws login. The idea is the same: a browser sign-in that yields temporary credentials, and no access keys on disk. Root user access keys were never created.

Pipelines: OIDC federation

A CI job is the caller with the most power and the least supervision: it runs automatically, on shared infrastructure, and can push images or change infrastructure. It also cannot type a password. The old answer was an access key stored in the CI system's secrets. The modern answer is OpenID Connect (OIDC) federation: the CI platform vouches for the job, and the cloud decides whether to trust that.

Four pieces make it work:

1. The job asks for a token. GitHub signs a JSON Web Token (JWT) for each job that has permissions: id-token: write. Without that line, no token is issued.
2. The token describes the job. Its iss (issuer) is GitHub's token service. Its aud (audience) is whatever the job asks for: the AWS login action, aws-actions/configure-aws-credentials, asks for sts.amazonaws.com, while a token fetched without an audience carries the repository owner's URL and fails the trust policy. Its sub (subject) names the repository and either the git ref that triggered the job or the GitHub environment the job runs in.
3. AWS trusts the issuer once per account. An IAM OIDC provider entry tells AWS to accept tokens signed by GitHub's published keys.
4. A role decides which jobs it will serve. Its trust policy allows sts:AssumeRoleWithWebIdentity only when aud and sub match. A different repository, branch, tag or environment is refused.

# This repo issues GitHub's "immutable subject" claim, which embeds the numeric owner and repo IDs
# (survives renames, blocks a re-created repo of the same name). Read it with:
#   gh api repos/Parzon/triage-assistant/actions/oidc/customization/sub
# A copy of this template has other ids: put its own prefix here (the same command, on its repo),
# or no role can be assumed. scripts/new-project.sh renames the repo, not these numbers.
SUB_PREFIX = "repo:Parzon@124113141/triage-assistant@1381921470"
ISSUER = "token.actions.githubusercontent.com"

# AWS trusts GitHub's signed tokens (one provider per account).
provider = iam.OpenIdConnectProvider(
    self, "GitHub", url=f"https://{ISSUER}", client_ids=["sts.amazonaws.com"]
)
role = iam.Role(
    self,
    "EcrPush",
    role_name="triage-gha-ecr-push",
    max_session_duration=Duration.hours(1),
    assumed_by=iam.WebIdentityPrincipal(
        provider.open_id_connect_provider_arn,
        conditions={
            "StringEquals": {f"{ISSUER}:aud": "sts.amazonaws.com"},
            # Only a release tag of this repo: not a PR, not a branch, not a fork. A tag
            # pushed on any branch runs that branch's release.yml, which could drop its
            # "tag is on main" check: the repository's tag ruleset, which lets only admins
            # create v* tags, is what makes this a release (using-this-template.md).
            "StringLike": {f"{ISSUER}:sub": f"{SUB_PREFIX}:ref:refs/tags/v*"},
        },
    ),
)
for repo in repos.values():
    repo.grant_pull_push(role)  # scoped to these repos, not ecr:*
# GetAuthorizationToken cannot be scoped to a repository.
role.add_to_policy(
    iam.PolicyStatement(actions=["ecr:GetAuthorizationToken"], resources=["*"])
)

One role per job, gated where the risk is

The reference build has two pipelines that touch AWS, and each has its own role. Copying images is low risk and happens on every release, so its role trusts any version tag. Deploying changes production, so its role trusts only jobs that run inside a GitHub environment named aws-demo.

That choice moves the approval rules out of the trust policy and into GitHub. When a job names an environment, its token's subject ends in :environment:aws-demo instead of the git ref. The environment's own protection rules then decide which jobs may run in it: here, only those from protected branches, and only after a required reviewer approves. That reviewer is what turns continuous deployment into deploy-on-approval, and adding or removing one never touches AWS.

jobs:
  # A job's `if` cannot read secrets: this one reads the secret and answers yes or no.
  enabled:
    if: >-
      github.event_name == 'workflow_dispatch' ||
      (github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push')
    runs-on: ubuntu-24.04
    outputs:
      deploy: ${{ steps.check.outputs.deploy }}
    steps:
      - id: check
        env:
          CONFIGURED: ${{ secrets.AWS_DEPLOY_ROLE_ARN != '' }}
        run: echo "deploy=$CONFIGURED" >> "$GITHUB_OUTPUT"

  deploy:
    needs: enabled
    if: needs.enabled.outputs.deploy == 'true'
    runs-on: ubuntu-24.04
    environment:
      name: aws-demo
      url: ${{ steps.smoke.outputs.url }}
    permissions:
      contents: read
      id-token: write

- uses: aws-actions/configure-aws-credentials@e1253824e5c10ff9df46874f81ed3ec929e19cfd # v6.3.0
  with:
    role-to-assume: ${{ secrets.AWS_DEPLOY_ROLE_ARN }}
    aws-region: ap-south-1
    # The account id also appears in what follows (CDK's output, resource ARNs).
    mask-aws-account-id: true

deploy = iam.Role(
    self,
    "Deploy",
    role_name="triage-gha-deploy",
    max_session_duration=Duration.hours(1),
    assumed_by=iam.WebIdentityPrincipal(
        provider.open_id_connect_provider_arn,
        conditions={
            "StringEquals": {
                f"{ISSUER}:aud": "sts.amazonaws.com",
                # A job that names an environment gets this subject instead of its ref.
                f"{ISSUER}:sub": f"{SUB_PREFIX}:environment:aws-demo",
            },
        },
    ),
)
deploy.add_to_policy(
    iam.PolicyStatement(
        actions=["sts:AssumeRole", "sts:TagSession"],
        resources=[
            f"arn:aws:iam::{self.account}:role/cdk-hnb659fds-*-{self.account}-{self.region}"
        ],
    )
)

Workloads: roles attached to the compute

A running service should never hold a key either. On AWS, a role is attached to the compute itself: a task role for an ECS task, an instance profile for an EC2 VM. The platform serves that role's credentials to the code at runtime and rotates them. Your code, using the standard AWS SDK, picks them up without any configuration.

ECS has two roles per task, and mixing them up is a common mistake. The task execution role is used by ECS itself, before your code runs, to pull the image and inject secrets. The task role is what your code may call once it is running, such as S3 or a queue. A service that needs no AWS API at runtime needs no task role at all.

Application secrets that are not AWS credentials, such as a database password or a model provider's API key, follow the same principle. They are generated by Secrets Manager, read by ECS when the task starts, and handed to the container as environment variables. They never appear in the image, the task definition, the CloudFormation template or the repository.

def generated_secret(scope: Construct, cid: str, username: str | None = None) -> sm.Secret:
    # No punctuation: the api builds a URL from these, and @ : / % would need encoding.
    return sm.Secret(
        scope,
        cid,
        generate_secret_string=sm.SecretStringGenerator(
            secret_string_template=json.dumps({"username": username} if username else {}),
            generate_string_key="password",
            exclude_punctuation=True,
            password_length=32,
        ),
        # Throwaway account: deleted with the stack. production: RETAIN (a destroyed secret
        # cannot be read back, and the database it opens may outlive the stack).
        removal_policy=RemovalPolicy.DESTROY,
    )


def secret(s: sm.ISecret, field: str) -> ecs.Secret:
    # Injected by ECS at start from Secrets Manager: never in the task definition or the template.
    return ecs.Secret.from_secrets_manager(s, field)

"""Stage 3: the one-VM deployment, done the way a platform team would accept it.

Compared with a hand-built VM:
- no SSH key and no port 22: operators use SSM Session Manager / Run Command, which the
  instance role allows; CloudTrail records who started a session or sent a command, and
  when, and the log group /triage/vm-operations keeps what ran: every session's input and
  output, and Run Command output (ops.sh vm-run);
- images come from ECR, authenticated by the instance role through the ECR credential helper,
  so no registry password exists anywhere;
- IMDSv2 only, encrypted disk, a security group that admits HTTP and nothing else;
- the host is built by cloud-init from code (adapted from infra/vm/cloud-init.yaml in the repo),
  so a replacement VM is one `cdk deploy` away.

A case from the reference build: the test that proved nothing

When the image-push role was first set up, the plan was to prove the trust policy worked by breaking it on purpose. The policy was edited to name the wrong repository, a release tag was pushed, and the job failed with Not authorized to perform sts:AssumeRoleWithWebIdentity, as expected. Then the correct repository name was restored and only the failed jobs were re-run. They failed again.

The policy had been written in the subject format most documentation shows, repo:owner/name:…. This repository's tokens use a form that embeds the numeric owner and repository IDs, which survives renames and stops a deleted-and-recreated repository with the same name from inheriting the trust. Reading the repository's OIDC subject setting showed the real value. With the prefix fixed, all three image copies succeeded, and the digests in ECR matched the ones published to GitHub's registry: the same bytes in a second place.

Two habits come out of this. Check the real claims first, before blaming anything else, because the error never says which condition failed. And re-run failed jobs rather than cutting a new release: it is faster, and it tests exactly the same thing.

Defaults, and when to choose otherwise

The same idea on Azure and Google Cloud

The names change; the mechanism does not. Each cloud trusts the CI platform's OIDC tokens and attaches identities to compute.

Traps

Questions

1. What does a CI job need before it can obtain cloud credentials through OIDC, and what does it receive?

   Answer

   The job needs permissions: id-token: write, so the CI platform will sign a token for it, and the cloud account needs an OIDC provider entry that trusts that platform. It presents the token to AWS STS and receives temporary credentials for a role, valid here for at most one hour.

2. Why does the deploy role trust a GitHub environment rather than a tag or branch?

   Answer

   A job that names an environment gets a subject ending in :environment:NAME. Trusting that subject means GitHub's environment protection rules, such as allowed branches and required reviewers, decide which jobs can reach production. Approvals can then change without editing AWS.

3. You break a trust policy on purpose and the job fails. What must you check before concluding that the policy works?

   Answer

   That the correct policy actually passes. A failure only proves something if a success was seen first. Otherwise the policy may never have matched, as in the case in this chapter.

4. What is the difference between an ECS task execution role and a task role?

   Answer

   The execution role is used by ECS before your code runs: to pull the image and to inject secrets. The task role is the identity your code uses to call AWS services while it runs. A task that calls no AWS API at runtime needs no task role.

5. Why is removing SSH access a security improvement rather than only an inconvenience?

   Answer

   There is no key to leak, no open port to attack, and every session or command is authorized by IAM, and CloudTrail records who opened or sent it, to which instance and when. The commands themselves are kept by session logging, which streams each session and Run Command's output to CloudWatch Logs, as the reference VM does.

6. Which permission in the image-push role cannot be limited to the three repositories, and how is that handled?

   Answer

   ecr:GetAuthorizationToken, which is account-wide by design. It is granted on * with a comment saying why, while push and pull are granted only on the three repositories.

The ladder

Rungs 0 and 1 are where many teams started: log in, copy, restart. Rung 2 is the first that is reproducible, because the server runs an image built by CI. Rung 3 adds capacity and redundancy but leaves N machines to update without taking the fleet down. Rung 4 is the big change in thinking: instead of running commands, you declare what should be running, "two copies of version 0.8.2", and an orchestrator such as Kubernetes or ECS works continuously to make reality match, replacing a few copies at a time. Rung 5 removes the machines from view entirely.

"Serverless" does not mean one very reliable server. It means many small, disposable units that the platform replaces when they are unhealthy. Fargate and Cloud Run run your container images; Lambda runs a zip file or an image in lightweight virtual machines. A Python virtual environment isolates packages but not the operating system's libraries or the interpreter build, which is why a platform that wants an image gains nothing from a virtual environment alone.

One VM, without ceremony

A single VM is the right first home for a pilot with few users, where a short night-time maintenance window is acceptable and the team is new to operations. Where ECS asks a new team to learn roles, task definitions, load balancers and networking at once, a VM is one box, one script and logs in one place. The reference service ran on a VM before it ran on ECS, with the same images.

• The image goes to a registry, not to Git. The VM pulls it by version, authenticated by its own role on AWS, and never builds anything.
• The deploy definition matches the version. The compose file and deploy script must match the images, so they come from the same release: shipped by CI with each deploy, or checked out at the release tag. Never a working copy someone edited.
• The deploy script never stops what works: it pulls the version, runs migrations as a one-off container, starts the new api beside the old one and waits for it to be healthy, drains the old one, and only then records the version, so later commands run it too. If the new api never becomes healthy, the script removes it and stops; the old one was never stopped.
• Rollback is the same script with the previous version.

#!/usr/bin/env bash
# Deploy a release to this host without refusing requests.
#
#   make deploy tag=1.4.0          images from $IMAGE_PREFIX (ghcr.io/..., see .env)
#   PULL=0 make deploy tag=local   images already on this host (local testing)
#
# `docker compose up -d` replaces a container by stopping it first: the old
# api drains its in-flight requests while nothing accepts new ones (measured:
# 502s for 6.7s, and for as long as the longest answer in flight). Instead:
#   1. migrate (the old api is still serving: migrations must be backward
#      compatible - add first, remove in a later release)
#   2. start the new api NEXT TO the old one and wait until it is healthy
#   3. wait for nginx to resolve it (resolver valid=10s in default.conf)
#   4. stop the old api: it finishes its in-flight requests (streams
#      included, up to graceful_timeout) while nginx sends new ones to the new
#   5. replace nginx: behind the TLS edge no request fails (the edge retries);
#      where nginx publishes the port itself, ~0.3 s of refused connections;
#      then the edge, only if its image changed (~2 s of refused connections)
# Rollback = deploy the previous tag. The database stays as it is (the
# code rolls back, the schema does not), so the previous tag must be able to
# run on it: the expand/contract rule.

The VM done properly

The reference build ran the same stack on an EC2 virtual machine defined in CDK. What makes it acceptable to a platform team, compared with a hand-built server:

• No SSH key and no port 22. Operators use Session Manager and Run Command, which the instance's role allows. CloudTrail records who opened a session or sent a command; the stack's session logging keeps what was typed, and the Run Command output, in a log group.
• No registry password. A credential helper gets ECR credentials from the instance's role.
• Built from code. cloud-init installs Docker, creates the deploy user, clones the repository for its compose files and scripts, and generates the .env with random passwords at first boot. Because cloud-init runs only once, a changed recipe replaces the VM rather than editing it.
• Small surface. The metadata service accepts only its token-based version (IMDSv2), the disk is encrypted, and the security group admits HTTP and nothing else.

role = iam.Role(
    self,
    "Role",
    assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"),
    managed_policies=[
        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonSSMManagedInstanceCore")
    ],
)
for repo in repos.values():
    repo.grant_pull(role)

# What operators do on the host, not only that they connected. Session Manager reads
# its preferences from one document per account and region, SSM-SessionManagerRunShell:
# this stack creates it (it must not exist already) and deletes it with the stack.
operations = logs.LogGroup(
    self,
    "Operations",
    log_group_name="/triage/vm-operations",
    retention=logs.RetentionDays.ONE_MONTH,  # production: what audits require
    removal_policy=RemovalPolicy.DESTROY,  # production: RETAIN
)

sg = ec2.SecurityGroup(self, "Sg", vpc=vpc, description="HTTP in; no SSH")
sg.add_ingress_rule(ec2.Peer.any_ipv4(), ec2.Port.tcp(80), "the app")

Results on the VM, with requests flowing throughout: the first start of an older release from ECR brought all six services up healthy; the upgrade to the next release migrated, started the new api beside the old one and drained it; and the rollback, across that migration, said "the database is ahead of 0.8.0: rolling back the code only, no migration". 139 requests arrived during the rollback, and none failed.

Making it boring for a team new to operations

The reference runbook for a demo VM even includes the request to send IT: Ubuntu Server LTS, 4 vCPU, 8 GB of memory and 40 GB of disk, with the reasoning. The full stack with monitoring idles at about 880 MiB but its memory limits add up to about 5 GiB; images take 2.9 GB, logs are capped per container, and two weeks of daily database dumps fit in about 1 GB.

Agents on hosts, and immutable infrastructure

A monitoring or log-shipping agent is a small background program that works for a remote service; nothing to do with AI agents. It is needed because the container runtime keeps only each container's standard output, and host metrics such as CPU and disk are a separate job. Often the runtime's logging driver is enough on its own, sending container output straight to CloudWatch.

How an agent reaches a host depends on the rung. On plain VMs it is baked into a golden image built by a tool such as Packer, or installed by the first-boot script. On ECS or Cloud Run it is a setting of the service, with no host to touch. On Kubernetes it is a DaemonSet: one copy per node. In every case the host is rebuilt from a versioned definition, never patched by hand. That is what immutable infrastructure means.

When to climb the next rung

Move off the single VM when there are paying customers, when releases must not drop even the brief gap of replacing the only proxy, or when the VM itself becomes a burden to patch and watch. If the habits of this chapter are kept (CI builds one image per version, and a deploy means "run this version"), the move to ECS in Chapter 9 changes where the images run, not how they are built.

Traps

Questions

1. What changes in your thinking between rung 3 and rung 4 of the ladder?

   Answer

   You stop running commands against machines and start declaring the state you want, such as two copies of a version. The orchestrator reconciles reality to that declaration continuously, including replacing failed copies.

2. Why does recreating a single container cause downtime, and how does a rolling deploy avoid it on one host?

   Answer

   Recreating stops the old container first, so nothing accepts new requests while it drains. A rolling deploy starts the new version beside the old one, waits until it is healthy and reachable, and only then drains the old one.

3. Why should a VM never run git pull and build?

   Answer

   A checkout can carry edits and drift, and builds on different hosts can differ. Pulling a released image means the host runs exactly what CI tested.

4. What makes the reference VM acceptable to a platform team?

   Answer

   No SSH key or open port 22, access through Session Manager, with every session recorded in CloudTrail, images pulled with the instance's own role, a single open port, IMDSv2 and an encrypted disk, and a host rebuilt from code at first boot.

5. When is it time to move from one VM to a managed container platform?

   Answer

   When there are paying customers, when releases must not drop any requests at all, or when the VM becomes a burden. If CI already builds versioned images and deploys mean "run this version", the move is small.

How the tools work

Every infrastructure-as-code tool has the same three parts: a description of the desired end state, a record of what it has already created, and an engine that turns the difference into API calls. The tools differ in the language you write and in where the record lives.

CDK (AWS Cloud Development Kit) lets you write infrastructure in Python, TypeScript and other languages. It synthesizes a CloudFormation template, and CloudFormation creates the resources and remembers them as a stack. Because AWS keeps that record, deleting the stack removes exactly what it created, and drift detection compares reality with it.

Terraform uses its own declarative language, HCL, and reaches any cloud or service through providers. It keeps its record in a state file. That file must live remotely (an S3 bucket, an Azure storage blob, a Google Cloud bucket) with a lock, so two people applying at once cannot corrupt it, and it must be protected: it can contain secrets. Terraform moved to the Business Source License; OpenTofu is its open-source fork. Pulumi offers real programming languages across clouds.

The reference infrastructure, as five stacks

The reference service's AWS infrastructure is about a thousand lines of Python in infra/aws/, split into five stacks by lifetime and cost. Things that are free and permanent (the registry, the CI roles, the network) live apart from the things that cost money by the hour (the VM, the ECS application), so the expensive parts can be destroyed and re-created at will.

app = cdk.App()
env = cdk.Environment(account=os.environ["CDK_DEFAULT_ACCOUNT"], region="ap-south-1")
registry = RegistryStack(app, "triage-registry", env=env)  # Stage 1
CicdStack(app, "triage-cicd", repos=registry.repos, env=env)  # Stage 2
network = NetworkStack(app, "triage-network", env=env)  # shared by 3-5, free
VmStack(app, "triage-vm", vpc=network.vpc, repos=registry.repos, env=env)  # Stage 3
AppStack(
    app, "triage-app", vpc=network.vpc, db_subnets=network.db_subnets, repos=registry.repos, env=env
)  # Stages 4-5
app.synth()

# Delete the database before its master secret. The template refers to the secret only
# inside a {{resolve:secretsmanager:...}} reference, and CloudFormation deleted both at
# once: the DB's delete handler then could not resolve the secret -> DELETE_FAILED
# (recovery: delete-stack --retain-resources for what is already gone).
db.node.add_dependency(owner.node.default_child)  # the secret only, not its attachment

The first attempt at that fix depended on the secret's whole construct, which includes an attachment that itself depends on the database: a cycle. It was caught when the template was synthesized, before anything was deployed, which is one reason to run cdk synth and a template linter in CI.

Drift: when reality and the code disagree

To test scaling, the reference ECS service was scaled from one task to two with a single command-line call: a change made outside the code. CloudFormation's drift detection reported it precisely. Left alone, the next deploy, which updates the service with a new task definition every time, would have set the count back to one without anyone noticing.

That reset depends on the deploy touching the drifted resource. A normal CloudFormation update compares the new template with the previous one, not with reality, so drift on a resource the update leaves alone stays in place, unseen until drift detection runs. Run detection regularly, and reconcile what it finds in code, or with a drift-aware change set (--deployment-mode REVERT_DRIFT), which compares with the actual state and brings drifted resources back. Terraform refreshes the real state before it plans, so its next plan shows the drift and applying it reverts the change.

The last mile: first-boot scripts

A virtual machine still needs something to run on its first boot. cloud-init (passed as user_data on AWS) runs once, at first boot: install the container runtime, log in to the registry with the machine's role, pull a specific version, run it with a restart policy, fetch secrets at runtime. It does not run again on later deploys. A new version either replaces instances (a new launch template version, rolled out by an Auto Scaling group's instance refresh) or runs a command on the existing ones through Run Command. The mature form bakes the runtime and agents into a golden image and keeps the first-boot script to "pull and run".

The same registry in Terraform, on three clouds

The language and the workflow carry over between clouds; the resources do not. An AWS container service has no Azure twin with the same fields, so it is rewritten. The image registry from this chapter's first stack, in Terraform on each cloud:

resource "aws_ecr_repository" "api" {
  name                 = "triage-assistant-api"
  image_tag_mutability = "IMMUTABLE"
  image_scanning_configuration { scan_on_push = true }
}

resource "azurerm_container_registry" "images" {
  name                = "triageassistant"          # globally unique, letters and digits only
  resource_group_name = azurerm_resource_group.app.name
  location            = "centralindia"
  sku                 = "Basic"
}

resource "google_artifact_registry_repository" "images" {
  repository_id = "triage-assistant"
  location      = "asia-south1"
  format        = "DOCKER"
}

Traps

Questions

1. What are the three parts of every infrastructure-as-code tool?

   Answer

   A description of the desired state, a record of what already exists (a CloudFormation stack or a Terraform state file), and an engine that turns the difference into API calls.

2. Why must a Terraform state file be remote and locked?

   Answer

   So the whole team shares one record, and two applies cannot run at once and corrupt it. It must also be access-controlled and encrypted, because it can contain secrets.

3. What is drift, and why is it dangerous even when the manual change was correct?

   Answer

   Drift is a difference between the real infrastructure and the code. If the next deploy updates that resource, it restores what the code says, so a correct manual change is silently undone. If it does not, the change stays unseen, and nobody reviewing the code knows it exists. Either way the code no longer describes what is running.

4. When would you choose Terraform over CDK?

   Answer

   When the organization runs several clouds, or a platform team has already standardized on Terraform. CDK fits AWS-only work by a team that prefers a general-purpose language.

A home router, split into AWS resources

A home router does three jobs in one box, and AWS turns each into a separate resource you configure on its own. The mapping is a good way in.

The parts

• VPC. Your own isolated network, spanning one region, with an address range written in CIDR notation.
• Subnets. Slices of that range, each pinned to one availability zone. Routing is decided per subnet by its route table, which is the real reason subnets exist. The usual minimum is a public and a private subnet in each of two zones, because load balancers and database subnet groups both require two.
• Internet gateway. The doorway. A subnet with a route to it is public.
• NAT gateway. Lets private subnets make outbound connections, to a model provider's API or a package mirror, without accepting inbound ones. It is billed per hour and per gigabyte processed.
• Security groups. Stateful, per-resource, default-deny allow-lists. Stateful means a reply to an allowed request is allowed automatically. A rule can name another security group as its source.
• Network ACLs. A coarser, stateless filter per subnet. Most teams leave them at their defaults unless compliance requires more.

The reference network

The reference stack's network is deliberately cheap, and the code says so. It has two zones with public subnets for the load balancer, the VM and the ECS tasks, and isolated subnets for the database, with no route to the internet in either direction. There is no NAT gateway, because nothing in a private subnet needs to reach out. The trade-off is named in the code: tasks get public IP addresses so they can pull images, and they rely on security groups, not subnets, to stay unreachable.

"""Stages 3-5: one VPC shared by the VM and the ECS stack. A VPC itself costs nothing.

Two availability zones (an ALB and an RDS subnet group both require two). Public subnets hold
the load balancer, the VM and the Fargate tasks; isolated subnets (no route to the internet in
either direction) hold the database. No NAT gateway: nothing in a private subnet needs to reach
out, and one would cost ~$0.06/hour plus data. The trade-off: tasks get public IPs (each
$0.005/hour) and rely on security groups, not subnets, to stay unreachable.
"""

# A constant, not read from the VPC: the app stack trusts proxies in this range
# (FORWARDED_ALLOW_IPS), and a token would make it import from this stack.
VPC_CIDR = "10.20.0.0/16"


class NetworkStack(Stack):
    def __init__(self, scope: Construct, cid: str, **kw) -> None:
        super().__init__(scope, cid, **kw)
        self.vpc = ec2.Vpc(
            self,
            "Vpc",
            ip_addresses=ec2.IpAddresses.cidr(VPC_CIDR),
            # Named, not looked up: synth works without credentials.
            availability_zones=["ap-south-1a", "ap-south-1b"],
            nat_gateways=0,
            subnet_configuration=[
                ec2.SubnetConfiguration(
                    name="public", subnet_type=ec2.SubnetType.PUBLIC, cidr_mask=24
                ),
                ec2.SubnetConfiguration(
                    name="data", subnet_type=ec2.SubnetType.PRIVATE_ISOLATED, cidr_mask=24
                ),
            ],
        )
        # Added after RDS refused 1a and 1b: "no subnets exist in Availability Zones with
        # sufficient capacity ... for db.t4g.micro; choose from ap-south-1c". Capacity is per AZ
        # and changes; an extra isolated subnet (main route table: local routes only) gives the
        # database a third zone without touching the subnets the VM already uses.
        self.data_1c = ec2.Subnet(
            self,
            "Data1c",
            vpc_id=self.vpc.vpc_id,
            availability_zone="ap-south-1c",
            cidr_block="10.20.10.0/24",
        )
        self.db_subnets = [*self.vpc.isolated_subnets, self.data_1c]

Address ranges in CIDR notation

CIDR notation writes a range as an address and the number of fixed bits. 10.20.0.0/16 fixes the first 16 of 32 bits and leaves 16 free: 65,536 addresses. A smaller number after the slash is a bigger range. AWS reserves five addresses in every subnet, so a /24 holds 251 usable ones. A subnet's range caps how many resources it can hold (one address each, not one per connection), which matters for tasks that each get their own network interface.

Toward a production network

The production version of this design moves the tasks into private subnets with no public addresses. Outbound traffic then needs a NAT gateway, and because it is billed per gigabyte, heavy traffic such as image pulls should go through VPC endpoints: private connections to ECR and S3 that skip the NAT gateway. If a security team restricts outbound traffic, the model provider's API host must be on the allowed list, or every assistant request fails. The public DNS record points at the load balancer as an alias, never at an IP address, because the load balancer's addresses change.

Traps

Questions

1. What makes a subnet public?

   Answer

   Its route table has a route to an internet gateway. Nothing about the subnet itself is public or private; routing decides.

2. Why reference security groups rather than IP ranges in rules?

   Answer

   Tasks and instances come and go with new addresses. A rule that names a group covers every current and future member, so autoscaling needs no rule changes, and the intent ("only the api may reach the database") is explicit.

3. How many usable addresses does a /24 subnet have on AWS, and why?

   Answer

   251: a /24 has 256 addresses, and AWS reserves five in every subnet.

4. Why does the reference stack put its tasks in public subnets, and what keeps them safe?

   Answer

   To pull images without paying for a NAT gateway. Their security groups admit traffic only from the load balancer's group, so a public address does not make them reachable. Production would move them to private subnets.

The pieces

Compose, translated to ECS

The Compose file stays as the local development environment, and the task definition is its production counterpart. Five Compose containers often become two or three ECS services plus managed services.

Rehearse the task locally first

Most cloud failures are configuration, and a Fargate task can be rehearsed on one machine before any money is spent: containers in a task share one network namespace, which docker run --network container:<first> reproduces, and a plain Postgres container with only an owner user stands in for RDS, which runs no setup scripts. The rehearsal for the reference build found four problems:

• The production guard worked. With production settings the api refused to start: plain-HTTP cookies and a fake model. Without a domain or a model provider, both were honest exceptions, named one by one in a waiver setting rather than turned off.
• The web image assumed Compose. Its nginx found the api through Docker's internal DNS, which Fargate does not have. On AWS the load balancer routes /api/* straight to the api and strips the prefix, and nginx only serves files.
• A rollback trap. Rolling back across a migration made the migration tool fail on a revision it did not know. The migration step was taught to recognize that case and do a code-only rollback (Chapter 13).
• Least privilege. A setup container creates the application's own database user, allowed to read and write rows but not to change the schema, and does it idempotently, because RDS will not run setup scripts.

How a task starts: init containers

Each api task first runs two short-lived init containers: one creates the least-privileged database user, the next runs migrations as the schema owner. Only when both have exited successfully, and the model sidecar is healthy, does the api container start. Every new task migrates before it serves, which is simple, but tasks can start together (the first deploy starts two), and two migration runs at once collided: one failed. So the runs take turns on a database lock, and a deploy also starts one new task at a time.

One detail of how ECS evaluates those dependencies cost an afternoon. When the database was down, a new task's first init container failed straight away; the second, which depended on it, then never ran; and the api, which depended only on the second, waited forever. The task stayed PENDING for 13 minutes, billed, until stopped by hand.

# Every init container is a direct dependency of the essential api: when one fails, the
# api gives up and ECS stops the task. Through a chain (api -> migrate -> dbinit) a dbinit
# that fails fast leaves migrate "stopped" without running, and the task PENDING forever
# (measured with one-off tasks: the chain stuck >4 min; direct: stopped in 1.5 min).
api.add_container_dependencies(
    ecs.ContainerDependency(
        container=dbinit, condition=ecs.ContainerDependencyCondition.SUCCESS
    ),
    ecs.ContainerDependency(
        container=migrate, condition=ecs.ContainerDependencyCondition.SUCCESS
    ),
    ecs.ContainerDependency(
        container=mock_llm, condition=ecs.ContainerDependencyCondition.HEALTHY
    ),
)

A release on ECS

1. CI copies the release images to ECR, authenticating with an OIDC role (Chapter 5).
2. The deploy registers a new task definition revision with the new image and updates the service.
3. ECS starts new tasks beside the old ones, waits for the load balancer's health check, shifts traffic, and stops the old tasks, which finish their in-flight requests first.
4. If the new tasks keep failing to start, the deployment circuit breaker rolls the service back to the previous revision, and the deploy fails.

Measured on the reference service, with a probe sending requests twice a second: a forward deploy took 3 minutes 17 seconds with 352 requests and none failed, and the deliberate bad release was rolled back in under 8 minutes while the old tasks kept serving. ECS Exec gives a shell into a running task, without SSH, for the rare case that needs one.

No readiness-only signal

The reference api follows the Kubernetes convention of two health endpoints. /health answers "is the process alive?" and checks nothing external. /ready answers "should traffic come here?" and requires the database. On Kubernetes they feed two separate probes, and only the liveness probe restarts anything. ECS has no readiness-only signal. It acts on two checks, and both end in a replaced task: the container health check in the task definition (the reference api's calls /health), and the load balancer's target health check. So a deep check behaves like a liveness check wherever it sits, and a database outage turns into task churn. A drill on the load balancer's check showed exactly that:

The rule on ECS: point the load balancer at a shallow check; rely on the init containers for readiness at start (migrations succeed only if the database answers); and detect outages with an alarm on 5xx responses. The api reconnected four seconds after the database returned, with no restart.

return ecs.FargateService(
    self,
    f"{name.title()}Service",
    service_name=name,
    cluster=cluster,
    task_definition=task,
    desired_count=2,
    # A public IP to pull images without a NAT gateway; the SG admits only the ALB.
    # production: private subnets, a NAT gateway or VPC endpoints for ECR and logs.
    assign_public_ip=True,
    vpc_subnets=public,
    security_groups=[sg],
    min_healthy_percent=100,  # start a new task before stopping an old one
    # 150% of 2 = one extra task at a time during a deploy. Tasks still start together
    # (the first deploy, a scale-out): their migrate init containers then take turns on
    # an advisory lock (apps/api/migrations/env.py).
    max_healthy_percent=150,
    circuit_breaker=ecs.DeploymentCircuitBreaker(enable=True, rollback=True),
    health_check_grace_period=Duration.seconds(120),
    # `aws ecs execute-command`: a shell without SSH. production: off, or limited by
    # IAM to the on-call role, with session logging.

# Liveness, not readiness. ECS replaces any task its load balancer calls unhealthy,
# so a deep check (/ready: the database) turns a database outage into task churn:
# measured, every api task marked unhealthy and replacements stuck in dbinit. Kubernetes
# separates the two (readinessProbe /ready, livenessProbe /health); ECS has no
# readiness-only signal: this check and the container health check both replace tasks.
# Readiness at start is covered by the init containers: migrate succeeds only if the
# database answers. A database outage still shows: /api/ready 503 and the 5xx alarm.
health_check=elbv2.HealthCheck(
    path="/health", interval=Duration.seconds(10), healthy_threshold_count=2
),
# ECS takes a task out of the ALB first, waits this long, and only then sends it
# SIGTERM; at the end of the wait the ALB drops whatever is still in flight. So the
# wait must cover the longest chat stream (LLM_STREAM_TIMEOUT_S, 120 s): at 30 s,
# any answer longer than that was cut during a deploy. Deploys wait up to this long
# per task only while requests are in flight.
deregistration_delay=Duration.seconds(130),

Scaling out, and the state that must be shared

The api's login endpoint is rate-limited per client: 30 attempts a minute, counted in Valkey. While the api ran with a private Valkey container beside it in each task, scaling to two tasks quietly doubled the limit, because each task counted on its own.

The general rule: a sidecar suits state that belongs to one task. Anything that must be counted once across copies (rate limits, sessions, locks) needs a shared store. The fix shipped through a pull request together with two tasks per service, one per zone, and autoscaling from two to four api tasks on requests per target, capped so that four tasks times one worker times ten pooled connections stays well inside the small database's connection limit (two workers of five at first, until a review matched the task's half CPU to one worker).

Autoscaling takes minutes. Under a steady 80 requests a second, the scaling alarm fired five minutes after the load started and four tasks were running a minute later: about six minutes from load to capacity, made of three one-minute data points, the metric's delay and the task start. The k6 load test that drove it sent 33,601 requests with none failed and a 95th-percentile latency of 58 ms. Autoscaling follows trends, so the minimum count must carry a sudden spike on its own. Stopping one task by hand, as a host failure would, caused no failed requests: ECS started a replacement at once.

Running a command in a task

ECS Exec opens a shell in a running container through Systems Manager, with no SSH and no open port. It is how an operator runs a one-off command on the service, such as the runbooks' "turn the assistant off" or "end this user's sessions". The shell gets the container's environment, not the environment of the process the start script launched. The api's start script builds DATABASE_URL from the separate secrets ECS injects, so in an exec shell the command-line tool found no database URL, and the runbooks had no path on AWS until the operator script built it the same way.

cli)
  shift; (($#)) || { echo "usage: ops.sh cli <command> [args], e.g. cli assistant --off --reason '...'" >&2; exit 2; }
  task=$(aws ecs list-tasks --cluster triage --service-name api --desired-status RUNNING --query 'taskArns[0]' --output text)
  [[ $task == arn:* ]] || { echo "no running api task" >&2; exit 1; }
  # container/api.sh composes DATABASE_URL before it starts gunicorn, so a shell opened with ECS
  # Exec has the parts but not the URL: compose it the same way. The arguments travel base64,
  # NUL-separated, so spaces and quotes in them (a reason) arrive as typed.
  args=$(printf '%s\0' "$@" | base64 -w0)
  aws ecs execute-command --cluster triage --task "$task" --container api --interactive --command \
    "sh -c 'export DATABASE_URL=postgresql://\${DB_USER}:\${DB_PASSWORD}@\${DB_HOST}:\${DB_PORT}/\${DB_NAME}; echo $args | base64 -d | xargs -0 python -m app.cli'" ;;

Traps

Questions

1. What is the difference between a task definition and a service?

   Answer

   A task definition is a versioned description of the containers to run. A service keeps a number of copies of one revision running, replaces failed ones, and rolls them to a new revision during a deploy.

2. Why should the load balancer's health check on ECS be shallow?

   Answer

   ECS replaces any task the load balancer calls unhealthy. If the check depends on the database, a database outage makes every task "unhealthy", and ECS churns tasks that cannot help. A shallow check keeps the tasks; the outage is detected by the 5xx alarm instead. The same holds for the container health check in the task definition, which also ends in a replaced task.

3. What do min_healthy_percent=100 and max_healthy_percent=150 mean for a service of two tasks?

   Answer

   Never fewer than two healthy tasks, and at most three running: one new task starts, becomes healthy, then one old task stops, and the cycle repeats.

4. Why did scaling to two tasks double the login rate limit?

   Answer

   Each task had its own Valkey container and counted attempts on its own, so each allowed the full limit. Shared limits need a shared store.

5. How long does autoscaling take to add capacity, and what does that imply?

   Answer

   About six minutes in the reference build, from the start of the load to the new tasks running. The minimum task count must absorb a sudden spike without help.

The model: declare, then reconcile

Kubernetes inverts the deploy script. Instead of a sequence of commands, you submit objects that describe the end state: a Deployment that says "three replicas of this image", a Service that gives them a stable name, a ConfigMap and Secret for configuration. Controllers then run a loop forever: observe the cluster, compare it with the objects, act on the difference. A node dies, three replicas become two, and the loop starts a third elsewhere without anyone being paged.

Two probes, two questions

The biggest practical difference from ECS for the reference service is health. Kubernetes asks two separate questions. A readiness probe decides whether a pod receives traffic: when it fails, the pod is removed from its Service's endpoints, but keeps running. A liveness probe decides whether the container is stuck: when it fails, the container is restarted. The reference api was written for exactly this split, with /ready and /health.

"""Liveness vs readiness - two different questions, two endpoints.

/health (liveness): "is this process alive and its event loop turning?"
No dependency checks: if the database is down, restarting the api does
not help, and a liveness probe that checks it would restart every replica
at once. `async def` on purpose: it runs on the event loop, so it fails
exactly when the loop is blocked, and never waits for a worker thread.

/ready (readiness): "should traffic be sent here right now?" Checks hard
dependencies. Postgres is one; Redis is not - the rate limiter fails open,
so a Redis outage is reported as "degraded" but keeps the instance ready.
Nor is the identity provider: signed-in users carry on without it. Its
state comes from the background check (app/oidc.py), never a call per
probe. Marking every instance unready over a soft dependency turns a
partial outage into a total one.
"""

@router.get("/health")
async def health() -> dict[str, str]:
    return {"status": "ok"}


@router.get("/ready")
async def ready(request: Request) -> JSONResponse:
    state = request.app.state
    database, redis = await asyncio.gather(
        probe(_ping_database(request), timeout_s=2.0),
        probe(state.redis.ping(), timeout_s=0.25),
    )
    idp = state.oidc.reachable
    body = {
        "status": "ready" if database else "not_ready",
        "checks": {
            "database": "ok" if database else "unavailable",
            "redis": "ok" if redis else "degraded",
            "identity_provider": "unknown" if idp is None else "ok" if idp else "degraded",
        },
    }
    return JSONResponse(body, status_code=200 if database else 503)

ECS and Kubernetes, side by side

Of the two difficulties the reference build met on ECS, Kubernetes handles one natively: the api's readiness and liveness split would have worked as designed. The other too, once its configuration stopped assuming Docker. The web image's nginx used to ask Docker's DNS address, 127.0.0.11, which a pod does not have. It now writes its resolver at start from /etc/resolv.conf, and takes the api's address from a setting, API_UPSTREAM: on Kubernetes the service's full name, because nginx's resolver ignores search domains. The problem was in the image's configuration, not in ECS, and the same images could now run on EKS Auto Mode.

Why orchestrators exist at all

Container networking on one host is easy: Compose creates a private network and its service names resolve. Across hosts, that stops. You need IP connectivity between machines and an orchestration layer on top: Kubernetes gives every pod a routable IP through a CNI plugin, and gives services stable names. Placement, health, rescheduling after a node failure, and rolling updates across many hosts are the other jobs a single host never needed.

Traps

Questions

1. What is the reconcile loop, and why does it make deploy scripts unnecessary?

   Answer

   Controllers repeatedly observe the cluster, compare it with the declared objects, and act on the difference. You change the declaration, and the controllers make it true, including after failures, so there is no script of steps to run.

2. What happens to running workloads if the control plane fails?

   Answer

   Running pods keep running, because each node's kubelet carries on. Deploys, autoscaling and recovery from a dead node stop until the control plane is back.

3. What is the difference between a readiness probe and a liveness probe?

   Answer

   Readiness decides whether a pod receives traffic; failing it removes the pod from rotation without restarting it. Liveness decides whether the container is stuck; failing it restarts the container.

4. When would you choose EKS over ECS?

   Answer

   When there are many services and a platform team, when portability across clouds matters, when the Kubernetes ecosystem (operators, GitOps, a mesh) is needed, or when the organization already runs Kubernetes.

The map

The reference service was built on AWS. Everything in this table that is not AWS comes from the providers' documentation rather than from running it, but the mapping is close enough that a team moving clouds changes names, not designs.

Three differences worth knowing

Revisions and traffic splitting

On Cloud Run and Container Apps, every deploy of a new image or new container settings creates an immutable revision, and traffic can be split between revisions by percentage. On Container Apps that takes multiple revision mode: the default, single revision mode, moves all traffic to the new revision once it is ready and retires the old one. Container Apps also keeps secret values and ingress settings app-wide, outside revisions, so changing them creates no revision, and moving traffic cannot roll them back. With splitting on, a canary (5% to the new revision) or a rollback (100% back to the old one) is a setting, not a redeploy. On ECS the equivalent is a rolling deployment with its circuit breaker, or the service's built-in blue/green, canary and linear strategies, which shift traffic between revisions behind a load balancer and roll back on an alarm. CodeDeploy is the older route to the same.

HTTPS out of the box

Cloud Run and Container Apps give each service an HTTPS address with a managed certificate. On AWS, a load balancer needs a certificate from ACM and a domain you control. That is why the reference build's AWS stage ran over plain HTTP, with that exception named explicitly in the service's production checks.

Scale to zero

Cloud Run and Container Apps can run no instance at all while a service is idle, and start one on the next request, at the cost of a cold start. A Fargate service keeps its minimum number of tasks running, and billing, at all times. For an internal tool used a few times a day, scale to zero changes the cost picture; for an interactive service, the cold start usually rules it out.

Terraform across clouds

The language, the plan-and-apply workflow and the state file are the same on every cloud, through each cloud's provider. The resources are not: an ECS service has no Azure twin with the same fields, so it is rewritten as a Container App. Remote state with a lock lives in an S3 bucket, an Azure storage blob (with a lease) or a Google Cloud bucket, and CI authenticates to each through that cloud's OIDC federation. Chapter 7 shows the same registry declared on all three.

Questions

1. How do Cloud Run and Container Apps make a rollback cheaper than a redeploy?

   Answer

   Every deploy of a new image or container settings is an immutable revision, and traffic is split between revisions by configuration (on Container Apps, in multiple revision mode). A rollback sets 100% of traffic back on the previous revision, without building or deploying anything. Settings kept outside revisions, such as Container Apps' secrets and ingress, are not rolled back this way.

2. Why did the reference AWS build run over plain HTTP, when the same service on Cloud Run would not have?

   Answer

   An AWS load balancer needs a certificate for a domain you control, and the build had no domain. Cloud Run and Container Apps give each service a managed HTTPS address by default.

3. What carries over between clouds in Terraform, and what does not?

   Answer

   The language, the plan-and-apply workflow and remote state carry over. The resource definitions do not: each cloud's resources are written for that cloud.

The stages

One change, from issue to production

Every change to the reference service takes the same path, and each step is held by a file in the repository or a setting on GitHub, not by anyone's memory. Chapters 2 to 5 explain the pieces one at a time. Here is the whole path in one place, with the file to read at each step. The timings are from a real release: the fixes in the version this handbook quotes took exactly this path.

Two properties make the path trustworthy. No step depends on someone remembering it: each gate is a file reviewed like code, or a setting only an administrator can override. And every step leaves evidence another person can read later: the issue, the pull request with its checks, the release with its notes, the deploy with its approval.

The infrastructure code, end to end

Everything AWS runs comes from one folder, infra/aws: a single CDK application of five stacks (Chapter 7). Three cost nothing or almost nothing and stay up between sessions: the image registry, the identity that lets CI reach AWS, and the network. Two are billed by the hour and are built for a session and destroyed after it: the single VM of Chapter 6, and the application stack, which holds everything a request touches. Read in the order a release travels, these files are a complete account of how a version tag becomes a running service.

internet -> ALB :80 -+- /api/metrics, /api/alerts/alertmanager -> 404
                     +- /api/*  -> rewritten to /* -> api task (target health: /health)
                     +- else    -> web task, nginx serving the built app (target health: /healthz)

api task (containers share localhost):
  dbinit, migrate   init containers: the app's least-privileged DB role, then migrations as owner
  mock-llm          sidecar
  api               starts after both init containers succeed and the sidecar is healthy
Valkey: one ElastiCache node shared by every api task (rate limits must be counted once).
2 tasks per service, one per AZ; the api scales to 4 on requests per target.

for name in IMAGES:
    self.repos[name] = ecr.Repository(
        self,
        name,
        repository_name=f"triage-assistant-{name}",
        image_scan_on_push=True,
        # Tags are immutable: a tag, once pushed, always means the same bytes.
        image_tag_mutability=ecr.TagMutability.IMMUTABLE,
        # Throwaway account: `cdk destroy` must leave nothing behind.
        removal_policy=RemovalPolicy.DESTROY,
        empty_on_delete=True,
        lifecycle_rules=[
            ecr.LifecycleRule(
                description="expire untagged images after 7 days",
                tag_status=ecr.TagStatus.UNTAGGED,
                max_image_age=Duration.days(7),
            ),
            ecr.LifecycleRule(description="keep the last 20 images", max_image_count=20),
        ],
    )

# --- Who may talk to whom: security groups reference each other, not IP ranges ---------
alb_sg = ec2.SecurityGroup(self, "AlbSg", vpc=vpc, description="internet to ALB port 80")
alb_sg.add_ingress_rule(ec2.Peer.any_ipv4(), ec2.Port.tcp(80), "HTTP")
api_sg = ec2.SecurityGroup(self, "ApiSg", vpc=vpc, description="ALB to api tasks port 8010")
api_sg.add_ingress_rule(alb_sg, ec2.Port.tcp(8010), "from the ALB only")
web_sg = ec2.SecurityGroup(self, "WebSg", vpc=vpc, description="ALB to web tasks port 8080")
web_sg.add_ingress_rule(alb_sg, ec2.Port.tcp(8080), "from the ALB only")
db_sg = ec2.SecurityGroup(
    self,
    "DbSg",
    vpc=vpc,
    description="api tasks to Postgres port 5432",
    allow_all_outbound=False,
)
db_sg.add_ingress_rule(api_sg, ec2.Port.tcp(5432), "from api tasks only")
cache_sg = ec2.SecurityGroup(
    self,
    "CacheSg",
    vpc=vpc,
    description="api tasks to Valkey port 6379",
    allow_all_outbound=False,
)
cache_sg.add_ingress_rule(api_sg, ec2.Port.tcp(6379), "from api tasks only")

# --- Database: managed Postgres in subnets with no internet route ---------------------
db = rds.DatabaseInstance(
    self,
    "Db",
    instance_identifier="triage-db",
    engine=rds.DatabaseInstanceEngine.postgres(version=rds.PostgresEngineVersion.VER_17),
    instance_type=ec2.InstanceType.of(
        ec2.InstanceClass.BURSTABLE4_GRAVITON, ec2.InstanceSize.MICRO
    ),
    vpc=vpc,
    vpc_subnets=ec2.SubnetSelection(subnets=db_subnets),
    availability_zone="ap-south-1c",  # where RDS had capacity (network.py)
    security_groups=[db_sg],
    credentials=rds.Credentials.from_secret(owner),
    database_name=DB_NAME,
    allocated_storage=20,
    storage_type=rds.StorageType.GP2,
    storage_encrypted=True,
    multi_az=False,  # production: True (a standby in the other AZ, ~2x the price)
    backup_retention=Duration.days(1),  # production: 7-35 days, and a restore rehearsed
    delete_automated_backups=True,  # production: False
    deletion_protection=False,  # production: True
    removal_policy=RemovalPolicy.DESTROY,  # production: SNAPSHOT
)

environment={
    **db_env,
    "APP_ENV": "prod",
    "APP_VERSION": tag,
    "ROOT_PATH": "/api",  # the ALB strips /api, as nginx does in compose
    # Whose X-Forwarded-For to believe: the ALB's addresses, i.e. the VPC. Not "*":
    # an ALB appends the client to what the client sent, so with "*" the left-most
    # entry, which the client wrote, became its address (the sign-in rate limit's key).
    "FORWARDED_ALLOW_IPS": VPC_CIDR,
    # One worker per usable CPU, rounded up (gunicorn.conf.py): a 0.5 vCPU task runs
    # one, and capacity comes from tasks. The pool keeps the connection budget.
    "WEB_CONCURRENCY": "1",
    "DB_POOL_SIZE": "10",  # x 1 worker x 4 tasks: inside a db.t4g.micro's ~80
    # Idle connections: the ALB keeps them 130 s (idle_timeout), so the api must keep
    # them longer, or the ALB reuses one the api has just closed and answers 502.
    "GUNICORN_KEEPALIVE": "140",
    # In-flight requests get 110 s after SIGTERM, inside the 120 s stop timeout.
    "GUNICORN_GRACEFUL_TIMEOUT": "110",
    "REDIS_URL": redis_url,

# The api reads one DATABASE_URL; ECS injects the secret as separate values, so compose
# it here. The generated passwords have no punctuation, so they need no URL-encoding.
export DATABASE_URL="postgresql://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}"
exec gunicorn app.asgi:app -c gunicorn.conf.py

elbv2.ApplicationListenerRule(
    self,
    "Internal",
    listener=listener,
    priority=5,
    conditions=[
        elbv2.ListenerCondition.path_patterns(["/api/metrics", "/api/alerts/alertmanager"])
    ],
    action=elbv2.ListenerAction.fixed_response(
        404, content_type="application/json", message_body='{"error":"not_found"}'
    ),
)
api_rule = elbv2.ApplicationListenerRule(
    self,
    "Api",
    listener=listener,
    priority=10,
    conditions=[elbv2.ListenerCondition.path_patterns(["/api/*"])],
    action=elbv2.ListenerAction.forward([api_tg]),
)
# URL rewrite (ALB, 2025): /api/ready -> /ready. Not in the CDK L2 yet, so set it on the
# CloudFormation resource directly (an "escape hatch").
api_rule.node.default_child.add_property_override(
    "Transforms",
    [
        {
            "Type": "url-rewrite",
            "UrlRewriteConfig": {"Rewrites": [{"Regex": "^/api/(.*)$", "Replace": "/$1"}]},
        }
    ],
)

# --- autoscaling: more api tasks while each serves over 1,000 requests a minute --------
# Bounded by the database: 4 tasks x 1 worker x DB_POOL_SIZE 10 = 40 connections, inside
# a db.t4g.micro's ~80. Scale-in waits longer than scale-out, so it does not flap.
scaling = api_svc.auto_scale_task_count(min_capacity=2, max_capacity=4)
scaling.scale_on_request_count(
    "Requests",
    requests_per_target=1000,
    target_group=api_tg,
    scale_out_cooldown=Duration.seconds(60),
    scale_in_cooldown=Duration.seconds(300),
)

# --- alarms (the first 10 are free); production: an SNS topic to the on-call ------------
cw.Alarm(
    self,
    "ApiUnhealthy",
    alarm_name="triage-api-unhealthy-targets",
    metric=api_tg.metrics.unhealthy_host_count(period=Duration.minutes(1)),
    threshold=0,
    comparison_operator=cw.ComparisonOperator.GREATER_THAN_THRESHOLD,
    evaluation_periods=2,
    treat_missing_data=cw.TreatMissingData.NOT_BREACHING,
)
cw.Alarm(
    self,
    "Api5xx",
    alarm_name="triage-api-5xx",
    metric=api_tg.metrics.http_code_target(
        elbv2.HttpCodeTarget.TARGET_5XX_COUNT, period=Duration.minutes(1)
    ),
    threshold=5,
    evaluation_periods=1,
    treat_missing_data=cw.TreatMissingData.NOT_BREACHING,
)

# Rolling deploy; CloudFormation waits until ECS is stable. A release whose tasks keep
# failing is rolled back by the ECS deployment circuit breaker, and this step fails.
- name: cdk deploy triage-app
  working-directory: infra/aws
  run: |
    CDK_DEFAULT_ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
    export CDK_DEFAULT_ACCOUNT
    uv sync --frozen
    npx --yes aws-cdk@2.1143.0 deploy triage-app --exclusively --require-approval never \
      -c tag="$TAG" --outputs-file outputs.json
- id: smoke
  name: Smoke test through the load balancer
  working-directory: infra/aws
  run: |
    url=$(jq -r '."triage-app".Url' outputs.json)
    echo "url=$url" >> "$GITHUB_OUTPUT"
    curl -fsS --retry 5 --retry-delay 5 --retry-all-errors "$url/api/ready"

From demonstration to production

The stack is sized for a throwaway account. Every place production differs is marked in the code with a production: comment or listed in the folder's README, so this list is the code's own:

Images: the same bytes in a second place

The release images already existed in GitHub's registry, built for two processor types. They were copied into ECR as whole multi-architecture indexes, and the index digest in ECR matched the original exactly. Then a deliberate test: pushing a different image over an existing version tag. ECR refused, because the repositories have immutable tags, and the digest behind the tag was unchanged. A tag in that registry always names the same bytes.

One surprise: the registry's scan-on-push never ran on the copied images. It fires for ordinary pushes, and the copy took another path. The lesson is general: scanning belongs in the pipeline, before publishing, not only in the registry (Chapter 4).

The first ECS deploy: two failures, both configuration

• No export named … found. The new application stack referenced the image repositories, which needed new exports from the registry stack, and the deploy had been limited to the new stack alone. Cross-stack references are ordered dependencies: deploy the producer first (Chapter 7).
• "No subnets exist in Availability Zones with sufficient capacity." The database instance type was listed as available in all three zones, yet two had no capacity at that moment. Capacity is per zone and temporary. An extra isolated subnet in the third zone fixed it without touching anything running (Chapter 8).

After those, the containers started in the designed order (setup, migrations, sidecars, then the api), the readiness endpoint reported the database and cache healthy and the identity provider degraded (there was none in this stage), and internal paths returned 404 at the load balancer as intended.

An outage that never happened

During the bad-release drill, a probe on a developer's machine sent a request twice a second and reported 15 failures. The timeline did not fit: failures began after the bad task was already gone and continued after the rollback. Splitting curl's timings showed why.

The whole loop, with no human step

With the deploy workflow in place, the only human action was pushing a version tag (the environment has since gained a required reviewer, so a release now also waits for one approval). Everything after it ran on its own: the release built, scanned and published the images, then smoke-tested them while copying them to ECR; the deploy workflow then ran in the gated environment, applied the infrastructure code with the new version, waited for ECS to be stable, and smoke-tested through the load balancer.

Three design choices made this loop safe to leave unattended. The infrastructure code always comes from main while the image comes from the release, so rolling the application back never rolls the infrastructure back. Deploys share a concurrency group without cancellation, so two deploys queue rather than race. And workflow inputs reach shell commands only through environment variables, never interpolated into the script text, which closes a script-injection route.

Outage drills

The first database-stop drill, run with the deep health check, is the one that changed the design: ECS replaced api tasks whose replacements could not start, and one replacement sat in PENDING for 13 minutes. Chapter 9 explains the health-check rule and the dependency fix that came out of it.

What the on-call looks at

Every container's output went to CloudWatch Logs, one stream per container, and CloudWatch Logs Insights queries them with a SQL-like language. Over the last four hours of the session:

filter @logStream like /^api\// | parse @message "\"level\": \"*\"" as level
| stats count(*) as n by level | sort n desc

Those 1,453 expected warnings are how alert fatigue starts. Alert on symptoms users feel (5xx responses, latency, unhealthy targets), not on log warnings. The load balancer is the source of truth for users, and its totals for the session were: 42,434 requests, no load balancer errors, no connection errors to targets, 281 server errors from the api, every one of them during the two deliberate database outages (honest 503s from readiness), and 107 client errors, which were the rate-limit tests' 429s.

Not built, but worth knowing: Container Insights for per-task CPU and memory, tracing through X-Ray or any OpenTelemetry backend (the api already exports OTLP), and CloudWatch Synthetics, which runs probes from inside AWS and would have avoided the false outage above.

A review against its own rules

The last stage read the whole service against what its own comments and chapters say. Eight gaps were on the path to AWS. Each was reproduced first, then fixed, then shown: on AWS where it could be, otherwise locally.

None of these failed a test or raised an alarm. Each was a rule written down, in a comment or a chapter, that the code did not keep. A team can run the same review: read the comments and documents as claims, and test them like code.

A second pass: the running system

Reading finds what the code says; running it finds what it does. The release was then tested end to end, locally and on a fresh stack: sign-in under attack (a replayed callback, a forged state, another provider, a cross-site sign-out, a reused cookie), team isolation, the rate limit with forged addresses, the load balancer's path rules against encoded and dotted paths, and every failure drill. All of that held. Four things did not:

Two anomalies appeared once each, while faults overlapped, and not again in a dozen targeted runs. They were recorded as an open issue, not "fixed" by a guess, and one was then traced. The database pool had stayed full after a network cut, with the database healthy. The cause was the api's address: Docker usually gives a container another address when it rejoins a network, and every pooled connection stayed bound to the old one. Data sent on them was never acknowledged, so each would hang for about 15 minutes, holding its slot. Reproduced on demand, it was fixed with a socket option, TCP_USER_TIMEOUT, that drops a connection whose data goes unacknowledged for 10 seconds; a new drill fills the pool and changes the address (Handbook 2, Chapter 15). The other, an HTTP 500 during a worker kill, never came back; the drills now keep both containers' logs, so it can be traced if it does. An anomaly that will not reproduce usually means a variable nobody has changed yet.

Traps

Questions

1. Trace a bug fix from issue to production. Which file or setting holds each gate?

   Answer

   The issue form; CONTRIBUTING.md for the branch and commit names; the pre-push hook for lint, types and unit tests; the pull request template and the code owners file; ci.yml, with branch protection requiring all six jobs on an up-to-date branch; the squash-only merge setting; the tag pattern that starts release.yml, and the ruleset that lets only admins create such a tag; release.yml for scanning, building, smoke-testing and copying to ECR; deploy.yml, gated by the aws-demo environment's reviewer, for cdk deploy and the smoke test. A rollback is deploy.yml run by hand with the previous version.

2. On ECS, why does the load balancer route /api rather than nginx?

   Answer

   In compose, nginx finds the api by its service name; this ECS stack gives the api no such name (no service discovery). So on ECS nginx only serves files, and the load balancer does the edge's routing: a rule sends /api/* to the api's target group and rewrites the path, and another answers 404 for the internal paths.

3. The load balancer keeps idle connections 130 seconds, and the longest answer takes 120. What must the api's keep-alive, the deregistration delay and gunicorn's graceful timeout be, and why?

   Answer

   Keep-alive above 130 seconds (140), so the load balancer never reuses a connection the api has just closed. A deregistration delay of at least 120 seconds (130), because the load balancer drops whatever is in flight when the delay ends, before ECS sends SIGTERM. A graceful timeout below the 120-second stop timeout (110), or ECS kills the process before its requests finish.

4. A synthetic check from your laptop reports an outage. What do you look at before declaring an incident?

   Answer

   The server-side metrics for the same window, starting with the load balancer's request count and error counts, and the client's own timings: a slow DNS lookup or connect points at the client's network, not the service.

5. Why does a rollback through the pipeline take about half as long as a release?

   Answer

   It runs only the deploy half. The older images were built, scanned and published when that version was released, so nothing is rebuilt.

6. Why does the deploy workflow take its infrastructure code from main rather than from the release being deployed?

   Answer

   So that deploying an older application version never rolls the infrastructure back. The application is versioned by its images; the infrastructure moves forward through its own pull requests.

Four ways to move traffic

The reference service uses rolling deploys everywhere, and for subtle regressions it currently relies on "deploy, watch the dashboard for half an hour, roll back if needed", which works because rollbacks are cheap and cross migrations. The next steps, in order of effort, are a canary on one host (a second api service and a weighted upstream in nginx), then the platform's own traffic splitting (on ECS, its built-in canary strategy) with automatic analysis against the service's objectives.

Rollbacks roll back code, never the schema

During a rolling deploy, the old and new versions run at the same time against the same database. So a schema change cannot break either of them, and it is done in steps that each leave both versions working. This is the expand and contract pattern.

The rule produced one of the reference service's decision records, after a rollback failed. The procedure had been "deploy the previous tag", and the previous tag's migration tool had never heard of the revision the newer release applied. The deploy stopped before touching anything, so it failed safely, but rollback was impossible across any migration: exactly when a rollback is most likely needed.

# Schema migrations as the owner, before the api starts (an init container).
# On a rollback the database can be ahead of this release: alembic then cannot find the
# current revision, and the right move is a code-only rollback, not a failed deploy.
# Any other error (wrong password, unreachable host) still fails the task.
export MIGRATIONS_DATABASE_URL="postgresql://${DB_OWNER_USER}:${DB_OWNER_PASSWORD}@${DB_HOST}:${DB_PORT}/${DB_NAME}"
if ! out=$(alembic current 2>&1); then
  if printf '%s' "$out" | grep -q "Can't locate revision"; then
    echo "the database is ahead of this release: code-only rollback, no migration"; exit 0
  fi
  printf '%s\n' "$out" >&2; exit 1
fi
exec alembic upgrade head

One rollback per kind of change

Decide, before shipping, how each kind of change will be undone, and practice it once.

Two more rules keep rollbacks honest. Releases are never skipped when one contracts what another expanded: the reference service's row-level-security release hides every row from a release two versions older, which never says who is asking, so a host must pass through the release in between. And a breaking API change is announced in the release notes, never discovered by a client. The same holds for settings. One release added production checks that stopped an older host's api until its settings changed, and its notes, a list of pull request titles, did not say so. The reference release notes now open with the settings added to or removed from .env.example since the previous release.

Handing a service to an infrastructure team

A service that can be released safely also has to be operable by someone else. These are the questions an infrastructure team asks, with the reference service's answers. Every answer points to something already in the repository.

The road to real users

Traps

Questions

1. What does a canary catch that a rolling deploy does not?

   Answer

   Regressions that do not crash anything but show in the numbers: higher error rates, slower answers, worse quality. A rolling deploy only waits for health checks, then moves all traffic within seconds.

2. Why is a column rename done across several releases?

   Answer

   Because during a rolling deploy the old code still runs against the database. Adding the new column, writing both, backfilling, switching reads while still writing both, then stopping writes to the old column, and only later dropping it keeps every running version working, and every rollback to the release before correct, at every step.

3. What should a deploy do when the database is ahead of the image it is deploying?

   Answer

   Treat it as a rollback: skip migrations, say so, and run the older code on the newer schema. Reversing a migration is a separate decision made by a person.

4. Why does each stage on the road to real users need a "done when"?

   Answer

   Each stage introduces one new kind of risk. A measurable exit (a timed restore, real sign-ins per role, the eval gate on the production model) proves that risk is handled before the next one is added.

What it cost, per hour

The reference ECS stack's cost is written into its own code, component by component. Roughly half is compute, and the rest is spread over the managed services that replaced containers.

Two numbers put this in perspective. The first full build of the reference deployment, a VM for about 45 minutes and the ECS stack for about three hours, came to roughly sixty cents; each later session on the ECS stack added cents more, about thirteen for the end-to-end test. And the permanent parts (image repositories holding about 1.3 GB, the CI roles, the network) cost cents a month, so they stay, and the pipeline keeps copying each release into the registry.

Teardown, in order

leftovers)
  echo "EC2:";  aws ec2 describe-instances --filters Name=instance-state-name,Values=pending,running,stopping,stopped --query 'Reservations[].Instances[].[InstanceId,InstanceType,State.Name]' --output text
  echo "RDS:";  aws rds describe-db-instances --query 'DBInstances[].[DBInstanceIdentifier,DBInstanceStatus]' --output text
  echo "ElastiCache:"; aws elasticache describe-replication-groups --query 'ReplicationGroups[].[ReplicationGroupId,Status]' --output text
  echo "ECS:";  aws ecs list-clusters --query clusterArns --output text
  echo "ALB:";  aws elbv2 describe-load-balancers --query 'LoadBalancers[].LoadBalancerName' --output text
  echo "NAT:";  aws ec2 describe-nat-gateways --filter Name=state,Values=available,pending --query 'NatGateways[].NatGatewayId' --output text
  echo "EIP:";  aws ec2 describe-addresses --query 'Addresses[].PublicIp' --output text
  echo "EBS:";  aws ec2 describe-volumes --query 'Volumes[].[VolumeId,Size,State]' --output text
  echo "Secrets:"; aws secretsmanager list-secrets --query 'SecretList[].Name' --output text
  # Every state but deleted: a stack stuck in DELETE_FAILED still holds what it could not delete.
  echo "Stacks:"; aws cloudformation list-stacks --query 'StackSummaries[?StackStatus!=`DELETE_COMPLETE`].[StackName,StackStatus]' --output text ;;

The teardown failed once. CloudFormation deleted the database and its password secret within two seconds of each other, because the template referred to the secret only inside a dynamic reference, which does not order deletes. The database was already gone, but its delete handler could not read the password, so the stack ended in DELETE_FAILED. It was finished by deleting the stack again while retaining the resources that no longer existed, and fixed in code with an explicit dependency (Chapter 7). A fuller check afterwards, beyond what the script covers, found nothing left: no instances, databases, clusters, load balancers, addresses, volumes, caches, alarms, secrets or snapshots.

Lessons from a real build

None of these is dramatic. Together they cost more time than any of the cloud's own surprises.

The shared-checkout incident deserves one more sentence, because coding agents make it common. The safe response was the same as for any surprise: stop, read git status, git log and the open pull requests before changing anything; follow the decision already merged rather than fighting it; recover files from Git's own history; and rebuild the change on a fresh branch from the current main.

Questions

1. Why is the first teardown step disabling the deploy workflow, not destroying the stack?

   Answer

   With the deploy workflow wired up, the next release tag would start a deploy that recreates the whole stack once someone approves it. Deleting the secret that lets the deploy run makes the teardown stick.

2. Which item of the reference stack's hourly cost surprises people most, and what does it suggest?

   Answer

   Public IPv4 addresses, the second-largest item. Production networks keep tasks on private addresses behind a load balancer, which also removes them from the internet.

3. What should happen after a teardown fails halfway?

   Answer

   Finish it (for example by retaining resources that are already gone), check for leftovers, and fix the cause in code so the next teardown works.