Handbook 2 of 3Modern Software Engineering

The map

The reference repository is grouped below by purpose rather than by folder. Each group answers one question a newcomer asks.

The way in

A newcomer should go from cloning the repository to a running stack in under an hour, ideally with one command, and know what to read next. Four files carry that:

• README says what the service is and how to run it.
• START_HERE is a reading path in four levels: how one question gets answered (the core), the production shape, operations, and the AI depth. Each level says what to read, what to run, and what you can explain afterwards.
• The code map gives one line per file, saying what it is for, with a star on every file a user's question passes through, in the order it passes.
• AGENTS.md is the contract for AI coding tools such as Claude Code and Codex: the commands, the conventions, and what they must not do. Tools read it automatically from the repository root; when an agent gets something wrong twice, the fix belongs in this file.

| File | What it is for |
|---|---|
| `asgi.py` | the entrypoint gunicorn and uvicorn load; process-wide setup (logging, tracing) |
| `main.py` | the application factory: creates the engine, clients and settings in the lifespan, mounts the routes |
| `config.py` | every setting, read from the environment and checked at startup; production refuses unsafe ones (ADR-0023) |
| `routes/chat.py` ★ | `POST /chat/stream`: checks the rate limit, then the off switch; gathers the context, streams the answer |
| `queries.py` ★ | the reads shared by routes: the alert list and the chat's context use the same visibility rule |
| `runbooks.py` ★ | runbooks split into sections, embedded, and searched (full-text and vector, the asker's teams only) |
| `redact.py` ★ | removes credentials from alerts, runbooks and questions before any model sees them |
| `triage.py` ★ | the prompt (`SYSTEM_PROMPT`), what the model is given, and its stream turned into client events |
| `llm.py` ★ | the seam to any OpenAI-compatible model: streaming, timeouts, typed errors, token counts |
| `sse.py` ★ | the Server-Sent Events wire format: one JSON line per event, heartbeats |

## What agents should NOT do
- Don't modify `infra/` without being explicitly asked.
- Don't add a new third-party service/dependency without flagging it in
  the PR description first.
- Don't edit a merged `docs/adr/*.md` file — write a new ADR that
  supersedes it instead.
- Don't make a scan pass by exempting its finding. An entry in
  `.trivyignore.yaml` needs a statement (why, who accepted) and an expiry
  at most 90 days out; a line in `.gitleaks.toml` or `.gitleaksignore`
  needs the value shown to be fake. A real secret is rotated first.
- Don't weaken, delete or re-label an eval case, a calibration answer or a
  retrieval question to make a run pass. A failing case is a claim to investigate: read the
  answer and the judge's reason, and change a check only when the check is
  shown wrong, in its own commit, saying why in the case's `notes`.

Documents, by job and lifetime

A team writes several kinds of document, and confusing them is how documentation rots. Each answers a different question, for a different reader, and lives for a different length of time.

Architecture decision records

An architecture decision record (ADR) is a one-page record of one decision: the context that forced it, the decision, the alternatives considered and rejected, and the consequences, good and bad. It is written when the decision is made and never edited afterwards. When the decision changes, a new ADR supersedes the old one, which keeps the history of why the system is the way it is. Teams keep them so that the next engineer, or the infrastructure team inheriting the service, can learn the reasoning without asking around.

The reference repository holds 26 ADRs, and most of them can be seen at work in the code. Take ADR-0015: it decides that a rollback deploys the old code and never reverses the database schema. You can see it carried out in the deploy script: before migrating, it asks whether the image knows the database's current revision; if not, the database is ahead, so this is a rollback, and it runs the older code on the newer schema without migrating.

# A rollback: the database already carries a newer release's migrations,
# which this image's Alembic has never heard of ("Can't locate revision").
# Run this release's code on the newer schema, without migrating. That is
# safe only if this release can run on it - down to the release before a
# contract migration, never past it (v0.3.0 -> v0.2.0 yes, -> v0.1.0 no).
# Migrations are never reversed automatically: `alembic downgrade` is a
# deliberate, separate step.
current=$("${COMPOSE[@]}" exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SELECT version_num FROM alembic_version" 2>/dev/null' || true)
ahead=no
if [ -n "$current" ] && ! shown=$("${COMPOSE[@]}" run --rm --no-deps -T migrate alembic show "$current" 2>&1); then
  # Only this message means "a newer release migrated it". Any other failure
  # (Docker unavailable, an image that does not start) is a failed deploy,
  # not a rollback: reading it as one would skip the migration.
  if ! grep -q "Can't locate revision" <<< "$shown"; then
    printf '%s\n' "$shown" >&2
    echo "could not check the database's revision against $TAG: deploy stopped" >&2
    exit 1
  fi
  ahead=yes
fi
if [ "$ahead" = yes ]; then
  step "the database ($current) is ahead of $TAG: rolling back the code only, no migration"
else
  step "migrate"
  "${COMPOSE[@]}" run --rm migrate
fi

Configuration as a contract

Twelve-factor applications take their configuration from the environment, and the reference service adds two safeguards around that. .env.example is committed: it lists what a first run sets, with placeholders and comments, and the settings reference in the repository's docs lists every variable, with its default. The real .env is generated from it with fresh secrets and is never committed. At startup, typed settings are validated, and in production mode the service refuses to start on unsafe values, naming each problem.

def production_problems(settings: Settings) -> dict[str, str]:
    """The production checks these settings fail: name -> what is wrong."""
    problems: dict[str, str] = {}
    public = _host(settings.public_url)
    if public in _LOCAL_HOSTS or public.endswith(".localhost"):
        problems["localhost_url"] = (
            f"PUBLIC_URL is {settings.public_url}: no other machine reaches it, and "
            "sign-in redirects there. Set the address users open."
        )
    if not settings.session_cookie_secure or settings.public_url.startswith("http://"):
        problems["insecure_cookies"] = (
            "session cookies would travel over plain HTTP: PUBLIC_URL must be https:// "
            "and SESSION_COOKIE_SECURE true."
        )
    if (
        _host(settings.llm_base_url) == "mock-llm"
        or settings.llm_model.startswith("mock")
        or (settings.embedding_model or "").startswith("mock")
    ):
        problems["mock_model"] = (
            "LLM_* or EMBEDDING_MODEL name the mock: set the provider's, and drop "
            '"mock" from COMPOSE_PROFILES.'
        )
    if _host(settings.oidc_metadata_url) == "keycloak":
        problems["demo_identity_provider"] = (
            "sign-in goes to the bundled Keycloak and its demo users: set OIDC_* to the "
            'organisation\'s provider, and drop "idp" from COMPOSE_PROFILES.'
        )

One command interface, and an overlay per environment

Every toolchain runs in a container, so the host needs only Docker, make and Git. A Makefile is the single entry point: its targets are thin wrappers over the real container commands, so anyone can read a recipe and run the underlying command directly. Running the default target lists the commands a newcomer needs in their first weeks.

Environments are Compose overlays on one shared base file, so development, tests and the production shape cannot drift apart without it showing in a diff.

Quality gates, as code

Every check a change must pass is a file in the repository, and so is every exception to a check.

• Pipelines: CI on every pull request, a release on every version tag, a weekly scan, a deploy (Handbook 3).
• Review rules: code owners name who must approve each path, and the pull request template asks for the evidence.
• Updates: Dependabot opens grouped update pull requests for packages, actions and base images.
• Exceptions with expiry: accepted vulnerabilities and known fake secrets live in files that code owners must approve, and accepted risks expire within 90 days.
• Monitoring: alert rules ship with unit tests that feed synthetic metrics and assert which alerts fire; dashboards are generated by a script, so an edit made in the dashboard tool cannot be lost on reload.

tests:
  - name: An unreachable identity provider fires after two minutes, on any worker
    interval: 30s
    input_series:
      - series: 'identity_provider_up{instance="api-1"}'
        values: "1 1 0x10"
      - series: 'identity_provider_up{instance="api-2"}'
        values: "1x12"
    alert_rule_test:
      - eval_time: 1m30s
        alertname: IdentityProviderDown
        exp_alerts: []
      - eval_time: 4m
        alertname: IdentityProviderDown
        exp_alerts:
          - exp_labels: {severity: high}
            exp_annotations:
              summary: "the identity provider does not answer: new sign-ins fail (signed-in users are unaffected)"
              runbook_url: "https://github.com/Parzon/triage-assistant/blob/main/docs/runbooks/alerts.md#identityproviderdown"

One shape for every project

The reference repository is also a template, and its first decision record explains why. The team's job is proving AI capabilities; services that prove out are handed to infrastructure engineers to run. For that handover to be fast, everything that is not the AI logic itself should look the same from project to project, so an engineer inheriting the seventh project already knows where everything is. Chapter 19 covers starting a new service from it: what to keep, what to replace and what to configure.

Traps

Questions

1. What is the difference between a PRD, an RFC and an ADR, and how long does each live?

   Answer

   A PRD says what to build and why, and lives until the feature ships. An RFC proposes how, and lives through its review. An ADR records one decision and its reasons, and is permanent: superseded by a later ADR, never edited.

2. Why is a merged decision record never edited?

   Answer

   Because its value is the record of what was decided and why at the time. A new record that supersedes it keeps both the old reasoning and the reason for the change, so the history reads in order.

3. Why does .env.example belong in Git while .env never does?

   Answer

   .env.example is the contract: what a first run sets, with placeholders and explanations, and no secrets (the settings reference lists every variable and its default). .env holds real values and secrets for one machine, generated from the example.

4. What do code owners add to branch protection?

   Answer

   They say whose approval each path needs, so a change to migrations, infrastructure, decision records or accepted risks cannot merge without the people responsible for them.

5. Why unit-test alert rules?

   Answer

   An alert that never fires, or fires on a blip, is discovered during an incident. A test with synthetic metrics proves when it fires and with which labels, and documents the rule's intent.

The system on one page

Four request paths cover almost everything the service does:

• Sign in. The browser goes to the api's login route, then to the identity provider's page, then back to the api's callback. The api exchanges the one-time code over its own back channel, verifies the ID token, copies the user's teams and roles from the token, and sets a session cookie (Chapter 8).
• Read alerts. Edge, nginx, api (cookie to user to teams, in one query), pooler, Postgres: only the caller's teams' alerts, in about 4 ms at 500 requests a second.
• Ask the assistant. The api finds the runbook sections that match the question and loads the asker's visible alerts, removes credentials, and streams the model's answer back as Server-Sent Events with a heartbeat every 15 seconds, then the sections it cited. Stop in the browser cancels the model call.
• The alert webhook. Alertmanager posts to the api with a bearer token, and the alert is stored in the team its label names.

One question, through the code

The core path is the one to know by heart. It touches about eight modules, and each does one thing.

# The model sees exactly what the asker may see: their teams' alerts,
# read with the same query as the alert list, and their teams' runbook
# sections, searched under the same rule. Loaded before the response
# starts, so a database failure is still a clean JSON 503; the session
# is closed when this function returns (DbSession's scope), so no
# connection is held while the model streams for tens of seconds.
retrieval: Retrieval | None = None
if settings.rag_context_chunks and state.llm.embedding_model is not None:
    # First: it embeds the question with no transaction open.
    retrieval = await search_runbooks(
        db,
        state.llm,
        settings,
        payload.message,
        principal.team_ids(),
        k=settings.rag_context_chunks,
    )
alerts = await newest_alerts(db, principal.team_ids(), limit=settings.chat_context_alerts)
sections = retrieval.hits if retrieval else []
messages = build_messages(payload.message, alerts, sections)

The data

A modular monolith, on purpose

The api is one process type, one deploy and one database, with visible internal boundaries: the AI logic sits behind one seam (llm.py), with the prompt in triage.py, retrieval in runbooks.py and the agent in agent.py and tools.py; identity is three files (one speaks the protocol, one turns a cookie into a user, one decides what that user may do), and the rest is the standard scaffold every service from the template shares. The web app is a separate artifact because it has a different toolchain and runtime, not because it is a microservice.

Splitting was considered and measured against. Every network call between services can hang, time out, or succeed after the caller gave up, and making a single dependency (Postgres through the pooler) fail cleanly took a whole decision record's worth of work. Latency adds up hop by hop, each service needs its own pipeline, dashboards and runbook, and data that used to join becomes an API call or a copy that drifts. Splitting pays off with several teams or a component with a very different load profile, and neither is true yet. Complexity should follow measured pain.

The pattern: the first splits are deployments of the same codebase, which buys independent scaling without network calls between your own components. And the scaling path comes in a fixed order, cheapest first: fix the query (the first real bottleneck was a missing index, p95 7 seconds down to 4 ms), give the api more CPU, add stateless replicas behind a load balancer, separate chat from reads, add database read replicas once the database is the measured bottleneck, and put bursty writes behind a queue.

Traps

Questions

1. Why does the chat route load the alerts and runbook sections before the response starts streaming?

   Answer

   So a database failure can still be answered with a clean JSON error, and so no database connection is held for the tens of seconds the model streams.

2. What makes the api safe to scale out to many replicas?

   Answer

   It is stateless: sessions live in Postgres and rate-limit counters in Valkey, so any replica can serve any request without sticky sessions. The database connection budget sets the practical limit.

3. When would you split the streaming chat out, and how?

   Answer

   When chat dominates the api's CPU or needs its own scaling policy. The first step is the same image deployed twice, with the load balancer sending only the chat route to the second deployment: no code split.

4. What is the first thing to try when the service is slow?

   Answer

   Find and fix the slow query. The reference service's first bottleneck was a missing index; no amount of scaling would have fixed it.

Exit codes and signals

A process's exit code is the first clue in any container post-mortem, and the important ones are few. Codes above 128 are normally not chosen by the program: the shell, or the container runtime, reports 128 plus the number of the signal that killed it. A program can still exit with such a code itself, so it is a strong clue, not proof.

Stopping a process gracefully

A graceful stop has three steps. The runtime sends SIGTERM; the server stops accepting new connections and finishes what it is doing; if it has not exited when the grace period ends, the runtime sends SIGKILL. docker stop waits 10 seconds by default; docker kill sends SIGKILL at once. For a service that streams model answers for up to two minutes, the default is far too short.

What holds a port, and which process is which

When something already listens on a port, ss -ltnp shows the listening socket and the process that owns it (an empty process column means another user owns it, so run it again with sudo). ps -p <PID> -o pid,ppid,user,lstart,cmd says who started it and when, and pstree -p <PID> shows its family. One parent with several identical children is the pre-fork worker model: one master that opened the socket, and workers that inherited it (Chapter 5).

The same tools explain the classic "port is already allocated" error from Docker: usually a forgotten stack still running. And when a network test gives a suspiciously perfect result, such as a 0.02 ms response or an instant success, the command probably ran on the wrong machine.

Limits the kernel enforces

A container is an ordinary process that the kernel isolates with two features. Namespaces give it a private view: its own process numbering, network stack and filesystem root. Control groups (cgroups) limit and account its resources: memory, CPU time, disk throughput and the number of processes. Disk space and GPUs are not cgroup limits; GPUs are handed to containers as whole devices.

Three more silent failures of memory limits are worth knowing. An out-of-memory kill of one worker inside a container leaves the container "healthy": only a log line from the parent, a cgroup counter and an alert record it. Docker's own OOMKilled flag can read false even after the main process was killed; the kernel's log (journalctl -k) is the authority. And lowering a live container's limit kills processes even when swap is allowed.

def available_cpus(cpu_max: Path = Path("/sys/fs/cgroup/cpu.max")) -> int:
    """CPUs this container may actually use.

    os.cpu_count() reports the host's cores even inside a CPU-limited
    container. A `--cpus` / compose `cpus:` / Kubernetes CPU limit is the
    cgroup v2 quota in cpu.max: "<quota> <period>", or "max <period>" when
    unlimited. sched_getaffinity() covers `--cpuset-cpus`. An ECS
    container's `cpu` is a relative share, not a quota: set WEB_CONCURRENCY
    there (infra/aws/service.py does).
    """
    try:
        quota, period = cpu_max.read_text().split()
        if quota != "max":
            return max(1, -(-int(quota) // int(period)))  # ceiling division
    except (OSError, ValueError):
        pass
    return len(os.sched_getaffinity(0))

How many containers fit on one host?

There is no fixed number. An idle container costs a few megabytes; the limit is the sum of what the containers actually use: memory (the working set, not the idle size), CPU, process counts, open files, disk and network connection tracking. Measure with docker stats, add up the working sets, add 20 to 30 percent headroom and the host's own needs, and give each container a memory limit so that one leak kills only itself. Memory is the resource that kills; the others mostly degrade: CPU starvation, process or file-descriptor exhaustion, a full disk from image layers and logs.

Distributions, and the images built on them

A distribution is the kernel plus the userland, a package manager and defaults. For Python services, the choice of base image matters more than it looks: wheels built for glibc (tagged manylinux) do not run on Alpine's musl library. Many popular packages now also publish musllinux wheels, but any package that does not is compiled from source, or fails to install. A slim Debian image is the safer default.

Debugging at the kernel boundary

• The kernel's own log, journalctl -k or dmesg, is the authority on who the out-of-memory killer killed and why.
• strace stops a process at every system call: extremely informative and extremely slow, for a single reproduced problem.
• Slim images have no ps: docker top from the host lists a container's processes.
• A sidecar debug container started in the target's network namespace sees exactly what the application sees, with tools the application image does not carry.

Traps

Questions

1. A container exits with 137 and nothing in its log. What happened, most likely, and where do you confirm it?

   Answer

   It was killed by SIGKILL, almost always the kernel's out-of-memory killer enforcing the container's memory limit. Confirm it in the kernel log and the cgroup's OOM counter; Docker's own flag is not always reliable.

2. Why must the application's drain time be shorter than the platform's stop grace period?

   Answer

   Because when the grace period ends, the platform sends SIGKILL. If the application is still finishing a long stream, that stream is cut off mid-answer.

3. Why can a 50 MB memory limit let a container use 95 MB?

   Answer

   With swap available and no swap limit, Docker allows as much again in swap. The container slows down instead of being killed. Setting swap equal to memory makes the limit a wall.

4. Why might a Python service fail to install its dependencies on an Alpine base image?

   Answer

   Alpine uses the musl C library, and wheels built for glibc (manylinux) do not run on it. A package without a musllinux wheel falls back to compiling from source, which needs compilers and headers, or fails.

A request, layer by layer

What TLS actually guarantees

Encryption solves three separate problems: confidentiality (nobody can read it), integrity (nobody can change it unnoticed) and authentication (the server is who it claims). TLS combines four tools to get all three.

Why a certificate cannot decrypt anything: it holds only the public key. The private key never leaves the server, and the session key does the encrypting. A machine in the middle cannot sign the handshake as example.com without that private key, so the client's checks fail. When both sides present certificates, the connection is mutual TLS, common between services.

Three related distinctions come up constantly. In transit means TLS, SSH or a VPN; at rest means disks, databases and object storage, usually with envelope encryption: a data key encrypts the data, and a master key in a key management service encrypts the data key. Encryption is reversible with a key; hashing is one-way (passwords are stored with a slow, salted password hash such as Argon2id or bcrypt, never encrypted); encoding, such as base64, is formatting, not secrecy.

HTTP: versions and status codes

HTTP/1.1 sends one request at a time per connection, so browsers open about six. HTTP/2 is binary and multiplexes many streams over one TCP connection, with header compression, but one lost packet stalls every stream on it (TCP's head-of-line blocking). HTTP/3 runs over QUIC, built on UDP: loss affects only its own stream, TLS 1.3 is built in, handshakes are faster, and connections survive a change of network. The common setup is HTTP/2 or 3 between clients and the edge, and HTTP/1.1 behind it. gRPC needs HTTP/2.

"I can't reach it": a method

The most common single cause is the bind address. 127.0.0.1 (the loopback interface) never leaves the machine, so a server bound there accepts only local connections; 0.0.0.0 listens on every interface. Inside a container, a server bound to loopback is unreachable even through a published port, because the published port arrives on the container's own network interface.

The opposite mistake is just as common in development: publishing a port on all interfaces. Docker's published ports are inserted into the host's packet filtering before the host firewall's rules, so a port published on 0.0.0.0 is reachable from the network even when the host firewall says otherwise.

# Development overrides, merged automatically by plain `docker compose ...`.
# Source is bind-mounted for hot reload. Ports bind to 127.0.0.1 unless
# BIND_ADDR says otherwise, so a dev stack is not reachable from the network
# (published ports bypass host firewalls such as ufw).
services:
  api:
    build:
      target: dev
      # Your UID and GID for the image's `dev` user (make exports them).
      args:
        DEV_UID: ${DEV_UID:-1000}
        DEV_GID: ${DEV_GID:-1000}
    ports:
      - "${BIND_ADDR:-127.0.0.1}:${API_PORT:-8010}:8010"

Private access without open ports

Two tools changed how small teams reach machines. Tailscale builds a private WireGuard mesh: every device gets a stable private address, and nothing is exposed publicly. Cloudflare Tunnel runs a small agent on the machine that opens an outbound connection to Cloudflare, which maps public hostnames to local ports, so the machine never listens on the internet at all.

Below HTTP, two transports remain. TCP is reliable and ordered, with a handshake, retransmission and congestion control. UDP is fire-and-forget datagrams with none of that: low overhead for DNS, video, games and QUIC.

Traps

Questions

1. What is the difference between "connection refused" and a timeout, and where does each point you?

   Answer

   Refused means a packet reached the machine and was rejected, so the path works and nothing listens on that port: look inside the host or container. A timeout means packets vanished: look along the path, at firewalls, routes, or a frozen process.

2. Why can a public certificate not be used to decrypt traffic?

   Answer

   It contains only the public key. The data is encrypted with a session key agreed per connection, and proving identity needs the private key, which never leaves the server.

3. What is forward secrecy, and what provides it?

   Answer

   The property that a later theft of the server's private key cannot decrypt traffic recorded earlier. It comes from ephemeral Diffie-Hellman key agreement: a new secret for every session, never stored.

4. A proxy returns 502 and later 504. What does each suggest about the backend?

   Answer

   502: the proxy got no valid response, so the backend is down or crashed. 504: the proxy timed out, either connecting (the backend is unreachable) or waiting for the reply (the backend is up but too slow); the proxy's error log says which.

Four things that are easy to confuse

How the event loop actually waits

A suspended coroutine is not a paused thread. Calling an async def function runs nothing; it returns a coroutine object. The event loop runs one coroutine at a time on one thread, and when a coroutine awaits I/O, the loop parks it and blocks in a single kernel call, epoll_wait(), until any registered socket is ready or the nearest timer is due. Three concurrent two-second sleeps make one two-second wait, and finish in two seconds, not six.

Two consequences show up in production. On a busy loop, wall-clock timeouts expire although the dependency answered in time: the reply waits in the loop's queue while the budget runs out. The reference service saw its rate limiter failing open and pool timeouts firing at about 46% CPU. So CPU percentage is a poor saturation signal for async Python; event-loop lag, how late a timer fires, measures it directly, and the service exports it as a metric. And cancellation is normal: a user closing the tab cancels the stream, and finally blocks must close the model connection, which stops the billing.

Workers: processes for CPU

One async process already holds thousands of waiting connections, so extra workers are for CPU: roughly one per core. They are processes, not threads, because each process has its own interpreter lock, so N workers really use N cores.

# Async workers: one per usable CPU. Each worker's event loop already
# serves many concurrent connections; the classic "2 x cores + 1" rule is
# for sync workers that block while waiting on I/O. On ECS/Kubernetes set
# WEB_CONCURRENCY=1 and scale with replicas instead.
workers = int(os.environ.get("WEB_CONCURRENCY") or available_cpus())
worker_class = "uvicorn_worker.UvicornWorker"

# A heartbeat, not a request deadline: an async worker is killed only when
# its event loop is blocked this long (a stuck synchronous call). Long SSE
# streams are unaffected.
timeout = int(os.environ.get("GUNICORN_TIMEOUT", "30"))

# On SIGTERM, in-flight requests get this long to finish. It must cover the
# longest LLM stream (LLM_STREAM_TIMEOUT_S) and stay below the container
# stop grace period (compose stop_grace_period, ECS stopTimeout, k8s
# terminationGracePeriodSeconds), which otherwise SIGKILLs first.
graceful_timeout = int(os.environ.get("GUNICORN_GRACEFUL_TIMEOUT", "120"))

# Idle keep-alive must outlive the proxy's upstream keep-alive (nginx
# keepalive_timeout 60s; an AWS ALB idle timeout likewise). Otherwise the
# proxy reuses a connection the worker has just closed: sporadic 502s.
keepalive = int(os.environ.get("GUNICORN_KEEPALIVE", "75"))

Dependencies and environments

A virtual environment is just a directory: bin/python is a link to the real interpreter, and only third-party packages are installed in it. Activating it merely puts its bin first on the shell's path, which is why venv/bin/python app.py works without activation, and why services and images run it by path. Modern Linux distributions refuse a global pip install because their own tools are Python programs pinned to specific package versions.

Compiled wheels are built per operating system and per processor: an Apple Silicon laptop builds arm64 images unless told otherwise, GPU builds of some libraries exist only for Linux on NVIDIA, and browser-automation tools need browser binaries and a dozen system libraries. The structural fix is to put system dependencies in the Dockerfile and develop inside the image that ships. The reliable check is to build and run the image.

# Only the manifests are mounted, so this layer is rebuilt only when
# dependencies change. The cache mount keeps downloaded wheels between
# builds without baking them into a layer.
RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    uv sync --frozen --no-dev --no-install-project

Traps

Questions

1. Why do three concurrent two-second sleeps on an event loop finish in two seconds?

   Answer

   Each await parks its coroutine, and the loop blocks once in the kernel until the nearest timer is due. All three timers expire together, so the waits overlap instead of adding up.

2. How many async workers should a service run, and why not more?

   Answer

   About one per usable CPU, read from the container's limit. Each worker already handles thousands of waiting connections; extra workers only add memory and database connections without adding CPU.

3. Why is CPU percentage a poor saturation signal for async Python?

   Answer

   Work queues on the event loop long before CPU reaches 100%, and timers start firing late. Event-loop lag measures that queueing directly.

4. What does uv.lock give you that pyproject.toml does not?

   Answer

   The full resolved dependency graph with exact versions and hashes. The manifest lists only the direct dependencies you chose; the lock makes every install identical.

Migrations: a history every database replays

Code and database must agree, and there are many databases: each developer's, CI's throwaway one, staging, production. A migration framework (Alembic for SQLAlchemy here) keeps them aligned. Each migration file has a random identifier and names its parent, so the history is a linked list, not a numbered sequence. Each database stores the revision it is at, and an upgrade runs only the files after it.

Changing a table that is in use

During a deploy the old release keeps serving while migrations run, so a migration must not break it and must not block it. Three techniques do most of the work, and the reference repository has a migration that uses each.

• Add, don't change. A nullable column, or one with a constant default, is instant (metadata only). A volatile default, such as gen_random_uuid() or clock_timestamp(), rewrites the whole table under an exclusive lock. Renames and type changes become expand and contract across releases (Handbook 3, Chapter 13).
• Build indexes concurrently. A plain CREATE INDEX blocks writes for the whole build. CONCURRENTLY does not, but cannot run inside a transaction, and if it fails midway it leaves an invalid index to drop before retrying.
• Add foreign keys in two steps. NOT VALID is instant and applies to new rows; VALIDATE CONSTRAINT in its own transaction then checks existing rows without blocking writes. The reference teams migration used all three patterns on 2 million rows in 1.5 seconds under load, with no failed request.
• Keep the lock timeout for the whole run. Alembic wraps a run in one transaction, and an autocommit block commits it. A SET LOCAL ends with that commit: the reference once set its timeout that way, and measured, it read 5 s at the start of a run and 0 inside the first block, after it, and in every later migration, so a run past the teams migration enabled row-level security on alerts, which takes an exclusive lock, with no timeout. It now uses a plain SET on the run's own connection, which lasts for the session: 5 s at every step, CREATE INDEX CONCURRENTLY included (one that gives up leaves an invalid index, to drop before retrying). Every test run now checks it: probe migrations through the real migration setup, failing if the timeout is ever not 5 s.
• Name every constraint, or the tool cannot write a downgrade that drops it.
• Migrations run as the schema owner, directly against Postgres. The application's own role can read and write rows but cannot change the schema.
• One run at a time. Where every copy of the service migrates as it starts (an init container per task), two can start together, and two runs at once collided: one failed, three times in three. The fix is an advisory lock, and the first two versions failed in instructive ways. A transaction lock is released when a CONCURRENTLY migration commits midway. A blocking session lock makes the waiting run hold a snapshot that CREATE INDEX CONCURRENTLY waits for, and Postgres reports a deadlock. What works is a session lock on a separate connection, polled, so the waiting run holds nothing.

"""add alert external id

Zero-downtime shape, the pattern for any index on a table in use:
- ADD COLUMN ... NULL is instant (metadata only, no table rewrite).
- A plain CREATE INDEX blocks every INSERT/UPDATE/DELETE until it
  finishes; CONCURRENTLY builds without blocking writes, but cannot run
  inside a transaction - hence the autocommit block. If it fails midway
  it leaves an INVALID index that must be dropped before retrying.

def upgrade() -> None:
    op.add_column("alerts", sa.Column("external_id", sa.Text(), nullable=True))
    with op.get_context().autocommit_block():
        op.create_index(
            "uq_alerts_external_id",
            "alerts",
            ["external_id"],
            unique=True,
            postgresql_concurrently=True,
        )

Indexes

An index is a separate, ordered structure that lets the database find rows without reading the whole table. It costs disk and memory and slows every insert and update, so the right number is the smallest set that meets a measured need. EXPLAIN shows whether a query uses one.

The first bottleneck was a missing index

Under a load test of only 40 requests a second, the dashboard's two alert queries reached a 95th-percentile latency of 7 seconds, and 45% of requests failed: every database connection was busy with a slow query, and the connection pool timed out. The query statistics showed two queries averaging over a second and a half; their plans were parallel sequential scans of 2 million rows, reading about 235 MB each. On an idle database the same plans said 43 ms and 152 ms, which looks harmless. Forty copies at once, each scanning the whole table, is a different workload.

One index shipped. It met the target fifty times over, and the second would have doubled the write cost for a query that was already fast. Its trigger was written down instead: add it when the severity query shows up in the slow-request alert. After the fix, under the same load: a 95th percentile of 4.07 ms and no failures.

The planner can choose badly as the data grows

Reading the newest alerts across a user's teams looks like a simple WHERE team_id = ANY(...). Measured on 2.5 million rows, the planner sometimes walked the global time index and filtered, which cost 13 ms when one of the user's teams was large but quiet (103,036 rows read and discarded), and the bad case grows with the table. Reading each team through its own index and merging the results bounds the cost, whatever the data: 0.05 to 0.1 ms in every case.

async def newest_alerts(
    db: AsyncSession,
    team_ids: Sequence[int] | None,
    *,
    limit: int,
    severity: Severity | None = None,
    before: tuple[datetime, int] | None = None,
) -> list[AlertOut]:
    """Newest first among `team_ids` (None: every team), optionally only one
    severity, optionally only older than a keyset cursor.

    Several teams are read one team at a time and merged - a LATERAL join,
    each team through the (team_id, created_at, id) index - so the cost is
    bounded by teams x limit index entries whatever the data looks like.
    The obvious `WHERE team_id = ANY(...)` lets the planner walk the global
    time index and filter instead, which depends on the data: measured on
    2.5M rows, 0.015ms to 13ms (103k rows filtered out when one of the
    user's teams is large but quiet), against 0.05-0.1ms here in every case.
    """

Analytical databases do it differently

Warehouses such as Snowflake, BigQuery and ClickHouse scan huge tables and mostly do without B-trees. Their main "index" is columnar storage: each column stored and compressed separately, so a query reads only the columns it names. Zone maps keep a minimum and maximum per block so blocks that cannot match are skipped; partitioning or clustering by a date or tenant skips whole files; sort keys make range filters and joins cheap; bitmap indexes and Bloom filters help with low-cardinality columns and "definitely not here" checks; materialized views trade storage for precomputed results. A transactional index jumps to a few rows; an analytical layout skips past almost all of them.

Traps

Questions

1. Why do migration tools use random identifiers linked to a parent rather than sequential numbers?

   Answer

   So two branches can each add a migration without colliding. If both are merged, the tool reports two heads, and a small merge migration joins them.

2. What does autogenerate get wrong that a person must fix?

   Answer

   Renames, which it writes as a drop plus an add and so loses data; data migrations, which it never writes; some constraint and default changes; and index builds, which it writes as blocking statements.

3. Why does a migration need a lock timeout?

   Answer

   A schema change waiting for a lock makes every later query on the table queue behind it, so a short wait becomes an outage. With a timeout the change gives up and can be retried.

4. The severity query was 3.8 ms after the first index. Why not add a second index that made it 0.063 ms?

   Answer

   The query already met its target, and the second index would have roughly doubled the cost of every insert and added 78 MB. The trigger for adding it was written down instead.

The connection budget

Transaction pooling has one hard consequence: a server connection is handed to a different client at the end of every transaction, so nothing may rely on session state. Session SET commands, advisory locks held across transactions, LISTEN/NOTIFY and temporary tables all break. That is why the reference service sets its statement and idle timeouts on the database role itself, and why it tells Postgres who is asking in every transaction rather than once per connection.

A pool that traps itself

Load tests of the reference service showed two tools disagreeing by a factor of 40: about 90 ms per request under one load generator and 2 ms under another, at the same rate. The difference was the shape of the arrivals, and what it did to the connection pool.

Row-level security: the second barrier

The application already filters every read by the caller's teams and checks every write's role. That is one line of code per query, and the next endpoint, a refactor or a reporting query could forget it, silently returning another team's data, perhaps into the assistant's prompt. Row-level security moves the same rule into Postgres: each table holding team data has policies, one per command, that read who is asking from transaction-local settings. A query that forgets the filter returns nothing instead of leaking.

def _set_config(values: dict[str, str]) -> Select[Any]:
    return select(*(func.set_config(k, v, True) for k, v in values.items()))


@event.listens_for(ContextSession, "after_begin")
def _apply_transaction_settings(
    session: Session, transaction: SessionTransaction, connection: Connection
) -> None:
    values: dict[str, str] | None = session.info.get("transaction_settings")
    if values:
        connection.execute(_set_config(values))


async def set_transaction_settings(session: AsyncSession, values: dict[str, str]) -> None:
    """Applied to the open transaction, if any, and to every later one."""
    session.info["transaction_settings"] = values
    if session.in_transaction():
        await session.execute(_set_config(values))

ORG_ADMIN = "(SELECT current_setting('app.org_admin', true)) = 'on'"
SERVICE = "(SELECT current_setting('app.service', true)) = 'alertmanager'"


def teams(setting: str) -> str:
    # The cast keeps ANY in its array form: `= ANY ((SELECT ...))` without
    # it is the subquery form, comparing bigint with bigint[].
    return f"team_id = ANY ((SELECT tenant_team_ids('app.{setting}'))::bigint[])"


def upgrade() -> None:
    op.execute(
        """
        CREATE FUNCTION tenant_team_ids(setting text) RETURNS bigint[]
        LANGUAGE sql STABLE PARALLEL SAFE
        AS $$ SELECT coalesce(nullif(current_setting(setting, true), ''), '{}')::bigint[] $$
        """
    )
    op.execute("ALTER TABLE alerts ENABLE ROW LEVEL SECURITY")
    op.execute(
        f"CREATE POLICY alerts_select ON alerts FOR SELECT "
        f"USING ({ORG_ADMIN} OR {SERVICE} OR {teams('read_team_ids')})"
    )
    op.execute(
        f"CREATE POLICY alerts_insert ON alerts FOR INSERT "
        f"WITH CHECK ({ORG_ADMIN} OR {SERVICE} OR {teams('write_team_ids')})"
    )
    op.execute(
        f"CREATE POLICY alerts_delete ON alerts FOR DELETE "
        f"USING ({ORG_ADMIN} OR {teams('admin_team_ids')})"
    )

The companion decision, ADR-0005, set the two-role model and the pooler: the schema owner is used only by migrations, which run directly against Postgres as a one-off step before the api starts, never on api startup, where several workers and replicas would race to migrate. The Compose deploy does exactly that. The reference ECS stack instead migrates in an init container in every api task, so replicas that start together did race, and an advisory lock now makes them take turns (Chapter 6).

Traps

Questions

1. What breaks when an application moves behind a transaction-mode pooler?

   Answer

   Anything relying on session state between transactions: session SET, advisory locks held across transactions, LISTEN/NOTIFY, and temporary tables. Each transaction may run on a different server connection.

2. Why must the tenant setting be transaction-local?

   Answer

   Because the pooler hands the same server connection to another client after the transaction ends. A session-level setting would still be there for them; a transaction-local one ends with the transaction.

3. What does "metastable" mean, and how did the reference service escape it?

   Answer

   A slow state that keeps itself going after its trigger is gone. Discarded overflow connections forced new logins, which slowed requests and kept concurrency high. A fixed pool with no overflow broke the loop.

4. If the application already filters by team, what does row-level security add?

   Answer

   A second barrier that holds when the application forgets: a missing filter returns an empty result instead of other teams' rows, and a query that says nothing about the caller sees nothing.

The sign-in flow

What each check stops

async def verify(self, id_token: str, *, nonce: str) -> Identity:
    try:
        header = jwt.get_unverified_header(id_token)
    except jwt.PyJWTError as exc:
        raise InvalidToken("malformed ID token") from exc
    algorithm = header.get("alg")
    if algorithm not in ALGORITHMS:
        raise InvalidToken(f"ID token algorithm {algorithm!r} is not accepted")
    key = await self._signing_key(header.get("kid"))
    client_id = self._settings.oidc_client_id
    try:
        claims: dict[str, Any] = jwt.decode(
            id_token,
            key=key.key,
            algorithms=[algorithm],
            audience=client_id,
            issuer=self._settings.oidc_issuer,
            leeway=LEEWAY_S,
            options={"require": ["exp", "iat", "iss", "aud", "sub"]},
        )
    except jwt.PyJWTError as exc:
        raise InvalidToken(f"ID token rejected: {exc}") from exc
    audiences = claims["aud"] if isinstance(claims["aud"], list) else [claims["aud"]]
    if claims.get("azp", client_id) != client_id or (
        len(audiences) > 1 and "azp" not in claims
    ):
        raise InvalidToken("ID token was issued to another client")
    if not hmac.compare_digest(str(claims.get("nonce", "")), nonce):
        raise InvalidToken("ID token nonce does not match this sign-in")

Server sessions, not JWT sessions

The cookie uses the __Host- prefix, which makes the browser refuse it unless it is Secure, has path / and no domain, so no subdomain can set or overwrite it. SameSite=Lax, not Strict: the provider's redirect back to the callback is a cross-site navigation, and a Strict cookie would be left behind on it. Passwords never reach the application: the provider stores them salted and hashed. Salts protect secrets that people choose and attackers can guess; a 256-bit random session token has nothing to guess, so a plain SHA-256 is enough and a slow salted hash would only slow every request.

Roles, and 404 versus 403

class Role(IntEnum):
    VIEWER = 1
    RESPONDER = 2
    ADMIN = 3

def require_role(principal: Principal, team_id: int, needed: Role, what: str) -> None:
    """404 when the caller cannot see the team - the same answer as for a
    resource that does not exist, so ids cannot be probed - and 403 when
    they can see it but their role is too low."""
    role = principal.role_in(team_id)
    if role is None:
        raise HTTPException(status_code=404, detail=f"{what} not found")
    if role < needed:
        raise HTTPException(status_code=403, detail=f"needs the {needed.label} role in this team")

Traps

Questions

1. In the code flow with a confidential client, what does the browser ever hold?

   Answer

   A one-time authorization code on the way back from the provider, and then only the application's session cookie. The access token is dropped; the ID token stays on the server until sign-out, when it rides once in the provider's sign-out URL as id_token_hint (OIDC RP-Initiated Logout).

2. What do state, the nonce and PKCE each protect against?

   Answer

   state stops login CSRF by tying the provider's answer to this browser; the nonce stops ID token replay; PKCE stops an intercepted code from being exchanged by anyone without the verifier.

3. Why does the reference service prefer server-side sessions to JWTs?

   Answer

   A server session can be revoked with a delete, which also makes a role change take effect at once; otherwise its roles are as of the last sign-in. A JWT cannot be ended before it expires without a revocation list, so its stale roles last until it expires; it also exposes its claims and adds a key to manage.

4. When should a request answer 404 rather than 403?

   Answer

   When the caller cannot see the resource at all. A 403 would confirm it exists. 403 is for a caller who can see it but whose role is too low.

Where secrets live

Each secret proves one thing to one party. The identity provider's client secret proves the api is the registered application; the database passwords identify the api, the schema owner and the metrics exporter; the model key identifies the service to its provider; the alert webhook's token identifies Alertmanager, a service, which is why it is a bearer token compared in constant time rather than a session. On a public repository, a secret pushed in a commit has been copied the moment it was pushed, so rotation comes first.

Least privilege, layer by layer

The host matters too. Membership of the docker group is root-equivalent, so only the deploy user has it; SSH takes keys only, never passwords, and root login is off; security updates install themselves. A debugger port is remote code execution, so the debugging overlay is a separate Compose file bound to loopback and never present in production. A profiler that needs to read another process's memory runs as a separate container borrowing the api's namespaces, on demand, so the api itself never holds that capability.

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" \
  -v db="$POSTGRES_DB" -v app_user="$APP_DB_USER" -v app_password="$APP_DB_PASSWORD" <<'SQL'
CREATE ROLE :"app_user" LOGIN PASSWORD :'app_password';

-- Server-side limits for every connection the app opens. They live on the
-- role because PgBouncer's transaction pooling discards session-level SET.
ALTER ROLE :"app_user" SET statement_timeout = '10s';
ALTER ROLE :"app_user" SET idle_in_transaction_session_timeout = '30s';

GRANT CONNECT ON DATABASE :"db" TO :"app_user";
GRANT USAGE ON SCHEMA public TO :"app_user";

-- Applies to tables/sequences the owner creates later, i.e. every migration.
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO :"app_user";
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT USAGE, SELECT ON SEQUENCES TO :"app_user";
SQL

Personal data: an inventory, retention and erasure

A service that holds data about people needs a document that answers a data protection officer, a lawyer, or a person asking what is held about them. The reference service's inventory lists every place personal data lives, and the most important line is the one saying what is not stored: questions and answers.

• Access. An operator command exports, as JSON, everything held about a person, across every identity provider, and records that it did so.
• Erasure. A dry run shows what would be erased; the real run removes accounts, memberships and sessions and records it. The person must also be removed at the identity provider, or their next sign-in recreates them. The audit trail keeps their events under an id that no longer resolves to anyone: pseudonymous, not anonymous, and whether that is allowed is a legal decision to write down.
• Retention. A scheduled job deletes expired sessions, users who have not signed in for a year, and old alerts; an empty setting keeps that kind forever. Backups keep rows until they rotate out.
• Rectification. Name and email come from the identity provider at each sign-in, so they are corrected there.

Some questions belong outside engineering and should be asked early: the legal basis for each use; a data-processing agreement with the model provider (no training on your data, how long prompts are kept, where they are processed); a data protection impact assessment; whether a works council must agree to a system that can show what an employee asked; how a breach would be detected in time to report it; and the AI-transparency rule that people must be told they are dealing with an AI system, which the reference chat states above the question box.

Traps

Questions

1. Where should a production secret come from, and where must it never appear?

   Answer

   From a secret store or generation at first boot, injected into the environment at runtime. Never in Git, an image, logs, chats or tickets; if it does, rotate it.

2. Name one least-privilege control at each of the database, container and network layers.

   Answer

   Database: an application role that can change rows but not the schema. Container: non-root with a read-only root filesystem and all capabilities dropped. Network: only the edge published, internal endpoints unreachable from outside.

3. What makes an erasure request complete in the reference service, and what remains?

   Answer

   Erasing the accounts, memberships and sessions, and removing the person at the identity provider. Audit events remain under an id that no longer resolves (a legal decision), and backups hold the rows until they rotate out.

Choosing a framework, and what not to add

State is two different things

Streaming to the browser

The reference chat streams its answers as Server-Sent Events, and its wire format was redesigned once a real model replaced the placeholder. Plain-text events broke on the first markdown answer: a blank line inside a token ended the event early. A real model can think for tens of seconds before its first token, while proxies and load balancers cut idle connections at around a minute. And a user who closes the tab must stop the provider from generating tokens that are still being paid for.

/**
 * Parses a Server-Sent Events byte stream (the parts of the spec the api
 * uses). Written by hand because the browser's EventSource can only GET,
 * and the chat request carries a JSON body.
 *
 * Network chunks do not line up with events: one read can hold half an
 * event, several events, or end in the middle of a multi-byte UTF-8
 * character. So bytes are decoded with `stream: true` (a split character
 * waits for its other half) and text is buffered until whole lines arrive.
 * Lines end in \n, \r\n or \r; a blank line dispatches the event; lines
 * starting with ":" are comments (the api's keep-alive heartbeats).
 *
 * Stopping early (break, or an aborted fetch) cancels the body, which
 * closes the connection - that is how the server learns the user left.
 */
export async function* parseSSE(body: ReadableStream<Uint8Array>): AsyncGenerator<SSEEvent> {
  const reader = body.getReader()
  const decoder = new TextDecoder()
  let buffer = ''
  let event = 'message'
  let data: string[] = []
  let id: string | undefined

  try {
    while (true) {
      const { done, value } = await reader.read()
      buffer += done ? decoder.decode() : decoder.decode(value, { stream: true })

Traps

Questions

1. When does a meta-framework such as Next.js earn its place, and when does it not?

   Answer

   It earns it for public content where search ranking or first paint matter, through server rendering. For an internal tool or dashboard behind sign-in, it adds complexity without a benefit; a single-page app built by Vite is enough.

2. What is the difference between server state and client state, and which tool suits each?

   Answer

   Server state is owned by the backend and needs fetching, caching and invalidation: a library such as TanStack Query. Client state belongs to the UI, such as an open dialog: usually the framework's own state, with a small store only when it is shared widely.

3. Why does the reference chat use Server-Sent Events rather than WebSockets?

   Answer

   The answer flows one way, from server to browser. SSE is plain HTTP, works through existing proxies and load balancers, and is simpler to run. WebSockets are for when the client must also stream continuously.

4. What has to happen, end to end, when a user presses Stop?

   Answer

   The browser aborts its fetch, which closes the connection; the server notices the disconnect, cancels the task reading from the model, and closes the provider's stream, so generation and billing stop.

Two proxies, two jobs

# Streaming (SSE): every event must reach the browser the moment the
# api writes it. Buffering is on by default in nginx and would deliver
# the whole answer at once.
location = /api/chat/stream {
    proxy_pass http://api_backend/chat/stream;
    include /etc/nginx/snippets/proxy.conf;
    proxy_buffering off;
    proxy_cache off;
    # Max silence between two reads from the api, not total duration.
    # The api sends a keep-alive comment while the model is thinking.
    proxy_read_timeout 120s;
}

# Everything else goes to nginx: the React build, /api, the SSE stream.
handle {
	reverse_proxy web:8080 {
		# Pass every chunk on the moment it arrives: answers stream as SSE.
		flush_interval -1
		# While nginx is being replaced (make deploy), hold a request for up
		# to 5s and retry, instead of failing it. Only requests that could
		# not reach nginx at all are retried, so nothing is sent twice.
		lb_try_duration 5s
		lb_try_interval 250ms
	}
}

The client's address, through two proxies

Behind a proxy, every connection the application sees comes from the proxy. If the application used the connection's address, every user would share one address, and one rate limit. The fix is the X-Forwarded-For header, which is only as trustworthy as the chain that sets it.

Proxies differ in what they do with the header. The edge here overwrites it with the address it saw. A cloud load balancer appends that address to whatever the client sent. Behind an appending proxy, the application must trust only the proxy's addresses and take the right-most entry it does not trust. Trusting every address (*) takes the left-most entry, and the client wrote that one. The reference service's AWS stack had exactly this bug until a review: behind the load balancer, any client could choose its own address, and with it its own sign-in rate limit.

Timeouts and keep-alive, in the right order

# Written at start-up by /docker-entrypoint.d/25-runtime-conf.sh, from where
# the container runs: the resolver (from /etc/resolv.conf: `resolve` below
# re-resolves the api at runtime, so a recreated api container is picked up
# without an nginx restart, and nginx starts even if the api does not exist
# yet), the trusted proxies (REAL_IP_FROM, else this container's networks)
# and the api_backend upstream (API_UPSTREAM, else api:8010).
include /tmp/nginx/resolver.conf;
include /tmp/nginx/upstream.conf;

# Behind the TLS edge or a cloud load balancer, the connection comes from
# the proxy: take the client's address from the X-Forwarded-For it sets, or
# every user shares one address - and one rate limit. Believed only from the
# trusted proxies; the edge overwrites whatever X-Forwarded-For a client
# sends, so it cannot be forged through it.
include /tmp/nginx/real-ip.conf;
real_ip_header X-Forwarded-For;
real_ip_recursive on;

set -eu
out=/tmp/nginx
mkdir -p "$out"

ns=$(awk '$1 == "nameserver" { printf "%s ", ($2 ~ ":" ? "[" $2 "]" : $2) }' /etc/resolv.conf)
echo "resolver ${ns:-127.0.0.11} valid=10s ipv6=off;" > "$out/resolver.conf"

trusted=${REAL_IP_FROM:-$(ip -o -f inet route show scope link | awk '{ print $1 }')}
: > "$out/real-ip.conf"
for cidr in $trusted; do
  echo "set_real_ip_from $cidr;" >> "$out/real-ip.conf"
done

cat > "$out/upstream.conf" <<CONF
upstream api_backend {
    zone api_backend 64k;
    server ${API_UPSTREAM:-api:8010} resolve;
    keepalive 16;
    # Below gunicorn's keepalive: the side that closes idle connections
    # first must be the proxy, never the backend.
    keepalive_timeout 60s;
}
CONF

Traps

Questions

1. Why must the proxy close idle keep-alive connections before the backend does?

   Answer

   If the backend closes first, the proxy may send a new request on a connection that was just closed. nginx does not retry a POST on a reset connection, so the user gets a 502.

2. How does the application get the real client address through two proxies, safely?

   Answer

   The outermost proxy overwrites X-Forwarded-For with the address it actually saw; inner proxies accept the header only from the addresses of the proxies in front of them, not from every private range, because internal clients have private addresses too, and pass on exactly one address.

3. What two proxy settings does a streamed answer need?

   Answer

   Buffering turned off, so each event is passed on immediately, and a read timeout longer than the gaps between events, with heartbeats from the server shorter than every idle timeout in the path.

Images, layers and the cache

# ---- production -------------------------------------------------------------
FROM python:3.13-slim@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS production
# Numeric UID: Kubernetes' runAsNonRoot can only verify a numeric USER.
# pip goes: the virtualenv below is complete, and the base image's pip
# carries its own vendored libraries with their own vulnerabilities
# (msgpack, setuptools' pkg_resources: make scan) - and it lets whoever gets
# code execution install more.
RUN useradd --system --uid 10001 --no-create-home --shell /usr/sbin/nologin app \
 && python -m pip uninstall --yes --root-user-action ignore pip
WORKDIR /api
# Same path as in the deps stage: a venv's scripts hard-code its location.
COPY --from=deps /api/.venv /api/.venv
# Code stays owned by root and read-only to the app user, so a compromised
# process cannot rewrite it.
COPY alembic.ini gunicorn.conf.py ./
COPY migrations/ ./migrations/
COPY app/ ./app/
# PROMETHEUS_MULTIPROC_DIR: one metrics file per worker, summed on scrape
# (app/metrics.py). Must be set before start, never set to "" (the client
# checks only that the variable exists). /tmp is a tmpfs in compose.prod.yaml.
# PYTHONBREAKPOINT=0: a breakpoint() left in the code is a no-op here. With
# the default (pdb) it would stop a worker waiting for keyboard input that
# never comes - the worker hangs until gunicorn kills it.
ENV PATH="/api/.venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONFAULTHANDLER=1 \
    PYTHONBREAKPOINT=0 \
    PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus
USER 10001:10001
EXPOSE 8010
HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --retries=3 \
  CMD ["python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8010/health', timeout=2).status == 200 else 1)"]
CMD ["gunicorn", "app.asgi:app", "-c", "gunicorn.conf.py"]

Ports and container networks

Two error messages tell you where to look. Connection refused means a packet arrived and was actively rejected: the path works, nothing listens there. A timeout means the packet vanished: a firewall, a frozen process, a wrong route. With a correct -p, a refusal points inside the container, at an application bound to 127.0.0.1 instead of all interfaces, or not started. A stopped container also disappears from Docker's DNS, so its clients see a name-resolution error rather than a refusal, and should treat it as "unavailable".

Volumes, restarts and health

Compose: one base, an overlay per environment

Before this layout, the production shape existed only as two build commands in a document, so it was never run as a whole, and the production-only bugs (a proxy buffering the stream, one rate limit for every user behind the proxy) lived exactly there. Two complete files were rejected because the shared majority drifts apart silently; Compose profiles were rejected because they switch services on and off but cannot change a service's build target, mounts or ports.

      # "-", not ":-": an empty value in .env must stay empty (a real provider).
      OIDC_DISCOVERY_URL: ${OIDC_DISCOVERY_URL-http://keycloak:8080/auth/realms/triage/.well-known/openid-configuration}
      OIDC_CLIENT_ID: ${OIDC_CLIENT_ID:-triage-web}
      OIDC_CLIENT_SECRET: ${OIDC_CLIENT_SECRET}
      # The api's other settings (app/config.py; the handbook's settings
      # reference): passed when .env or the shell sets them, and absent
      # otherwise, so the code's default applies - for these first ones, a
      # default that differs in production (APP_ENV=prod, ADR-0023). A
      # variable compose does not list never reaches the container, whatever
      # .env says.
      DOCS_ENABLED:
      OTEL_TRACES_SAMPLER:
      OTEL_TRACES_SAMPLER_ARG:
      CHAT_RATE_LIMIT_FAIL_CLOSED:
      PROD_CHECKS_WAIVED:

Traps

Questions

1. Why are dependency manifests copied and installed before the application code?

   Answer

   Layers are cached, and a changed instruction invalidates every layer after it. With dependencies first, a code change rebuilds only the code layer instead of reinstalling everything.

2. What do EXPOSE, the application's bind address and -p each do?

   Answer

   EXPOSE is metadata. The bind creates the listener inside the container's network namespace. -p creates NAT rules from the host into the container, and is the only one that opens anything on the host.

3. Why does the base Compose file declare no ports?

   Answer

   Port lists from overlays are concatenated, not replaced, so a port in the base would be published in every environment. Each overlay adds the ports its environment needs.

4. Does Docker restart an unhealthy container?

   Answer

   No. Restart policies act only when the process exits. Health matters to depends_on and to orchestrators, which replace unhealthy tasks or pods, but by their own checks (Kubernetes probes, an ECS task-definition health check); the image's HEALTHCHECK alone does not reach them.

The layers, and what each catches

Real services, fakes, and what not to mock

The integration suite starts its own Compose project, with its own containers, network and volumes, so tests never see development data and everything is deleted afterwards. Before the tests run, the migrations build the schema, and a check fails if the models and the migrations disagree. The test database runs with durability turned off, because throwaway data does not need it and every commit then skips the disk flush. Timeouts and limits are set small (a 1-second model read timeout, 0.2-second heartbeats, five requests a minute) so failure paths are fast to test.

• Do not mock the database. Real SQL through the real driver and pooler; a mocked session would have missed both of the big database bugs.
• Do not mock the provider's SDK. The mock model is an HTTP server speaking the provider's API, so tests exercise the real SDK, its retries, its streaming and its exceptions; it can also hang, drop mid-stream or return 429 on demand, and it counts canceled streams.
• Do not fake time when the test is about time. Heartbeat and timeout tests use small real durations, because the bugs live in the interaction with the event loop.
• Prefer fakes to mocks. A fake is a small working implementation; a mock records calls and ties the test to the implementation.

async def test_each_user_sees_only_their_teams(cast: dict[str, AsyncClient]) -> None:
    await post(cast["payments"], "payments", "pay-1")
    await post(cast["platform"], "platform", "plat-1")
    assert await listed(cast["payments"]) == ["pay-1"]
    assert await listed(cast["platform"]) == ["plat-1"]
    assert await listed(cast["viewer"]) == ["plat-1"]
    assert await listed(cast["org"]) == ["plat-1", "pay-1"]
    assert await listed(cast["newcomer"]) == []


async def test_other_teams_alerts_are_not_found_not_forbidden(
    cast: dict[str, AsyncClient],
) -> None:
    secret = await post(cast["platform"], "platform")
    # Same answer as for an id that does not exist: ids cannot be probed.
    for response in (
        await cast["payments"].get(f"/alerts/{secret}"),
        await cast["payments"].delete(f"/alerts/{secret}"),
        await cast["payments"].get("/alerts", params={"team": "platform"}),
        await cast["newcomer"].get(f"/alerts/{secret}"),
    ):
        assert response.status_code == 404
        assert response.json()["error"]["code"] == "not_found"

The minimum for a new endpoint

Code style that linters cannot enforce

Coverage, flaky tests and review

Coverage. The api's gate is 85% of lines and branches, and it measures 92.6%. Coverage needs to know about the concurrency library in use: without it, the lines after every awaited database call were reported as never run, and the same tests read 89.7%. A line that ran is not a behavior that was checked.

Flaky tests. Nothing is retried, in pytest or in Playwright. The causes met in practice were all races: asyncio runs a task's done-callbacks one loop iteration after it finishes, so a test asserted too early; two quick cancellations merged into one; and tests shared a per-user rate-limit budget. A model's answers are the one exception, because they vary by design; evals measure a pass rate over repeated runs instead (Handbook 1, Chapter 7).

Review. Formatting and lint are settled by machines in CI, so review is for design and behavior. A good description says what changed and why, how it was verified with the commands and the numbers, one line per new dependency, and what the reviewer should read first. When CI and a laptop disagree, suspect state the laptop has and CI does not: stale images, an old dependency volume, a different user id. CI's user id once exposed a HOME=/ problem the developer's had hidden.

Gates for generated code

When an agent writes most of the code, the failure mode measured across hundreds of repositories is structural rather than functional: more lines, more duplication, higher complexity, more swallowed errors, fewer refactors, while the tests still pass. The answer is not a longer instructions file, which the model weighs against the task, but gates arranged in four layers by how binding they are and how fast they run. The reference build's make check is the command behind the middle layers: an agent runs it before it may say "done", and the pipeline runs it again.

Two numbers to keep in mind. Coverage is not test quality: one published suite reached 100% line coverage and killed 4% of the mutants introduced into the code it covered, which is why the mutation step sits beside patch coverage rather than instead of it. And generated package names are wrong between one in twenty and one in five times depending on the model, so an install that is not from a lockfile is an install of whatever a typo resolves to. The review that follows the gates is in chapter 18 of the AI handbook: one simplification pass given the diff and the surrounding code, then two reviewers asking different questions, then a person reading the tests first.

Traps

Questions

1. Why does the reference service avoid mocking the database and the model SDK?

   Answer

   A mock tests your assumptions about a dependency. The real failures were in the driver, the pooler and the SDK's behavior, which only real services (or an HTTP server speaking the real API) exercise.

2. Name two kinds of bug that unit and integration tests cannot find.

   Answer

   Bugs that appear only under load, such as a missing index or a design that collapses at high request rates; and bugs that appear only during a failure, such as connections leaked after a database froze. Rare model behavior is a third, found only over many runs.

3. Why put decisions in pure functions?

   Answer

   They take and return plain values, so they can be tested exhaustively and fast, with no database, network or mocks; the I/O stays in the routes and clients that call them.

4. What should a pull request description contain besides what changed?

   Answer

   Why it changed, how it was verified with the commands and the numbers, one line per new dependency, and what the reviewer should read first.

Three signals, one id

What to measure

Two details of Python servers matter. With several worker processes, a normal metrics client answers each scrape from whichever worker received it: six scrapes after 40 requests returned 15, 15, 5, 6, 14 and 5. In multiprocess mode, each worker writes its samples to files and the endpoint sums them: 40 every time. And every distinct combination of label values is a separate series in Prometheus's memory, so values come only from fixed sets: route templates, outcomes and error codes, never messages, user ids, addresses or a tool name chosen by a model.

# Accept an upstream id (nginx sets one per request) only if it looks like
# one; anything else is replaced, so clients cannot inject log content.
_VALID_REQUEST_ID = re.compile(r"^[A-Za-z0-9._-]{8,128}$")


def route_template(scope: Scope) -> str:
    """ "/alerts/{alert_id}", never "/alerts/123": raw paths as log or metric
    labels grow without bound (one series per id)."""
    route = scope.get("route")
    return getattr(route, "path", None) or "__unmatched__"

Dashboards and alerts as code

The dashboard is generated by a Python script and committed as JSON, which Grafana loads at start; a check fails if the committed JSON is not what the generator produces, and an edit made in the UI lasts only until the next reload. The first version of the dashboard showed "No data" in four of its thirty panels on a healthy system, for reasons worth knowing: a ratio whose numerator does not exist yet (no 5xx has happened) is "no data", not zero, unless written with or vector(0); rate() needs two samples, so panels blank briefly after a restart, and the very first increment of a new series is invisible to it; and one long-lived streaming route flattened a latency panel until it got its own row.

      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            - (sum(rate(chat_refusals_total{reason="assistant_disabled"}[5m])) or vector(0))
          ) / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels: {severity: high}
        annotations:
          summary: "more than 5% of api requests fail with 5xx"
          runbook_url: "https://github.com/Parzon/triage-assistant/blob/main/docs/runbooks/alerts.md#higherrorrate"

Debugging: which tool for which question

A request that a user complains about is followed from the outside in. The request id finds its lines in the proxy's log and the api's log; if the proxy's total time is much larger than the time it waited for the api, the time went into the client connection, not the service. The trace then shows each step: retrieval, each SQL statement, the model call with its tokens and time to first token. Traces carry ids, counts and timings, never the question or the answer, unless content capture is explicitly turned on in development.

Traps

Questions

1. Which question does each of metrics, logs and traces answer?

   Answer

   Metrics: is the system healthy over time. Logs: what happened in one request or event. Traces: where one request's time went across steps and processes.

2. Why is CPU a poor saturation signal for an async Python service, and what is better?

   Answer

   Work queues on the event loop, so wall-clock timeouts start firing while CPU is around half used. Event-loop lag, how late a timer fires, measures the queueing directly.

3. Why can a metric label never be a user id or a raw URL path?

   Answer

   Each distinct combination of label values is a separate time series held in memory. Unbounded values create unbounded series; labels must come from small, fixed sets such as route templates and error codes.

4. Roughly how long does the reference stack take from a failure to a notification, and why?

   Answer

   About two minutes: up to a scrape interval, the rule's one-minute for, evaluation, and Alertmanager's grouping wait. A fault shorter than a scrape can leave no trace in metrics at all.

Decide what happens when a dependency fails

Where timeouts belong

Put the cap where the work happens: the database's statement_timeout, the pooler's query timeout. The server then stops the work and returns a clean error. A client-side timeout only stops waiting, and if its cleanup needs the remote side to cooperate, a dead remote turns that cleanup into a hang. That was the reference service's worst bug.

      # Fail fast when Postgres is down or frozen (make drills d="db-stop
      # db-hang"). PgBouncer defaults in brackets.
      # A client waits this long for a server connection, then gets an error [120s].
      QUERY_WAIT_TIMEOUT: 5
      # Cancels a query running this long and closes its connections [off].
      # A backstop for when Postgres cannot enforce the role's 10s
      # statement_timeout itself (frozen, partitioned): keep it above 10s.
      QUERY_TIMEOUT: 15
      # A new server connection must be up within this [15s].
      SERVER_CONNECT_TIMEOUT: 5
      # After a failed login to Postgres, retry after this long [15s] - which
      # is also how long clients keep getting "server login has been
      # failing" once Postgres is back.
      SERVER_LOGIN_RETRY: 2
      # A stopped container vanishes from Docker's DNS, and PgBouncer caches
      # that "no such host" answer [15s]: Postgres would be back while
      # PgBouncer still failed every login from its cache.
      DNS_NXDOMAIN_TTL: 1

Three more rules about waiting

• Cancel or abandon. A timeout cancels once, and cleanup after it can block again. To answer on time no matter what, run the work as a separate task, wait with a deadline, and abandon it (keep a reference, let it finish) when canceling would trigger a cleanup that can hang.
• Propagate deadlines. Pass the remaining budget down with each call (gRPC deadlines, a per-transaction statement timeout), so no layer works longer than its caller will wait. Otherwise a timed-out client leaves work running, and a retried write can commit twice: hence idempotency keys on writes.
• Keep the sum under the outer limit. The longest a user can wait is the sum of the timeouts along the path; the proxy's must be above it (Chapter 11).

Liveness and readiness

Failure drills

The reference service's drill script injects one fault at a time into the production-shaped stack with plain Docker: stop for "down", pause for "frozen" (the process is alive, its connections stay open, nothing answers), a network disconnect, a kill from the host for a crash, a lowered memory limit, and the mock model's admin interface for provider failures. It records what a signed-in user sees on each path, restores the stack, and checks recovery. An injection that fails is reported as a failure, never as a result: an early version reported "rate-limited model, answer fine" because the mock was still starting.

Two findings shape deployment. A single container replaced by "stop, then start" refuses new requests for as long as the longest in-flight answer takes to drain, plus boot time; a rolling deploy starts the new instance beside the old one and drains the old one only once the new one is ready (Handbook 3, Chapter 13). And memory limits on a host with swap were soft: the api put 48 MiB in swap and kept serving without a single error, until every limited service set its swap limit equal to its memory limit, which made an out-of-memory kill loud and alerting.

One failure appeared once and would not come back. During a full drill run the api's pool stayed at 40 of 40 connections while the database was healthy, and every request got a 503 until the api was restarted. Repeating the faults that were active changed nothing; the variable nobody had changed was the api's address. When a container rejoins a network, Docker usually gives it another address, and every pooled connection stays bound to the old one. Data sent on such a connection is never acknowledged, the kernel retransmits for about 15 minutes, and the pool's health check, which has no timeout of its own, hangs at each one. The fix is a socket option, TCP_USER_TIMEOUT, which drops a connection whose data goes unacknowledged for 10 seconds; it never fires for a slow or frozen database, whose kernel still acknowledges. A new drill fills the pool and brings the api back on another address: ready again in 50 seconds, instead of never (ADR-0027 in the repository).

Traps

Questions

1. When should a rate limiter fail open, and when closed?

   Answer

   Open when the limit is a courtesy on cheap requests and its store is a soft dependency; closed when the limit is itself the security control, or when it guards spend because each request is expensive, as with a paid model call.

2. Why is a client-side timeout not enough to protect a pool?

   Answer

   It stops waiting, not the work. If the driver's cleanup needs the server's cooperation, a frozen server turns the cleanup into a hang and the connection is held forever. A server-side cap stops the work itself.

3. What is the difference between liveness and readiness?

   Answer

   Liveness asks whether the process is alive and checks nothing external, because failing it restarts the container. Readiness asks whether traffic should come here: it fails on hard dependencies and reports soft ones as degraded.

4. Why is freezing a dependency a harsher drill than stopping it?

   Answer

   A stopped dependency refuses connections at once. A frozen one keeps its connections open and never answers, so only timeouts protect you, and code paths that wait without a bound hang.

Open and closed load

export const options = {
  scenarios: {
    // Open model: iterations start at a fixed rate whatever the response
    // time, like independent users. A closed model (a fixed number of VUs
    // looping) slows its own arrival rate when the server slows down, and
    // hides the very latency it is supposed to measure ("coordinated
    // omission"). maxVUs caps how many can be in flight at once.
    reads: {
      executor: 'constant-arrival-rate',
      rate: Number(__ENV.RATE || 50),
      timeUnit: '1s',
      duration: __ENV.DURATION || '60s',
      preAllocatedVUs: 50,
      maxVUs: 1000,
    },
  },
  thresholds: {
    http_req_failed: ['rate<0.01'],
    'http_req_duration{kind:critical}': ['p(95)<200'],
    'http_req_duration{kind:recent}': ['p(95)<200'],
  },
  summaryTrendStats: ['avg', 'p(50)', 'p(95)', 'p(99)', 'max'],
}

How much one service served

The reference service was measured on one box: the production-shaped stack with two api workers on two CPUs, two million seeded alerts, and load generated through nginx by k6 on the same host. Because the generator competes for the same CPUs, the absolute numbers are that box's; the ratios are the lesson.

The bottlenecks, in the order they bite

A saturation cascade

At 200 to 500 concurrent streams, errors came from everywhere at once: connection resets in nginx, thousands of rate-limiter fail-opens, Redis pool exhaustion and database pool timeouts. Each had its own cause, and they amplified each other. Workers were being recycled every ten thousand requests, closing keep-alive connections nginx was about to reuse, which nginx does not retry for a POST; the limiter's 50-millisecond budget was wall-clock time, so it included the time a busy loop took to read the reply; the Redis pool was smaller than the number of concurrent requests; and the database pool was churning. After four changes: zero errors in 10,056 answers, and more answers per second. The general lesson is that on a busy event loop wall-clock timeouts expire while finished replies wait to be read, so the first symptoms of saturation are fail-opens and pool timeouts at moderate CPU, not slow responses.

The method, as a checklist

• A target with a number, and seeded data at production volume: an idle EXPLAIN said 43 ms for a query that took seconds under load.
• An open-model load, a warm-up, then a steady state long enough for the p99 to settle.
• The dashboard open: rate, errors, p95 and p99, loop lag, pools, CPU per container; and the load generator's own CPU.
• Database statistics reset before the run and read after; a profile during the steady state if the api's CPU is the limit.
• One change at a time; before and after numbers in the pull request; a decision record if it changes a default.

Traps

Questions

1. What is coordinated omission, and how does an open model avoid it?

   Answer

   In a closed model, a stalled request delays the requests that would have been sent meanwhile, so their latencies are never recorded and the percentiles look better than reality. An open model sends at a fixed rate regardless, so the stall shows up as queueing latency.

2. Why can the same average request rate give very different latencies?

   Answer

   Arrival shape: clumped arrivals queue behind each other on one event loop, so the last request of a burst waits for the rest, even though the average rate is the same.

3. In the reference service, what was the capacity ceiling for reads, and what does that imply?

   Answer

   The api's CPU, not the database, which used under one core at the limit. More capacity means more CPUs or replicas; a faster database would not help.

4. Why were the first signs of saturation errors rather than slow responses?

   Answer

   On a busy event loop, a timeout measured in wall-clock time can expire although the reply has arrived, because it includes the time the reply spends waiting for the loop. Limiter fail-opens and pool timeouts appeared at moderate CPU.

Service levels and error budgets

Runbooks

The reference repository has three kinds of runbook. The alert runbook has one section per alert rule, and each rule's runbook_url links straight to it. An operational runbook covers a procedure, such as setting up and running a demo host. A situation runbook covers a decision under pressure, such as turning the assistant off. Every section answers three questions in the same order, because the reader is under stress and needs the first action first.

## HighErrorRate

**Users:** more than 5% of requests fail with 5xx.
**Check:** the dashboard's "Requests / s by route and status" (which
route, which status), then `make trace id=<id>` on one failing request.
**Seen before:**
- 503 `database_unavailable`: see PostgresDown / PgBouncerDown.
- 502/504 `upstream_unavailable` from nginx: the api is down or stuck
  (ApiDown, EventLoopLagHigh).
- Bursts of 502 on POSTs under load came from gunicorn worker recycling
  (now off).

An off switch for the riskiest feature

Incidents and game days

Drills inject faults into a stack nobody depends on; a game day adds the people. One person injects a scenario while the on-call engineer responds without knowing it in advance, and the team records the time to detect (did an alert fire, or did someone notice?), the time to mitigate, what users saw, and what the runbook got wrong, then fixes the runbook the same day. Scenarios worth rehearsing for a service like the reference one: the host is gone and must be restored from off-host backups; the identity provider is down; the model provider is down or rate limiting; the assistant gives harmful answers; the disk fills; a bad release is rolled back under load; and a leaked credential is rotated to see what breaks.

Backups are only real once restored

Only Postgres needs backing up: the rate limiter's counters are disposable, and runbook vectors can be rebuilt from runbook text, but the text cannot. Backups are copied off the host, because a dump on a server's disk dies with the server, and a restore is rehearsed at production size before it is trusted. A restore also signs out everyone who signed in after the dump, since sessions live in the database.

The honest list

The reference service's production guide ends with what has never been tested, and why each matters: certificate renewal over months, the organization's real identity provider, a hosted model's latency and limits, browsers other than Chromium, more than one host, real traffic, a full-size restore, months of operation, a third-party security review, an accessibility audit, and answer quality on real questions. Each needs its own run before the stage that depends on it. A list like this is what makes the rest of a production guide credible.

Traps

Questions

1. What is an error budget, and how does it change what a team works on?

   Answer

   The amount of failure the SLO allows over its window, such as 3.6 hours of errors a month at 99.5%. While budget remains, the team ships; when it is spent, reliability work comes first until it recovers.

2. Why are burn-rate alerts better than a fixed error threshold?

   Answer

   They page on how fast the budget users care about is being spent, with a short window for fast burns and a long one for slow leaks, so blips do not page and slow degradations still get noticed.

3. What three things does each alert's runbook section answer?

   Answer

   What users experience, what to check first (exact panel and command), and what caused it before.

4. Why does an AI feature deserve an off switch rather than a rollback?

   Answer

   It can start doing harm without any code change, through a poisoned document or a change at the provider, so there may be nothing to roll back; and a switch takes effect in seconds on every server.

The map

Triggers for adding a piece

**Valkey** ◐ · when more than one process shares a limit, a cache or a
lock<br>
Rate-limit counters shared across workers. A counter kept in each
worker's memory is a classic bug: with N workers, a user gets N times the
limit. Valkey is the open-licence fork of Redis and speaks the same
protocol (ADR-0002). Decide what happens when it is down: here limits
fail open (ADR-0004), except the chat's, which fails closed in
production because each question costs money (ADR-0023). Its
`allkeys-lru` eviction suits counters, not data you must keep.

The same service at three sizes

Adding a technology

1. Justify it in one line in the pull request: what it does that the stack cannot.
2. Check its license, that it is maintained, its security record, that it publishes builds for both CPU architectures, and its size.
3. Pin it in a lockfile, or by image tag and digest, so the update bot and the scanners cover it.
4. Price it, and write a decision record if a reader would ask "why this?".
5. Add it to the catalog with its trigger, and add a line to the traps list when it first bites.

Traps

Questions

1. What are the three kinds of trigger that justify adding a component?

   Answer

   A measurement with the command that produced it, a written requirement, or a failure users would see, shown by a drill.

2. Name four costs of adding even a small component.

   Answer

   Any four of: an image to patch and scan, settings and secrets, a failure mode to decide and drill, a backup if it holds state, an upgrade path, monitoring with alerts and a runbook section, a decision record and every new engineer's reading time.

3. When should a component be removed?

   Answer

   When nobody can say what happens when it fails, when a managed service does the same job, when two things do one job, or when it was added "in case" and nothing has needed it since.

Before the first feature

Every project a team builds should look the same outside its domain logic, so that whoever inherits it, often an infrastructure team taking a proven service to scale, knows where everything is with no ramp-up. The reference repository's first decision record set that shape: the api and web apps in fixed places, the AI logic in its own clearly named module, infrastructure in its own directory, decisions as numbered records, the same CI shape, and configuration from environment variables so the secrets source can change without code changes (Chapter 1).

From a template: keep, replace, configure

#!/usr/bin/env bash
# Make a fresh copy of this template your own project:
#   scripts/new-project.sh <name> <github-owner>      e.g. claims-helper acme-corp
#
# Replaces the template's identifiers everywhere outside the docs:
# - "Parzon/triage-assistant" (repository links: runbook URLs in the alert
#   rules, the security-advisory link, the VM bootstrap's clone URL);
# - "@Parzon" (CODEOWNERS);
# - "triage-assistant" (compose projects, image names, the Grafana
#   folder, the page heading, the api's title);
# - the cookie names' prefix (COOKIE_PREFIX in app/sessions.py). Cookies
#   are scoped by host, not port: two projects from this template on
#   localhost would otherwise sign each other out.
#
# Left alone on purpose:
# - the docs, which describe the template and are yours to rewrite;
# - the database role names in .env.example;
# - the demo realm ("triage", client "triage-web"), which your
#   organisation's identity provider replaces;
# - app/triage.py, the AI module a new project rewrites anyway.
#
# Run it once, on a fresh copy, then `make setup && make check`.
# Portable: GNU and BSD (macOS) sed both accept `sed -i.bak`.

Replacing the domain, in order

What not to change without a decision record

Some rules are what stop copies of a template drifting into different platforms. In the reference template they are the access model (roles, 404 for invisible data, row-level security), the deploy path (rolling deploys, expand-and-contract migrations), the timeout chain, the split between liveness and readiness, and the eval gate for prompt and model changes. Change one only on purpose, and write down why. To take a later fix from the template, add it as a remote and cherry-pick one commit at a time; never merge it wholesale, because the histories are unrelated and it would bring the template's domain back.

Team defaults for development

Review standards matter more, not less, when much of the code is written with AI coding tools. No merge without a human review; CI green is mandatory; tests are the real gate, and untested generated code is as unmergeable as untested human code; pull requests stay small, because generated diffs are large and large diffs get rubber-stamped; and a reviewer must be able to explain why the code works. Watch for the typical failure patterns: invented APIs or parameters, silently swallowed exceptions, clever one-liners, and tests that assert something trivial (Handbook 1, Chapter 18).

Day one with an agent

A new service that an agent will mostly write needs its guardrails before its first feature, for the same reason the pipeline comes before the first feature: everything the agent writes afterwards is shaped by what the repository enforces on day one. The hour it takes is the cheapest hour of the project.

The contract stays short because the research on repository instruction files found that longer ones did not raise success and did raise cost; it holds what a tool cannot express. The hooks are where "the agent runs the checks before it says done" stops being a request. The owners file is what stops a pull request from quietly loosening a rule. And the problem-statement template is the engineer's main artifact from now on: the agent is only as good as the outcome it was given and the check that proves it.

Traps

Questions

1. What should a new service's first deliverable be, and why?

   Answer

   A health endpoint built by CI and deployed by CD to the real environment. It proves the whole path (access, networking, TLS, DNS, the pipeline) before any feature depends on it.

2. Starting from a template, what is kept, what is replaced, and what is configured?

   Answer

   Kept: the platform (build, deploy, sign-in, data access, streaming, observability, evals harness, safety controls). Replaced: the domain (data model, routes, prompt, eval cases, UI, tests). Configured: the providers, URLs, secrets, registry and hosts.

3. Why do eval cases come before the prompt?

   Answer

   They define what a good answer looks like before anyone is tempted to adjust the definition to fit the output, the same reason tests come before a refactor.

4. What is the default development database for a team, and what never goes into it?

   Answer

   A database per developer in containers, seeded and migrated. Raw production data never does; use anonymized snapshots in staging, synthetic data, or a minimal row under a documented exception.