Production Delivery Principles

▸ Shift-left: never break production

Every change must be proven safe before it reaches production. "It works on my machine" is not evidence. "CI passed" is necessary but not sufficient — CI catching something a local hook missed is itself a gap, not a success.

The principle

Nothing that can break the production build, test suite, or deployment should be catchable only in CI. Every failure class CI checks for needs a local equivalent that catches it before git push — otherwise the feedback loop is minutes-to-hours instead of seconds.

How it's enforced, not just documented

• A pre-commit hook runs the same checks as CI: formatting, tests, linting, YAML validation.
• A pre-deploy script verifies the full production surface — not just "tests pass," but Docker build, release boot, infra config validity.
• When CI catches something the hook missed, the tooling gap gets fixed first, then the bug — otherwise the same class of failure recurs.
• When a new CI check is added, its local equivalent lands in the same commit. A CI-only check is a check that only fires after the round-trip cost has already been paid.

▸ Pre-deploy verification checklist

Tests passing is necessary but not sufficient before a deploy. The full checklist, automated as a single script rather than a manual list to remember:

1. Tests pass with zero failures
2. Formatting, linting, static analysis clean
3. The production Docker image actually builds
4. The built image actually boots and runs a basic sanity command — a build that succeeds but produces a container that crash-loops on start is a build success and a deploy failure
5. Infrastructure config (YAML, Dockerfiles, compose files) validates
6. If clustering/distribution changed, verify nodes actually discover each other post-deploy — not just that each node individually boots

▸ Expand / contract migrations

The running application must never break during a deploy. Old code must keep working against the new schema; new code must keep working against the old schema, for the whole rollout window between old and new instances being live simultaneously.

▸ Feature-flag lifecycle

Every new user-facing feature deploys behind a flag, off by default. The lifecycle has five stages:

1. Develop behind a flag check (e.g. flags.enabled?(:feature, for: user))
2. Deploy — the code ships to production, invisible to every user
3. Validate — enable for a small set of test users via an admin surface
4. Release — enable globally
5. Contract — remove the flag check entirely in a follow-up change

▸ Test isolation is non-negotiable

Tests must be hermetic. No test should be able to affect another test's outcome. Global shared state — ETS tables, Redis, the filesystem, process registries — persists across tests by default; a flag like async: false only serializes tests within one module, not across modules.

A worked example: the ETS cross-contamination bug

# Test A writes an incomplete map to a shared ETS cache
Test A writes %{id: 1} to shared ETS (incomplete map)
    │
# Test B's own setup reads from that same cache, unaware A ran first
Test B's GenServer.init reads from ETS cache
    │
Test B gets %{id: 1}, tries to process it
    │
# crashes on a field that a *complete* record would always have
Crashes on missing :content key
    │
# only reproduces when A and B happen to run in this order
Flaky test failure (only when A and B interleave)

The fix isn't a defensive nil-check in the code under test — that would hide a real test-isolation bug behind a runtime guard. The fix is cleaning the shared cache before and after every test:

# in a shared test-case template, not scattered across individual test files
DataCase.setup_sandbox/1:
  :ets.delete_all_objects(:cache_table)  # before
  on_exit(fn -> :ets.delete_all_objects(:cache_table) end)  # after

▸ Root cause, not workarounds

Never hide the underlying issue with a defensive fix. Defensive code is a legitimate production safety net — but it must never be the primary fix, or the real bug just waits for the next unguarded path to it.

When a test fails

1. Find the actual root cause
2. Fix the root cause
3. Add a defensive fallback as a safety net, optionally
4. Document why the defensive code exists — otherwise a future cleanup pass deletes it as "dead code" and the original bug resurfaces

When a deploy fails

1. Fix the deploy issue
2. Add the check to the pre-deploy script so this exact failure can't recur silently
3. Update the deployment-discipline docs with the lesson learned
4. If the failure class was catchable locally, add it to the pre-commit hooks too

▸ Summary checklist