Staged rollouts

Purpose: Roll a delivery-path change (e.g. clean → watermarked, or worker-path → edge-redirect) out across a cohort of third-party hosts without a broad incident or a runaway bill.

Applies to: any change where one shared edge surface serves many distinct hosts at different volumes, the variant decision is made upstream of a long-lived edge cache, and a mistake is served per request until cache expiry. Concrete hosts, ids, domains, and account ids live in the internal decoder (not published) — this doc stays generic.

1. Operating principles

Order by incident exposure, not dollar cost. Exposure = volume × fraction-served-wrong. A high-volume host that is mostly already correct can rank below a smaller host whose every request is wrong. Cost ranking and exposure ranking are different orderings — use exposure for sequencing and treat cost as a separate gate (§6).
Smallest live blast radius first. Canary the host with the smallest exposure and an observable change — prefer a host whose delivery actually flips (mixed/live) over one already in the target state, so the canary exercises the real code path.
Highest-volume host last and alone, ramped one id at a time. The largest host is a batch of one, flipped after every smaller host has soaked. For it, ramp by id: top id first, soak, then add the rest.
Every change reversible — and reversibility may be multi-lane. If the variant decision lives in two places (e.g. an edge snippet and an upstream worker cohort), turning off one lane does not restore baseline. Full rollback = both lanes backed off plus any pinned cache purged. Confirm which lanes exist before you start (§7).
A committed pre-flip revision is a hard deploy gate. If there is no one-command content rollback, git is the rollback. Log the pre-flip commit hash before deploying.
Smoke is synthetic and single-id; it is not a volume health signal. If the flipped path is a redirect that skips the worker, worker analytics undercount flipped traffic. Confirm with real-traffic proxies (object-store GET volume on the target key, coverage reports), not worker invocation counts.
Don’t reformat the artifacts your safety checks parse. If gates string-scan a hand-formatted set/list (e.g. const HOT = new Set([…])), a formatter run mid-rollout can blind the parser and produce false-green gates. Freeze formatting on those files during a rollout.

2. Order the cohort by exposure

Build a per-host risk table before sequencing. Decode roles/code names in the internal decoder (not published).

Column	Meaning
`req/mo`	Traffic volume — one factor in exposure.
Incident exposure	`volume × fraction-served-wrong` — the sort key.
Catalog shape	Stable / single-hero / rotating. Drives decay risk.
Decay	How fast the host’s live ids rotate out of the pinned set.
Subdomain trap	Does a subdomain need its own cohort/allowlist entry? (Normalizers often strip only `www.`/`m.`.)
Visibility	Is the asset on an operator-visible surface, and is the rendered id known? Unverified = gated.

Representative role ordering (smallest exposure first):

Batch	Role	Soak	Why this slot
0 canary	the canary host (smallest exposure, live/mixed delivery)	48h	only host whose delivery actually flips; smallest blast radius; exercises any subdomain trap once
1	a single-hero host already in the pinned set	24h	lowest surprise — the one id is already covered
2	a batch of stable low-volume hosts	24h	each independently smoked; revert all on any one regression
3	a rotating-catalog host, high decay	48h	isolated; decay-monitored alone; blocked until the decay alarm is wired
4	first >1M host (+ its second subdomain)	72h	subdomain census; each subdomain is its own entry
5	the highest-volume host, alone, last	72h	highest exposure; ramp by id — top id first, soak, then add
6 tail	bundle of trivial-volume hosts	final	each gated on operator-visibility first

No per-host percentage knob. If “staging” means editing a set and redeploying, then a deploy’s blast radius = every host added since the last deploy. Add hosts in the batches above, never in bulk.

3. Pre-flight gates

Split gates into global (run once, program-blocking), per-deploy-batch, and per-host.

Global G0 (run once — program-blocking)

Gate	Check	Fail action
Operator visibility / live variant	Record the current live variant per host (probe a non-pinned id with the host’s referrer and a cache-buster). Establishes whether the flip is a genuine baseline→target change or just a delivery-path change.	document per host
Auth probes	Object-store / CDN API credentials work before any source edit. Failing auth after editing the pinned set strands you mid-flip.	fix creds first
No unprotected paying host	Every paying customer’s bypass entitlements are populated. A paying host with empty entitlements is structurally UNPROTECTED and a hard program blocker.	ABORT until populated or waived in `docs/`
Fail-closed dev/test scope	Any “exempt dev/test referrer” branch either exists (with tests) or is explicitly descoped. Don’t assume it exists.	implement or descope
Decay alarm wired	A coverage-floor check exits non-zero below floor (e.g. 0.85 rotating / 0.80 stable). Report-only tooling is not an alarm.	wire it or accept written manual daily re-verify
Policy sign-offs	Any policy that the flip depends on (e.g. no-referrer handling) has a dated approval doc in `docs/`.	fail-closed
Cost estimate committed	If spend could shift by ≥ the org threshold, the estimate is committed to `docs/` before flipping. See cost-safety-gates.	commit first
Redirect cacheability	If the flip emits a redirect, confirm it carries `Cache-Control: no-store` (or treat revert latency as the redirect’s cache TTL) before flipping any high-volume host.	add `no-store` first

Per-deploy-batch (G1–G7)

G1 unit/contract tests — add a case per flipped host.
G2 artifact-size + forbidden-pattern lint — if the edge artifact has a hard byte cap, assert projected < cap − margin.
G3 repo lint (bun run lint:touched && bun run lint).
G4 fresh rollback snapshot + committed pre-flip revision — assert the snapshot timestamp is this batch; log the pre-flip hash.
G5 target-asset existence + parse-count cross-check — verify every pinned id’s target object exists (0 missing) and that the parsed count equals an independent rg -c count equals pre-edit + added. The cross-check catches a reflow-blinded parser (see principle 7).
G6 baseline-bypass block, multi-id — the baseline variant must NOT be publicly fetchable for a flipped id from ≥1 id per host. Any success = hard ABORT.
G7 predeploy route guard — if this release also edits upstream (worker) source, run the route/collision guard.

Per-host (H0–H3)

H0 operator visibility (fail-closed for high-volume + tail): render the host’s key pages, capture evidence the asset lands on an operator-visible surface and the actual rendered id(s). For high-volume hosts, assert each rendered id is pinned with its target object 200 from ≥3 colos; block if rendered id ≠ assumed id.
H1 allowlist↔pinned-set consistency: host’s normalized domain in the bypass/flip allowlist AND every current live id (re-pulled same-day for rotating hosts) in the pinned set. Subdomains are separate entries.
H2 not a paying host: host ∉ paying domains and shares no paying IP.
H3 top id has a live target object (200). If 404-but-exists, purge that one URL and re-HEAD.

4. Per-batch rollout procedure

Ordering is load-bearing — each step removes a way the next step can land traffic on the wrong path.

backfill → verify+count-check → remove from upstream cohort (same release)
  → size guard → snapshot+commit → deploy → soak+multi-colo smoke
  → per-host real-id smoke → purge → close gate → observe

Backfill the target object before anything can redirect to it. Edit the pinned set (+ allowlist, subdomains as their own entries) and ensure each target object exists. Guard against double-processing: backfill from the baseline source object, not from a path that may already be transformed.
Verify + parse-count cross-check (G5): 0 missing.
Remove each flipped host from the upstream cohort, in the SAME release. Otherwise every non-pinned id re-pins the wrong variant after purge, and the single-lane “disable” rollback never reaches baseline. Accepting residual leak on non-pinned ids requires a written docs/ waiver.
Size guard (G2): post-edit artifact under cap.
Rollback snapshot + commit (G4).
Deploy. If deploy is non-atomic (content PUT then rule PUT, no retry), gate on the live artifact: live content hash == local hash AND rule enabled with the expected expression, before smoke. Mismatch → targeted disable, redeploy, re-snapshot.
Propagation soak ≥10 min, then multi-colo smoke BEFORE any purge. A colo where the rule hasn’t propagated still routes upstream and re-pins the wrong variant right after a purge. Confirm the flipped response from a multi-colo quorum (≥10–20 colos spanning regions) — a 2-colo sample is not all-colo proof.
Per-host smoke with the host’s REAL id (not ids[0] — see the warning below). Assert: top-3 ids serve the target variant; a current host id not in the pinned set does not flip and its served variant matches intent; no-referrer → baseline variant; baseline asset blocked. Run subdomains twice. Re-confirm paying + no-referrer controls each batch.
Purge the wrongly-pinned bare keys ONLY after deploy-live + cohort-removed + smoke-green. Per id, just-in-time re-HEAD the target object (require 200) before purging its bare key; skip+flag 404s to avoid purging into a 404 storm. Rate-limit and retry with backoff (see edge-cache-and-purge). Write a manifest per id.
Close gate — re-probe each bare key with a cache-buster → variant baseline for every id (treat a tiered-fill header as a hit). Also re-probe a known non-pinned id with referrer — if it serves the wrong variant, the upstream cohort still overlaps → fix the cohort, do not advance.
Observe through soak (§5). Re-run smoke at T+15m and T+24h (+48h for fast-rotating hosts). For >1M hosts, a human loads one real embedding page and visually confirms before advancing.

Zone-id footgun: if the bare key and the redirect target live in different zones, a wrong-zone purge silently no-ops. Pin both zone ids explicitly; a purge tool that validates the host but not zone ownership will appear to succeed while doing nothing.

5. Observability + numeric ABORT criteria

#	Signal	ABORT threshold	Action
1	Scheduled per-host smoke (primary)	any regression, or a top-3 id doesn’t serve target, or a non-pinned host id flips	disable + cohort backoff
2	Multi-colo target-object HEAD	any non-200 on a flipped pinned id from ≥3 colos	purge stale URL; if missing, disable + re-backfill
3	Object-store GET volume on the target key (positive confirm the flip lands)	does NOT rise above baseline within the first soak hour	disable, investigate
4	Coverage / decay	below floor (0.85 rotating / 0.80 stable) or drop >5 abs pts	hold + refresh the pinned set
5	Paying control	any paying domain/IP request for a flipped id serves the wrong variant	disable immediately
6	No-referrer control	any no/malformed-referrer request serves non-baseline	disable + purge bare key
7	Wrong-variant share	> 0.5% of an id’s sampled requests	disable

Advance gate (the return-to-baseline rule): promote host N+1 only when host N is green at smoke T+15m AND T+24h, the object-store GET rose, coverage is above floor, and (for >1M hosts) a human visually confirmed. A one-time re-warm spike decays; a structural leak is a flat plateau — they are only distinguishable after two consecutive return-to-baseline polls. Do not advance on the single decaying-spike reading alone. Pin the pre-flip baseline as an immutable captured value, not a rolling re-pull, so a leak can’t be absorbed into “the new normal.”

Why worker analytics lie here. If the flipped path is a redirect, it skips the worker entirely, so worker invocation/subrequest counts undercount flipped traffic and may read flat while real traffic climbs. Treat them as informational only. Use object-store GET volume and coverage reports as the real-traffic proxies.

Synthetic smoke is single-id. A smoke tool that always probes ids[0] (or a fixed sample) is non-validating per host — it never exercises the host’s real id. Require a per-host real-id override before trusting smoke as a flip gate.

6. Cost ordering vs incident ordering

Exposure ordering (§1) and cost ordering are different. Sequence by exposure; gate by cost separately.

Cloudflare (and most CDNs) offer no hard spend cap on paid plans — every native control is notify-after-the-fact (~24h lag). The real brakes are self-built.
The dangerous cost vector is usually not volume — it’s a non-pinned, cache-busting request (/{id}?v=NNN) that misses the edge on every request forever, or a purge that runs faster than the deploy propagates and re-warms a synchronized flood.
Because the at-risk metric (worker invocations + KV reads) often lags in billing and isn’t measured by default tooling, rate caps + trickle-purge are the primary safety; monitoring confirms, it does not prevent.

Full cost threat model, per-wave caps, the fail-closed cost gate, and circuit-breakers live in cost-safety-gates. Do not purge until those gates exist.

7. Rollback (may be multi-lane)

If the variant decision lives in two lanes (edge snippet + upstream cohort), turning off one lane does not restore baseline — and the “off” action can even invert cost (disabling the redirect can route the full flipped volume back through the expensive worker). Confirm the lane topology before you trust any kill switch.

Full rollback = edge lane OFF + upstream cohort backed off + bare keys purged.

(a) Per-host fast revert (~3–6 min + propagation): remove host from the flip allowlist + pinned set AND from the upstream cohort (or set its rollout to 0) → test/lint → deploy (+ upstream redeploy) → rollback-verify with a real-id smoke asserting the baseline variant. If it still serves the wrong variant, the upstream lane is still active — fix the cohort. Revert the highest-volume host first if a global signal is ambiguous; rotating hosts are the most likely culprits.
(b) Global kill + restore: disable the edge lane (fastest stop), restore rules from the snapshot, and git checkout <pre-flip-commit> for the content rollback. If you disable the edge lane, also back off the upstream cohort or the hosts keep getting the wrong variant.

Residual risk: cached redirects linger until expiry unless no-store was set (G0) — for >1M hosts, treat revert latency as the redirect cache TTL. Never delete a still-pinned id’s warm target object (you’ll 404 the host). Purges are not auto-undone.

See reverting-deploys for the general revert workflow.

8. Pinned-set decay + artifact budget

Cadence by catalog shape: rotating hosts daily; stable hosts weekly (or on coverage drop); single-hero / pinned-once hosts once.
A report-only coverage metric is not an alarm — wire a coverage-floor check that exits non-zero below floor (G0).
If a daily cron already syncs the upstream cohort but not the edge set, that gap is real: a rotating host’s edge pins decay silently. Close it with a refresh job that splices the live-id delta into the edge set (preserving any per-id comments the parsers depend on), re-backfills, audits, lint+tests, and fails below floor — deploy stays manual.
Artifact byte budget: if the edge artifact has a hard cap, a refresh must prune, not just append (cap per-host id count for high-rotation hosts), and must fail before writing if it would exceed the cap minus margin.

9. Quick checklist

Cohort ordered by exposure, canary is live/mixed + smallest, highest-volume host last and alone
Global G0 gates clear (no unprotected paying host; decay alarm wired; cost estimate committed)
Per batch: backfill → cohort-removed (same release) → deploy-live verified → multi-colo soak → real-id smoke → purge → close gate
Smoke uses the host’s real id, not a fixed sample
Advance only on two-poll return-to-baseline
Rollback rehearsed on both lanes; redirect carries no-store
Concrete hosts/ids/zones decoded only in the internal decoder (not published)