Reverting deploys

Purpose: revert a watermark/delivery change safely when delivery is split across an edge snippet (watermark router) and a backing worker. The core rule is dual-lane: a full revert needs the snippet turned off and the worker referrer cohort backed off. Turning off only the snippet routes traffic back to the watermarking worker — it does not restore clean delivery and can increase cost.

Applies to: any host whose thumbnails are delivered through the edge snippet’s HOT/redirect set (307 → R2 wm.jpg) while the same host sits in the worker’s referrer rollout cohort at rollout=1, variant=wm. For the concrete host/customer/zone identities behind every role below, see the internal decoder (not published).

Sibling runbooks: flipping-hosts-to-watermarked, purge-cost-safety, hot-set-decay.

Why revert is dual-lane

There are two independent delivery surfaces, and they fail back differently:

Surface	What it does	Disabling it alone
Edge snippet (watermark router)	Matches `WM_DOMAINS` + a HOT id set, 307s to `cache.<zone>/thumbnails/{id}/wm.jpg`	Routes traffic back to the worker, which is still in-cohort and re-watermarks
Worker referrer cohort	Watermarks in-path and re-pins `wm` on the bare `/{id}.jpg` key under the 31-day referrer-blind page-rule cache	Returns the host to the clean/R2-cheap path for non-HOT ids

Consequences that make single-lane reverts wrong:

Snippet disable does NOT restore clean. While a host stays at rollout=1, disabling the snippet just moves its traffic from the snippet’s 307 path onto the watermarking worker. Viewers still get watermarked images.
Snippet disable can INCREASE cost. The snippet 307 path is served from R2 (cheap). The worker path bills worker invocations + KV reads (a much higher blended rate). Disabling the snippet on a high-volume cohort host swings the whole HOT-307 volume back onto the worker — the single largest spend swing in this system. Never reach for snippet-disable as a cost brake.
Reverse-leak purges don’t stick while the host is in-cohort. Any non-HOT id re-pins wm on its bare key on the next request after a purge. Clean is only durable once the host is out of the cohort.

Rule of thumb: snippet off = “stop redirecting”; cohort off = “stop watermarking.” You usually need both, and the order matters (see Kill action ordering).

Revert tiers

Pick the smallest tier that addresses the incident.

Tier 0 — Global kill switch (fastest stop, one API call)

Disable the snippet rule. This is the fastest way to stop the snippet 307 path, but it is not a clean rollback and it is cost-increasing while hosts remain in the cohort.

CONFIRM_DISABLE=YES \
CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_ID=<bare-zone> \
  bun scripts/cloudflare/snippet-rollback.ts disable

What it does: fetches all snippet rules, flips enabled=false on the watermark-router rule only (unrelated rules preserved), and PUTs the full list back. It refuses unless CONFIRM_DISABLE=YES and aborts if no matching rule is found.

After a disable you must also back off the worker cohort, or every affected host keeps getting watermarked by the worker (and now bills the worker path). Disable is a stop-gap, never the whole revert.

Tier 1 — Per-host fast revert (~3–6 min + propagation)

The correct full revert for a single host. Both lanes, same release:

Snippet lane — remove the host from WM_DOMAINS and from the HOT/redirect set in the snippet source. Remove subdomain entries separately (domain normalization strips only www./m., so a regional or sub-brand subdomain — e.g. eu., shop. — is its own entry).
Worker lane — remove the host from the referrer rollout cohort (referrer-rollout-cohort.ts) or set its rollout=0.
Gate — run unit/contract tests + lint; verify the snippet size budget still passes; capture a fresh rollback snapshot (Tier 3).
Deploy both — redeploy the snippet and redeploy the worker. Neither alone is sufficient.
Verify with a real id — smoke with the host’s actual id (not a generic hotIds[0] placeholder) and assert the served variant is clean:
- a current host id → does not 307, served variant clean;
- a known non-HOT id with the host referrer → if it still serves wm, the worker is still watermarking → the cohort backoff didn’t land; fix and re-verify before calling it reverted.

# after editing snippet source + cohort source
bun --filter @repo/snippet test && bun run lint:touched && bun run lint
bun --filter @repo/snippet deploy          # snippet lane
bun --filter @repo/thumbnail deploy        # worker lane (cohort change)
# verify a REAL id serves clean:
WM_REFERER=https://<host>/<page> HOT_REDIRECT_ID=<real-host-id> \
  bun scripts/cloudflare/smoke-watermark-snippet.ts

Revert highest-volume / rotating-catalog hosts first when a global signal is ambiguous — they carry the most exposure and the most decay. The canary and the fast-rotating auction-catalog host are the most likely revert targets.

Tier 2 — Content rollback = version control

There is no one-command content rollback. The snippet source is not under Worker Version Management. Restoring routing rules (the snapshot restore-rules command) restores which rules are enabled and their expressions — it does not restore snippet source content. Content rollback is a git operation against a committed pre-flip revision:

# 1) restore the rule list (enabled state + expressions) from the latest snapshot
CONFIRM_RESTORE=YES \
CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_ID=<bare-zone> \
  bun scripts/cloudflare/snippet-rollback.ts restore-rules \
  artifacts/cloudflare/snippet-rollback.latest.json

# 2) restore the SOURCE from the pre-flip commit, then redeploy
git checkout <pre-flip-commit> -- apps/snippet/watermark-router.snippet.js
bun --filter @repo/snippet deploy

Because git is the only content rollback path, a committed pre-flip revision is a hard deploy gate — capture the snapshot and commit the snippet + cohort edits before deploying, and log the pre-flip hash.

Do not biome format / reflow the snippet during any revert. The safety parsers string-scan const HOT = new Set([…]);; reformatting can blind the parser and produce false-green gates.

Tier 3 — Rollback snapshot (capture before you need it)

The snapshot captures Cloudflare state git cannot represent: the deployed snippet content hash and the full snippet rule list, plus the current git head/patch for the source.

CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_ID=<bare-zone> \
  bun scripts/cloudflare/snippet-rollback.ts snapshot
# → artifacts/cloudflare/snippet-rollback.latest.json

Before any deploy that you might revert, assert the snapshot’s generatedAt is from this change (a stale snapshot restores the wrong rules). The deploy itself is non-atomic (content PUT then rule PUT, no retry) — after deploy, verify live deployedContentSha256 == local sha and the rule is enabled with the expected expression before trusting it.

Kill action ordering

When stopping an in-flight incident, order matters because snippet-disable is cost-inverting:

Step	Action	Why this order
1	Stop the bleed — if a purge loop is running, stop it first	Reversible and cheap
2	Back off the cohort — remove the host from `referrer-rollout-cohort.ts` + redeploy the worker	Returns the host to the non-watermarked, R2-cheap path; this is what actually restores clean
3	Disable the snippet — LAST, only if still needed	Snippet-disable alone neither restores clean nor reduces worker cost; doing it first routes the full HOT-307 volume back onto the worker (the largest spend swing)

A true global stop = stop purge + cohort backoff + (only if needed) snippet disable as the final step. Never disable the snippet first.

Residual risk after revert

Reverting source + rules does not instantly restore every viewer. Track these:

Cached 307s linger until TTL. A reverted host keeps redirecting already-cached viewers to wm.jpg until the cached redirect expires — unless the 307 carries Cache-Control: no-store. If the redirect does not set no-store, treat revert latency as the redirect cache TTL (effectively unbounded for high-volume hosts). Add no-store to the 307 before flipping any >1M/mo host so revert latency stays bounded.
Never delete a still-referenced variant object. Do not delete a warm wm.jpg that a still-HOT id is redirecting to — deleting it 404s the host. Remove the id from HOT (so nothing redirects to the object) before considering any object deletion.
Purges are never auto-undone. A bare-key purge is one-directional; there is no rollback of a purge. If you purge into a colo set that hasn’t picked up the revert yet, those colos re-pin from the worker. Confirm the snippet/cohort change has propagated (multi-colo quorum, not a 2-colo sample) before purging anything during a revert.
Close-gate re-probe. Re-probe each bare /{id}.jpg?permcheck=<ts> and require variant clean for every id; re-probe a known non-HOT id with the referrer — if it serves wm, cohort overlap remains. Do not declare the revert complete until both are clean.

Special cases

Unprotected paying customer. A paying customer with empty entitlements (no domains/IPs populated) is not structurally protected by the paying-bypass check and is a rollout blocker, not a revert concern — but if one slipped through, a revert (cohort backoff + snippet removal) is the immediate mitigation. See the internal decoder (not published) for the affected customer role.
Extensionless ids. Hosts that embed extensionless ids bypass the *.jpg* page rule and re-warm at default TTL, not 31 days. Do not purge extensionless paths during a revert (there is no stale wm key to clear).
Redirect target on the cache zone. The *.jpg* 31-day page rule lives on the bare zone and does not match the cache zone. A wm.jpg that 404s on the cache zone is not edge-pinned and re-fetches R2 on every request — verify wm.jpg is 200 (R2 HEAD on the bucket binding, authoritative) before purging its bare key.

Quick reference

Stop only            → snippet disable (Tier 0)  [NOT a clean revert; cost-increasing]
Restore one host     → snippet remove + cohort backoff + deploy both + verify real id clean (Tier 1)
Restore source       → restore-rules (rules) + git checkout pre-flip commit + deploy (Tier 2)
Before any deploy    → capture snapshot + commit pre-flip revision (Tier 3)
Incident order       → stop purge → cohort backoff → snippet disable LAST

Decode any concrete host, customer, zone, or id role in the internal decoder (not published).