Safe deploys

One-line purpose: deploy the edge watermark router snippet (and the thumbnail worker) without serving 404s, leaking clean assets, breaking paying/no-referrer traffic, or losing the ability to roll back.

Applies to: anyone deploying apps/snippet/watermark-router.snippet.js via the Cloudflare Snippets API, or the thumbnail worker behind the same zone. For concrete hosts, ids, zone ids, and customer identities, see the internal decoder (not published) — this doc carries none of them. Sibling runbooks: watermark-system (architecture), referrer-cohort-and-delivery-policy, no-referrer-traffic, reverting-deploys, an internal entitlements runbook (not published).

System model (read this first)

The edge snippet is a referrer-aware router that sits in front of the thumbnail worker on the bare-image path (/{id}.jpg). It holds two string-scanned data structures the safety tooling depends on:

HOT set — const HOT = new Set([ "<id>", ... ]); — the redirect-eligible ids.
WM_DOMAINS — the referrer cohort (self-referred hosts) whose traffic is eligible for the watermark redirect.

On a request the snippet decides, in this order: paying-bypass (evaluated before WM_DOMAINS — structurally safe) → watermark-referrer + HOT id → 307 to the public R2 object …/thumbnails/{id}/wm.jpg. Anything else (no referrer, non-watermark referrer, non-HOT id) passes through to the worker.

Key invariants the gates protect:

Invariant	Why it matters
Every HOT id has a public `wm.jpg` in R2	A 307 into a missing object is a 404 storm.
The bare `/{id}.jpg` key ends clean-only; `clean.jpg` in R2 is not public	Determinism + no clean-asset bypass.
Snippet stays under the size cap and is never reformatted	Safety parsers string-scan the HOT block; a formatter reflow blinds them → false-green gates.
A committed pre-flip git revision exists	Git is the only content rollback (see Rollback).
Deployed content-hash == local, rule enabled with expected expression	The deploy is non-atomic (content PUT then rule PUT, no retry).

A 31-day referrer-blind page-rule cache sits over the bare key. Consequence: a reverted host keeps redirecting cached viewers until the cached response (the 307, and the pinned variant) expires. The snippet is not under Worker Version Management — snippet rollback is a separate procedure from worker version rollback.

The gate ladder

Run top-to-bottom. Each gate is fail-closed: a failure blocks the deploy. Add a test/HOT case per newly-flipped host before starting.

G1 — Unit / contract tests

bun --filter @repo/snippet test

Cover the router’s redirect, pass-through, paying-bypass, and no-referrer contracts. One case per host you are flipping.

G2 — Size guard (32KB) + projected-size budget + NEVER-REFORMAT

The Snippets API rejects payloads over 32,768 bytes (MAX_SNIPPET_BYTES); the deploy script asserts this before upload. But the cap is a hard wall, not a target — budget against it:

bun --filter @repo/snippet lint   # forbidden-pattern + size lint
wc -c apps/snippet/watermark-router.snippet.js

Fail if projected post-edit size > 28,672 bytes (keep a ~4KB margin under the 32KB cap for the next batch).
NEVER run a formatter on the snippet during a deploy. The safety parsers locate the HOT set by string-scanning the literal const HOT = new Set([ … ]); and pull ids with a quoted-string regex (see Parser fragility). A reflow that wraps, re-indents, or re-quotes that block produces gates that pass while validating the wrong content. Formatting this file is forbidden during rollout.

G3 — Repo lint

bun run lint:touched && bun run lint

G4 — Fresh rollback snapshot + committed pre-flip revision

Git is the only content rollback, so this gate is mandatory and cannot be skipped.

CLOUDFLARE_API_TOKEN=… CLOUDFLARE_ZONE_ID=… \
  bun scripts/cloudflare/snippet-rollback.ts snapshot

The snapshot captures what git cannot: deployed content sha256, the full snippet-rule list, and current git state. Then:

Assert the snapshot’s generatedAt is this batch (a stale snapshot restores stale rules).
Commit the snippet (and any worker/cohort) edits.
Log the pre-flip commit hash — this is your content-rollback target.

G5 — HOT-set `wm.jpg` existence verify + parse-count cross-check

bun scripts/cloudflare/verify-watermark-snippet-hotset.ts verify   # 0 missing

verify HEADs every HOT id’s public wm.jpg and exits non-zero on any missing/non-200. Use verify as the gate; ensure (write path, CONFIRM_ENSURE=YES) is for backfill, not gating.

Cross-check the parse count to catch a reflow-blinded parser (G2):

parsed HOT count  ==  rg -c '"[^"]+"' (quoted ids in the HOT block)  ==  pre-edit count + ids added

If the parser reports fewer ids than rg sees, the HOT block was reformatted and the verify is validating a subset. Stop and re-check G2.

G6 — Multi-id clean-bypass block check

For ≥1 id per host, HEAD the public R2 clean.jpg:

HEAD https://<r2-cache-domain>/thumbnails/{id}/clean.jpg

Any 2xx = clean asset is publicly reachable = bypass of the watermark = hard ABORT. (The verify script’s CHECK_CLEAN_BLOCK=true does a single-id version; the gate wants multiple ids across hosts.)

G7 — Predeploy route guard

Required whenever the release also edits worker source (e.g. cohort changes). Fail-closed on a stale snapshot, missing expected route, drift, collision, or normalization mismatch:

bun scripts/cloudflare/predeploy-thumbnail-guard.ts

Confirms the worker’s declared routes match live Cloudflare routes and that the route snapshot is fresh. A stale snapshot blocks — refresh it first.

Deploy (non-atomic — assert before smoke)

The deploy writes content, then writes the rule. Two separate PUTs, no retry, no transaction. A failure between them leaves the snippet content updated but the rule unchanged (or vice versa).

CLOUDFLARE_API_TOKEN=… CLOUDFLARE_ZONE_ID=… \
  bun --filter @repo/snippet deploy

The deploy upserts the snippet’s rule while preserving all other rules in the zone (it replaces the matching snippet_name in place, or prepends if absent).

Post-deploy, BEFORE smoke — assert the deploy actually landed:

Content hash: fetch live deployed content, sha256 it, assert it equals the local source sha. (snippet-rollback.ts snapshot computes both localSourceSha256 and deployedContentSha256 — they must match.)
Rule state: the matching rule is enabled: true with the expected expression (host match).

If either fails → the non-atomic deploy half-applied. Targeted disable, redeploy, re-snapshot. Do not smoke a half-applied deploy.

Allow a short propagation soak before smoke — a colo where the rule hasn’t taken yet still routes to the worker.

Smoke (the real flip gate)

Smoke runs against live traffic. health.ts (often auto-run by deploy) does not assert flip success — smoke is the only gate that does.

EDGE_BASE_URL=https://<edge-host> R2_BASE_URL=https://<r2-cache-domain> \
WM_REFERER=https://<host>/page \
  bun scripts/cloudflare/smoke-watermark-snippet.ts

Asserts, per the snippet contract:

Case	Expectation
HOT id + watermark referrer	307 → R2 `…/{id}/wm.jpg`, and that object is a live 2xx `image/*`
HOT id + size/numeric suffix (`{id}_large`, `{id}_12345`)	307 to the base id’s `wm.jpg`
HOT id + no referrer	no redirect → continues to worker (`X-Thumbnail-*` headers present)
HOT id + non-watermark referrer	no redirect, served variant `clean`
Non-HOT id + no referrer	no redirect → worker
R2 `clean.jpg`	not publicly accessible

Per-host smoke caveat: by default the smoke picks hotSample[0] ?? hotIds[0] for the redirect case regardless of WM_REFERER — so a per-host smoke is non-validating unless you pin the host’s real id (e.g. a HOT_REDIRECT_ID override, if available; otherwise order the HOT set so the host’s id is first, or smoke that id directly). Run subdomain entries twice (each subdomain is its own WM_DOMAINS/HOT entry). Re-confirm the paying-bypass and no-referrer controls on every batch.

Rollback

Snippet rollback is separate from Worker Version Management and is dual-lane when a flipped host is also in the worker referrer cohort: disabling the snippet rule only routes traffic back to the worker, which may still watermark. Full rollback = snippet off and worker cohort backed off and bare keys handled.

Fastest stop (snippet rule off):

CONFIRM_DISABLE=YES CLOUDFLARE_API_TOKEN=… CLOUDFLARE_ZONE_ID=… \
  bun scripts/cloudflare/snippet-rollback.ts disable

Restore the full rule list from a snapshot:

CONFIRM_RESTORE=YES … \
  bun scripts/cloudflare/snippet-rollback.ts restore-rules \
  artifacts/cloudflare/snippet-rollback.latest.json

Content rollback = git (there is no one-command content restore):

git checkout <pre-flip-commit> -- apps/snippet/watermark-router.snippet.js
bun --filter @repo/snippet deploy

Residual risk: cached 307s and the pinned variant linger under the 31-day referrer-blind page-rule cache until expiry unless the 307 carries Cache-Control: no-store. Treat revert latency for high-volume hosts as the redirect cache TTL. Never delete a still-HOT id’s warm wm.jpg (that 404s the host). Purges are not auto-undone.

See reverting-deploys for the worker-side version rollback.

Parser fragility (why NEVER-REFORMAT exists)

Three independent scripts string-scan the snippet rather than parse it:

The hotset verifier and the smoke checker both locate the literal tokens const HOT = new Set([ and ]);, slice between them, and extract ids with a "([^"]+)" regex.
The size guard counts raw bytes.

None of this survives a formatter pass that re-wraps, re-indents, splits, or re-quotes the HOT block. A reflow can leave a passing verify that validated a subset (or zero) ids — a false green. The G5 parse-count cross-check (parsed == rg-count == expected) exists specifically to catch this. Do not format the snippet during a deploy.

Quick checklist

G1 tests · G2 size + no-reformat · G3 lint · G4 snapshot + committed pre-flip hash
G5 hotset verify (0 missing) + parse-count cross-check
G6 multi-id clean-bypass HEAD (no 2xx) · G7 predeploy route guard (if worker edited)
Deploy → assert deployed-hash == local AND rule enabled w/ expected expression → soak
Smoke green (with host’s real id) before any purge
Rollback is dual-lane; content rollback = git; concrete entities in the internal decoder (not published)