Watermark delivery system

Purpose: explain the freemium watermark delivery pipeline end to end — how a thumbnail request becomes a clean or watermarked image, why delivery can drift, and how to operate it deterministically.

Applies to: the thumbnail image edge (snippet router + page-rule cache + variant worker + R2), the referrer cohort, and anyone rolling a host into or out of watermarking.

Concrete entities (which hosts are flipped, paying-customer ids/IPs, code-name decode, the live page-rule id) live in the internal decoder (not published). This doc stays generic on purpose.

TL;DR

Watermarking is additive, not destructive: it is a 307 redirect to a distinct wm.jpg key. The bare /{id}.jpg key only ever holds clean bytes.
A 2022 referrer-blind 31-day page-rule cache pins one variant per URL per colo for ~31 days. First requester wins → the cache lottery.
Two control planes can emit wm for the same URL: the snippet (deterministic, pre-cache) and the worker (cohort decision, on cache miss). They overlap, which is why a snippet-only rollback never reaches clean and a purge re-pins wm.
Reverse leak: once a URL is pinned wm, no-referrer / direct users get the watermark too, violating the developer-friendly policy.

Components

Component	Role	Where
Edge snippet (watermark router)	Pre-cache request router. Runs before the worker. Redirects eligible requests to a distinct `wm.jpg` key. No bindings, 5ms CPU, 32KB.	`apps/snippet/watermark-router.snippet.js`
Referrer-blind page-rule cache	Zone page rule, `.jpg`, edge TTL ~31 days, keyed without referrer. Pins one variant per URL per colo.	zone config (snapshot in `docs/watermarks/initial-rules.json`)
Variant worker	On cache miss, decides clean vs `wm` for the referrer cohort, builds the watermark, writes both variants to R2.	`apps/thumbnail/worker.ts`, `lib/cache.ts`
Referrer cohort	The list of referrer hosts the worker watermarks. Hand-synced from analytics; decays.	`apps/thumbnail/lib/referrer-rollout-cohort.ts`
Snippet HOT / redirect set	Pre-generated `wm.jpg` ids the snippet is allowed to 307 to, plus the WM referrer domains. Decays as catalogs rotate.	`HOT` + `WM_DOMAINS` in the snippet
R2 bucket	Holds `clean.jpg` and `wm.jpg` at distinct keys per id. Public so redirects resolve.	`thumbnails/{id}/{clean,wm}.jpg`
Smart tiered cache	Enabled on the zone; accelerates origin fetch on cold colos. Does not change variant logic.	zone setting

Variant via redirect (why it’s additive)

The core mechanic: watermarking = a 307 redirect to a separate URL, never a mutation of the bare key.

GET /{id}.jpg                      -> always clean bytes (bare key)
307 -> /thumbnails/{id}/wm.jpg     -> watermarked bytes (distinct key)

R2 stores the two variants at distinct keys (thumbnails/{id}/clean.jpg, thumbnails/{id}/wm.jpg). Consequences:

The bare /{id}.jpg key is deterministically clean when only the snippet is in play — watermarking adds a redirect hop rather than overwriting bytes.
Payment-status changes need no purge: the next request simply routes to the other key.
Clean variants must stay non-public at the R2 layer (a WAF/host rule blocks …/clean.jpg), otherwise the watermark is trivially bypassed by requesting the clean key directly.

The cache lottery

The 31-day page rule caches one response per URL per colo, keyed without referrer. Whoever lands the cache miss first pins that variant — clean or wm — for everyone on that colo for up to a month.

Mixed-traffic hosts (majority no-referrer / non-cohort) tend to pin clean → cohort users never see the watermark.
Self-referred hosts (their own pages are the referrer) tend to pin wm → and then every no-referrer/ambiguous user of that URL also gets wm.
Net effect: delivery is nondeterministic per URL × colo × month. Any conversion read taken before delivery is deterministic is noise in both directions.

Custom (referrer-aware) cache keys are Enterprise-only, so the supported fix on this plan is the snippet redirect — it runs before the cache and sends eligible requests to a distinct key.

The reverse leak

Because the cache is referrer-blind, a single self-referred (or cohort) request that pins wm on the bare URL causes no-referrer / direct / dev users to receive the watermark for the rest of the TTL. That violates the developer-friendly “ambiguous traffic stays clean” policy.

Fix shape: serve no-referrer → clean explicitly, keep watermarking on the distinct wm.jpg key, and purge the bare URLs that are currently pinned wm for the affected hosts.

The worker-cohort overlap (the rollback trap)

This is the subtlety that breaks naive rollouts and rollbacks.

The worker also watermarks the cohort on cache miss. So for a host that is in the worker cohort:

A snippet 307 only fires for ids in the snippet HOT set. A non-HOT id misses the snippet, hits the worker, gets watermarked, and re-pins wm on the bare URL.
Therefore “non-HOT id → snippet passes through → served clean” is false for any cohort host.
A snippet-only rollback does not restore clean: the worker keeps watermarking and re-pins wm after any purge.
HOT-set decay does not fall back to clean — it erodes delivery determinism (some ids snippet-redirect, others get lottery’d by the worker).

Rollout / rollback rules

Flipping a host ON (deterministic wm): add it to snippet WM_DOMAINS and add its current top ids to HOT. If it is also in the worker cohort, that’s belt-and-suspenders; the snippet just makes it deterministic and skips a worker invocation.
Flipping a host OFF (back to clean) is dual-lane:
1. Remove from snippet WM_DOMAINS (+ its HOT ids).
2. Remove from the worker referrer-rollout-cohort.ts.
3. Purge the pinned bare URLs (otherwise non-HOT ids re-pin wm). Doing only step 1 leaves the worker re-pinning wm.

See reverting-deploys for the general deploy-rollback procedure; the watermark case is the exception that needs the cohort + purge steps above, not just a redeploy.

Request flow

flowchart TD
    A["Client → /{id}.jpg"] --> G["Snippet guards<br/>method · / · favicon · '://' · /.jpg→308 · /thumbnails/ · extract id"]
    G --> PAY{"paying IP / domain?"}
    PAY -->|yes| PASS["pass-through → clean"]
    PAY -->|no| NOREF{"has Referer?"}
    NOREF -->|no| PASS
    NOREF -->|yes| WMDOM{"referrer ∈ WM_DOMAINS?"}
    WMDOM -->|no| PASS
    WMDOM -->|yes| HOT{"id ∈ HOT?"}
    HOT -->|no| PASS
    HOT -->|yes| WMR["307 → /thumbnails/{id}/wm.jpg<br/>(distinct key, edge-cached)"]

    subgraph DOWN["Downstream cache + worker"]
      PASS --> PR["Page-rule edge cache<br/>*.jpg* · ~31d · referrer-blind<br/>(the lottery)"]
      PR -->|HIT| SERVE["served bytes (pinned variant)"]
      PR -->|miss| WK["Worker: cohort variant decision<br/>(watermarks bare URL → re-pins wm)"]
      WK --> R2["R2: clean.jpg / wm.jpg at distinct keys<br/>→ origin on miss"]
      R2 --> SERVE
    end
    WMR --> R2W["R2 public wm.jpg (distinct cache key)"]

Labels are generic. The snippet’s special-case loop redirect (one malformed self-referring WordPress fetch) and the paying-customer allowlists decode in the internal decoder (not published).

Operating the system

Keep the lists fresh

Both the worker cohort and the snippet HOT set are hand-synced from analytics and decay to ~0% coverage within ~8 weeks as catalogs rotate. Rotating-catalog hosts (auction / gray-market / large dynamic libraries) decay fastest; single-hero-video hosts are stable.

Re-sync cohort + HOT from current analytics on a schedule.
Keep the snippet under its 32KB script limit when expanding HOT.
Verify with the snippet test + hotset verifier before deploy.

Verify delivered bytes, not just headers

The historical miss was checking worker headers and R2 contents but never the eyeball-delivered bytes at scale. Add a periodic check: sample N cohort URLs, assert the delivered variant matches intent. The page-rule cache between worker and user is the fourth control plane and is the one that silently overrides the decision.

Purge

Single-URL purge tooling exists (purge the bare URL when fixing a reverse leak). Purge both the bare key and any internal ?_wm=1 variant the worker uses.
cache.put() is per-PoP; for a global purge use the Cloudflare purge-by-URL API, not a local delete.
Deploys do not clear the page-rule cache. Stale wm.jpg after a watermark-asset change needs an R2 delete + URL purge or a TTL wait.

Cost guardrail

Do not blanket-delete the 31-day edge TTL rule. Inverting it pushes tens of millions of edge-served requests back through the worker path (illustrative blended rate ≈ $1/1M requests moved onto the worker), which can add real monthly spend. Scope the rule (exempt hot paths or lower TTL) and write a cost estimate in docs/ first per repo policy. The snippet redirect path is cheaper than status quo because it skips the worker entirely.

Known traps

Trap	Why it bites	Guard
Snippet-only rollback	Worker re-pins `wm` for cohort hosts	Dual-lane: cohort removal + purge
Purge without cohort removal	Non-HOT id re-pins `wm` on next miss	Remove from cohort before purging
HOT decay	Erodes determinism, not safety (never falls to clean)	Scheduled re-sync
Header-only verification	Page-rule cache overrides the decision silently	Sample delivered bytes
Unprotected paying customer	Empty entitlements → still watermarked (rollout blocker)	Verify entitlements before flipping a batch; see the internal decoder (not published)
Public clean key	Watermark bypassable by requesting `…/clean.jpg`	WAF/host rule blocking clean keys
Auditing the pricing/funnel with static fetch	JS-only surfaces read empty	Verify in a rendered browser

reverting-deploys — general rollback; watermark needs the dual-lane exception above.
An internal entitlements runbook (not published) — entitlement source for paying-customer bypass.
The internal decoder (not published) — private decode of hosts, code names, IPs, ids, and the live page-rule.
Source: apps/snippet/watermark-router.snippet.js, apps/thumbnail/lib/referrer-rollout-cohort.ts, apps/thumbnail/worker.ts, apps/thumbnail/lib/cache.ts, docs/watermarks/initial-rules.json.