This is the full developer documentation for Vumbnail # Vumbnail > Add an image extension to a video URL and get its thumbnail back. No API keys, no SDK, no server. These docs describe the product and double as its build spec Vumbnail is a real, running service. Every page is badged with its true status — **Live** (serving production traffic), **Flagged** (built but gated off), or **Planned** (designed here, not built yet). The machine-readable spec a build agent reads lives at [`/llms-full.txt`](/llms-full.txt). What it is A thumbnail image host for video. Take any Vimeo or YouTube URL, swap it for a `vumbnail.com/.jpg` URL, and get the poster frame back as a real JPEG — cacheable, hotlinkable, and globally fast. How it makes money Thumbnails are free up to a monthly request allowance. High-volume sites that cross the line are served a watermarked image until they subscribe; paying customers are allow-listed back to clean images automatically. ## Start here [Section titled “Start here”](#start-here) [Introduction ](/overview/introduction/)What Vumbnail is and the problem it solves. [Who it's for ](/overview/who-its-for/)Site owners, CMS users, and the operator. [How it works ](/overview/how-it-works/)One request, end to end, through the edge. [Thumbnail URL API ](/features/thumbnail-api/)The URL shapes and what each returns. [Freemium watermarking ](/features/freemium-watermark/)How free, watermarked, and clean are decided. [Roadmap ](/project/roadmap/)What's live, flagged, and next. # Data model > Where Vumbnail's data lives — R2 objects, KV keys, D1 tables, and the committed allow-list. Vumbnail has very little “database” in the traditional sense. Its real data is cached images in R2, a small live allow-list in KV, and a billing record in D1. The Postgres schema is mostly auth boilerplate. ## R2 — the thumbnail store (`vumbnail-thumbnails`) [Section titled “R2 — the thumbnail store (vumbnail-thumbnails)”](#r2--the-thumbnail-store-vumbnail-thumbnails) All objects live under the `thumbnails/` prefix: ```plaintext thumbnails//clean.jpg clean variant (paying / under-threshold) thumbnails//wm.jpg watermarked variant thumbnails/_shared/error.jpg one shared graphic for all failures thumbnails/_shared/locked.jpg one shared graphic for password-locked thumbnails/_negative/.json negative-cache record (see below) thumbnails/_resolver-success/.json memoized Vimeo resolution ``` Clean images on the public `cache.vumbnail.com` domain are blocked by a WAF rule; only `wm.jpg` and the shared objects are publicly fetchable. See [Caching & delivery](/features/caching-and-delivery/) for the record shapes. ## KV — `THUMBNAIL_CONFIG` [Section titled “KV — THUMBNAIL\_CONFIG”](#kv--thumbnail_config) The hot-path config and live allow-list, shared between the thumbnail Worker (read) and the accounts Worker (write): | Key | Purpose | | ---------------------------------- | ------------------------------------------------------------------------------------ | | `billing-entitlements:projection` | the live allow-list (domains/IPs/keys/UAs) read on the hot path | | `billing-entitlements:mode` | `static` / `shadow` / `runtime` (currently `runtime`; hardcoded out of the hot path) | | `youtube:playlist-resolver-url(s)` | optional external playlist-resolver overrides | A separate KV namespace, `PROCESSED_EVENTS_STORE`, de-duplicates Stripe webhook events (\~90-day TTL). ## D1 — `vumbnail-billing-entitlements` [Section titled “D1 — vumbnail-billing-entitlements”](#d1--vumbnail-billing-entitlements) The billing record of truth, written by the accounts Worker: ```sql -- latest entitlement per subscription billing_entitlements_current ( stripe_subscription_id TEXT PRIMARY KEY, stripe_customer_id TEXT, stripe_price_id TEXT, stripe_product_id TEXT, customer_email TEXT, domains_json TEXT NOT NULL, -- referer hosts → clean ips_json TEXT NOT NULL, -- client IPs → clean added_at TEXT NOT NULL, updated_at TEXT NOT NULL ); -- append-only audit of every processed event billing_entitlement_events ( stripe_event_id TEXT PRIMARY KEY, stripe_event_type TEXT NOT NULL, stripe_subscription_id TEXT NOT NULL, domains_json TEXT NOT NULL, ips_json TEXT NOT NULL, observed_at TEXT NOT NULL ); ``` D1 is regenerated into the KV `projection`, which is what the hot path actually reads. See [Paid access](/features/paid-access/). ## Committed allow-list — `apps/thumbnail/lib/customers.json` [Section titled “Committed allow-list — apps/thumbnail/lib/customers.json”](#committed-allow-list--appsthumbnaillibcustomersjson) A third copy of entitlements, committed to git and bundled at deploy time as the fallback when KV is unavailable. Each entry mirrors the [`CustomerAllowlistRecord`](/features/paid-access/). It is kept in sync automatically by the accounts Worker via the GitHub API (visible as `chore(thumbnail): sync Stripe allowlist for sub_…` commits). ## Postgres (Neon) — auth only [Section titled “Postgres (Neon) — auth only”](#postgres-neon--auth-only) The Drizzle schema is Better Auth + organization-plugin boilerplate: `user`, `session`, `identity`, `verification`, `organization`, `member`, `team`, `team_member`, `invitation`, `passkey`. It backs operator-dashboard sign-in. **No Vumbnail product entity lives here** — thumbnails are R2, entitlements are D1/KV. ## Shared policy constants — `@repo/core` [Section titled “Shared policy constants — @repo/core”](#shared-policy-constants--repocore) Single source of truth for numbers that several apps depend on: vumbnail-pricing.ts ```ts VUMBNAIL_FREE_TIER_MONTHLY_REQUESTS = 50_000 VUMBNAIL_OUTREACH_GRACE_PERCENT = 20 VUMBNAIL_OUTREACH_MONTHLY_THRESHOLD = 60_000 // billing-entitlements.ts BILLING_ENTITLEMENTS_PROJECTION_KEY = "billing-entitlements:projection" BILLING_ENTITLEMENTS_MODE_KEY = "billing-entitlements:mode" // projection "kind": "billing-entitlements/v1" ``` # Deployment topology > The Workers, Snippet, domains, and request path that make up the running system. Everything runs in the `vumbnail.com` Cloudflare zone. The product is a small set of Workers plus an edge Snippet; several supporting and legacy Workers sit around them. ## Workers & routes [Section titled “Workers & routes”](#workers--routes) | Worker | Route | Role | | -------------------------------------- | --------------------------------- | ---------------------------------------------------------------- | | `vumbnail-thumbnail` | `vumbnail.com/*` (apex catch-all) | the product: resolve + watermark + serve thumbnails | | *(Snippet)* watermark-router | runs ahead of the apex Worker | hot-set short-circuit & edge redirects (cost) | | `vumbnail-accounts` | `accounts.vumbnail.com/*` | Stripe webhook → entitlement sync (KV + D1 + GitHub) | | `vumbnail-astro-site` | `astro.vumbnail.com/*` | marketing site; thumbnail Worker passes non-thumbnail paths here | | `vumbnail-animated-thumbnail-renderer` | service binding | WASM animated rendering (flagged) | | `vumbnail-video-metadata` (consumer) | queue | metadata observation (flagged) | | `playlist-resolver(-worker)` | external resolver | resolves unknown YouTube playlists | | API / dashboard | starter-kit routes | operator metrics + auth (mostly internal) | Public R2 is exposed at **`cache.vumbnail.com`**; the edge Snippet redirects hot watermarked traffic there, and a WAF rule blocks `…/clean.jpg`. ### Test & staging hosts [Section titled “Test & staging hosts”](#test--staging-hosts) * `watermark-test.vumbnail.com` — thumbnail Worker with animated thumbnails and debug enabled. * `relay-test.vumbnail.com` — forces Vimeo relay-only mode, separate R2 prefix. * `accounts-staging.vumbnail.com`, `accounts-preview.vumbnail.com` — webhook staging/preview. ### Legacy (pre-Cloudflare) [Section titled “Legacy (pre-Cloudflare)”](#legacy-pre-cloudflare) `legacy-vumbnail-vercel-api`, `legacy-vumbnail-video`, and `legacy-vumbnail-worker-canary` are retained from the Vercel era; moving the last `/6*` canary routes fully onto Cloudflare is a roadmap item. ## Bindings [Section titled “Bindings”](#bindings) **Thumbnail Worker:** `THUMBNAIL_BUCKET` (R2), `THUMBNAIL_CONFIG` (KV), `ASSETS` (watermark.png + fallbacks), `IMAGES` (legacy transform), `SITE_PASSTHROUGH_SERVICE` (→ Astro site), `ANIMATED_THUMBNAIL_RENDERER` (service binding), `VIDEO_METADATA_QUEUE` (producer), plus `VERSION_METADATA`. **Accounts Worker:** `PROCESSED_EVENTS_STORE` (KV), `THUMBNAIL_CONFIG` (KV, shared), `BILLING_ENTITLEMENTS_DB` (D1); secrets `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `GH_TOKEN`. ## Request path (clean vs. watermarked) [Section titled “Request path (clean vs. watermarked)”](#request-path-clean-vs-watermarked) ```plaintext Browser → vumbnail.com/.jpg │ ▼ Cloudflare edge Snippet (watermark-router) ├─ paying domain/IP? → pass through ──┐ ├─ /.jpg malformed? → 308 error.jpg │ ├─ Borås loop? → 307 wm.jpg │ ├─ hot id + wm referrer? → 307 cache.vumbnail.com/.../wm.jpg └─ else → pass through ──┤ ▼ vumbnail-thumbnail Worker ├─ parse id/size/password ├─ watermark decision (clean | wm) │ └─ KV entitlement projection (paid bypass) ├─ edge cache → R2 variant → R2 clean→wm └─ miss: resolve source (Vimeo/YouTube/relay) → store R2 → cache → respond ``` ## Deploy & safety [Section titled “Deploy & safety”](#deploy--safety) * Per-app deploys via Wrangler (`bun --filter @repo/ deploy`); the Snippet has its own `deploy` / `verify-hotset` / `smoke` / `rollback:snapshot`. * The thumbnail Worker ships with a predeploy route-safety guard and ramps by percentage (`0 → 1 → 5 → 10 → 25 → 50 → 100`) with a health-check suite. * Multi-worker deploy safety is enforced by `scripts/cloudflare` checks so one app’s deploy can’t clobber another’s routes. * The hard cost guardrail applies to every deploy: ≥ $1/month of new Cloudflare spend requires a written estimate in `docs/` first. # Tech stack > The platform, runtime, and libraries Vumbnail is built on. Vumbnail is Cloudflare-native end to end. Compute, storage, config, billing state, and the edge cost-control layer are all Cloudflare primitives, with a Bun * TypeScript monorepo on top. ## Platform & runtime [Section titled “Platform & runtime”](#platform--runtime) | Layer | Choice | Role | | ------------------------- | -------------------------------------------------------- | --------------------------------------------- | | Runtime / package manager | **Bun** (≥1.3) | dev, build, test, scripts | | Language | **TypeScript 5.8** | everything | | Compute | **Cloudflare Workers** | thumbnail serving, accounts/webhooks, API | | Edge pre-filter | **Cloudflare Snippet** | hot-set watermark short-circuit (cost) | | Persistent store | **Cloudflare R2** (`vumbnail-thumbnails`) | clean/wm variants, negative cache, memos | | Hot config / allow-list | **Cloudflare KV** (`THUMBNAIL_CONFIG`) | entitlement projection, processed-event dedup | | Billing record of truth | **Cloudflare D1** (`vumbnail-billing-entitlements`) | subscription entitlements + audit | | Async | **Cloudflare Queues** | video-metadata observation (flagged) | | Image transform | **Photon (WASM)**, default; Cloudflare **Images** legacy | watermark overlay | | Deploy | **Wrangler** | per-app Workers deploys | ## Application frameworks [Section titled “Application frameworks”](#application-frameworks) * **Thumbnail Worker** (`apps/thumbnail`) — the product core; plain Worker fetch handler with the cache/source/watermark libraries. No web framework on the hot path. * **Accounts Worker** (`apps/accounts`) — **Hono** app handling the Stripe webhook and entitlement sync. * **API Worker** (`apps/api`) — **Hono + tRPC** with **Better Auth**; serves the operator dashboard’s metrics endpoints. Most generic starter-kit routers (user/organization) are unused stubs. * **Marketing site** (`apps/astro`, `apps/web`) — **Astro** static site at `astro.vumbnail.com`, reached via a service binding for non-thumbnail paths. * **Operator dashboard** (`apps/local-dash`) — **React 19 + TanStack Router + Jotai + Tailwind v4 + shadcn/ui**. * **Shopify app** (`apps/shopify`) — **React Router 7** embedded app + theme extension (planned). ## Integrations & libraries [Section titled “Integrations & libraries”](#integrations--libraries) * **Stripe** — subscription billing; Checkout custom fields carry the domains/IPs to allow-list. API version `2026-02-25.clover`. * **`@repo/videos`** — provider metadata resolution (Vimeo/YouTube oEmbed, player config, password handling, size mapping). * **External resolvers** — a playlist resolver service and a Vimeo relay (`noembed.com` by default) for fallback resolution. * **Drizzle ORM + Neon Postgres** — auth tables (Better Auth) for the dashboard; **not** where billing lives (that’s D1). * **GitHub API** — the accounts Worker commits `customers.json` as a third entitlement store. ## Tooling [Section titled “Tooling”](#tooling) * **Vitest** (+ Happy DOM) for tests, including Worker and Snippet contract tests. * **Biome** for lint/format, with a stricter **agentic overlay** (`biome.agentic.jsonc`) run on touched files via `bun run lint:touched`. * **Opt-in stricter TypeScript** presets per package (`packages/typescript-config/agentic-*.jsonc`). * Repo conventions: trunk-based development, scripts written in the `docs/tiger` style, and a hard **Cloudflare cost guardrail** (any change ≥ $1/month needs a written estimate in `docs/` first). ## Notable monorepo apps [Section titled “Notable monorepo apps”](#notable-monorepo-apps) Beyond the core, the repo contains `animated-thumbnail-renderer-worker` / `anim-worker` (animated rendering), `video-metadata-worker` (queue consumer), `playlist-resolver(-worker)`, `outreach`, `email`, and three `legacy-vumbnail-*` apps retained from the pre-Cloudflare (Vercel) era. See [Deployment topology](/architecture/deployment-topology/). # Design principles > The handful of rules that decide Vumbnail's trade-offs. These are the rules that settle arguments when two designs both look reasonable. They explain why the system is shaped the way it is. ## 1. The URL is the product [Section titled “1. The URL is the product”](#1-the-url-is-the-product) The entire integration surface is a URL shape. Anything that would force a user to sign up, fetch a key, install an SDK, or read documentation before getting their first thumbnail is a regression. New capabilities should be reachable by changing the path or a query parameter, not by adopting an API. ## 2. Fail open, never blank [Section titled “2. Fail open, never blank”](#2-fail-open-never-blank) A thumbnail that fails should degrade to a clean image or a small fallback graphic — never an HTTP 500 or a broken ``. The watermark transform fails open to the clean image; unknown and blocked videos fall back to a shared `error.jpg` / `locked.jpg`; provider failures are cached as negative results so they don’t repeat. A visitor should always see *something* reasonable. ## 3. Monetize without breaking free usage [Section titled “3. Monetize without breaking free usage”](#3-monetize-without-breaking-free-usage) Free traffic is never cut off with an error or a quota wall. The lever is the **watermark**: over-threshold free sites keep working, just with a visible mark that prompts a subscription. Conversion pressure comes from the image, not from downtime. Paying customers must *never* see a watermark, so the clean path is the safe default whenever the entitlement signal is ambiguous. ## 4. Cost discipline is a feature [Section titled “4. Cost discipline is a feature”](#4-cost-discipline-is-a-feature) Vumbnail runs on metered Cloudflare primitives, and the operator treats spend as a first-class constraint: * Any change likely to add **$1/month or more** of Cloudflare spend requires a written cost estimate saved in `docs/` *before* it ships. * The cheapest layer that can answer a request should answer it. That’s why a free edge Snippet short-circuits hot watermarked traffic before the metered Worker runs, and why hot config values are hardcoded out of the KV read path. * Hot-path KV reads, Worker invocations, and subrequests are measured and actively reduced. ## 5. Edge-first, resolve once [Section titled “5. Edge-first, resolve once”](#5-edge-first-resolve-once) Resolving a video’s poster from the provider is the expensive, fragile step, so it happens as rarely as possible. A resolved thumbnail is persisted to R2 and served from cache thereafter; successful and failed resolutions are both memoized. The provider is the source of last resort, not the hot path. ## 6. Trunk-based and always deployable [Section titled “6. Trunk-based and always deployable”](#6-trunk-based-and-always-deployable) Work lands on `main`. Incomplete features hide behind flags or branch-by-abstraction (KV-driven modes, rollout percentages, `*-test` subdomains) rather than long-lived branches, so a half-finished change never blocks deploying the rest. Rollouts ramp by percentage with health checks at each step, and risky edge changes ship with a rollback snapshot. ## 7. Observe before optimizing [Section titled “7. Observe before optimizing”](#7-observe-before-optimizing) Optimizations are driven by sampled production attribution, not hunches. The Worker emits cache-layer hit/miss counters on a small request sample; the operator ranks error buckets and miss paths before deciding what to fix. A plan without a measured baseline doesn’t ship. # Animated thumbnails > Short animated preview frames for a video, rendered on demand — built but gated off. Animated thumbnails return a short looping preview of a video instead of a single still frame. The rendering path exists in the codebase and runs on the test host, but it is **gated off in production** behind a flag and an allowed-hosts list. ## Behavior [Section titled “Behavior”](#behavior) When enabled, a thumbnail request can be served as a short animated image rendered by a dedicated renderer Worker (a WASM-based animation pipeline) instead of a static JPEG. The thumbnail Worker calls the renderer over a service binding, caches the result in R2 like any other variant, and returns it. Today: * In production, `ANIMATED_THUMBNAIL_ENABLED = 0`, so requests always get the static JPEG. * On the `watermark-test.vumbnail.com` host the feature is enabled for testing, with debug headers on. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** a thumbnail request whose host is on the animated allowed-hosts list, with the feature flag enabled; the renderer service binding. **Outputs:** an animated image variant stored in R2 (under a configurable namespace) and returned with the normal caching behavior; renderer error detail is exposed only to allow-listed hosts. Controlling env vars (thumbnail Worker): * `ANIMATED_THUMBNAIL_ENABLED` — `0` (off) in production. * `ANIMATED_THUMBNAIL_ALLOWED_HOSTS` — comma-separated hostnames permitted to request animated output. * `ANIMATED_THUMBNAIL_ERROR_DETAIL_HOSTS` — hosts allowed to see renderer error detail. * `ANIMATED_THUMBNAIL_R2_NAMESPACE` — R2 path override for animated outputs. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Flag off (production default)** — the animated path is never taken; static JPEG is served. * **Host not allow-listed** — even with the flag on, a non-listed host gets the static image. * **Renderer failure** — fails open to the static thumbnail; error detail is suppressed except for the error-detail hosts. * **Cost gating** — turning this on broadly is a Cloudflare-spend change and therefore requires a written cost estimate first (see [Design principles](/design/principles/)); container/WASM size has been a focus of research to keep it affordable. ## Data shape [Section titled “Data shape”](#data-shape) ```ts type AnimatedThumbnailConfig = { enabled: boolean // ANIMATED_THUMBNAIL_ENABLED allowedHosts: string[] // ANIMATED_THUMBNAIL_ALLOWED_HOSTS errorDetailHosts: string[] // ANIMATED_THUMBNAIL_ERROR_DETAIL_HOSTS r2Namespace: string // ANIMATED_THUMBNAIL_R2_NAMESPACE } // Rendered output is cached in R2 alongside clean/wm variants, // under the configured animated namespace. ``` ## Open questions [Section titled “Open questions”](#open-questions) * **Productization.** Whether animated previews become a paid add-on, which path shape exposes them (`.gif`/`.webp` vs. a query flag), and what the per-render cost ceiling is — all undecided. * **Demand gate.** Broad rollout is blocked on demonstrated demand and a cost estimate. See the global [Open questions](/project/open-questions/). # Cache purge API > A paid-only way to force-refresh a cached thumbnail when the source video changes. A cache purge API lets a paying customer force a specific thumbnail to be re-resolved when the underlying video’s poster changes. This feature is **planned** — it is designed here but not built. ## Behavior [Section titled “Behavior”](#behavior) A paying customer requests a purge for one of their video IDs. Vumbnail invalidates the cached clean and watermarked variants for that ID (edge cache and R2), so the next request re-resolves the current poster from the provider and re-populates the caches. The purge is scoped to IDs the customer is entitled to. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs (proposed):** an authenticated paid request (API key) naming one or more video IDs (or a playlist) to purge. **Outputs (proposed):** a confirmation that the named variants were invalidated, and a count of objects purged. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Not a paying customer** — purge is rejected; this is a paid-only capability. * **Unknown / never-cached ID** — treated as a no-op success (nothing to purge). * **Negative-cached ID** — the negative record is cleared so a recovered video can resolve again. * **Rate limiting / abuse** — purge must be bounded so it can’t be used to force expensive re-resolution at scale; the limit is undecided. * **Partial failure** — if edge invalidation succeeds but R2 deletion fails (or vice versa), the response must report partial success rather than claim a full purge. ## Cost-safe purging model [Section titled “Cost-safe purging model”](#cost-safe-purging-model) Every purge — customer-requested or operational (for example, a watermark flip) — runs under the same cost-safety model. On this platform there are exactly two ways a purge produces a surprise bill: purging faster than the edge snippet has propagated, or purging a URL that isn’t covered by the HOT list, so the miss lands on the worker instead of a cached redirect. The model below closes both. The full procedure lives in the [cost-safe cache purging runbook](/runbooks/cost-safe-cache-purging/). ### Sequencing: purge last [Section titled “Sequencing: purge last”](#sequencing-purge-last) A purge is always the **last** step of a rollout, never a lever to force one: 1. **Snippet live** — the edge snippet release is deployed. 2. **Propagated** — confirmed at multiple colos, not just one. 3. **IDs HOT** — every id being purged has 100% HOT coverage. 4. **Cohort removed** — the affected host is out of the worker referrer cohort in the same release, so the worker can’t re-pin the old variant on miss. 5. **Smoke green** — post-deploy smoke checks pass. 6. **Then trickle-purge.** Purging earlier in the sequence doesn’t speed anything up — it just re-warms the cache with the state you were trying to leave. ### Trickle rate and per-wave caps [Section titled “Trickle rate and per-wave caps”](#trickle-rate-and-per-wave-caps) Purges run as a slow trickle of **20–30 URLs per minute**, in waves capped by the host’s volume tier: **50 / 25 / 15 / 5** URLs per wave, smallest caps for the highest-volume tiers. **Planned** — the batch purge wrapper that enforces these caps is designed but not built; the current purge tool is single-URL and uncapped, so no wave runs until the wrapper exists. ### Fail-closed pre-wave cost gate [Section titled “Fail-closed pre-wave cost gate”](#fail-closed-pre-wave-cost-gate) Before each wave, a cost gate (**Planned**) estimates the wave’s exposure and refuses to run unless all three ceilings hold: * **\~$0.25** re-warm cost for the wave, * **\~$0.10/month** permanent (recurring) cost added, * **\~$1.00** total program cost. The gate is fail-closed: if cache-busting data for the wave is missing, it refuses rather than assumes zero. This matters because Cloudflare has **no hard spend cap** — billing alerts only, with roughly a 24-hour lag — so every real brake here is self-built. The cache-busting permanent-cost trap A purged URL that carries a cache-busting query string and is **not** on the HOT list bypasses the cached redirect and hits the worker on every request — a permanent, recurring cost, and the one unbounded vector in the whole model. This is why 100% HOT coverage for the affected ids is a hard precondition of any purge, and why the cost gate refuses when cache-busting data is missing. ### Kill action [Section titled “Kill action”](#kill-action) If a wave misbehaves, the kill action is: **stop the purge and back the host off in the worker cohort**. Disabling the snippet is not a brake — cached redirects are what keep requests off the worker, so snippet-disable **increases** cost. Run correctly, the program nets out as a monthly **savings**, because redirects served from cache skip the worker entirely (illustrative blended worker rate: \~$1 per 1M requests). ## Data shape [Section titled “Data shape”](#data-shape) ```ts // Proposed type PurgeRequest = { videoIds: string[] variants?: ("clean" | "wm")[] // default: both } type PurgeResult = { requested: number purged: number notFound: string[] failed: { videoId: string; reason: string }[] } ``` ## Open questions [Section titled “Open questions”](#open-questions) * **Contract & auth surface.** Is purge a path (`//purge`), a separate endpoint, or part of a future account API? API-key only, or dashboard too? * **Rate limits and cost.** What per-customer purge budget prevents abuse while staying useful? Operational purges already run under the [cost-safe model](#cost-safe-purging-model) above; the customer-facing budget is undecided. * **Priority.** Rated a medium-value “support and trust” multiplier rather than a direct revenue unlock; sequenced accordingly on the [roadmap](/project/roadmap/). See the global [Open questions](/project/open-questions/). # Caching & delivery > The three-layer cache, negative caching, and graceful fallbacks that make thumbnails fast and cheap. Resolving a poster frame from a provider is the expensive, fragile step, so Vumbnail does it as rarely as possible. Every thumbnail is resolved once and then served from progressively cheaper caches. ## Behavior [Section titled “Behavior”](#behavior) On each request the Worker checks, in order: 1. **Cloudflare edge cache** (`caches.default`). A warm hit returns immediately with no further work. Clean and watermarked variants are cached under separate keys (the watermarked key carries an internal `?_wm=1` marker that is hidden from clients). 2. **R2 persistent store** (`vumbnail-thumbnails` bucket). The resolved variant is looked up at `thumbnails//clean.jpg` or `thumbnails//wm.jpg`. If the watermarked variant is missing but the clean one exists, the watermark is applied from the stored clean image rather than re-resolving the source. 3. **Source resolution.** Only on a full miss does the Worker fetch the poster from the provider (see below), store the clean variant to R2, derive the watermark if needed, populate the edge cache, and return the image. A successfully resolved Vimeo source is **memoized** to R2 so repeat misses reuse the discovered image URL instead of re-walking the provider’s metadata. ### TTLs [Section titled “TTLs”](#ttls) * **Browser** cache: \~1 day for both variants. * **Edge/CDN** cache: up to \~30 days for the watermarked variant (it changes rarely and is the most cost-sensitive). * **Fallback graphics** (`error.jpg` / `locked.jpg`): served `no-cache, must-revalidate` so a temporary failure doesn’t pin a bad image. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** the parsed [`ThumbnailRequest`](/features/thumbnail-api/), the resolved `variant` (clean vs. watermarked, from the [freemium decision](/features/freemium-watermark/)), and a `debug` flag that disables cache reads/writes. **Outputs:** the image bytes plus the variant-appropriate cache headers, and — on a sampled fraction of requests — internal attribution counters describing which layer answered. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Negative cache.** A failed resolution is recorded in R2 under `thumbnails/_negative/.json` with a state and a backoff-based expiry, so a deleted, blocked, or rate-limited video isn’t re-fetched on every hit. States: `not-found`, `password-protected`, `blocked`, `rate-limited`, `unknown`. Backoff roughly doubles the retry window up to a cap. * **Shared fallbacks.** `error.jpg` and `locked.jpg` are single shared R2 objects reused across all failing IDs, so failures cost almost nothing to store. * **Watermark transform failure** — fails open: the clean image is served and the failure is logged as a structured signal rather than thrown. * **Debug mode** (`?debug`, `?_debug`, or a `watermark-test.` host) — bypasses edge cache reads and writes and returns `no-store`, so probes always exercise the live path. * **Cache key fragmentation** is a known risk: the clean/watermarked split and query handling must not accidentally create multiple keys for one image. Miss- path attribution (below) exists to catch this. ## Data shape [Section titled “Data shape”](#data-shape) R2 layout and the cached records: ```ts // R2 object keys under the `thumbnails/` prefix // /clean.jpg clean variant // /wm.jpg watermarked variant // _shared/error.jpg shared failure graphic // _shared/locked.jpg shared password graphic // _negative/.json negative cache record // _resolver-success/.json memoized Vimeo resolution type NegativeCacheRecord = { state: "not-found" | "password-protected" | "blocked" | "rate-limited" | "unknown" reason: string // e.g. "source-http-404", "vimeo-block-detected" backoffLevel: number // drives the retry window (≈ 2^min(level,3)) softExpiresAt: string // ISO; serve-stale-while-revalidate boundary hardExpiresAt: string // ISO; record is discarded after this } type ResolverMemo = { url: string // discovered upstream image URL resolvedAt: string softExpiresAt: string hardExpiresAt: string } ``` Cache-attribution counters sampled on \~1% of requests (configurable via `THUMBNAIL_INTERNAL_ATTRIBUTION_SAMPLE_RATE`) include `cdnMatchHit/Miss`, `r2VariantHit/Miss`, `r2CleanForWmHit/Miss`, `r2NegativeHit/Miss/Stale`, and `r2SharedFallbackHit/Put`. ## Open questions [Section titled “Open questions”](#open-questions) * **Miss-path ranking (P1).** Stage-1 attribution is deployed but a long-enough sample to rank edge-cache misses vs. R2 misses vs. negative-cache misses hasn’t been collected. The optimization (cache-key fragmentation fix, removing redundant `wm → clean` recursion, or reordering negative-cache reads) is gated on that data. * **Miss coalescing via Workers Cache.** Cloudflare’s new pre-Worker [Workers Cache](/research/workers-cache-for-thumbnails-2026-07-07/) provides request collapsing (one Worker run per key per colo, free on any plan), which is the single-flight this layer otherwise lacks. It keys on URL not referrer, so it is **not** safe to front the clean/watermark variant path until the variant is encoded in the URL (see [Edge watermark router](/features/edge-watermark-router/)). Direction: pilot on fixed-variant surfaces (shared fallback graphics, a test route) first; adopt on the image path only after the Snippet URL-encodes the variant, then it can replace the hand-rolled `caches.default` layer. The [savings estimate](/research/workers-cache-savings-estimate-2026-07-07/) finds this is **not a cost play** (\~$1–$12/mo, ceiling \~$13/mo, negative if naive), so sequence it for collapsing/latency — not savings. * **Projection/cache TTL tuning** for the entitlement allow-list interacts with freshness — see [Paid access](/features/paid-access/) and the global [Open questions](/project/open-questions/). # Edge watermark router > A Cloudflare Snippet that short-circuits hot watermarked traffic before the metered Worker runs. The watermark router is a tiny Cloudflare **Snippet** that runs ahead of the thumbnail Worker. It is the **deterministic watermark control point**: a watermark is never written into the bare URL’s cache — it is a `307` redirect to a distinct `wm.jpg` key, so the bare URL only ever holds clean bytes. Flipping a host onto watermarks is **additive**: add the host to `WM_DOMAINS` and pin its current ids into the hot set. Because the redirect also skips the metered Worker (and its KV reads), every flip is a cost saving, not a cost. Everything else passes through untouched. ## Behavior [Section titled “Behavior”](#behavior) The Snippet evaluates a shallow decision tree (it has a \~5 ms CPU budget, so no heavy logic): 1. **Paying customer?** If the client IP or referrer domain is in the paying set, pass through to the Worker (which serves a clean image). Paying customers are never short-circuited. 2. **Malformed path `/.jpg`?** `308` redirect to the shared `error.jpg`. 3. **Borås WordPress loop?** If the exact loop predicate matches (video ID, UA, self-referrer, ASN `14061`), `307` to that video’s `wm.jpg`. See [Redirects](/features/redirects/). 4. **Hot-set short-circuit?** If the request is `GET`/`HEAD` for a **hot** video ID from a **whitelisted watermark referrer** (and not a paying domain/IP), `307` to `https://cache.vumbnail.com/thumbnails//wm.jpg`. 5. **Otherwise** pass through to the Worker. ### Watermark via redirect [Section titled “Watermark via redirect”](#watermark-via-redirect) The router never rewrites bytes. Watermarking a request means answering it with a `307` to a **distinct** `wm.jpg` cache key — the bare thumbnail URL is never asked to hold watermarked bytes, so no shared edge cache can pin `wm` where a clean image belongs. This is what makes the mechanism deterministic: the decision is re-evaluated per request at the edge instead of being frozen into a referrer-blind cache entry, and a host flip (or unflip) takes effect as fast as the Snippet deploy propagates. ### The hot set is a performance pin, not a policy boundary [Section titled “The hot set is a performance pin, not a policy boundary”](#the-hot-set-is-a-performance-pin-not-a-policy-boundary) Cold ids from a flipped host still pass through to the Worker, which applies the **same** referrer-cohort decision on its miss path — the same watermark outcome, just metered and cache-dependent. The hot set only decides *where* the decision is served (a deterministic, unmetered redirect at the edge), not *whether* the host is watermarked. Worker-cohort overlap Every flipped host in `WM_DOMAINS` is also in the Worker’s referrer cohort, so the Worker watermarks that host’s **non-hot** ids on miss — and can re-pin `wm` at the bare URL. Three consequences: * **A purge does not stick** unless the host’s cohort removal ships in the **same release**; otherwise the next miss re-pins `wm`. * **Disabling the Snippet alone never restores clean delivery.** Rollback is dual-lane: Snippet off **plus** cohort backoff **plus** purge. See [reverting-deploys](/runbooks/reverting-deploys/). * **Hot-set decay erodes determinism, not coverage.** An id that ages out of the hot set falls back to the Worker’s cohort decision — still watermarked, just metered and cache-dependent. See [watermark-system](/runbooks/watermark-system/). ### Why a Snippet instead of Worker code [Section titled “Why a Snippet instead of Worker code”](#why-a-snippet-instead-of-worker-code) Snippets run before Workers in Cloudflare’s pipeline and are far cheaper per request. The hot set is a hand-maintained list of the most-requested watermark-eligible video IDs whose `wm.jpg` is pre-generated in R2. Serving those via a redirect skips a metered Worker invocation and its KV reads. At current volume the blended saving is on the order of **$1.07 per million** requests diverted; the Snippet pays for itself immediately. ### Protecting clean images [Section titled “Protecting clean images”](#protecting-clean-images) Because `cache.vumbnail.com` is a public R2 domain, a WAF rule blocks direct fetches of any `…/clean.jpg` path, so the public bucket can only ever serve the watermarked or shared objects. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** request method, path, `Referer`, `CF-Connecting-IP` / `True-Client-IP` / `X-Forwarded-For`, `User-Agent`, and Cloudflare ASN. **Outputs:** a `307`/`308` redirect to a public R2 object, or pass-through to the thumbnail Worker. The Snippet itself never renders or fetches an image. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Hot ID without a pre-rendered `wm.jpg`** — the `verify-hotset` step ensures every hot ID has its watermarked object in R2 before deploy; a missing object would redirect to a 404, so the hot set and R2 are kept in sync. * **Hot ID ages out of the set** — the request falls through to the Worker, which makes the same cohort decision on miss. Coverage holds; determinism (and the cost saving) is what decays. Rotating-catalog hosts decay fastest — see [hot-set-management](/runbooks/hot-set-management/). * **Dev/local referrer** — dev and local traffic is served **clean by policy**, detected by **referrer host** (`localhost`, `127.0.0.1`, `[::1]`, `*.local`, `*.test`, RFC-1918 ranges, tunnel hosts) — never by user-agent. The explicit early return (`isDevCleanReferrer`) is **Planned**; today these referrers simply never match `WM_DOMAINS` and pass through. * **Size / bundle limits** — the deployed Snippet must stay within a 32 KB bundle and 5 ms CPU; it uses plain `Set` lookups and avoids regex. Current usage is \~10.5 KB. * **Paying customer on a watermark referrer** — the paying check runs first, so they pass through to a clean image even if the referrer is whitelisted. * **Rollback** — deploys take a snapshot first; `rollback:snapshot` restores the previous Snippet and rule. Restoring **clean delivery** for a flipped host is the dual-lane procedure above, not a Snippet rollback alone — see [reverting-deploys](/runbooks/reverting-deploys/). * **Deploy / verify / smoke** — `bun --filter @repo/snippet deploy`, `verify-hotset`, and `smoke` are the operational commands; `lint` enforces the size budget. ## Data shape [Section titled “Data shape”](#data-shape) ```ts // Hand-maintained constants in watermark-router.snippet.js const HOT_VIDEO_IDS: Set // ~68 most-requested watermark-eligible ids const WM_DOMAINS: Set // referrers that trigger watermarking: // the always-on self-referred hosts // S01–S04, plus flipped rollout hosts // (H01, …) — a flip is additive const PAYING_DOMAINS: Set // e.g. C01's domain → always pass through const PAYING_IPS: Set // e.g. C01's entitled IP → always pass through const BORAS_WORDPRESS_VIDEO_IDS: Set // the two pinned loop ids const BORAS_ASN = 14061 type SnippetOutcome = | { action: "passthrough" } | { action: "redirect"; status: 307 | 308; location: string } ``` ## Open questions [Section titled “Open questions”](#open-questions) * **How far to expand the hot set.** Larger hot sets (e.g. a `top200-per-host` slice) save more Worker/KV spend but grow the bundle toward the 32 KB ceiling (\~22.5 KB for the largest candidate). The coverage-vs-size trade-off is open, and the hot-set refresh automation is not built — rotating-catalog hosts decay toward \~0% hot coverage in \~8 weeks without it. See [hot-set-management](/runbooks/hot-set-management/). * **`Cache-Control: no-store` on the `307`** — required before flipping any host above \~1M requests/month, so intermediaries cannot pin the redirect itself. * **`isDevCleanReferrer` early return** — the explicit dev/local clean short-circuit is Planned, not built. * **No-referrer policy at the edge** — no-referrer → clean is decided; the written sign-off document is still open. See [Freemium watermarking](/features/freemium-watermark/). * **Keeping `PAYING_DOMAINS`/`PAYING_IPS` in sync** with the Worker’s KV entitlement projection (the Snippet list is currently hand-maintained). Tracked on the global [Open questions](/project/open-questions/). # Freemium watermarking > How Vumbnail decides whether a request gets a clean or a watermarked thumbnail. The watermark is the business model. Free traffic is never cut off; instead, high-volume free sites are served a **watermarked** poster frame — a visible prompt to subscribe — while everyone else gets a clean image. This page is the decision logic; [Paid access](/features/paid-access/) is how a site earns its way back to clean. ## Behavior [Section titled “Behavior”](#behavior) For every thumbnail request the Worker resolves a single `variant`: `clean` or `wm`. The decision runs in this order: 1. **Paid bypass?** If the request matches the entitlement allow-list (domain, IP, API key, or user-agent), `variant = clean`. Paying customers stop here and never see a watermark. See [Paid access](/features/paid-access/). 2. **Non-domain watermark cohort?** If the request matches a reviewed non-domain signal (a branded `X-Requested-With` app identifier, a specific IP, or a branded user-agent that has been graduated into the cohort), `variant = wm` at 100%. 3. **No `Referer` header?** `variant = clean`, by design. No-referrer traffic is dominated by datacenter renderers and SDK defaults; real browsers are a minority, so marking it would punish exactly the direct users the developer-friendly policy protects. (Historically this defaulted to a blanket watermark; it was narrowed to targeted cohorts.) 4. **Dev/local referrer?** `variant = clean`. Development traffic is detected by the **referrer host** — `localhost`, `127.0.0.1`, `[::1]`, `*.local`, `*.test`, RFC-1918 addresses, and tunnel hosts — never by user-agent. The explicit early-return branch in the edge snippet (`isDevCleanReferrer`) is **Planned**; until it lands, dev hosts get the same clean result by being absent from the cohort. 5. **Referrer in the target cohort?** If the `Referer` host is in the reviewed high-volume referrer cohort, apply the rollout percentage. The percentage is driven by a stable hash of the video ID so a given video is consistently marked or not. The rollout is currently pinned to **100%** (hardcoded out of the KV hot path for cost), so a cohort referrer reliably gets `wm`. 6. **Otherwise** `variant = clean`. The threshold that decides which referrers belong in the cohort is a policy number, not a live per-request counter: **50,000 requests/month** is the free clean allowance, with a **20% grace buffer** (so outreach starts at **60,000/ month**). Hosts crossing it are reviewed and added to the watermark cohort and the [outreach](/features/outreach-toolkit/) list. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** `Referer` host, `CF-Connecting-IP`, `User-Agent`, `X-Requested-With`, the video ID (for the stable rollout hash), the entitlement allow-list, and the `FORCE_ROLLOUT` override env var. **Outputs:** a `variant` of `clean` or `wm`, consumed by [Caching & delivery](/features/caching-and-delivery/) to select or render the right image. In debug mode the decision is exposed via `X-Thumbnail-Rollout`, `X-Thumbnail-Rollout-Reason`, and the bypass/variant headers. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Ambiguous entitlement** — when the allow-list signal can’t be read, the safe default is `clean`; a paying customer must never be watermarked. * **Watermark render failure** — fails open to the clean image (see [Caching & delivery](/features/caching-and-delivery/)). * **Reverse leak** — a referrer-blind long-TTL edge cache can pin `wm` bytes on the bare URL, so no-referrer / direct traffic receives `wm` where the policy says `clean`. The policy is explicit: **ambiguous traffic stays clean.** Under the snippet delivery model the bare URL always stays clean (the watermark is a 307 redirect to a distinct `wm.jpg` key), and any pinned-`wm` bare URLs that violate the policy are purged as part of every host flip — coupled in the same release with worker-cohort removal so the miss path can’t re-pin `wm`. * **Funnel gap (known issue)** — a watermarked image is only a conversion lever if it points the viewer to `/pricing`. Wiring the watermark to the pricing funnel is the binding constraint on converting watermarked hosts. * **Very high-volume referrers** — reviewed individually as a separate expansion rather than flipped in bulk, because delivery surface and audience vary widely at that scale. ## Data shape [Section titled “Data shape”](#data-shape) ```ts type WatermarkDecision = { variant: "clean" | "wm" reason: | "paid-bypass" | "non-domain-cohort" | "no-referrer-default-clean" | "dev-clean-referrer" // Planned: explicit isDevCleanReferrer branch | "target-referrer-rollout" | "default-clean" rollout: number // 0.0–1.0; currently pinned 1.0 for the cohort cohortMatch: boolean } // Policy thresholds (shared from @repo/core/vumbnail-pricing.ts) const FREE_TIER_MONTHLY_REQUESTS = 50_000 const OUTREACH_GRACE_PERCENT = 20 const OUTREACH_MONTHLY_THRESHOLD = 60_000 // ceil(50_000 * 1.20) ``` ## Open questions [Section titled “Open questions”](#open-questions) * **No-referrer policy sign-off.** No-referrer → clean is decided (see [Decisions](#decisions)), but the written policy sign-off document is still required before the first purge wave runs. * **Cohort graduation rules.** When should a non-domain signal auto-graduate into the watermark cohort (volume, persistence, confidence), and which signals are always excluded (browsers, generic runtimes, crawlers, shared social/ search referrers)? These are tracked on the global [Open questions](/project/open-questions/) page. ## Decisions [Section titled “Decisions”](#decisions) * **2026-06-14 — No-referrer requests are served clean, explicitly.** Evidence: no-referrer traffic is dominated by datacenter renderers plus SDK defaults, with real browsers a minority. Watermarking it would hit the direct users the developer-friendly policy exists to protect. A written policy sign-off document is still required before the first purge (open). * **2026-06-14 — Dev/local traffic is served clean, detected by referrer host.** `localhost`, `127.0.0.1`, `[::1]`, `*.local`, `*.test`, RFC-1918 ranges, and tunnel hosts match by referrer host — never by user-agent. The explicit snippet early-return (`isDevCleanReferrer`) is **Planned**, not yet built. * **2026-06-14 — The edge snippet is the deterministic watermark control point.** A watermark is a 307 redirect to a distinct `wm.jpg` cache key; the bare URL stays clean. Flipping a host = adding it to `WM_DOMAINS` plus its current ids to `HOT`. * **2026-06-14 — Ambiguous traffic stays clean (reverse-leak policy).** A live re-probe found delivery had flipped wm-dominant at the probed colo, with no-referrer users receiving `wm` from the referrer-blind long-TTL cache — a policy violation. Pinned-`wm` bare URLs that violate the policy are purged as part of any host flip, in the same release as worker-cohort removal so the worker’s miss path cannot re-pin `wm`. # Operator dashboard > The internal dashboard the operator uses to watch cache health, storage, and watermark decisions. The operator dashboard is an internal, authenticated SPA that surfaces the health of the thumbnail service. It is **not** customer-facing — embedders never see it — but it is a real, shipped part of the system, so it’s documented here. ## Behavior [Section titled “Behavior”](#behavior) After signing in, the operator lands on a metrics view (the dashboard index; `/dashboard` redirects to it) that shows: * **R2 storage size** — total and per-group breakdown (thumbnails, error logs, suffix-cleanup), so storage growth and cleanup are visible. * **Telemetry time-series** — requests over time, cache hit/miss rates, and watermark-decision counts. * **Production health** — request rates, error rates, and latency for the thumbnail Worker. * Supporting views for analytics, reports, settings, and about, with search, sort, and refresh controls. The data is pulled from a small set of metrics endpoints on the API Worker; the dashboard itself holds no business logic beyond presentation. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** an authenticated operator session (Better Auth); metrics requests to `/api/local-dashboard/metrics/*`. **Outputs:** rendered charts and tables of R2 size, telemetry summaries/ time-series, and production-health metrics. Backing endpoints (on the API Worker): * `GET /api/local-dashboard/metrics/r2-bucket-size` * `GET /api/local-dashboard/metrics/telem-r2-timeseries` * `GET /api/local-dashboard/metrics/telem-summary` * `GET /api/local-dashboard/metrics/prod-health` ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Unauthenticated** — `/login` is the entry; an already-authenticated user hitting `/login` is redirected to the sanitized `returnTo` path (or `/`). * **Metric source stale** — some panels depend on the [video-metadata pipeline](/project/roadmap/), which has been intermittently stale; affected panels can show gaps until that pipeline is repaired. * **Boilerplate routes** — the underlying starter kit ships `users`, `organization`, and similar tRPC routers as **stubs** (TODO-marked, not wired to product logic). The live dashboard value is the metrics views above, not those stubs. ## Data shape [Section titled “Data shape”](#data-shape) ```ts type R2BucketSize = { totalBytes: number groups: { name: "thumbnails" | "error-logs" | "suffix-cleanup"; bytes: number }[] sampledAt: string } type TelemetrySummary = { windowHours: number requests: number cacheHits: number cacheMisses: number watermarkDecisions: { clean: number; wm: number } } type ProdHealth = { requestRate: number // req/s errorRate: number // 0.0–1.0 p50LatencyMs: number p99LatencyMs: number } ``` ## Open questions [Section titled “Open questions”](#open-questions) * **Metric provenance & richer charts.** Adding provenance labels and pie/share charts is parked until the underlying operational signals (metadata pipeline, miss-path attribution) are repaired — polishing on top of stale data isn’t worthwhile yet. * **Customer-facing usage view.** Whether any of this graduates into a customer-visible usage dashboard is undecided; today billing is allow-list based with no end-user metering UI. See the global [Open questions](/project/open-questions/). # Outreach toolkit > An internal, evidence-driven workflow for converting over-threshold free domains to paid. The outreach toolkit is an internal operator workflow that turns first-party usage evidence into targeted, personalized conversion emails for domains that are clearly over the free allowance. It is deliberately **not** a bulk marketing tool — it only targets domains with proven, over-threshold Vumbnail usage. ## Behavior [Section titled “Behavior”](#behavior) The operator runs a short script-driven pipeline: 1. **Prepare local inputs** (gitignored): an `entry-tier.csv` of domains already over threshold plus contacts, and a `sites-meta.json` of per-domain overrides (industry, contact role, send priority, notes). 2. **Fetch usage evidence** (`bun outreach:usage`) — exports referrer, direct/ no-referrer, user-agent, and IP summaries from production analytics as NDJSON. 3. **Build sequences** (`bun outreach:build-sequences`) — generates a personalized three-part email sequence per domain into `docs/sites/.md`, merging evidence + templates + the policy thresholds. 4. **Configure sender** (gitignored `sender-config.json`) — from/reply-to and a compliant `List-Unsubscribe` header. 5. **Export & validate** — `bun outreach:export-mail-merge` produces a Thunderbird-compatible `mail-merge.csv`; `bun outreach:validate-mail-merge` gate-checks it before any send. The actual send is done by a human through Thunderbird Mail Merge — the toolkit prepares and validates, it does not send. ### The threshold [Section titled “The threshold”](#the-threshold) Targeting is gated on the shared policy numbers: free allowance **50,000/month**, **20%** grace, so outreach only fires at **60,000/month** and above. These come from `@repo/core/vumbnail-pricing.ts` so the toolkit, the watermark cohort, and the docs can’t drift. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** `entry-tier.csv`, `sites-meta.json`, production usage exports, templates in `docs/templates/`, `sender-config.json`, `suppression-list.json`. **Outputs:** per-domain sequence markdown, a validated `mail-merge.csv`, and a pass/fail validation report. No outreach artifacts or contact data are committed to git. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Under threshold** — a domain below 60,000/month is not eligible; the toolkit refuses to target it. * **Suppressed recipient** — validation fails if a recipient’s email or domain is on `suppression-list.json`. * **Duplicate / missing recipient** — validation fails on duplicate addresses or a row missing an email or domain. * **Missing compliance headers** — validation fails if sender headers or `List-Unsubscribe` aren’t populated. * **Pilot batches** — a `--max-send-priority` flag restricts a run to the highest- priority contacts for a small first send. ## Data shape [Section titled “Data shape”](#data-shape) ```ts type CsvEntryRow = Record // domain + contact columns from entry-tier.csv type SiteMeta = { domain: string contact_name?: string contact_email?: string contact_role?: string industry?: string send_priority?: number // lower = earlier notes?: string } type MailMergeRow = { to: string contact_name: string domain: string monthly: number // measured monthly requests monthly_requests_over_threshold: number free_tier_monthly_requests: 50000 outreach_monthly_threshold: 60000 from_email: string from_name: string list_unsubscribe: string part_1_subject: string; part_1_body: string part_2_subject: string; part_2_body: string part_3_subject: string; part_3_body: string } ``` ## Open questions [Section titled “Open questions”](#open-questions) * **Send-lane automation.** A pilot send lane is in progress but has been blocked on local sender configuration; how much of the send step to automate vs. keep human-in-the-loop is open. * **Contact selection.** The toolkit encodes role-preference guidance (web owner → marketing ops → … → generic alias), but the right primary/backup contact per archetype is still being tuned. See the global [Open questions](/project/open-questions/). # Paid access & entitlements > How a paying customer is allow-listed back to clean thumbnails, kept current from Stripe. A paying customer is one whose requests match the **entitlement allow-list**. When they match, the [watermark decision](/features/freemium-watermark/) short-circuits to `clean`. The allow-list is kept current automatically from Stripe subscription webhooks, so onboarding a customer does not require a code change or a deploy. ## Behavior [Section titled “Behavior”](#behavior) ### Earning clean images [Section titled “Earning clean images”](#earning-clean-images) A customer subscribes through Stripe Checkout and, in a checkout custom field, lists the **domains** and/or **IPs** their thumbnails are served from. On a successful checkout, Vumbnail records the entitlement and the customer’s traffic starts getting clean images within the allow-list’s propagation window. ### How a request is matched [Section titled “How a request is matched”](#how-a-request-is-matched) For each request the Worker checks the allow-list in this order and stops at the first match: 1. **API key** — from the `X-Vumbnail-Key` header or `?key=` query param (keys look like `vb_live__`). 2. **Client IP** — from `CF-Connecting-IP`. 3. **Referer host** — lowercased, `www.`/`m.` prefixes ignored. 4. **User-agent substring** — for mobile apps that can’t set a referrer. Any match ⇒ `variant = clean`. ### Keeping the allow-list current [Section titled “Keeping the allow-list current”](#keeping-the-allow-list-current) The allow-list has two read paths and one write path: * **Write:** the `accounts` Worker (`accounts.vumbnail.com`) receives Stripe webhooks (`checkout.session.completed`, `checkout.session.async_payment_succeeded`), parses the domains/IPs, and upserts a customer record. It writes to **three** stores: a committed `customers.json` (via the GitHub API), a **D1** database (the record of truth), and a regenerated **KV projection**. * **Read (runtime):** the thumbnail Worker reads the KV projection (`billing-entitlements:projection`) on the hot path, cached briefly in the isolate. This is the live path — new customers propagate without a deploy. * **Read (fallback):** if KV is unavailable, the Worker falls back to the `customers.json` bundled at deploy time. The entitlement system runs in `runtime` mode (KV projection preferred), having graduated from `static` (bundled JSON) → `shadow` (compare-and-log) → `runtime`. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** Stripe webhook events with a `domains` custom field; per-request `X-Vumbnail-Key`/`?key=`, `CF-Connecting-IP`, `Referer`, `User-Agent`. **Outputs:** an updated entitlement projection in KV + D1 + `customers.json`; a per-request bypass boolean feeding the watermark decision; in debug mode, `x-thumbnail-billing-entitlements-mode` and bypass headers. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Duplicate webhook** — events are de-duplicated in KV (`PROCESSED_EVENTS_STORE`, \~90-day TTL) so a re-delivered event doesn’t double-write. * **KV read failure** — falls back to the bundled `customers.json`; the request is still served, defaulting to clean only if a static entry matches. * **Propagation delay** — a newly paid customer can briefly still see watermarks until the projection refreshes (KV cache TTL). The acceptable window is an open question (see below). * **Revocation** — an entitlement removed from the projection stops bypassing at the next cache refresh; there is no instant sub-minute revoke path today. * **Unmatched paid traffic** — if a customer serves from a domain/IP they didn’t list, those requests are treated as free and may be watermarked; the fix is to add the domain/IP to their record. ## Data shape [Section titled “Data shape”](#data-shape) ```ts // Per-subscription customer record (customers.json + D1 + KV projection source) type CustomerAllowlistRecord = { stripeSubscriptionId: string // primary key stripeCustomerId: string | null stripePriceId: string | null stripeProductId: string | null customerEmail: string | null domains: string[] // referer hosts → clean ips: string[] // client IPs → clean addedAt: string updatedAt: string } // The hot-path projection the thumbnail Worker reads from KV type BillingEntitlementsProjection = { kind: "billing-entitlements/v1" version: string generatedAt: string customerCount: number domains: string[] ips: string[] apiKeys: string[] userAgents: string[] } ``` KV keys: `billing-entitlements:mode`, `billing-entitlements:projection`. D1 tables: `billing_entitlements_current` (latest per subscription), `billing_entitlement_events` (append-only audit). ## Open questions [Section titled “Open questions”](#open-questions) * **Self-serve onboarding.** Today the paid set is small and partly hand-onboarded. The end-to-end “subscribe → list domains → clean in minutes” self-serve flow and its quickstart docs are planned, not shipped. * **Projection freshness vs. cost.** Raising the KV cache TTL (e.g. 60s → 300s) cuts KV reads but widens the propagation window. Is 5-minute propagation acceptable for normal entitlement changes, and does any support flow need a sub-minute revoke? * **Pricing tiers.** The free-tier threshold is defined; the paid price points and tier structure are not documented here. See the global [Open questions](/project/open-questions/). # Redirects & compatibility > The redirects that keep legacy and playlist URLs working as stable thumbnail paths. Some video URLs don’t map cleanly to a single thumbnail path. Rather than break them, Vumbnail canonicalizes them with HTTP redirects so the eventual cached URL is consistent. The repo keeps a single source-of-truth **Redirect Inventory** in the root `README.md`; this page is its product-level description. ## Behavior [Section titled “Behavior”](#behavior) ### YouTube playlist compatibility (Worker) [Section titled “YouTube playlist compatibility (Worker)”](#youtube-playlist-compatibility-worker) A request to `/playlist` or `/playlist/` carrying a known `list=` value is redirected to the mapped video’s thumbnail. * `GET /playlist?list=PLzc4VjwEUvFg928S7nLEABR9qzd-tdsSK.jpg` → `307` to `/.jpg`, cached `public, max-age=3600`. * Unknown playlists are instead **resolved** (not redirected) to their first video via YouTube oEmbed, falling back to an external resolver service. ### Vimeo legacy password canonicalization (Worker) [Section titled “Vimeo legacy password canonicalization (Worker)”](#vimeo-legacy-password-canonicalization-worker) Older embeds pass the Vimeo password as a `?h=` query parameter. Vumbnail moves it into the canonical `:password` path form. * `GET /637615910?h=27057b8971.jpg&foo=bar` → `302` to `/637615910:27057b8971.jpg?foo=bar`. * Unrelated query parameters are preserved; only `h` is removed. Cached `public, max-age=3600`. ### Malformed path fallback (edge Snippet) [Section titled “Malformed path fallback (edge Snippet)”](#malformed-path-fallback-edge-snippet) The legacy artifact path `/.jpg` (a full URL accidentally pasted into the path) is caught at the edge. * `GET /.jpg` → `308` to `https://cache.vumbnail.com/thumbnails/_shared/error.jpg`. ### Borås WordPress loop escape (edge Snippet) [Section titled “Borås WordPress loop escape (edge Snippet)”](#borås-wordpress-loop-escape-edge-snippet) A specific WordPress site re-requests its own thumbnails in a self-referential loop. A tightly-scoped rule breaks the loop by redirecting those exact requests to the watermarked image. All conditions must match: video ID in the Borås set, path `/.jpg`, the exact WordPress user-agent, a self-referrer, and origin ASN `14061`. * Matching request → `307` to `https://cache.vumbnail.com/thumbnails//wm.jpg`. This case is deliberately narrow: a path-only rule would match \~267k requests where the scoped predicate matches \~134k, so the scope is the spec. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** request path, query string, and (for the Snippet rules) `User-Agent`, `Referer`, and Cloudflare’s ASN signal. **Outputs:** a `302`, `307`, or `308` `Location` redirect with a `Cache-Control` header, or pass-through to normal resolution when no rule matches. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Unknown playlist** — no hardcoded mapping, so the Worker resolves the first video live; if resolution fails, it negative-caches a `playlist-resolve-failed` reason and serves the error fallback. * **`?h=` with extra query params** — preserved across the redirect except `h`. * **Near-miss on the Borås rule** (wrong UA, missing referrer, different ASN) — intentionally *not* redirected; falls through to normal handling to avoid over-redirecting unrelated traffic. * **Redirect consolidation** — these rules live in Worker code and the edge Snippet; the standalone Cloudflare dynamic-redirect deployer has been retired in favor of consolidating in the Snippet (decision recorded 2026-05-26). ## Data shape [Section titled “Data shape”](#data-shape) ```ts // Hardcoded YouTube playlist → first-video mapping (Worker) type PlaylistRedirect = { listId: string; videoId: string } // Borås loop-escape predicate (edge Snippet) — every field must match type BorasEscapeRule = { videoIds: string[] // e.g. ["1173997793", "1173997754"] userAgent: string // exact: "WordPress/6.1.1; https://www.borasdjurpark.se" referrerHostMatches: "self" asn: 14061 redirectTo: "https://cache.vumbnail.com/thumbnails/{videoId}/wm.jpg" status: 307 } ``` ## Open questions [Section titled “Open questions”](#open-questions) * **Where new redirects live.** The current decision is to consolidate edge redirects in the Snippet rather than Cloudflare Single Redirects / dynamic rules. Whether any future case justifies reintroducing a separate redirect surface is open. * **Playlist mapping growth.** The hardcoded playlist map is maintained by hand; whether to back it with a resolver cache instead is undecided. See the global [Open questions](/project/open-questions/). # Shopify app > An embedded Shopify app and theme block for product-page video thumbnails — scaffolded, not launched. The Shopify app puts Vumbnail thumbnails into Shopify storefronts via a theme app extension, with an embedded admin app for setup and billing. It is **scaffolded and researched but not launched** — the surface exists from the Shopify app template and the design direction is set, but it is not in the roadmap’s near-term ship lane. ## Behavior [Section titled “Behavior”](#behavior) When live, a merchant installs the Vumbnail app from the Shopify App Store, opens the embedded admin (App Bridge), and adds a **Vumbnail Product Video** block to a product template in the theme editor. The block renders a poster-first video player: it shows the Vumbnail thumbnail and only swaps in the YouTube/Vimeo iframe after a click, keeping product pages fast. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs:** a Shopify shop session (OAuth/App Bridge), a product video URL or ID configured in the block, and theme block settings. **Outputs:** a rendered poster-first video block on the storefront product page; admin views of shop, plan, and configuration. Entry routing: the app index route redirects to `/app` (preserving the `shop` search params) when a `shop` parameter is present. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Not installed / unauthenticated** — standard Shopify OAuth install flow. * **App uninstalled / scope change** — handled by `app.uninstalled` and `app.scopes_update` webhooks. * **No video configured** — the block renders nothing (or a placeholder) rather than a broken player. * **Billing** — Shopify Billing is the intended system of record for merchant charges; the existing Stripe entitlement system would at most be a downstream sync target, not the merchant-facing biller. ## Data shape [Section titled “Data shape”](#data-shape) ```ts // Theme block setting (storefront) type ProductVideoBlock = { videoUrl: string // Vimeo/YouTube URL or id posterUrl: string // derived Vumbnail thumbnail URL autoplayOnClick: boolean } // Admin (embedded app) — shop context from Shopify GraphQL type ShopContext = { name: string domain: string plan: string } ``` ## Open questions [Section titled “Open questions”](#open-questions) * **Launch decision.** The app is in a research/scaffold state; whether and when to push it through Shopify review is deferred behind the core revenue work. * **Billing integration depth.** Exactly how Shopify Billing maps to clean-image entitlement (per-shop allow-list vs. account-level) is undecided. * **Hosting & session storage** for production (Prisma/SQLite locally) needs a production target. See the global [Open questions](/project/open-questions/). # Thumbnail URL API > The URL shapes Vumbnail accepts and the image each one returns. The thumbnail API is a set of URL shapes served from the apex domain `vumbnail.com`. There is no JSON API and no key — the path encodes the request and the response is an image. ## Behavior [Section titled “Behavior”](#behavior) A user takes a video and rewrites its URL as a Vumbnail path: * **YouTube** `https://www.youtube.com/watch?v=zFJ3yNVQ3v4` → `https://vumbnail.com/zFJ3yNVQ3v4.jpg` * **Vimeo** `https://vimeo.com/637615910` → `https://vumbnail.com/637615910.jpg` The Worker parses the path into a provider, video ID, optional password, and optional size, resolves the poster frame, and returns it as `image/jpeg`. The same path is stable and cacheable, so it can be dropped into an `` tag, an Open Graph tag, or any client that fetches a URL. ### Supported path shapes [Section titled “Supported path shapes”](#supported-path-shapes) | Path | Meaning | | ---------------------------------- | --------------------------------------------------------------------- | | `/.jpg`, `/.jpeg`, `/` | Default thumbnail (extension optional, case-insensitive) | | `/_.jpg` | A specific size variant, e.g. `/zFJ3yNVQ3v4_hqdefault.jpg` | | `/.jpg` | Vimeo (all-numeric IDs route to Vimeo; alphanumeric route to YouTube) | | `/:.jpg` | Password-protected Vimeo video | | `/playlist?list=.jpg` | The first video of a YouTube playlist | * **Provider detection** is by ID shape: all-numeric → Vimeo, otherwise YouTube. * **Default size**: YouTube resolves to `hqdefault` unless a size is given. * **ID canonicalization**: a trailing numeric suffix is treated as a size hint and stripped for the cache key (`1101804103_123456.jpg` and `1101804103_.jpg` both cache as `1101804103`). Non-numeric suffixes are preserved. ## Inputs & outputs [Section titled “Inputs & outputs”](#inputs--outputs) **Inputs** (all from the HTTP request — there is no request body): * **Path** — encodes provider, video ID, optional `:password`, optional `_size`, optional `.jpg`/`.jpeg` extension. * **Query** — `?list=` for playlists; legacy `?h=` for Vimeo (see [Redirects & compatibility](/features/redirects/)); `?key=` for paid bypass; `?debug` / `?_debug` for diagnostic headers. * **Headers** — `Referer` (drives the watermark decision), `CF-Connecting-IP`, `User-Agent`, and an optional `X-Vumbnail-Key` header for paid bypass. **Outputs:** * A `200` `image/jpeg` body — the clean or watermarked poster frame. * `Content-Disposition: inline` with a filename hint. * `Cache-Control` headers tuned per variant (clean vs. watermarked vs. fallback). * Redirects (`302`/`307`) for the compatibility cases in [Redirects & compatibility](/features/redirects/). * In debug mode, a set of `X-Thumbnail-*` headers describing the decision. ## States & edge cases [Section titled “States & edge cases”](#states--edge-cases) * **Unknown / deleted video** — resolves to the shared `error.jpg` fallback, served `200` so the `` never breaks; the failure is negatively cached. * **Password-protected Vimeo** — if the password is supplied and valid, the poster is returned; otherwise a shared `locked.jpg` graphic is served with a restriction header. * **Playlist** — known playlists are redirected to a mapped video ID; unknown playlists are resolved to their first video via oEmbed or an external resolver (see [Redirects & compatibility](/features/redirects/)). * **Unsupported but thumbnail-shaped path** (8+ char id + optional suffix, `.jpg`/`.jpeg`) — routed to an `error.jpg` fallback rather than a 404. * **Non-thumbnail path** — passed through to the marketing site service binding (`astro.vumbnail.com`), or returns a `404` text response (debug JSON if `?debug` is set). * **Provider slow / rate-limiting** — served from cache if warm; otherwise the Vimeo relay or negative cache absorbs the failure. The viewer never waits on a hung upstream beyond the resolver timeout. ## Data shape [Section titled “Data shape”](#data-shape) The parsed request that the rest of the pipeline operates on: ```ts type ThumbnailRequest = { provider: "youtube" | "vimeo" videoId: string // canonical id used for the cache key password?: string // Vimeo password, from ":pw" path or legacy ?h= size?: string // e.g. "hqdefault", "sddefault"; provider default if absent extension: "jpg" | "jpeg" | "" playlistId?: string // present for /playlist?list=... requests } // What the client ultimately receives: type ThumbnailResponse = { status: 200 | 302 | 307 | 404 contentType: "image/jpeg" | "text/plain" variant: "clean" | "wm" | "fallback-error" | "fallback-locked" body: ArrayBuffer // the JPEG bytes } ``` ## Open questions [Section titled “Open questions”](#open-questions) * **Animated and non-JPEG output.** Short animated previews exist behind a flag (see [Animated thumbnails](/features/animated-thumbnails/)); WebP/AVIF output is not currently offered. Whether to expose them through the same path shape (e.g. `.gif`, `.webp`) is undecided. * **Explicit size vocabulary.** The set of accepted `_size` tokens per provider is provider-derived rather than a documented, stable list. A published size vocabulary is not yet committed to. * See the global [Open questions](/project/open-questions/) page for cross-cutting items. # How it works > One thumbnail request, from the URL to the returned image, end to end. A thumbnail request travels through three layers: a cheap edge **Snippet**, the main thumbnail **Worker**, and the upstream video **provider**. Most requests never get past the first two. ## The end-to-end flow [Section titled “The end-to-end flow”](#the-end-to-end-flow) 1. **A browser requests an image.** Somewhere a page contains ``. The browser fetches that URL. 2. **The edge Snippet looks first.** A tiny Cloudflare Snippet runs ahead of the Worker. For a small, hand-maintained set of *hot* videos requested by watermark-eligible referrers, it redirects straight to the pre-rendered watermarked image in public storage — the Worker never runs, which keeps cost down. Paying domains and IPs are excluded so they always reach the Worker for a clean image. Everything else passes through. See [Edge watermark router](/features/edge-watermark-router/). 3. **The Worker resolves the request.** The [thumbnail Worker](/features/thumbnail-api/) parses the URL into a provider, a video ID, an optional password, and a size. It then decides one thing: should this viewer get a **clean** image or a **watermarked** one? That decision is the whole business model and is covered in [Freemium watermarking](/features/freemium-watermark/) and [Paid access & entitlements](/features/paid-access/). 4. **Caching is checked, cheapest first.** The Worker looks in the Cloudflare edge cache, then in an R2 bucket for the already-resolved clean and watermarked variants. A warm thumbnail is returned immediately. See [Caching & delivery](/features/caching-and-delivery/). 5. **On a miss, the source is fetched.** If nothing is cached, the Worker resolves the real poster image from the provider (Vimeo or YouTube, directly or via a relay), stores the clean variant in R2, applies the watermark if needed, populates the caches, and returns the image. 6. **Failure degrades gracefully.** Unknown IDs, blocked or deleted videos, and password-protected videos resolve to small shared fallback graphics (`error.jpg` / `locked.jpg`) instead of an error, and the negative result is cached so the failing upstream isn’t hammered. ## The money path [Section titled “The money path”](#the-money-path) Vumbnail never returns “quota exceeded.” Instead: * A **free** site under its monthly allowance gets clean images. * A **high-volume free** site (over the threshold, matched by referrer or other signals) gets a **watermarked** image — a visible nudge to subscribe. * A **paying** site is matched against an allow-list of domains, IPs, API keys, and user-agents, and is served clean images. The allow-list is kept current automatically from Stripe subscription webhooks. The operator watches all of this on the [dashboard](/features/operator-dashboard/) and reaches out to over-threshold domains with the [outreach toolkit](/features/outreach-toolkit/). ## Where it runs [Section titled “Where it runs”](#where-it-runs) Everything is Cloudflare-native: Workers for compute, R2 for the persistent thumbnail store, KV for the live entitlement allow-list, D1 for the billing record of truth, and a Snippet at the very edge for cost control. The full map is in [Deployment topology](/architecture/deployment-topology/). # Introduction > What Vumbnail is, the problem it solves, and what makes it different. Vumbnail turns a video URL into a thumbnail **image URL**. You take a Vimeo or YouTube link, rewrite it as `https://vumbnail.com/.jpg`, and the service returns the video’s poster frame as a real JPEG — something a browser, CMS, RSS reader, or `` tag can use directly. There is no API key to request, no SDK to install, and no server to run. The URL *is* the API. Drop it into an ``, an Open Graph tag, an email, or a WordPress block, and it works. ## The problem it solves [Section titled “The problem it solves”](#the-problem-it-solves) Vimeo and YouTube both have a poster image for every video, but getting at it is annoying: * The official paths are inconsistent between providers, change over time, and differ by size. * Some require an API call, an oEmbed round-trip, or scraping a player page. * Password-protected and unlisted videos, playlists, and deleted videos each need special handling. * Hotlinking the provider’s own CDN is fragile and sometimes blocked. Vumbnail hides all of that behind one stable URL shape and serves the result from a global edge cache, so the image loads fast and keeps working even when the upstream provider is slow or rate-limiting. ## What makes it different [Section titled “What makes it different”](#what-makes-it-different) * **It’s a URL, not an API.** The integration is copy-paste. The smallest possible thing a user has to learn is “append `.jpg`.” * **Edge-native.** Thumbnails are resolved once and then served from Cloudflare’s cache and an R2 bucket, close to the viewer, for as long as they stay warm. * **Freemium by watermark, not by gate.** Free usage never breaks. Instead of cutting traffic off at a quota, Vumbnail keeps serving — it just switches high-volume free sites to a *watermarked* image, which converts them to paying customers without ever returning an error to their visitors. * **Fails open.** When anything goes wrong — a watermark transform error, an upstream block, an unknown ID — the worst case is a clean image or a small fallback graphic, never a broken ``. ## What it is not [Section titled “What it is not”](#what-it-is-not) * Not a video host or player. Vumbnail serves still images (and, when enabled, short animated previews). The video stays on Vimeo or YouTube. * Not a general image CDN. It is purpose-built for video poster frames. * Not a metered API with usage dashboards for end users. Billing is handled through a simple subscription that allow-lists your domain or IP; see [Paid access & entitlements](/features/paid-access/). # Who it's for > The real audiences Vumbnail serves — and the operator who runs it. Vumbnail has two kinds of users: the people who embed thumbnails, and the operator who keeps the service healthy and converts heavy free users to paid. ## Site owners and developers who embed video thumbnails [Section titled “Site owners and developers who embed video thumbnails”](#site-owners-and-developers-who-embed-video-thumbnails) The primary audience is anyone who needs a video’s poster image as a plain URL: * **CMS and WordPress sites** that paste a thumbnail URL into a block or a template instead of wiring up the Vimeo/YouTube API. * **Marketing and content teams** putting video poster frames into landing pages, Open Graph / Twitter card tags, and email. * **Schools, churches, and nonprofits** embedding lecture, sermon, and event video thumbnails (a large share of current traffic comes from `*.org` education and health domains). * **Ecommerce and SaaS sites** showing product or explainer-video posters, including via Shopify storefront blocks. * **Developers** who want a thumbnail in an `` tag without an API key, in any language, because the integration is just a URL. What they share: they want a poster image *now*, with zero setup, and they don’t care how it’s produced. They discover Vumbnail by needing exactly this and finding that appending `.jpg` to a video URL just works. ### Free users vs. paying customers [Section titled “Free users vs. paying customers”](#free-users-vs-paying-customers) * **Free users** — the long tail. They get clean thumbnails up to a monthly request allowance. They never have to sign up to start. * **High-volume sites** — once a domain’s traffic crosses the threshold, its thumbnails switch to a watermarked image. The watermark is the prompt to subscribe. * **Paying customers** — sites that subscribe (via Stripe) and have their domain, IP, or API key allow-listed back to clean images. Today this is a small, hand-onboarded set growing toward self-serve. ## The operator [Section titled “The operator”](#the-operator) The second audience is the person running Vumbnail (a solo operator today). The product includes internal tooling built for that role: * An [operator dashboard](/features/operator-dashboard/) showing cache health, R2 storage size, request/error rates, and watermark decisions. * An [outreach toolkit](/features/outreach-toolkit/) that turns first-party usage evidence into targeted, threshold-gated conversion emails for domains that are over the free allowance. These tools are documented here because they are real features of the system, but they are **internal** — not exposed to embedders. ## Not the audience [Section titled “Not the audience”](#not-the-audience) * Viewers of the embedding site never interact with Vumbnail directly; they just see an image. * People wanting to host or transcode video. Vumbnail only deals in the poster image. # Changelog > Notable shipped changes, newest first. Reflect design changes back here as they land. This changelog tracks notable, dated changes to the documented surface. Because these docs double as the spec, **reflect significant design changes back here** as they ship, so the spec stays anchored to reality. ## 2026-06-17 [Section titled “2026-06-17”](#2026-06-17) * Documentation site created (`apps/docs`) — this docs-as-spec site covering the live product, flagged features, and roadmap. ## 2026-06-14 [Section titled “2026-06-14”](#2026-06-14) * Watermark delivery audit: most cohort hosts now confirmed serving watermarked bytes; **reverse leak** (no-referrer traffic also receiving `wm`) confirmed and flagged as an open question. Tier-1 conversion targets identified. * Live re-probe finds delivery flipped **wm-dominant** with the reverse leak active — see [the delivery-flip research](/research/watermark-delivery-flip-2026-06-14/). * **Path-1 decided:** the edge Snippet is the deterministic control point for watermark routing. * Staged 10-host rollout runbook plus a cost-safety addendum authored (internal). * **11 reusable runbooks** published unlisted at [/runbooks/](/runbooks/), pending audit. * **Research** section added to the site. * Docs site custom domain planned: (cutover not yet shipped). ## 2026-05-26 [Section titled “2026-05-26”](#2026-05-26) * **Borås redirect** consolidated into the edge Snippet (narrow id + UA + self-referrer + ASN-14061 predicate); standalone dynamic-redirect deployer retired. ## 2026-05-13 [Section titled “2026-05-13”](#2026-05-13) * **P0 fix:** watermark/Photon transform failures now **fail open** to the clean/ fallback image (HTTP 200) instead of throwing 500. * **P1:** stage-1 miss-path attribution counters deployed (on the \~1% sample). ## 2026-05-12 [Section titled “2026-05-12”](#2026-05-12) * **Hot KV values hardcoded** out of the thumbnail hot path: `billing-entitlements:mode` pinned to `runtime`, target-referrer rollout pinned to `1.0`. Full ramp `0 → 1 → 5 → 10 → 25 → 50 → 100` completed with health checks. * Snippet test/health hardening and cost-reduction work scoped. ## 2026-03-11 [Section titled “2026-03-11”](#2026-03-11) * **Billing entitlements `runtime` mode** cutover complete — paid customers now propagate via the KV projection (D1 source of truth), removing the manual GitHub-commit + deploy cycle for onboarding. ## 2026-03-09 → 03-23 [Section titled “2026-03-09 → 03-23”](#2026-03-09--03-23) * Reliability/observability planning: durable miss-path, video-metadata queue recovery, observability capture, hotspot-ranking refactor, and HN observability research. ## 2026-03-07 [Section titled “2026-03-07”](#2026-03-07) * **Non-domain watermark cohorts**: default for no-referrer traffic narrowed from a blanket watermark to targeted, reviewed cohorts. * Developer-docs / API-DX direction set (Stripe-inspired, task-first). # Open questions & decisions > What's genuinely undecided. A build agent should ask before assuming any of these. This is the anti-hallucination surface. If a question here isn’t answered, a build agent should **ask the operator** rather than pick an answer. Items are grouped by area and link back to the feature they affect. ## Watermark policy [Section titled “Watermark policy”](#watermark-policy) * **No-referrer policy sign-off.** The routing decision is made — no-referrer traffic gets the clean image (see Decisions below) — but a written policy sign-off document is required before the first purge runs. Who signs it, and where does it live? → [Freemium watermarking](/features/freemium-watermark/) * **Worker-cohort removal for flipped hosts.** Every flip host is also in the Worker’s referrer cohort, so the Worker watermarks non-hot ids on miss and re-pins `wm` — a purge does not stick unless cohort removal ships in the same release, and disabling the Snippet alone never restores clean (rollback is dual-lane: Snippet off + cohort backoff + purge). What is the exact removal step per flipped host? * **Pricing-funnel wiring.** A watermark only converts if it points the viewer to `/pricing`. This is the binding constraint on converting watermarked hosts — what’s the exact funnel and CTA? * **Cohort graduation rules.** Volume/persistence/confidence thresholds for auto-graduating a non-domain signal into the watermark cohort; which signals are always excluded (browsers, generic runtimes, crawlers, shared social/ search referrers). * **Whale conversion vs. cost-control.** Which high-volume hosts have operator-visible watermarks (conversion targets) vs. embedded/app-only players where the mark is unseen (cost-control only)? Requires a manual render audit per host before adding. Specifically: operator-visibility is unverified for H08 (the highest-volume flip host, deliberately last and alone in the rollout) and for the tail batch (H10–H14) — those flips are gated on this verification. ## Billing & entitlements [Section titled “Billing & entitlements”](#billing--entitlements) * **C02 entitlements (flip-program blocker).** C02 is a paying customer whose domain and IP entitlement lists are both empty — zero bypass paths, so the flip program would watermark a paying customer. This must be resolved (fill the entitlements) or explicitly waived before any flip wave ships; the whole rollout is blocked on it. → [Paid access](/features/paid-access/) * **Projection freshness vs. cost.** Raise KV cache TTL 60s → 300s? Is 5-minute propagation acceptable for normal entitlement changes, and does any support flow need a sub-minute revoke path? → [Paid access](/features/paid-access/) * **Self-serve onboarding.** What is the end-to-end “subscribe → list domains → clean in minutes” flow, and what’s the first-wave quickstart? * **Pricing tiers.** The free threshold (50k/month) is defined; paid price points and tier structure are not committed here. * **Snippet allow-list sync.** The Snippet’s `PAYING_DOMAINS`/`PAYING_IPS` are hand-maintained and separate from the Worker’s KV projection — should they be generated from the same source? → [Edge watermark router](/features/edge-watermark-router/) ## Delivery & API [Section titled “Delivery & API”](#delivery--api) * **Output formats.** Expose animated (`.gif`/`.webp`) and still WebP/AVIF through the same path shape, or a query flag? → [Thumbnail URL API](/features/thumbnail-api/) * **Published size vocabulary.** Commit to a documented per-provider `_size` token list, or keep it provider-derived? * **Playlist mapping growth.** Keep the hardcoded playlist→video map, or back it with a resolver cache? → [Redirects](/features/redirects/) ## Cost & reliability [Section titled “Cost & reliability”](#cost--reliability) * **Purge tooling before any purge.** The current purge tool is single-URL and uncapped. Before the first trickle purge, three things must exist: a batch purge wrapper with per-wave URL caps (50/25/15/5 by volume tier, 20–30 URLs/min), a fail-closed pre-wave cost gate (refuses to run when cache-busting-URL data is missing), and invocations/KV polling to watch worker cost during a wave. Cloudflare has no hard spend cap — alerts only, with \~24 h lag — so every real brake is self-built. → [Cache purge API](/features/cache-purge/) * **Hot-set size vs. coverage.** How far to grow the Snippet hot set before the 32 KB bundle ceiling bites; whether to automate a daily hot-set regenerate. * **Hot-set refresh automation for rotating catalogs.** Hosts with rotating catalogs (e.g. H03) decay toward \~0% hot coverage in roughly 8 weeks, which erodes deterministic delivery (not coverage — misses fall back to the Worker). Deterministic delivery on those hosts needs automated hot-set refresh; cadence and trigger are undecided. * **`Cache-Control: no-store` on the watermark 307.** The redirect response must carry `no-store` before flipping any host above \~1M requests/month, so flip decisions stay revocable at the edge. Not yet shipped. * **Miss-path optimization target.** After enough P1 attribution samples: fix cache-key fragmentation, remove redundant `wm → clean` recursion, or reorder negative-cache reads? → [Caching & delivery](/features/caching-and-delivery/) * **Workers Cache adoption.** Cloudflare’s new pre-Worker Workers Cache (`"cache": { "enabled": true }`) offers free request collapsing + SWR, but serves before the Worker runs and keys on URL, not referrer — so it can’t front the clean/watermark variant path yet. Where and when do we adopt it? * *Direction (2026-07-07):* not globally now (it recreates the referrer-blind cache lottery). Pilot on fixed-variant surfaces (shared fallback graphics, a test route); adopt on the image path only after the Snippet encodes the variant in the URL, then let it replace the hand-rolled `caches.default` layer. The cost estimate is now written: **not a cost play** — \~$1–$12/mo saved (central \~$5), ceiling \~$13/mo, and negative if enabled naively; adopt for request-collapsing/latency, not savings, and ship the entitlement `cacheTtl 60→300s` tuning for the cheap KV win. See [Workers Cache research](/research/workers-cache-for-thumbnails-2026-07-07/), the [savings estimate](/research/workers-cache-savings-estimate-2026-07-07/), and [Caching & delivery](/features/caching-and-delivery/). * **Animated rollout gate.** Demand + cost estimate needed before turning animated thumbnails on broadly. → [Animated thumbnails](/features/animated-thumbnails/) ## Tools & integrations [Section titled “Tools & integrations”](#tools--integrations) * **Cache purge contract.** Path vs. endpoint vs. account API; API-key only or dashboard too; per-customer rate budget. → [Cache purge API](/features/cache-purge/) * **Shopify launch.** Whether/when to push the app through review; how Shopify Billing maps to clean-image entitlement; production hosting/session storage. → [Shopify app](/features/shopify-app/) * **Outreach send-lane automation.** How much of the send step to automate vs. keep human-in-the-loop. → [Outreach toolkit](/features/outreach-toolkit/) * **Runbooks audit before public listing.** The internal runbooks are published unlisted at [/runbooks/](/runbooks/); each one needs a scrub audit (no real host, customer, or billing identities) before being linked from public navigation. ## Decisions already made (do not relitigate) [Section titled “Decisions already made (do not relitigate)”](#decisions-already-made-do-not-relitigate) * **Entitlements run in `runtime` mode** (KV projection preferred, `customers.json` fallback) — not static. * **The Snippet redirect is the deterministic watermark control point** (2026-06-14). Watermark delivery = a 307 redirect to a distinct `wm.jpg` key; the bare URL always stays clean. Flipping a host = adding it to `WM_DOMAINS` plus its current ids to the hot set. This supersedes relying on the referrer-blind page-rule cache, where a live re-probe showed delivery had flipped wm-dominant (110/126 uncontaminated reads) and a reverse leak was serving `wm` to no-referrer/direct users. → [Edge watermark router](/features/edge-watermark-router/) * **No-referrer traffic gets the clean image** (2026-06-14). No-referrer volume is dominated by datacenter renderers (DC1/DC2) and SDK defaults that strip referrers; real browsers are a minority. Watermarking it would punish exactly the developer-friendly path. The written policy sign-off is still open above. * **Dev/local traffic gets the clean image, detected by referrer host** (2026-06-14). `localhost`, `127.0.0.1`, `[::1]`, `*.local`, `*.test`, RFC-1918 hosts, and tunnel domains route clean via an explicit `isDevCleanReferrer` early return — never user-agent sniffing. The implementation is Planned, not built. * **Edge redirects consolidate in the Snippet** — the standalone Cloudflare dynamic-redirect deployer is retired (2026-05-26). * **Watermark transforms fail open** to the clean image — never throw. * **Borås loop rule is intentionally narrow** (id + UA + self-referrer + ASN 14061) — a path-only rule is too broad. * **No Cloudflare spend ≥ $1/month without a written estimate in `docs/`.** # Roadmap > What's live, what's in flight, and what's planned — in priority order. Vumbnail’s core is live and serving production traffic. The active work is concentrated on **revenue activation**, **reliability**, and **cost reduction**, in that spirit. Dates reflect repo artifacts. ## Live (shipped & serving traffic) [Section titled “Live (shipped & serving traffic)”](#live-shipped--serving-traffic) * **Thumbnail URL API** — Vimeo/YouTube, sizes, passwords, playlists. → [Thumbnail URL API](/features/thumbnail-api/) * **Three-layer caching + negative cache + fallbacks**. → [Caching & delivery](/features/caching-and-delivery/) * **Freemium watermarking** with referrer/non-domain cohorts; rollout pinned to 100% and hardcoded out of the hot path (2026-05-12). → [Freemium watermarking](/features/freemium-watermark/) * **Paid access** via Stripe webhook → KV/D1/`customers.json`; entitlements in `runtime` mode (cutover 2026-03-11). → [Paid access](/features/paid-access/) * **Edge watermark router Snippet** — hot-set short-circuit, Borås escape, malformed-path fallback; test/rollback hardening in progress. → [Edge watermark router](/features/edge-watermark-router/) * **P0 watermark-error fix** — transform failures now fail open (HTTP 200) rather than 500 (deployed 2026-05-13). * **Operator dashboard** and **outreach toolkit** (internal). ## In progress [Section titled “In progress”](#in-progress) * **Snippet test & health hardening** — rollback safety + local contract tests done; post-deploy health checks underway. Gates the Snippet cost-reduction work. * **P1 miss-path attribution** — stage-1 counters deployed (2026-05-13); collecting a long-enough sample to rank edge-cache vs. R2 vs. negative-cache misses before optimizing. * **Snippet worker cost reduction** — expand the hot set / reviewed-referrer slice to divert more metered Worker+KV traffic (\~$1.07/M saved per diverted request); blocked on hardening above. * **KV projection cache TTL tuning** — 60s → 300s to cut entitlement-projection reads; pending a freshness decision. * **Non-domain watermark cohorts** — core shipped; known-user list generation + remote overlay publishing remain. * **Developer docs & API DX** — first-wave public docs (paid quickstart, auth patterns, “why still watermarked?”, Next.js image cache caveats) planned in an Astro Starlight `apps/docs` site. ## Staged watermark flip program [Section titled “Staged watermark flip program”](#staged-watermark-flip-program) **Status: Planned — blocked on the G0 gates below.** A 2026-06-14 live re-probe shows watermark delivery is now wm-dominant at the probed colo, with an active reverse leak: no-referrer/direct users get watermarked images from the referrer-blind page-rule cache — the opposite of the developer-friendly policy. The fix is decided: the edge Snippet is the deterministic control point (watermark = 307 redirect to a distinct `wm.jpg` key; the bare URL stays clean), no-referrer resolves to clean, and dev/local traffic is detected by referrer host — never user agent (implementation Planned). Flipping a host adds it to `WM_DOMAINS` + its current ids to HOT, **and removes it from the worker referrer cohort in the same release** — otherwise the worker re-pins the watermark on miss and purges don’t stick. G0 pre-flight gates — all currently open * **C02** — a paying customer with empty entitlements would be watermarked; resolve or waive before any flip. * **Purge tooling** — batch wrapper, fail-closed pre-wave cost gate, and invocations/KV poll must be built (current purge tool is single-URL, uncapped). * **No-referrer → clean policy sign-off** (written) before the first purge. * **HOT-refresh automation** for rotating catalogs (decay to \~0% in \~8 weeks). * **`Cache-Control: no-store` on the 307** before flipping any >1M/mo host. Rollout order, one batch at a time with 24–72h soak and numeric abort criteria: 1. **Canary — H01** (the only host still leaking clean). 2. **Single-host batches**, then the stable bundle **H04/H05/H06/H07**. 3. **H03 isolated** (rotating catalog; decay-monitored). 4. **H09** (plus its subdomain). 5. **H08 last and alone** — highest volume (\~4M/mo), ramped id-by-id, gated on operator visibility. 6. **Tail H10–H14**, also gated on operator visibility. Purges follow the sequencing rule (snippet live → propagated → ids HOT → cohort removed → smoke green → trickle-purge) with per-wave URL caps and hard cost ceilings; rollback is dual-lane (snippet off + cohort backoff + purge) — snippet-disable alone never restores clean. Net program effect is a monthly savings, since redirects skip the worker. Full procedure: [Staged rollouts](/runbooks/staged-rollouts/) and [Cost-safe cache purging](/runbooks/cost-safe-cache-purging/). ## Planned [Section titled “Planned”](#planned) * **Whale-watermark expansion** — open watermark serving for reviewed high-volume hosts; blocked on non-domain-cohort validation. Reverse-leak fix and pricing-funnel wiring are prerequisites. * **Cache purge API** (paid) — force-refresh a thumbnail. → [Cache purge API](/features/cache-purge/) * **Video-metadata queue recovery** — repair the metadata pipeline so the system can learn from new customers without manual support. * **Vimeo request reduction (P2)** — keep cutting the largest upstream-failure bucket. * **Canary host cutover** — move remaining `/6*` routes off Vercel onto Cloudflare (removes Vercel spend; needs a cost estimate). * **Durable auth-email delivery**, **observability capture**, **YouTube legacy path routing (P3)**. * **Animated thumbnails** productization (currently flagged off). → [Animated thumbnails](/features/animated-thumbnails/) * **Shopify app** launch (scaffolded/researched). → [Shopify app](/features/shopify-app/) ## Parked [Section titled “Parked”](#parked) Worthwhile but not the current bottleneck: page-rules centralization, dashboard metric provenance + pie charts, action-tracking substrate, Stripe email automation, DuckDB BI evaluation, function-declaration refactor, security-header audit, hotspot-ranking refactor tranches. ## How priorities are set [Section titled “How priorities are set”](#how-priorities-are-set) Work is scored on a running backlog and re-ranked against measured production signals (error buckets, subrequest breakdowns, cost reports). The bias is: unlock revenue, keep paying customers clean, fail open, and don’t add Cloudflare spend without a written estimate. See [Design principles](/design/principles/). # Research > Dated, source-backed research memos behind the spec's decisions. Research memos are point-in-time evidence — dated snapshots that inform the spec, not the spec itself. | Memo | Summary | | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Workers Cache for thumbnails (2026-07-07)](/research/workers-cache-for-thumbnails-2026-07-07/) | Cloudflare’s new pre-Worker Workers Cache keys on URL, not referrer — can’t front the variant path yet; value is request collapsing + SWR, sequenced after the Snippet URL-encodes the variant. | | [Workers Cache savings estimate (2026-07-07)](/research/workers-cache-savings-estimate-2026-07-07/) | HTMA cost estimate: \~$1–$12/mo (central \~$5), ceiling \~$13/mo, negative if enabled naively — Workers Cache is not a cost play. | | [Watermark delivery flip & leverage ranking (2026-06-14)](/research/watermark-delivery-flip-2026-06-14/) | Live re-probe found delivery flipped watermark-dominant; leverage ranking of flip candidates. | | [No-referrer traffic sources (2026-06-14)](/research/no-referrer-traffic-sources-2026-06-14/) | Taxonomy of missing-referrer traffic; why dev detection uses referrer host, not UA. | # Research: no-referrer traffic sources (2026-06-14) > Taxonomy of missing-referrer thumbnail traffic — mostly machines, not people; dev/local arrives with a localhost referrer. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Date: 2026-06-14 · Our-data analysis complete. The online-corroboration pass is **still open** — no web sources are captured yet. ## Summary [Section titled “Summary”](#summary) No-referrer ≠ real users. Decomposing the no-referrer cohort on the thumbnail image path, the volume is dominated by datacenter renderers spoofing browser UAs (\~4.5M/mo from two fixed IPs, `DC1` and `DC2`) and generic SDK/library default UAs (\~3.6M/mo). Real browsers with a stripped referrer are a **minority (\~1.3M/mo)**. A single CDN-internal IPv6 (`CFINT`, the CDN’s own tiered-cache fabric) shows 153M/mo and is excluded as infra noise. This de-risks defaulting no-referrer requests to the **clean** (unwatermarked) variant — most no-referrer requests are bots/servers/proxies that never render a buy page, so serving them clean costs \~no conversion. It also confirms dev/local traffic is separately identifiable: it mostly arrives *with* a `localhost` referrer, so a referrer-host dev rule is high-precision. The operational guidance that consumes this memo lives in the [no-referrer-traffic](/runbooks/no-referrer-traffic/) runbook; the flip design it feeds is an internal repo doc — not published. ## Method [Section titled “Method”](#method) Parsed a 7-day cohort-candidate pull (2026-05-05→05-12, thumbnail `.jpg` path, referrer = empty string; monthly = window × 4.286). Cross-referenced the `noReferrerUserAgentsOverThreshold`, `xRequestedWithOverThreshold`, `noReferrerIpsOverThreshold`, and `noReferrerIpUserAgentPairsOverThreshold` query sets. For the entity decode behind `DC1`/`DC2`/`CFINT`, see the internal decoder (not published). ## Ranked source taxonomy [Section titled “Ranked source taxonomy”](#ranked-source-taxonomy) Ranked by estimated monthly requests, with an eyeball classification: | # | Source bucket | \~req/mo | Eyeballs? | Evidence in our data | | - | ----------------------------------------------------------- | -------: | --------- | ---------------------------------------------------------------------------------------------------------------- | | 1 | **Datacenter / server-side renderer spoofing a browser UA** | \~4.5M | No | `Firefox/150.0` (Win) = 4.52M, of which `DC1` (2.68M) + `DC2` (1.82M) — two fixed datacenter IPs, same UA | | 2 | **Generic SDK / HTTP-library default UA** | \~3.6M | No | bare `Mozilla/5.0` 3.32M, `node` 0.07M, `okhttp/4.9.2` 0.07M, `Dart/3.9–3.11 (dart:io)` \~0.19M | | 3 | **Real browsers, referrer stripped** | \~1.3M | **Yes** | iPhone Safari 0.60M, Mac Safari 0.35M, Win Chrome 147 0.26M, Android Chrome 0.05M | | 4 | **Native mobile apps (iOS/Android)** | \~0.5M | partial | iOS `CFNetwork/Darwin` UAs: a branded installer app 0.19M, a basketball training app 0.16M; Android via `okhttp` | | 5 | **Search / social crawlers** | \~0.4M | No | `Googlebot-Image` 0.14M, `facebookexternalhit` 0.11M, `meta-externalads` 0.08M, `bingbot` 0.06M | | 6 | **Email / document image proxies** | \~0.17M | indirect | `...GoogleImageProxy` (Gmail) 0.09M, `ms-office; MSOffice 16` (Outlook/Office) 0.08M | | 7 | **Monitoring / uptime bots** | \~0.13M | No | `LogicMonitor SiteMonitor/1.0` 0.07M, another self-identifying monitoring bot 0.06M | | — | **CDN-internal tiered-cache (`CFINT`) — excluded** | 153M | n/a | single CDN-range IPv6, flagged `likelyInternalOrInfra`; the cache talking to itself, not traffic | ## The three questions, answered [Section titled “The three questions, answered”](#the-three-questions-answered) * **Is it mostly servers?** Yes — buckets 1, 2, 5, 7 (servers/libraries/bots) are the clear majority of legitimate no-referrer volume; real browsers are \~1.3M of \~11M over-threshold (after excluding `CFINT`). * **Is it clearly not an iPhone?** Mostly not an iPhone — the largest chunks are Windows/datacenter UAs and bare library UAs. Genuine iPhone Safari no-referrer exists (\~0.6M) but is a minority. * **Do apps mostly send referrers?** Yes — Android in-app webviews are NOT the no-referrer problem. `X-Requested-With` apps (facebook 1.75M, instagram 0.68M, etc.) mostly **do** send a referrer — their no-referrer subset is \~0. The lone exception is an education platform app (0.08M, 100% no-referrer), a native app that strips it. ## Conclusion: detect dev by referrer host, not UA [Section titled “Conclusion: detect dev by referrer host, not UA”](#conclusion-detect-dev-by-referrer-host-not-ua) Dev/local traffic is referrer-**bearing**, not no-referrer. Local dev mostly arrives with a `localhost` (\~220k/mo) or `127.0.0.1` (\~69k/mo) **referrer** — seen in the referrer-host candidates, not the no-referrer set. A referrer-host rule therefore catches dev precisely without touching the no-referrer bucket, while UA-based dev detection collides head-on with bucket 2 (server/library default UAs). Current direction: * Treat no-referrer as **safe to serve clean by default** for conversion purposes (the eyeball minority is small; the bulk is non-buyers/bots). * Detect **dev/local via referrer host**, not user-agent. Where no-referrer is handled today: `apps/snippet/watermark-router.snippet.js` (`shouldWatermarkRequest` returns false on missing `Referer`). Paying-customer identifiers live in `apps/thumbnail/lib/customers.json` and are referenced, not copied. Open: online corroboration The web-research pass has not run. Candidate topics: default `Referrer-Policy: strict-origin-when-cross-origin` behavior, HTTPS→HTTP downgrade referrer loss, native-app referrer stripping, Gmail/Office image-proxy fetch patterns. Also open: an ASN/owner lookup on `DC1`/`DC2` (one embedder’s server-side renderer vs. a scraper), and a fresh data pull — these figures are the 2026-05-12 window. ## Related [Section titled “Related”](#related) * [No-referrer traffic](/runbooks/no-referrer-traffic/) — the runbook that operationalizes this taxonomy * [Watermark delivery flip & leverage ranking (2026-06-14)](/research/watermark-delivery-flip-2026-06-14/) — the host ranking this informs * Source data and the flip design plan are internal repo docs — not published # Research: watermark delivery flip & leverage ranking (2026-06-14) > Live re-probe shows watermark delivery flipped wm-dominant and a reverse leak went live; leverage ranking of flip hosts by buyer-fit × operator-visibility. Date: 2026-06-14 · Owner: thumbnail-worker · Status: Research memo (point-in-time) Host identities in this memo use the code-name scheme (H01–H14 flip hosts, H20–H24 cost-control hosts, S01–S04 self-referred institutions, K01 gray-market network). Decodes live in the internal decoder (not published). ## TL;DR [Section titled “TL;DR”](#tldr) * The “we’re not watermarking the buyers” problem from the 06-12 audit (internal repo doc — not published) mostly fixed itself — and overshot. Re-probing production two days later shows the cohort hosts, including **H22** (the audit’s flagship “pinned clean for 29 days” example), now serve **watermarked** bytes on long-aged, uncontaminated edge pins at the probed colo. * Across 126 probes: **110 `wm`, 16 `clean`**. A purge/rollout around the 06-12–13 snippet hardening appears to have flipped the per-URL cache lottery, and because the worker watermarks every cohort referrer on a miss, re-pinning resolved toward `wm`. * The same flip introduced a **reverse leak**: no-referrer/direct users now receive `wm` from edge cache (confirmed on H01, H08, H20, H22) — a violation of the developer-friendly delivery policy in [referrer-cohort-and-delivery-policy](/runbooks/referrer-cohort-and-delivery-policy/). * The binding constraint is therefore no longer delivery. It is the **funnel** (the now-visible mark still routes nobody to the pricing page) and the **reverse leak** (over-watermarking ambiguous traffic). * The cost worry (“invalidating cache is expensive”) is largely moot: the blended per-request rate only applies to *blanket page-rule removal*. A targeted purge re-pins `wm` and stays edge-cached (≈$0 ongoing); the snippet redirect path is **cheaper** than the worker. * So leverage ranking collapses to one axis — **buyer-fit × operator-visibility** — because the cheapest hosts to flip are also the best conversion targets, and the expensive ones (H20, H21, H08) are poor conversion targets anyway. ## Method [Section titled “Method”](#method) * **Cost rates** from the snippet/worker cost-reduction plan (internal repo doc — not published): worker `$0.504/1M`, KV `$0.503/1M`, `1.13` KV reads/request, blended `≈$1.07/1M` skipped-or-added worker requests. These are illustrative per-million unit rates, not bill totals. * **Volumes** from the 2026-05-12 cost-candidates pull and per-host top-video-id extract (internal repo docs — not published). * **Cohort membership** from `apps/thumbnail/lib/referrer-rollout-cohort.ts` (266 domains; the worker watermarks all of these on a cache miss). * **Live probes**, 2026-06-14: bare-URL `GET https://vumbnail.com/{id}.jpg` with the candidate host as `Referer`, reading `cf-cache-status`, `age`, and `x-thumbnail-variant`. Self-poisoning: only trust aged reads Only reads with `age > 3600s` are trusted. Anything younger may be **self-poisoned by an earlier probe**: the worker normalizes its cache key, and a single probe carrying a cohort referrer pins `wm` on every miss — so a fresh read can be measuring the probe, not production. Mechanics in [watermark-system](/runbooks/watermark-system/). All probes egress a single colo — see [Caveats](#caveats). ## Findings [Section titled “Findings”](#findings) ### Delivery flipped watermark-dominant [Section titled “Delivery flipped watermark-dominant”](#delivery-flipped-watermark-dominant) Uncontaminated pins (`age > 1h`, never touched this session), by host: | Host | wm | clean | Read | | --------------------------------- | ----: | ----: | ---------------------------------------------------------------- | | H22 | 4 | 0 | **flipped** — audit found its top id clean at 29d | | H21 | 7 | 0 | wm | | H07 | 9 | 0 | wm (oldest pin 5.2d) | | H04 | 9 | 1 | wm (oldest pin 3.4d) | | H06 | 4 | 0 | wm | | H03 | 3 | 0 | wm | | H20 | 3 | 0 | wm | | H08 | 2 | 0 | wm | | H01 | 3 | 4 | **mixed — still leaking clean** | | H24 | 1 | 1 | mixed | | H02 | 1 | 1 | mixed (its hero id is in the snippet HOT set → deterministic wm) | | H09 / H05 | 1 / 1 | 0 | wm | | one further cohort host (uncoded) | 3 | 0 | wm | Across 126 probes: **110 `wm`, 16 `clean`**. The set of “hosts we’re not watermarking” is now **smaller and colo-dependent**, not the broad clean-pinned commercial cohort the 06-12 audit described. The genuinely-still-mixed buyer-fit host at this colo is **H01**. ### The reverse leak is live [Section titled “The reverse leak is live”](#the-reverse-leak-is-live) Policy violation: direct users are being watermarked Re-requesting the same URLs with **no referrer** returned `wm` from edge cache on one long-aged pinned id each at H01 (age 17.4h), H08, H20, and H22 (age 14h). Direct/no-referrer users — the population the developer-friendly policy explicitly protects — are now being watermarked. Fixing or purging the wm-pinned entries serving ambiguous traffic is the highest-priority correctness issue the flip introduced. Policy: [referrer-cohort-and-delivery-policy](/runbooks/referrer-cohort-and-delivery-policy/). ## Ranking [Section titled “Ranking”](#ranking) Leverage = **buyer-fit** (commercial, brand-conscious, has budget) × **operator-visibility** (the mark lands on a page the operator/marketing team actually looks at — not buried in an app/player/LMS) × **mark-legibility**, divided by cache-invalidation cost. Since the recommended flip mechanisms cost ≈$0 or less (see [Cost framing](#cost-framing-corrected)), the denominator drops out and the ranking is buyer-fit × operator-visibility. Targets must also be cheap to keep deterministic: stable catalog ⇒ low snippet HOT churn. Full method: [monetization-leverage-method](/runbooks/monetization-leverage-method/). ### Tier 1 — Convert (high buyer-fit, plausibly operator-visible, cheap to flip) [Section titled “Tier 1 — Convert (high buyer-fit, plausibly operator-visible, cheap to flip)”](#tier-1--convert-high-buyer-fit-plausibly-operator-visible-cheap-to-flip) | # | Host | Volume | Why | Delivery now (probed colo) | | -- | ----------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- | | 1 | **H01** | low | Brand-conscious consumer commerce; stable product-video catalog | **mixed (still leaking clean)** — best live target | | 2 | **H02** | low-mid | Commerce; single dominant hero video already in the HOT set — nearly deterministic already | mixed | | 3 | **H03** | low | Commerce with staff who browse the catalog pages the embeds live on; catalog rotates | wm | | 4 | **H04** | low | Commerce; **confirmed operator-visible** (embeds referenced in landing-page static HTML); stable catalog | wm | | 5 | **H05** | low | Design-goods commerce; brand-conscious; stable catalog | wm | | 6 | **H06** | low | Public company; marketing-site embeds; stable | wm | | 7 | **H07** | low | Commerce; stable YouTube-style ids; already wm-pinned | wm | | 8 | **H08** | **high** (largest flip host) | Global brand, high budget — but high volume + likely product-page embeds; verify operator-visibility before spending the purge | wm | | 9 | **H09** | mid | Commerce, seasonal catalog; both subdomains are separate cohort entries | wm | | 10 | **H10–H14** | tail (near-free each) | Good buyer-fit, tiny volume — verify operator-visibility | varies | ### Tier 2 — Do NOT chase for conversion; cost control only [Section titled “Tier 2 — Do NOT chase for conversion; cost control only”](#tier-2--do-not-chase-for-conversion-cost-control-only) | Host | Why not a conversion target | Action | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | | **H20** | High volume; audit confirmed embeds live in the player/app — **the operator never sees the mark**; ids rotate (expensive to keep in the snippet HOT set) | Snippet-redirect to cut worker/KV spend | | **H21** | High volume; back-office surface, operator-invisible; heavy malformed-path (`/.jpg`) traffic | Snippet-redirect / invalid-path short-circuit | | **H23, H24, and a creator-tipping platform (uncoded)** | App/creator embeds — the mark shows to end-users, not the buyer | Cost control only | | **S01–S04, H22, K01** | Viewers ≠ buyers (the audit’s audience-mismatch thesis: institutions, nonprofits, gray-market networks) | Snippet-redirect (cost); already mostly wm | ## Cost framing (corrected) [Section titled “Cost framing (corrected)”](#cost-framing-corrected) Three mechanisms for changing which variant a URL serves, three cost signs: | Mechanism | Effect | Cost | | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | **Targeted purge** of clean-pinned URLs | next request re-pins `wm`, stays edge-cached for the 31-day page-rule TTL | one-time miss only ≈ **$0 ongoing** | | **Snippet redirect** (add host to `WM_DOMAINS` + current HOT ids in `apps/snippet/watermark-router.snippet.js`) | 307 → prebuilt R2 `wm.jpg`, runs **before** cache, **skips the worker** | **negative** — saves ≈$1.07/1M blended | | **Blanket page-rule removal** | every request hits the worker every time | **≈$1.07/1M ongoing** at full program volume — the only expensive path; do not | Any naive “worst case” per-host dollar figure is volume × the blended rate under blanket removal — an anchor, not a plan. The recommended mechanisms cost ≈$0 or save money, so **cost does not gate the conversion targets**. Details: [cost-safe-cache-purging](/runbooks/cost-safe-cache-purging/). ## Recommendation [Section titled “Recommendation”](#recommendation) 1. **Re-confirm per-host delivery before any purge** — the lottery is colo-local. H01 is the only Tier-1 host still visibly leaking clean at the probed colo; most others already serve `wm`, so the action there is funnel + legibility, not delivery. 2. **Make Tier 1 deterministic via the snippet** (add to `WM_DOMAINS` + current HOT ids), not via page-rule changes. Cheaper than the status quo and removes the lottery. Rollout mechanics: [staged-rollouts](/runbooks/staged-rollouts/). 3. **Fix the reverse leak first or in parallel** — the flip’s policy violation outranks its conversion upside. Scope or purge the no-referrer-pinned entries. 4. **Wire the funnel** — every watermark now visible is wasted until the mark points at `/pricing` and `/pricing` is crawlable. This gates conversion for *all* tiers and outranks adding more watermarked hosts. 5. **Verify operator-visibility for Tier 1** by rendering each host’s key pages in a real JS browser and confirming where the thumbnail lands (homepage/PDP/blog = good; player/app = demote). ## Caveats [Section titled “Caveats”](#caveats) * **Single vantage.** All probes egress one colo. Pin state is per-colo × URL × month; global coverage is a patchwork. The wm-dominance finding is directional, not a census. * **Warm-URL bias.** Trusted (`age > 1h`) reads over-sample high-traffic URLs, which are likelier to have been pinned by a cohort referrer. The long tail may hold more `clean` than this sample implies. * **Stale volumes.** Volumes are the 2026-05-12 pull; two hosts from the March cohort fell below the May threshold — re-pull before acting on them. ## Sources [Section titled “Sources”](#sources) * Watermark monetization audit, 2026-06-12 (internal repo doc — not published) * `apps/thumbnail/lib/referrer-rollout-cohort.ts`, `apps/snippet/watermark-router.snippet.js`, `apps/thumbnail/lib/customers.json` * Snippet/worker cost-reduction plan and 2026-05-12 volume artifacts (internal repo docs — not published) * The 2022 31-day edge-TTL `*.jpg*` page rule and cache baseline (internal repo artifacts — not published); mechanics in [watermark-system](/runbooks/watermark-system/) * Live production probes, 2026-06-14 (this memo) # Research: Workers Cache for thumbnails (2026-07-07) > Cloudflare's new Workers Cache serves a response before the Worker runs and keys on URL, not referrer — so it can't front the variant path yet; its value is request collapsing + stale-while-revalidate, sequenced after the Snippet encodes the variant in the URL. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Date: 2026-07-07 · Primary Cloudflare docs + the launch blog validated against the current Worker. No code changed. ## Summary [Section titled “Summary”](#summary) Cloudflare shipped a **new Workers Cache** on 2026-07-06 — distinct from the Cache API (`caches.default`) the Worker already uses as [edge-cache layer 1](/features/caching-and-delivery/). It’s a one-line wrangler flag (`"cache": { "enabled": true }`) that serves a cached response **before the Worker runs**, keyed on **URL path+query — not on `Referer`**. For the thumbnail Worker, whose job is deciding clean-vs-watermark **from the referrer at request time**, enabling it globally on the image path would **recreate the referrer-blind cache lottery** the product already moved off of (the first requester’s variant pins per-URL-per-colo — the exact failure the [delivery-flip research](/research/watermark-delivery-flip-2026-06-14/) and the [Snippet-as-control-point decision](/features/edge-watermark-router/) resolved). The genuinely valuable part is its **request collapsing** (single-flight) and **stale-while-revalidate**, which are free on any plan and answer the open miss-coalescing question in [Caching & delivery](/features/caching-and-delivery/). The clean move: **do not enable it globally now; sequence it after the Snippet encodes the variant in a distinct URL.** Once the URL fully determines the variant, caching-before-the-Worker stops being a lottery and collapsing + SWR + zero-CPU hits become upside. ## What the evidence says [Section titled “What the evidence says”](#what-the-evidence-says) ### The Worker already uses a Workers cache — just not this one (confirmed) [Section titled “The Worker already uses a Workers cache — just not this one (confirmed)”](#the-worker-already-uses-a-workers-cache--just-not-this-one-confirmed) Layer 1 today is the **Cache API** (`caches.default`) with a variant-aware key: the watermarked variant appends an internal `?_wm=1`. This is the standard-plan way to get custom keys without the Enterprise `cf.cacheKey`. The new feature is a different mechanism at a different position (in front of the Worker, not called from inside it). ### The new Workers Cache is “cache in front of the Worker” (confirmed — Cloudflare docs + blog, 2026-07-06) [Section titled “The new Workers Cache is “cache in front of the Worker” (confirmed — Cloudflare docs + blog, 2026-07-06)”](#the-new-workers-cache-is-cache-in-front-of-the-worker-confirmed--cloudflare-docs--blog-2026-07-06) * Enable: `"cache": { "enabled": true }` + a current `compatibility_date`; **any plan**, available now. * **Hit → the Worker does not run.** Cached response served directly, **zero CPU billed**. Miss → the Worker runs; the response is stored if cacheable per its `Cache-Control`. * **Request collapsing** (single-flight): many concurrent requests for the same cache key run the Worker **once**, per key **per data center**; streaming responses join the in-flight stream. Cloudflare calls this the most significant difference from the Cache API, which does **not** collapse concurrent requests. * **Stale-while-revalidate** via `Cache-Control` (`max-age=…, stale-while-revalidate=…`): serve stale instantly, refresh in the background. * **Only GET/HEAD** are cached (they share one entry); other methods always invoke the Worker. * **Cache key** = path + query (order and trailing slash matter) + target entrypoint + a few anti-poisoning headers + `Cloudflare-Workers-Version-Key` + `ctx.props` (service-binding/RPC). **Excluded:** HTTP method, host, body, and standard headers — **including `Cookie`, `Authorization`, and `Referer`.** * **Version-scoped by default:** each deployed version has its own cache; share across versions with `"cache": { "cross_version_cache": true }`. * **Variance** is via the response `Vary` header (a separate variant per exact header value; `Vary: *` disables caching). Off-Enterprise custom keys now exist via `cf.cacheKey` **when invoked through `ctx.exports`**, and caching can be toggled **per entrypoint**. ### Why this collides with the watermark model (confirmed mechanism) [Section titled “Why this collides with the watermark model (confirmed mechanism)”](#why-this-collides-with-the-watermark-model-confirmed-mechanism) The Worker serves **different bytes for the same URL** depending on the requester’s referrer (cohort → watermark, else clean). Workers Cache keys on URL and **the Worker doesn’t run on a hit**, so it can’t make that per-request decision — the first variant generated for a URL is served to everyone at that colo until it expires. `Vary: Referer` can’t rescue it (inference, well-grounded): `Referer` is per-source-URL high-cardinality, so it explodes into near-infinite variants with \~0% hit rate, and it’s exact-value — not host- or cohort-aware, which is what the clean/watermark split needs. This matches the standing finding that referrer-keyed caching needs a control point that runs **before** any Cloudflare cache — which is the Snippet. ### It answers an open problem we already have (confirmed the gap; inference the fit) [Section titled “It answers an open problem we already have (confirmed the gap; inference the fit)”](#it-answers-an-open-problem-we-already-have-confirmed-the-gap-inference-the-fit) [Caching & delivery](/features/caching-and-delivery/) leaves miss-path coalescing / single-flight open, and it’s flagged as possibly needing a new paid primitive. Workers Cache’s **request collapsing gives exactly that, free, on any plan.** On a cold or freshly-purged URL today, N concurrent misses each fetch the source, run the watermark transform, and write R2 — duplicate expensive work that collapsing would dedupe to one. ### Billing changes — the spend rule applies (confirmed rule + mechanism; numbers TBD) [Section titled “Billing changes — the spend rule applies (confirmed rule + mechanism; numbers TBD)”](#billing-changes--the-spend-rule-applies-confirmed-rule--mechanism-numbers-tbd) With caching enabled, **every** request bills at the standard Workers **request** rate whether served from cache or the Worker, but **CPU is billed only on miss/bypass**. Net direction is numbers-dependent: watermark renders are CPU-heavy, so zero-CPU hits push cost **down**; but any hits currently served for free by the edge before the Worker would convert into **billed** Worker requests, pushing cost **up**. Per the standing rule, a change likely to move Cloudflare spend ≥ $1/month needs a **written estimate first** — required before rollout. ## What works (scoped adoption candidates) [Section titled “What works (scoped adoption candidates)”](#what-works-scoped-adoption-candidates) 1. **Shared fallback graphics** (`_shared/error.jpg`, `_shared/locked.jpg`): one object serves many IDs and is variant-stable per URL → safe to front; and collapsing helps during error storms. 2. **A fixed-variant endpoint** (a test route, or a future always-clean / always-watermark path): URL determines variant → safe, and gains collapsing + SWR + zero-CPU hits. 3. **Post-Snippet main path:** once the Snippet encodes the watermarked variant as a distinct URL and the bare URL is always clean ([Edge watermark router](/features/edge-watermark-router/)), variant ∈ URL → Workers Cache becomes safe *and* high-value everywhere, and can largely **replace** the hand-rolled `caches.default` layer. 4. **SWR for negative/fallback refresh** — overlaps the existing R2 soft/hard-TTL negative cache, so treat as a possible simplification, not an addition. ## What to avoid [Section titled “What to avoid”](#what-to-avoid) * **Do not** set `"cache": { "enabled": true }` globally while the referrer-based clean/watermark decision still lives inside the Worker and the variant is not in the URL — it reintroduces the cache lottery and undoes deterministic delivery. * **Do not** try to preserve variance with `Vary: Referer` — cardinality blowup, \~0% hit rate, not cohort-aware. * **Do not** stack Workers Cache on top of an unresolved referrer-blind edge cache rule; resolve that first. * **Do not** ignore version-scoping: gradual rollouts start cold per version unless `cross_version_cache` is set — decide deliberately. * **Do not** roll out without the written cost estimate. ## Recommendation [Section titled “Recommendation”](#recommendation) 1. **Now:** don’t enable globally; record this reasoning. 2. **Cheap win, low risk:** pilot Workers Cache on a **fixed-variant surface** (shared fallback graphics and/or a test route) to bank request-collapsing + SWR and measure the billing delta on a small slice — which produces the required cost note. 3. **Sequenced main-path adoption:** fold Workers Cache into the Snippet URL-encoded-variant rollout; then enable it on the image path, set `cross_version_cache` as appropriate, and retire the manual `caches.default` layer for that path. 4. **Keep the Snippet as the variant control point.** Workers Cache complements it; it does not replace referrer determinism. ## Open questions [Section titled “Open questions”](#open-questions) * Exact billing delta for this traffic mix (needs the current request vs edge-hit vs CPU-render split) — gates rollout. * Does resolving the referrer-blind edge rule plus Workers Cache double-count or simplify the edge layer? Confirm before enabling both. * Should the negative-cache/fallback path move to SWR semantics, or stay on its bespoke R2 soft/hard TTLs? ## Sources [Section titled “Sources”](#sources) Primary (Cloudflare): * Workers Cache overview (new): * Workers Cache — cache keys: * Launch blog, “Your Worker can now have its own cache in front of it” (2026-07-06): * Cache API runtime reference (the API the Worker already uses): * How the cache works: Internal (not published): the Worker cache implementation (`apps/thumbnail/lib/cache.ts`), the miss-path plan, and the Snippet variant-flip plan are repo docs — pointed to, not copied. # Research: Workers Cache savings estimate (2026-07-07) > How much Workers Cache saves the thumbnail worker: ~$1–$12/mo (central ~$5), ceiling ~$13/mo, and negative if enabled naively — not a cost play. Unlisted internal dev doc — audit before treating as public Internal engineering note with **infrastructure cost figures**, published unlisted for review only. Not audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Authoritative copy is the repo The tracked source of truth for this cost estimate is the repo file `docs/workers-cache-savings-estimate-2026-07-07.md` — it’s the git-tracked artifact the purge cost-gate and the repo’s “spend ≥ $1/mo needs a written estimate in `docs/`” rule depend on. This page is the published/reviewable copy; edit the repo file and redeploy. Date: 2026-07-07 · HTMA calibrated measurement brief. No code changed. ## Summary [Section titled “Summary”](#summary) Adopting Cloudflare’s new [Workers Cache](/research/workers-cache-for-thumbnails-2026-07-07/) on the thumbnail worker saves an estimated **\~$1–$12/month (central \~$5/mo)** in the realistic scoped, post-Snippet-URL-encoding case, with a hard ceiling near **\~$13/mo**. Near-term (fixed-variant surfaces only) it’s **<$2/mo**. Enabled naively/globally it **increases** cost by **$5–$15+/mo** (it converts page-rule-served “free” hits into billed requests) and re-creates the referrer-blind variant lottery. **This is not a cost play** — adopt Workers Cache for request-collapsing/latency/resilience, not to save money. ## Question made precise [Section titled “Question made precise”](#question-made-precise) * **Decision:** adopt Workers Cache for cost savings, and if so how (scoped vs global, before vs after the Snippet encodes the variant in the URL)? * **Quantity:** net recurring Cloudflare spend change (positive = savings). **Unit:** USD/mo. * **Horizon:** steady-state monthly at current traffic. * **Threshold:** repo rule flags any change ≥ ±$1/mo (this memo satisfies it); the “worth-doing-alone” bar is \~$10/mo. ## Why the saving is small (the mechanism) [Section titled “Why the saving is small (the mechanism)”](#why-the-saving-is-small-the-mechanism) A Workers-Cache **hit skips the worker**, which removes the worker’s **KV read** (~~$0.50/1M) and any worker-side R2 read — but the **request line still bills** (~~$0.30/1M) and **CPU is already free** (worker averages 0.58 ms/req ⇒ \~17M CPU-ms/mo, under the 30M free tier). So the only real saving is eliminated KV reads, capped by the whole KV bill. Confirmed anchors (rates are Cloudflare public pricing; the blended model matches the [cost-safe cache-purging runbook](/runbooks/cost-safe-cache-purging/)): * Worker-path blended \~**$1.07/1M** = \~$0.504/1M request + 1.13 × \~$0.50/1M KV. * R2 Class B GET \~**$0.36/1M**. * Current KV read bill \~**$12.28/mo** (\~34.6M reads/mo after 10M free). * Derived: worker runs **\~30M×/mo** (34.6M KV ÷ 1.13). CPU ≈ **$0** (under free tier). ## Decomposition — realistic scoped saving (post-URL-variant, \~40–70% capture) [Section titled “Decomposition — realistic scoped saving (post-URL-variant, \~40–70% capture)”](#decomposition--realistic-scoped-saving-post-url-variant-4070-capture) | Component | Low | Central | High | Basis | | ---------------------------------- | ---------: | -------: | --------: | --------------------------------------- | | Eliminated KV reads | $2 | $5 | $9 | 30M inv × 1.13 KV × $0.50/1M × capture% | | Eliminated worker R2 Layer-2 reads | $0.5 | $1.5 | $3 | subset of R2 GET × $0.36/1M | | Eliminated CPU | $0 | $0 | $0.5 | already under free tier | | Request-line change (scoped) | $0 | $0 | $0 | hits still bill the request | | **Net (scoped, post-URL-variant)** | **\~$2.5** | **\~$6** | **\~$12** | sum | * **Ceiling (100% capture, unrealistic):** \~$13–15/mo. * **Near-term (fixed-variant surfaces only):** \~$0.3–$2/mo (central \~$1) — those surfaces are already cheap. * **Downside (naive/global enable):** converting even 20% of \~150M/mo page-rule-served requests to billed requests ≈ **+$9/mo**; range **−$5 to −$15+/mo**. ## Calibrated range [Section titled “Calibrated range”](#calibrated-range) 90% CI for the realistic scoped saving is **$1–$12/mo, central \~$5**. The swing is dominated by an unmeasured **capture share**, not by the (known) unit rates. >90% confident the saving is **< \~$13/mo** (structural ceiling) and **> $0 only if scoped correctly** — a naive enable can be negative. ## Value of information [Section titled “Value of information”](#value-of-information) 1. **Capture share + page-rule billing interaction** (biggest swing) → resolve with a **1-week Workers Cache pilot on a fixed-variant surface**, tracking `Cf-Cache-Status` hit rate and the KV-read / invocation / billed-request deltas. This settles the whole question. 2. Actual worker-invocation metric (tightens the \~30M/mo derivation ±30%). 3. Skip external pricing lookups — the rates are known and not the bottleneck (VoI ≈ 0). **Stop rule:** if the pilot shows a scoped saving under \~$3/mo (likely), the cost case is settled — adopt for non-cost reasons or not at all. ## Recommendation [Section titled “Recommendation”](#recommendation) **Do not adopt Workers Cache to save money.** Central \~$5/mo (CI $1–$12) is below the \~$10/mo worth-it-alone bar, CPU savings are \~$0, and a naive global enable can raise the bill while breaking deterministic delivery. Instead: 1. For the cheap KV win now: ship the already-planned entitlement **`cacheTtl 60→300s`** tuning — same headroom, no new feature. 2. Adopt Workers Cache **after** the Snippet encodes the variant in the URL, for request-collapsing/latency/resilience; take the \~$5/mo as a byproduct. 3. Confirm sign and size with the 1-week fixed-variant pilot before any enable. ## Sources [Section titled “Sources”](#sources) * Feature behavior & billing model: [Workers Cache research](/research/workers-cache-for-thumbnails-2026-07-07/) * Blended cost model / unit rates: [cost-safe cache-purging runbook](/runbooks/cost-safe-cache-purging/) * Cloudflare Workers pricing (public): * KV volume/bill + `cacheTtl` tuning, R2 GET volume, and worker avg CPU are internal repo artifacts — pointed to in the tracked repo copy, not reproduced here. ```json { "quantity": "recurring monthly Cloudflare cost saving from adopting Workers Cache (scoped, post-URL-variant)", "unit": "USD/month", "low_90": 1, "central": 5, "high_90": 12, "confidence": "90%", "decision_threshold": 10, "threshold_implication": "Below the ~$10/mo worth-it-alone bar (ceiling ~$13/mo); naive global enable is negative. Not a cost play — adopt post-URL-variant for collapsing/latency; ship cacheTtl 60->300s for the cheap KV win.", "top_uncertainty_driver": "capture share of worker invocations converted to Workers-Cache hits, plus whether enabling converts page-rule-served free hits into billed requests", "estimate_status": "estimated", "next_measurement_step": "1-week Workers Cache pilot on a fixed-variant surface measuring Cf-Cache-Status hit rate and KV-read / invocation / billed-request deltas" } ``` # Runbooks (internal, unlisted) > Reusable operational runbooks distilled from real rollout work — unlisted internal dev docs pending audit. Unlisted internal dev docs — audit before treating as public These runbooks are **published unlisted for review only**. They are internal engineering notes, **not yet audited for public release**. They are not linked from public navigation and must not be cited, indexed, or treated as public-facing product docs until a human review signs off. Concrete site/customer data is deliberately kept out of these files and lives only in a local-only decoder (the internal decoder — not published, gitignored — never deployed). Reusable, abstracted runbooks distilled from real operational work. No customer, pricing, or vendor specifics — concrete targets are referenced only via the local-only decoder. ## Index [Section titled “Index”](#index) | Runbook | What it covers | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | [Watermark system](/runbooks/watermark-system/) | Edge-cache variant delivery, the referrer-blind page-rule “lottery”, probe methodology | | [Referrer cohort & delivery policy](/runbooks/referrer-cohort-and-delivery-policy/) | How a referrer cohort drives the variant decision; no-referrer/dev defaults; reverse-leak policy | | [Safe deploys](/runbooks/safe-deploys/) | Pre-flight gates, atomic-vs-non-atomic deploys, post-deploy verification | | [Staged rollouts](/runbooks/staged-rollouts/) | Blast-radius ordering, canary → batches, soak windows, advance gates | | [Reverting deploys](/runbooks/reverting-deploys/) | Dual-lane rollback, kill switches, residual-risk cleanup | | [Cost-safe cache purging](/runbooks/cost-safe-cache-purging/) | Trickle-purge, pre-wave cost gate, the cache-busting permanent-cost trap, circuit-breakers | | [HOT-set management](/runbooks/hot-set-management/) | Redirect-set freshness, decay cadence, size budgets | | [No-referrer traffic](/runbooks/no-referrer-traffic/) | Classifying missing-referrer traffic (servers vs eyeballs); dev/local detection | | [Monetization leverage method](/runbooks/monetization-leverage-method/) | Ranking which embedding hosts to watermark for conversion vs cost control | | [Internal docs publishing](/runbooks/internal-docs-publishing/) | Keeping internal trees out of public builds | > 🔒 The local-only decoder (the internal decoder — not published) maps code names to real sites/customers and is **never** committed or deployed. Keep it that way. # Cost-safe cache purging > Trickle purges, pre-wave cost gates, and the cache-busting permanent-cost trap. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: invalidate (purge) edge cache for a high-volume CDN zone without triggering a surprise compute bill, by sequencing the purge last and capping its rate. Applies to: any zone fronted by an **edge snippet / router** that 307-redirects a HOT id set to cheap object-storage (R2), backed by a **worker** miss path, behind a **referrer-blind page-rule cache** (e.g. a 31-day `*.jpg*` `edge_cache_ttl`). Read [reverting-deploys](/runbooks/reverting-deploys/) and [snippet-rollout-sequencing](/runbooks/staged-rollouts/) alongside this. For any concrete host/customer/id, see the internal decoder (not published) — this doc is deliberately entity-free. *** ## TL;DR [Section titled “TL;DR”](#tldr) * A purge forces a **re-warm**. Re-warm is cheap (object-GET path) **only if** the id is in the snippet’s redirect set AND the live snippet is serving the 307. Otherwise re-warm lands on the **worker path** and costs per request. * **One vector is unbounded:** cache-busting unique-query URLs (`/{id}.jpg?v=NNN`) miss the edge forever and hit the worker on every request, permanently. A single host with any cache-busting can cross a $1/mo line and never decay. **Measure it first or do not purge.** * The **caps + trickle are the primary safety**, not monitoring: the at-risk metric (worker invocations + KV reads) lags hours-to-a-day, and there is **no hard CDN spend cap** on paid plans. * **Never disable the snippet as a kill action** — with the cohort populated, disabling the snippet routes the full 307 volume back into the worker and *increases* cost. Kill = stop the purge loop, then cohort-backoff. *** ## 1. Cost model: why a purge can cost money [Section titled “1. Cost model: why a purge can cost money”](#1-cost-model-why-a-purge-can-cost-money) A purge evicts an edge key. The next request re-warms it. Where it lands decides the cost: | Re-warm lands on | When | Unit cost (illustrative blended) | | ----------------------------------- | --------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Object-storage GET (R2 Class B) | id is in the **live deployed** redirect set AND snippet 307 is serving | \~`$0.36 / 1M` | | Worker path (invocation + KV reads) | id is NOT in the live redirect set, or snippet isn’t live/propagated, or cohort still re-pins | \~`$1.07 / 1M` steady-state; use \~`$1.26 / 1M` for cold-isolate bursts (KV ratio \~1.5, not steady \~1.13) | Two consequences: 1. **HOT classification must come from the LIVE deployed snippet**, never a plan/manifest file. A manifest-HOT / snippet-non-HOT divergence purges an id you *believe* is the cheap path straight into the worker. Assert `manifest-HOT ⊆ deployed-snippet-HOT` before any wave; abort on any id HOT-in-plan but absent from the live Set. 2. A bare single-URL purge (`{files:[url]}`) cannot evict a **cache-busting** key (`?v=NNN`) — each distinct query is a separate edge key — so cache-busting re-warm is not a one-time cost, it is **permanent**. *** ## 2. The sequencing rule (load-bearing order) [Section titled “2. The sequencing rule (load-bearing order)”](#2-the-sequencing-rule-load-bearing-order) Re-warm must land on R2 (HOT) or a cohort-removed worker → clean — **never** the watermarking worker re-pinning under the referrer-blind page rule. The purge wrapper **refuses to run** until this exact order is green, per host: ```plaintext snippet-live → propagated (multi-colo quorum) → ids in redirect set → cohort-removed → smoke-green → THEN trickle-purge ``` 1. **snippet-live** — `deployedContentSha256 == local sha` AND the rule is enabled with the expected expression. Deploy is non-atomic (content PUT, then rule PUT, no retry) — verify the **live artifact**, not just an API 200. 2. **propagated** — soak ≥10 min AND confirm the 307 from a **multi-colo quorum (≥10–20 colos spanning regions)**, not 2 colos. A 2-colo sample is not all-colo proof; a lagging colo routes to the worker and re-pins on purge. 3. **ids in redirect set** — the host’s top ids are present in the **live deployed** redirect (HOT) set and confirmed live. Until this lands, those ids are worker-path — size against 100%-worker re-warm. JIT re-HEAD each redirect target (e.g. `wm.jpg`) → 200 required; skip + flag 404s (never purge into a 404 storm, see §6 T4). 4. **cohort-removed** — host removed from the worker cohort and the worker redeployed. Without this, every non-HOT id re-pins after purge and rollback never reaches clean. 5. **smoke-green** — per-host smoke: top ids 307 → redirect target, a non-HOT id does NOT 307, no-referrer → clean, the clean asset stays blocked. 6. **THEN trickle-purge** — only now, purge the bare `/{id}` keys, capped and rate-limited (§3), then re-probe the close-gate (§7). Reorder any step earlier and a purge lands into the worker path. *** ## 3. Wave sizing, trickle rate, and per-host URL caps [Section titled “3. Wave sizing, trickle rate, and per-host URL caps”](#3-wave-sizing-trickle-rate-and-per-host-url-caps) A **wave** = one uninterrupted purge run for one host’s id set (or a sub-slice). The binding constraint is **re-warm blast radius**, not the CDN API ceiling (which is far higher, \~1–2k files/req). ### Per-host URL caps [Section titled “Per-host URL caps”](#per-host-url-caps) `URLs_per_wave = min(host_distinct_ids, volume_tier_cap)`. The tier cap shrinks as host volume rises (more colos touched → bigger re-warm multiplier). Size the worst case against **100%-worker** at a worst-case fan-out (e.g. `COLOS = 330`). | Host volume / mo | Per-wave URL cap | Worst-case (every capped id non-HOT) | | -------------------------------------- | -----------------------------------------------------------: | ------------------------------------ | | < \~0.35M | 50 ids/wave | `50 × 330 × $1.07e-6 ≈ $0.018` | | \~0.35M–0.75M | 25 ids/wave | `≈ $0.009` | | \~0.75M–2M | 15 ids/wave | `≈ $0.0053` | | > \~2M (incl. the highest-volume host) | 5 ids/wave — ramp the top id first, soak, then the remainder | `≈ $0.0018` | Most hosts fit one wave (cardinality < cap). No single mistaken wave crosses the $1/mo line even if every assumption inverts. ### Trickle rate (the recoverability guarantee) [Section titled “Trickle rate (the recoverability guarantee)”](#trickle-rate-the-recoverability-guarantee) The single-URL purge tool has no rate-limit/retry/sleep. The wrapper enforces: * **Hard ceiling:** 500 URLs/min/zone, sleep ≥100 ms between calls, single in-flight request. * **For a cost-sensitive flip, run a TRUE TRICKLE:** 1 URL every 2–3 s (\~20–30 URLs/min). A 50-id wave is \~2–3 min, leaving the in-wave watcher time to react between ids. * **>\~1M hosts:** purge ONE id, watch the worker-MISS tail decay to baseline, THEN purge the next. The highest-volume host is single-id-waves only. * **429/5xx → exponential backoff** (250 ms → 2 s → 8 s, max 3 retries) then **hard stop**. Backoff must **pace** (sleep then continue forward); it must **never re-issue an already-sent purge** — re-purging re-evicts a just-warmed key = extra re-warm. The manifest marks each id purged-once; resume skips already-purged ids. * **Worst-case-mistake bound:** at 30 URLs/min a wrong target = `30 × $1.07e-6 ≈ $0.000032/min`; a runaway 5-minute trickle < $0.001. * **Manifest per id** (purged url, timestamp, pre-purge redirect-target HEAD status) → every wave auditable and reversible. *** ## 4. Fail-closed pre-wave cost gate [Section titled “4. Fail-closed pre-wave cost gate”](#4-fail-closed-pre-wave-cost-gate) Run a hard gate **before every wave** that computes worst-case $ and **refuses (non-zero exit) over ceiling**. It does not purge — it gates the wrapper. Reuse the `DRY_RUN` / `process.exitCode` convention of the purge tool. ### Worst-case formula [Section titled “Worst-case formula”](#worst-case-formula) ```plaintext COLOS = 330 # worst-case fan-out, not real (tens–hundreds) KV_BURST = 1.5 # cold-isolate ratio, NOT steady-state 1.13 RATE = $0.504e-6 + KV_BURST × $0.503e-6 # ≈ $1.26e-6/req during a burst/cache-bust rewarm_$ = nonHOT_urls × COLOS × RATE # one-time (decays in TTL window IF the page rule pins) rewarm_$ += nonHOT_COLD_misses × 4 × $0.36e-6 # object-GET fan-out across the miss-ladder permanent_$mo = cachebusting_nonHOT_reqPerMo × RATE # the unbounded vector — recurring forever wave_$ = rewarm_$ + permanent_$mo × 12 # annualize the permanent term so it dominates ``` `nonHOT_urls` = wave urls whose id is **NOT in the LIVE deployed redirect set**. Parse HOT classification from the **live deployed snippet artifact**, never from a plan/opportunity file. Hard precondition: assert `manifest-HOT ⊆ deployed-snippet-HOT`; abort on any divergence. ### The cache-busting / non-HOT check (fail-closed) [Section titled “The cache-busting / non-HOT check (fail-closed)”](#the-cache-busting--non-hot-check-fail-closed) ```ts // CACHEBUST_REPORT = per-host count of .jpg requests carrying a non-empty query string, // from a fresh analytics pull grouped by request-query, with a query limit large enough // to defeat the low-row floor. A path/referrer dataset blind to the query dimension // (0 rows carry '?') CANNOT supply this — it is structurally blind. if (!cachebustReportPresent || cachebustReportStale) { console.error("REFUSE: CACHEBUST_REPORT missing/stale — the one unbounded vector is unmeasured."); process.exitCode = 1; return; // FAIL-CLOSED. Never PASS-with-warning. } // Non-HOT steady-state: score extensionless / non-.jpg ids as permanent-by-default // (no page-rule pin), measured from an origin-source header on bare ids — not just '?'. ``` ### Per-wave ceilings [Section titled “Per-wave ceilings”](#per-wave-ceilings) | Gate constant | Value | Rationale | | ---------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `WAVE_REWARM_CEIL_USD` | $0.25 one-time | \~14× a realistic worst case; trips only on genuinely anomalous high-cardinality (e.g. auction-lot explosion on a rotating-catalog host). | | `WAVE_PERMANENT_CEIL_USD_MO` | $0.10/mo | Fires *before* the $1/mo line. **Non-overridable by an estimate** — a permanent leak must be neutralized at the cache layer (query-normalization in the snippet, or a CDN “Ignore Query String” rule), never accepted on a savings rollout. | | `WAVE_TOTAL_CEIL_USD` | $1.00 | The org backstop. A one-time breach may be overridden only by `OVERRIDE_WITH_ESTIMATE=docs/plans/-cost-estimate.md` (gate verifies the file exists and is git-tracked). | On any breach the gate prints the exact estimate path the operator must commit first. *** ## 5. The one unbounded trap: cache-busting [Section titled “5. The one unbounded trap: cache-busting”](#5-the-one-unbounded-trap-cache-busting) > **A single host with any cache-busting must be 100%-HOT-covered before any purge.** A zone caching at `cache_level=aggressive` keys each distinct query string separately. A cache-busting URL (`/{id}.jpg?v=NNN`, `?t=timestamp`) therefore **misses the edge on every request** → hits the worker (when non-HOT) **forever** → never re-pins. A single-URL purge **cannot** evict it (each query is its own key). * One host at 1M cache-busting req/mo ≈ **$1.07/mo recurring** — alone it crosses the $1/mo line. * This is the **only** vector that grows the bill forever; every other vector is bounded to cents-to-low-dollars and one-time. Required handling: * **GATE-CB blocks the flip.** Run a fresh per-host query-string pull (large query limit). ANY sustained nonzero `?query` on the asset → the host is **gated out** unless *every* emitting id is proven present in the LIVE deployed redirect set (so it 307s to stable storage). No HOT-coverage proof → no flip. Re-run per batch — a host can adopt `?v=timestamp` between snapshot and flip. * **If you cannot measure it, do not purge.** The cost gate fails closed on a missing/stale cache-busting report (§4). A path/referrer dataset that is blind to the query dimension does **not** prove “zero cache-busting.” *** ## 6. Other re-warm cost vectors (bounded, but watch them) [Section titled “6. Other re-warm cost vectors (bounded, but watch them)”](#6-other-re-warm-cost-vectors-bounded-but-watch-them) | # | Vector | Mechanism | Bound | | -- | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | | T2 | **Purge faster than propagation** | Snippet/cohort deploy is non-atomic across colos. A fast full-id purge evicts every colo at once; lagging colos route to the still-cohort worker and re-pin. With no trickle, a synchronized re-warm + 429 tail on a >1M host can cost a few unbudgeted dollars. Most likely *operational* spike — the brake is the trickle. | one-time | | T3 | **Non-HOT steady-state miss (no `?` needed)** | Extensionless / non-matching ids bypass the `*.jpg*` page rule → no long-TTL pin → re-warm at default TTL, recurring. Slips under a `?`-only gate. Score extensionless ids permanent-by-default. | recurring, small per host, stacks | | T4 | **404-storm on the redirect target** | If the 307 target (e.g. `wm.jpg`) 404s, the negative response often has no positive Cache-Control → not edge-pinned → every HOT request re-fetches storage forever. The asset page rule is on the primary host and does **not** match the storage subdomain. The worker is never invoked, so a worker/subrequest proxy **cannot see it**. | recurring, can cross $1/mo | Pre-purge gates for these: * **GATE-404:** verify the redirect target via an authoritative origin HEAD (storage binding), not a single-colo edge HEAD, AND poll the storage subdomain for `edgeResponseStatus=404`. Never purge a bare key whose target is not 200 at origin. * **GATE-EXT:** purge only the page-rule-matching ids; do NOT purge extensionless paths (no stale key to clear; they re-warm at default TTL). * **GATE-ORIGIN:** probe origin health (one known-clean fetch) before purging; **abort on origin 5xx** — re-warming into a sick origin turns one-time re-warm into a sustained storm and corrupts the decay-vs-plateau signal. * **GATE-RULE:** verify against the **live zone** (read-only API) that the long-TTL page rule actually exists and keys per-query as assumed. The “re-warm decays, one-time” claim depends on it — do not assume it from the repo. *** ## 7. Circuit-breakers [Section titled “7. Circuit-breakers”](#7-circuit-breakers) ### There is no hard CDN spend cap [Section titled “There is no hard CDN spend cap”](#there-is-no-hard-cdn-spend-cap) Every native control on a paid plan is notify-after-the-fact: * **Budget Alerts** = email only, \~24h lag, fire once/cycle, never throttle (pay-as-you-go accounts only — confirm the tier). * `request_limit_fail_open` is dormant on paid plans — not a cost lever. * `limits.cpu_ms` bounds cost **per request**, not request **count**, and is often unset — set it (config-only, no spend increase). So every real brake is self-built. The CDN is the outermost, slowest ring only. ### Fast signals that beat billing lag (ranked, with failure modes) [Section titled “Fast signals that beat billing lag (ranked, with failure modes)”](#fast-signals-that-beat-billing-lag-ranked-with-failure-modes) | Signal | Freshness | Failure mode — do NOT trust blindly | | -------------------------------------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Filtered logpush (`*.jpg?*`, origin-source header) | seconds, **records unattended** | Preferred in-wave watcher. | | Attached `wrangler tail` MISS watch | seconds, attached-only | **Drops messages above \~100 req/s** — exactly when a flood hits. Live spot-check only, never the record. | | Worker-subrequest breakdown, 1h window | \~60–75 min | A **proxy** (subrequests, not invocations/KV). A cache-bust served from storage may read **flat while invocations climb**. | | Object-storage cost report, 1-day window | \~hours | **Cheap path ONLY.** Structurally blind to worker/KV — a green storage gate is **NOT** a worker-cost all-clear. Force a 1-day window; a default 7-day window dilutes a spike \~7×. | | Budget alert / billable usage | \~24h | Reconciliation only. Never the trip. | **Token-scope tripwire:** a token missing Account Analytics Read returns **all-zeros silently**. Before wave 0, pull a 24h baseline and require `allStatuses > 0`; fail the rollout if the baseline is zero, so no later zero reads as “clean.” ### Numeric ceilings [Section titled “Numeric ceilings”](#numeric-ceilings) Convert $/mo to a same-day MISS count you can watch: a sustained `+$0.30/mo ≈ 280k extra MISS/mo ≈ 9,300/day ≈ 390/hr`. * **Per-wave sustained-delta KILL:** +$0.30/mo (smaller hosts) to +$0.50/mo (largest host). * **Daily MISS ceiling per active wave:** WARN 5,000/day, KILL 12,000/day. * **Hourly tripwire (fastest):** KILL if incremental MISS > 600/hr for two consecutive 1h polls *after* the first 60-min re-warm window (which is exempt/expected-elevated). * **Trip discriminator:** a one-time re-warm **decays** within the TTL window (one spike, then decline); a cache-busting/T3/T4 leak is a **flat non-decaying plateau**, only distinguishable after **two** consecutive 1h polls. Do not advance to host N+1 on the decaying-spike signal alone — require two post-rewarm polls return-to-baseline for host N first. ### The kill action (order matters) [Section titled “The kill action (order matters)”](#the-kill-action-order-matters) **Stop the bleed (purge loop) first — it is reversible and cheap. Pull the cohort second. NEVER disable the snippet first.** > Disabling the snippet is a cost-**increasing** action here: with hosts in the cohort, it routes the full 307 volume back into the worker (the single largest spend swing in the rollout). | Trip | Immediate KILL | Then | | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | MISS > 600/hr × 2 polls (purge-induced plateau) | **STOP the purge loop** | Run the KV/invocations poll to size; if permanent, next row. | | Plateau persists after purge stopped (organic cache-busting / T3 / T4) | **Pull the host from the COHORT** (remove + worker redeploy) → returns it to the non-watermarked cheap path. **Snippet stays enabled.** | Snapshot, root-cause; for cache-busting deploy the “Ignore Query String” cache rule (needs a `docs/` estimate). | | Incremental MISS > 12,000/day | **STOP purge + HOLD rollout**; snippet + cohort unchanged | Re-pull cardinality un-truncated. | | Storage cost over budget (1-day window) | **HOLD** (cheap path — lower urgency than a worker trip) | Investigate before next wave. | | `purge_cache` 429 in stdout | **STOP the purge loop** (no throttle in either path) | Resume with manual pacing. | A true global stop = stop purge + cohort backoff + (only if needed) snippet-disable **as the last step** — snippet-disable alone neither restores clean nor reduces worker cost. *** ## 8. Go/no-go checkpoint between waves [Section titled “8. Go/no-go checkpoint between waves”](#8-gono-go-checkpoint-between-waves) The wrapper pauses for a detection window between waves because the dangerous metric lags. Promotion is blocked until: 1. **Close gate (immediate):** re-probe every `/{id}.jpg?permcheck=` → variant clean (100%); re-probe a known NON-HOT id with referrer → if it returns the watermark, cohort overlap remains → STOP. 2. **Two-poll plateau clear (T+1h, T+2h):** the KV/invocations poll (or the subrequest proxy) for the just-flipped host shows **return-to-baseline on two consecutive polls**. Pin the pre-flip baseline as an **immutable captured value**, not a rolling re-pull, so a leak can’t be absorbed into “the new normal.” 3. **Storage confirm (T+24h):** 1-day-window storage cost gate passes under budget AND daily MISS delta < 5,000. Label the storage gate “cheap-path only, NOT a worker-cost gate.” 4. Record projected monthly storage cost; review on >20% batch-over-batch rise. Because true worker/KV detection lags hours-to-a-day, **the caps + trickle (§3) are the primary safety; the checkpoint confirms, it does not prevent.** *** ## 9. Pre-purge checklist (per host) [Section titled “9. Pre-purge checklist (per host)”](#9-pre-purge-checklist-per-host) * [ ] GATE-CB: fresh query-string pull, large limit; cache-busting either zero or 100% HOT-covered. * [ ] GATE-RULE: long-TTL page rule verified live, keys per-query as assumed. * [ ] HOT classification parsed from the **live deployed snippet**; `manifest-HOT ⊆ deployed-HOT` asserted. * [ ] snippet-live → propagated (≥10–20 colos) → cohort-removed → smoke-green, in order. * [ ] GATE-404 + GATE-ORIGIN: redirect target 200 at origin; origin healthy. * [ ] Cost gate run, fail-closed, under all three ceilings (or estimate committed). * [ ] Wave cap + trickle rate set for the host’s volume tier; single-id mode for >1M. * [ ] In-wave watcher (logpush) running; named operator assigned for the wave window. * [ ] Baseline pull non-zero (token-scope tripwire cleared). *** See the internal decoder (not published) for the concrete host/customer/id behind any role referenced here (canary host, rotating-catalog host, highest-volume host, the unprotected paying customer that blocks rollout, self-referred school/health hosts, gray-market network, datacenter no-referrer sources). # HOT-set management > Redirect-set freshness, decay cadence, and size budgets. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: how to manage the snippet’s redirect / HOT set — the per-id pins that make the edge snippet `307` deterministically to a pre-rendered variant object, instead of falling through to the worker. Applies to: the watermark-router edge snippet (`apps/snippet/watermark-router.snippet.js`), its `HOT` set, and the generate / verify / deploy / purge tooling under `scripts/cloudflare/`. For any concrete host/customer/id, decode the code names (H01–H14, C01/C02, S01–S04, K01, …) in the internal decoder (not published). ## What the HOT set is [Section titled “What the HOT set is”](#what-the-hot-set-is) The snippet is a referrer-blind, binding-less edge function that runs before the worker. For a request it can do one of three things: 1. Pass through to the worker (`fetch(request)`) — the default, and the source of truth for clean-vs-watermarked. 2. `307` to the public variant object (`cache.vumbnail.com/thumbnails/{id}/wm.jpg`), skipping the worker entirely. 3. Short-circuit malformed paths to a shared error asset. The `307` redirect path fires only when **both** conditions hold: ```js const watermark = shouldWatermarkRequest(request); // referrer host in WM_DOMAINS, not paying if (!watermark) return fetch(request); if (!HOT.has(videoId)) return fetch(request); // <-- the HOT-set gate return Response.redirect(buildWatermarkRedirectLocation(videoId), 307); ``` So `HOT` is the **per-id allowlist that decides which ids get the cheap deterministic edge redirect** vs. which fall through to the worker. It is a pure cost / determinism lever — it is **not** a correctness boundary (the worker still decides clean-vs-wm for anything that passes through). ## Why it exists [Section titled “Why it exists”](#why-it-exists) Every HOT hit is a worker request (and \~1.13 KV reads) avoided. At illustrative blended rates of \~`$0.50/1M` worker requests and \~`$0.50/1M` KV reads, the avoided-worker path is worth roughly `$1.07/1M` skipped requests. The HOT set concentrates that savings on the highest-volume reviewed referrer ids. See [cost-safe-cache-purging](/runbooks/cost-safe-cache-purging/) for the billing-shift caveat (worker+KV → R2 Class B). ## Decay semantics (read this before pinning anything) [Section titled “Decay semantics (read this before pinning anything)”](#decay-semantics-read-this-before-pinning-anything) A HOT id is just a string pin. When a catalog rotates and an id stops being requested, it is eventually pruned out of `HOT`. **Decay does not fall back to clean.** | Belief | Reality | | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ”Rotated-out id leaves HOT → snippet passes through → served clean” | FALSE for any host still in the worker referrer cohort. | | What actually happens | The id silently leaves the redirect set and falls back to the **worker path**. If the worker still watermarks that referrer cohort, the id is still watermarked — by the worker, not the edge. | | What decay actually erodes | **Delivery determinism**, not coverage. The mark stays; the cheap deterministic edge `307` is what is lost. | Consequences: * Determinism (and the cost savings) holds **only for HOT ids of hosts that have been removed from the worker cohort**. While a host stays in the cohort, a non-HOT id re-pins `wm` via the worker. See [referrer-cohort-and-delivery-policy](/runbooks/referrer-cohort-and-delivery-policy/). * “Cost looks fine” is **not** a success signal for a rotating host. A decayed HOT set bills cleanly while determinism rots. Track coverage (`currentHotCoveragePct`), not dollars, as the health metric. * Disabling the snippet does **not** restore clean — it routes traffic back to the still-cohorted worker. HOT-set work never substitutes for cohort changes. ## Refresh cadence by catalog churn [Section titled “Refresh cadence by catalog churn”](#refresh-cadence-by-catalog-churn) Cadence is driven by how fast the host’s catalog rotates, not by its volume: | Catalog type | Cadence | Rationale | | --------------------------------------------------- | -------------------------------- | ----------------------------------------------------- | | Rotating / fast-decay (e.g. auction lots, seasonal) | **Daily** | Top ids churn day-to-day; stale pins decay out fast. | | Stable catalog | **Weekly** (or on coverage drop) | Top ids are durable; refresh only when coverage dips. | | Single-hero / single-video host | **Pin once** | One id carries \~all traffic; no recurring refresh. | Reference instances (decode in the internal decoder (not published)): a rotating-catalog host like H03 → daily; stable hosts → weekly; a single-hero host like H02 (and a pin-once host like H07) → pin once. Coverage floors that gate a refresh / hold: * Rotating host: hold and refresh if `currentHotCoveragePct < 0.85` or it drops `>5` absolute points batch-over-batch. * Stable host: floor `0.80`. `scripts/cloudflare/analyze-whale-hotset.ts` computes `currentHotCoveragePct` per host but is **report-only** today — it does not exit non-zero below the floor. Until a coverage-floor exit code exists, a rotating host kept in HOT requires either a wired alarm or a written manual daily re-verify. ## The 32KB package budget — PRUNE, do not append [Section titled “The 32KB package budget — PRUNE, do not append”](#the-32kb-package-budget--prune-do-not-append) The snippet package has a hard **32 KB (32,768 B)** Cloudflare limit. The HOT set is the dominant variable-size payload, so it must be actively pruned, never blindly grown. Rules: * **Prune, don’t append.** A daily refresh splices in the new top ids for a host *and removes* the decayed ones. Appending forever blows the budget. * **Per-host id caps.** Cap each rotating host at roughly **20–30 ids**. Past the cap you are pinning long-tail ids that decay before they pay off. * **Keep ≥4 KB margin.** Fail any generated/`--write` output that projects `> 28,672 B` (4 KB margin under the 32 KB cap). Several KB of headroom must stay free for comments, tests, and small safety checks. * **Per-id referrer comments are load-bearing.** The verify/analyze parsers string-scan `const HOT = new Set([…]);` and the trailing `// host` comments. Any writeback must preserve them. * **Never reformat the file during a refresh/flip.** `bunx biome format` on the snippet reflows the `HOT` set and blinds the string-scan parsers → false-green gates. Forbidden during any HOT-set change. Approximate per-id cost: `id.length + 6` bytes (quotes, comma, indent). Use that to project a candidate before writing. ### Size-budget sizing reference [Section titled “Size-budget sizing reference”](#size-budget-sizing-reference) Illustrative scenario sizing from a prior expansion (your numbers will differ): | Scenario | Est. source size | Headroom under 32 KB | | --------------------- | ---------------: | ------------------------: | | Baseline snippet | \~12,000 B | \~20,700 B | | `top50-per-host` | \~15,200 B | \~17,500 B | | `top100-per-host` | \~18,300 B | \~14,500 B | | `top200-per-host` | \~22,500 B | \~10,300 B | | `coverage50-per-host` | \~25,200 B | \~7,600 B | | ”everything at once” | \~32,100 B | \~600 B — **do not ship** | ## Generation + verify + purge workflow [Section titled “Generation + verify + purge workflow”](#generation--verify--purge-workflow) HOT-set changes are externalized and gated. Never hand-edit ids straight into a deploy. ### 1. Analyze / select [Section titled “1. Analyze / select”](#1-analyze--select) Pull fresh path-referrer and cohort artifacts, then rank ids per host: ```bash bun scripts/cloudflare/analyze-whale-hotset.ts \ --path-referrer-json \ --cohort-report-json \ --previous-path-referrer-json \ --out-json artifacts/cloudflare/whale-hotset-opportunity..json ``` This reports per-host concentration, coverage targets, top-id overlap vs. the previous snapshot (stability), how many ids each scenario adds, and whether the resulting snippet still **fits the 32 KB limit**. Use top-set overlap to judge churn: low overlap = rotating = shorter cap + daily cadence. ### 2. Splice into HOT (prune + cap) [Section titled “2. Splice into HOT (prune + cap)”](#2-splice-into-hot-prune--cap) Apply the delta for rotating hosts — add the new top ids, drop the decayed ones, honor the per-host cap, preserve referrer comments. A `--write` / `CONFIRM_HOTSET_WRITE=YES` mode must fail **before writing** if the projected source exceeds the 28,672 B budget. ### 3. Backfill the variant object BEFORE it can be redirected to [Section titled “3. Backfill the variant object BEFORE it can be redirected to”](#3-backfill-the-variant-object-before-it-can-be-redirected-to) A HOT id that has no `wm.jpg` in R2 would `307` users straight into a 404. So **ensure the object exists before the pin can go live**: ```bash # verify only (gate — exits non-zero on any missing wm.jpg) bun scripts/cloudflare/verify-watermark-snippet-hotset.ts verify # backfill missing objects (renders wm.jpg from the clean source) CONFIRM_ENSURE=YES CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ACCOUNT_ID=... \ bun scripts/cloudflare/verify-watermark-snippet-hotset.ts ensure ``` `verify` parses every quoted id out of the `HOT` set and HEADs each `…/wm.jpg`; any non-`200` fails. `ensure` pulls the clean source (R2 `clean.jpg`, or the worker with a clean-forcing referer), applies the watermark, and writes `wm.jpg`. For a rotating host, **source the clean image from R2 `clean.jpg`, not a worker fetch** — a worker fetch for a cohorted host can return an already-watermarked image and bake a double mark that the `200`-check won’t catch. Backfill spends money / writes R2 objects — keep estimates under the repo cost-estimate threshold or commit a written estimate (see [cost-safe-cache-purging](/runbooks/cost-safe-cache-purging/)). ### 4. Cross-check the parse count [Section titled “4. Cross-check the parse count”](#4-cross-check-the-parse-count) Catch a reflow-blinded parser: the parsed HOT count must equal the `rg -c` quoted-id count must equal `pre-edit + added`. A mismatch means the file was reformatted and the gates are lying. ### 5. Deploy, then purge public-cache-lag 404s [Section titled “5. Deploy, then purge public-cache-lag 404s”](#5-deploy-then-purge-public-cache-lag-404s) After deploy goes live (content sha matches, rule enabled), purge the bare cached keys whose prior `404`/clean cache entries would otherwise mask the new pins: * Just-in-time re-HEAD each `wm.jpg` and require `200` **before** purging its key; skip + flag any `404` (do not purge into a 404 storm). * A newly-backfilled object can return a cached `404` at the edge from a prior miss — purge those specific public URLs so the first real viewer gets the object, not the stale miss. * Respect purge rate limits and use the correct zone (a wrong-zone purge silently no-ops). Full purge mechanics live in [cost-safe-cache-purging](/runbooks/cost-safe-cache-purging/). ## Pre-deploy gates (HOT-set specific) [Section titled “Pre-deploy gates (HOT-set specific)”](#pre-deploy-gates-hot-set-specific) * [ ] `verify-watermark-snippet-hotset.ts verify` → **0 missing** `wm.jpg`. * [ ] Parse-count cross-check passes (parsed == `rg -c` == pre-edit + added). * [ ] Projected source size `< 28,672 B` (4 KB margin under 32 KB). * [ ] Per-id referrer comments intact; file **not** biome-reformatted. * [ ] Every new id maps to a public `wm.jpg` (`200`) from multiple colos. * [ ] Coverage above floor for every rotating host being kept (or written manual re-verify). * [ ] Rollback snapshot regenerated for this batch (see [safe-deploys](/runbooks/safe-deploys/)). ## Common failure modes [Section titled “Common failure modes”](#common-failure-modes) | Symptom | Likely cause | Fix | | ---------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | HOT id `307`s to a 404 | `wm.jpg` never backfilled, or purged while still HOT | Re-`ensure`; never delete a still-HOT id’s warm `wm.jpg`. | | Gates green but flip didn’t take | File was biome-reformatted → parsers string-scanning `HOT` blinded | Restore formatting; re-run parse-count cross-check. | | Cost looks fine but determinism eroding | Decayed HOT set on a rotating host (silent worker fallback) | Track `currentHotCoveragePct`, not dollars; refresh HOT. | | Snippet over 32 KB after refresh | Appended instead of pruned; per-host cap exceeded | Prune decayed ids; enforce \~20–30/host cap. | | Non-HOT id still watermarked after purge | Host still in worker cohort → worker re-pins `wm` | Remove host from cohort in the same release ([referrer-cohort-and-delivery-policy](/runbooks/referrer-cohort-and-delivery-policy/)). | ## Related runbooks [Section titled “Related runbooks”](#related-runbooks) * [referrer-cohort-and-delivery-policy](/runbooks/referrer-cohort-and-delivery-policy/) — the worker cohort that the HOT set’s determinism depends on. * [cost-safe-cache-purging](/runbooks/cost-safe-cache-purging/) — purge mechanics, rate limits, zone selection, cost estimates. * [staged-rollouts](/runbooks/staged-rollouts/) — flipping hosts in batches; HOT/cohort ordering. * [safe-deploys](/runbooks/safe-deploys/) — snippet rollback snapshots and the git-is-the-only-content-rollback rule. * The internal decoder (not published) — decoder for every host / customer / id code name used above. # Internal docs publishing > Keeping internal trees and local-only decoders out of public builds. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: keep internal engineering docs (plans, research, rollout strategy, customer-identifying notes) out of every public-facing surface, while still allowing a deliberately redacted external page when sharing is required. Applies to: anyone editing `docs/` (the internal VitePress site), `apps/docs/` (the public Starlight product-spec site), or standing up an external share page. For concrete entities referenced abstractly here, see the internal decoder (not published — never committed to a public surface). *** ## Why this matters [Section titled “Why this matters”](#why-this-matters) Internal docs contain PID-class material: customer names/emails, host/brand domains, IPs, billing-provider ids, monthly bill totals, specific video ids, infra topology, and rollout sequencing. None of it may reach a public URL, a public AI-readable manifest, or a co-located build that carries the real product brand. Code names (H01, C02, S01, …) do **not** survive co-location with the real brand: a reader who sees the brand plus a descriptor (“the highest-volume rotating-catalog host”) can re-identify the underlying entity. Treat descriptors as PID too. *** ## The two doc surfaces [Section titled “The two doc surfaces”](#the-two-doc-surfaces) | Surface | Path | Build model | Public? | AI manifest | | ------------------------- | ------------ | --------------------------------------------------------- | ------------- | ----------------------------------------------------- | | Internal docs (VitePress) | `docs/` | Globs **every** `.md` under `srcDir` into a routable page | Internal only | none | | Product-spec (Starlight) | `apps/docs/` | Builds **only** its own content collection | Public | emits `llms.txt` / `llms-full.txt` / `llms-small.txt` | ### Surface 1 — internal VitePress build [Section titled “Surface 1 — internal VitePress build”](#surface-1--internal-vitepress-build) VitePress turns every `.md` under `srcDir` into a routable page by default. **The sidebar only controls navigation, not URL reachability** — an un-listed page is still served at its slug. So an internal tree that is merely “not in the sidebar” is still publicly fetchable if the site is deployed. Containment is `srcExclude`, not the sidebar. The internal trees are excluded from the build entirely: docs/.vitepress/config.ts ```ts srcExclude: [ "plans/**", "research/**", "progress/**", "watermarks/**", ], ``` Rules: * Any new internal-only tree (anything containing plans, research, progress notes, or customer/host-specific material) **must** be added to `srcExclude` in the same change that introduces it. * Do not rely on omitting a page from the sidebar to hide it. Sidebar omission ≠ exclusion. * After changing `srcExclude`, verify the served bytes (see [Verify the live bytes](#verify-the-live-bytes)) — not just the source config. ### Surface 2 — public Starlight product-spec site [Section titled “Surface 2 — public Starlight product-spec site”](#surface-2--public-starlight-product-spec-site) The product-spec site builds **only** the docs content collection (`docsLoader()` over its own `src/content/docs`), so a stray `.md` elsewhere in the repo is not swept in the way VitePress globs. But it has the opposite risk: it is **public by design** and emits machine-readable manifests via `starlight-llms-txt`: * `/llms.txt`, `/llms-small.txt` — indexes for AI agents * `/llms-full.txt` — the full machine-readable concatenation of the collection Anything you place in the public collection is, by design, served to humans **and** flattened into `llms-full.txt` for AI agents. There is no “internal” page in this site. Rules: * **Never copy or summarize an internal plan/research doc into the public collection.** Not the file, not a paraphrase, not a “lightly cleaned” version. Co-location with the real brand re-identifies code-named entities via their descriptors. * New public pages are present-tense product spec only (Live / Flagged / Planned status badges) — no customers, no hosts, no infra costs, no rollout timing. * Assume every public page ends up verbatim in `llms-full.txt`. Redact accordingly before, not after. *** ## DO-NOT-PUBLISH banner [Section titled “DO-NOT-PUBLISH banner”](#do-not-publish-banner) Every internal plan/research/progress doc carries a banner at the top so a future editor cannot mistake it for shareable material: ```md > **DO NOT PUBLISH.** Internal working doc. Contains customer-identifying and > infra/cost detail. Excluded from the VitePress build via `srcExclude`. Never > copy, summarize, or migrate into the public product-spec collection. ``` The banner is a backstop, not the control. The control is `srcExclude` (Surface 1) and not-co-locating (Surface 2). *** ## Sharing externally: a separate, redacted, brand-detached worker [Section titled “Sharing externally: a separate, redacted, brand-detached worker”](#sharing-externally-a-separate-redacted-brand-detached-worker) When something genuinely needs to go to an external reader, do **not** loosen either doc site. Stand up a **separate worker** serving a single redacted page: * Brand-detached: no product domain, no product name, no logos that re-identify the system. * `noindex`: send `X-Robots-Tag: noindex, nofollow` (and a ``) so it stays out of search and AI crawls. * Separate deploy unit: its own worker/route, never a sub-path of either doc site, so it cannot inherit their content or be reached by editing a shared build. * Fully redacted before deploy (checklist below). See [reverting-deploys](/runbooks/reverting-deploys/) for pulling a share page back down, and the internal decoder (not published) for the decode of any code-named entity you are tempted to mention. *** ## Redaction checklist [Section titled “Redaction checklist”](#redaction-checklist) Run this against the **rendered output**, not just the source. A single surviving real entity fails the page. * [ ] **Customer / contact names** — removed. Replace with a role (“the unprotected paying customer”, “a self-referred school/health host”). * [ ] **Emails** — removed (including in examples, mailto links, and code samples). * [ ] **Brand / host / product domains** — removed. No real hostnames, even partial or obfuscated. * [ ] **IP addresses** — removed (no host IPs, no datacenter no-referrer source IPs, no CDN-internal IPs). * [ ] **Billing-provider ids** — removed (subscription / customer / price ids). * [ ] **Monthly bill totals** — removed. Public unit rates and mechanics are fine; the real blended bill is not. * [ ] **Specific video ids** — removed. * [ ] **Re-identifying descriptors** — generalized. “The \~4M/mo highest-volume host rolled last and alone” → “a high-volume host”. If you must disambiguate across the doc, use a code name and decode it only in the internal decoder (not published). * [ ] **Rollout sequencing / entitlement gaps** — omitted from public surfaces (these re-identify which customer was a blocker). Prefer **roles over code names**; prefer **code names over real entities**; never the real entity. *** ## Verify the live bytes [Section titled “Verify the live bytes”](#verify-the-live-bytes) Source config and rendered output drift. Always confirm what is actually served. ```bash # Surface 1 — an excluded internal tree must NOT be reachable on the deployed site. # Expect 404 for each excluded slug. curl -s -o /dev/null -w "%{http_code}\n" https:///plans/ curl -s -o /dev/null -w "%{http_code}\n" https:///research/ # Surface 2 — the public AI manifests must contain ZERO internal material. # Each grep below should return nothing. curl -s https:///llms-full.txt | grep -iE '||' curl -s https:///llms.txt | grep -iE 'plan|research|progress|rollout' # Share worker — must be noindex and brand-detached. curl -sI https:/// | grep -i 'x-robots-tag' # expect: noindex, nofollow ``` Notes: * Test against the **deployed** host, not the local dev server — build-time exclusion can differ from a dev server that serves everything. * A page can be absent from the HTML sidebar yet still 200 at its slug. The 404 check is the real test for Surface 1. * `llms-full.txt` is the highest-risk artifact: it concatenates the whole public collection into one fetch. Grep it explicitly. *** ## Quick decision table [Section titled “Quick decision table”](#quick-decision-table) | You want to… | Do | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | Add an internal plan/research/progress doc | Put it under an excluded tree; add a DO-NOT-PUBLISH banner; confirm the tree is in `srcExclude` | | Add a new internal tree | Add it to `srcExclude` in the same change; verify 404 on the deployed host | | Document the product publicly | Add a present-tense spec page to the Starlight collection — no customers, hosts, costs, or rollout detail | | Share findings with an outside reader | Stand up a separate, redacted, `noindex`, brand-detached worker; run the redaction checklist; verify live bytes | | Reference a specific entity | Use a role; if you must disambiguate, a code name decoded only in the internal decoder (not published) | *** ## See also [Section titled “See also”](#see-also) * [reverting-deploys](/runbooks/reverting-deploys/) — taking a share page or doc deploy back down * The internal decoder (not published) — private decode of all roles / code names # Monetization leverage method > Ranking which embedding hosts to watermark for conversion vs cost control. Unlisted internal dev doc — audit before publishing publicly Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: a reusable method for ranking which embedding hosts to watermark for **conversion** vs which to leave alone (or redirect) for **cost control**, when a free image/thumbnail service uses a watermark as soft dunning. Applies to: any service that (a) serves an asset that third parties hot-link/embed, (b) can deliver a watermarked vs clean variant per request, and (c) wants the watermark to drive upgrades. Concrete per-host targets, code names, and decoded entities live in the internal decoder (not published) — never inline them here. See also: [reverting-deploys](/runbooks/reverting-deploys/), [edge-cache-variant-lottery](/runbooks/watermark-system/), [snippet-redirect-cost-control](/runbooks/cost-safe-cache-purging/). ## TL;DR [Section titled “TL;DR”](#tldr) * A watermark only converts when **the person who can buy actually sees it**. Viewers are not buyers. * Rank targets by **leverage = buyer-fit × operator-visibility × mark-legibility ÷ cache-invalidation cost**. * Cost rarely gates the good targets: a targeted purge is ≈free, a snippet redirect is *cheaper* than the worker, only blanket page-rule removal is expensive. * So in practice leverage collapses to **buyer-fit × operator-visibility** — the cheapest hosts to flip are usually the best conversion targets, and the expensive-to-flip hosts are usually poor targets anyway. * Hosts whose audience is watermark-tolerant non-buyers (self-referred institutions, gray-market video networks) go to **cost control via snippet redirect**, not conversion. ## The Leverage Formula [Section titled “The Leverage Formula”](#the-leverage-formula) ```plaintext leverage = (buyer_fit × operator_visibility × mark_legibility) / cache_invalidation_cost ``` | Factor | High score | Low score | | --------------------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | **buyer-fit** | commercial, brand-conscious, has budget, self-serve-purchasable | institution, nonprofit, gray-market, no budget | | **operator-visibility** | mark lands on a page the operator/marketing team looks at (homepage, PDP, blog, marketing embed) | mark buried in an app, player, LMS, or back-office only end-users see | | **mark-legibility** | mark renders large enough and high-enough contrast to read at the host’s actual display size | mark downscaled to a thumbnail where it’s illegible | | **cache-invalidation-cost** | targeted purge or snippet redirect (≈$0 or negative) | requires blanket page-rule removal (ongoing per-request worker cost) | The denominator almost never dominates (see [Cost framing](#cost-framing-cost-rarely-gates-the-target)). Score the numerator first; only let cost break ties or veto a high-volume host that needs blanket removal. ## Why Buyer-Fit And Operator-Visibility Are Separate Axes [Section titled “Why Buyer-Fit And Operator-Visibility Are Separate Axes”](#why-buyer-fit-and-operator-visibility-are-separate-axes) A watermark-as-dunning mechanism converts only when the **buyer** sees the mark on a surface they own and act on. Two independent failure modes: 1. **Audience mismatch (buyer-fit fails).** The host’s audience is non-buyers. Examples of the pattern: self-referred school/health institutions, religious-media nonprofits, gray-market/app-network video sites. These audiences are *watermark-tolerant* — they keep using the free asset indefinitely and never upgrade. Watermarking them is wasted pressure. 2. **Visibility mismatch (operator-visibility fails).** The host *is* a good buyer, but the embed lives where the operator never looks — inside a player, a logged-in app, or an LMS — so the mark reaches end-users while the buyer/webmaster never sees it. A real-world version of this: a host’s homepage displayed watermarked thumbnails for weeks-to-months with zero conversion, because the mark was on a viewer surface, not a buyer surface. A host must score on **both** axes to be a conversion target. Score one axis high and the other low → it is not a conversion target. ### The non-buyer escape hatch [Section titled “The non-buyer escape hatch”](#the-non-buyer-escape-hatch) Watermark-tolerant non-buyers (institutions, gray-market networks) are still **cost** on the worker/KV path. The right move for them is not conversion and not “leave clean” — it is a **snippet redirect** that serves the watermarked variant before the worker runs, cutting compute spend while keeping the mark on (which is harmless for non-buyers and mildly discouraging for the few who might churn off). See [Cost framing](#cost-framing-cost-rarely-gates-the-target). ## Cost Framing: Cost Rarely Gates The Target [Section titled “Cost Framing: Cost Rarely Gates The Target”](#cost-framing-cost-rarely-gates-the-target) There are three ways to change what variant a URL serves, and they have three different cost signs. Know which one a given action triggers before you worry about cost. | Mechanism | Effect | Cost sign | | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | | **Targeted purge** of specific pinned URLs | next request re-pins the intended variant, stays edge-cached for the page-rule TTL | one-time miss only ≈ **$0 ongoing** | | **Snippet redirect** (add host to the snippet’s watermark-domain set + current HOT ids) | redirect to the prebuilt `wm` variant in R2, runs **before** cache, **skips the worker** | **negative** — *saves* roughly a blended worker+KV request rate per request | | **Blanket page-rule removal** | every request hits the worker every time | **positive, ongoing** — the only expensive path; avoid | Implications: * A targeted purge and a snippet redirect are the recommended levers. Both are ≈$0 or negative, so **cost does not gate the conversion targets**. * The only expensive lever (blanket removal of the referrer-blind page-rule cache) is also the one you almost never need. Reach for it last, and per repo policy write a cost estimate in `docs/` before any change that could add ≥$1/mo (blanket removal of a high-traffic page rule easily clears that). * Use illustrative blended unit rates (public Cloudflare per-million rates) only to *anchor* the comparison; do not put real monthly bill totals in this doc. ## Procedure [Section titled “Procedure”](#procedure) 1. **Pull volumes + referrer cohort.** Get est requests/30d per host and the current set of cohort referrers the worker watermarks on a miss. Note that hand-synced cohort/HOT lists decay (observed: \~0% coverage in \~8 weeks as video ids rotate) — re-pull before ranking. 2. **Score buyer-fit** per host: commercial intent, brand-consciousness, budget, self-serve purchasability. Demote institutions, nonprofits, gray-market, app/player networks. 3. **Verify operator-visibility**, do not assume it. Render each candidate’s key pages in a **real JS browser** and confirm where the embed lands: * homepage / product-detail page / blog / marketing embed → **good** (operator sees it) * player / logged-in app / LMS / back-office → **demote to cost-control** * Static fetches miss JS-loaded embeds and miss JS-rendered pricing — do not audit visibility (or your own funnel) with `curl`/static fetch. 4. **Check mark-legibility** at the host’s *actual* display size, not the asset’s native size. A mark legible at 640px may be illegible when the host renders the thumbnail at 256×144. 5. **Classify cache-invalidation cost** per host using the table above. Almost everything resolves to purge or snippet (≈$0). Flag any host that would need blanket page-rule removal. 6. **Rank** by the leverage formula. Split into two tiers. ## Output: Two Tiers [Section titled “Output: Two Tiers”](#output-two-tiers) **Tier 1 — Convert.** High buyer-fit, verified operator-visibility, cheap to flip (purge or snippet). Action: make delivery **deterministic** for these via the snippet (add to the watermark-domain set + current HOT ids), then purge any clean-pinned URLs. Prefer stable-catalog hosts — a rotating catalog means high snippet HOT churn and is more expensive to keep deterministic. **Tier 2 — Do NOT chase for conversion; cost-control only.** Watermark-tolerant non-buyers, or good buyers whose embeds are operator-invisible (player/app/LMS), or rotating-catalog hosts that are expensive to keep deterministic. Action: snippet-redirect to cut worker/KV spend (and short-circuit malformed-path traffic where present). Do not spend purge effort chasing conversion here. ## Preconditions That Gate The Whole Method [Section titled “Preconditions That Gate The Whole Method”](#preconditions-that-gate-the-whole-method) Ranking targets is wasted effort if the funnel and delivery underneath are broken. Confirm these first: * **Delivery is deterministic.** A referrer-blind edge cache (e.g. a long-TTL `*.jpg*` page rule) can pin one variant per URL/colo/month *ahead of* the worker’s per-request decision, making delivery a lottery in both directions. Trust only aged, uncontaminated reads (`age` well over the worker’s own normalization window) when probing, and remember a single probe with a cohort referrer can self-poison a URL. See [edge-cache-variant-lottery](/runbooks/watermark-system/). * **The funnel is wired.** The mark must point at the offer (render the upgrade path *in the mark text*), and the pricing page must be crawlable (server-render a price summary, not just a JS-only pricing-table iframe). A visible watermark with no route to a purchasable, crawlable offer converts nobody — this gates **all** tiers and outranks adding more watermarked hosts. * **The reverse leak is fixed.** When a self-referred host’s traffic pins `wm` on a shared URL, no-referrer/ambiguous users of that same URL get watermarked too — a policy violation for developer-friendly defaults. Scope or purge wm-pinned entries serving ambiguous traffic. * **Watch for unprotected paying customers.** A paying customer with empty entitlements can be silently watermarked by a blanket rollout — a rollout blocker. Reconcile the customer/entitlements list before flipping delivery broadly. ## Anti-Patterns [Section titled “Anti-Patterns”](#anti-patterns) * Judging “watermarks don’t convert” on data gathered before delivery is deterministic — it is contaminated in both directions. * Treating viewers as buyers. The audience that *reliably* sees the mark (institutions, gray-market networks) is usually the audience that *can’t* buy. * Making the mark more hostile instead of more **legible**. Fix legibility and delivery before touching aggression. * Auditing visibility, pricing, or your own funnel with static fetches — JS-loaded embeds and JS-rendered prices are invisible to them. * Relying on hand-synced cohort/HOT lists as the backbone; they rot to \~0% coverage as ids rotate. Automate the sync. * Reaching for blanket page-rule removal when a targeted purge or snippet redirect achieves the same delivery change at ≈$0. ## Cross-References [Section titled “Cross-References”](#cross-references) * Concrete host code names (flip hosts, non-buyer cost-control hosts, self-referred institutions, gray-market networks, protected/unprotected paying customers, datacenter no-referrer sources, excluded CDN-internal IPs) and their decodes: the internal decoder (not published). * Cache mechanics and probe methodology: [edge-cache-variant-lottery](/runbooks/watermark-system/). * Snippet redirect setup and cost math: [snippet-redirect-cost-control](/runbooks/cost-safe-cache-purging/). * Rolling back a delivery flip: [reverting-deploys](/runbooks/reverting-deploys/). # No-referrer traffic > Classifying missing-referrer traffic and detecting dev/local requests. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: explain what no-referrer image traffic actually is, so you can decide whether to default it to the clean (unwatermarked) variant without bleeding conversions. Applies to: the edge snippet / watermark router decision for requests that arrive with an empty/missing `Referer`, on the thumbnail image path. For concrete entity decodes (specific hosts, IPs, customer ids), see the internal decoder (not published). Sibling runbooks: [reverting-deploys](/runbooks/reverting-deploys/), [referrer-cohort-allowlist](/runbooks/referrer-cohort-and-delivery-policy/). ## TL;DR [Section titled “TL;DR”](#tldr) No-referrer does **not** mean “a real person with privacy settings.” On the thumbnail path the no-referrer cohort is dominated by machines: datacenter renderers spoofing browser UAs and generic SDK/library default UAs. Real browsers with a stripped referrer are a **minority** of the cohort. Therefore defaulting no-referrer to **clean** is **low conversion-risk** — the bulk will never convert. Detect dev/local separately (it arrives **with** a localhost referrer, not as no-referrer). ## Why no-referrer is not “privacy-conscious humans” [Section titled “Why no-referrer is not “privacy-conscious humans””](#why-no-referrer-is-not-privacy-conscious-humans) The intuition is that a missing `Referer` is a real browser that stripped it (`Referrer-Policy`, HTTPS→HTTP downgrade, etc.). The data says the opposite: that real-browser bucket exists but is small relative to non-human sources. Most no-referrer volume is servers, libraries, and bots that fetch the image directly and never render a buy page. ## Source taxonomy (ranked by volume) [Section titled “Source taxonomy (ranked by volume)”](#source-taxonomy-ranked-by-volume) Ranked high→low by share of no-referrer requests on the thumbnail path. Treat the counts as illustrative orders of magnitude from one window, not fixed truth — re-pull before acting (see [Data caveats](#data-caveats)). | Rank | Source bucket | Eyeballs? | How to recognize it | | ---- | ----------------------------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | **Datacenter / server-side renderer spoofing a browser UA** | No | A handful of fixed datacenter IPs all sharing one desktop-browser UA (e.g. a current Firefox/Windows string). Largest single bucket. Two IPs alone can be \~40% of it. | | 2 | **Generic SDK / HTTP-library default UA** | No | Bare `Mozilla/5.0` with nothing after it; `node`; `okhttp/`; `Dart/ (dart:io)`; other language/runtime defaults. | | 3 | **Real browsers, referrer stripped** | **Yes (minority)** | Full, well-formed mobile/desktop UAs (iPhone Safari, Mac Safari, Windows Chrome, Android Chrome) with no referrer. The only conversion-relevant slice — and it is small. | | 4 | **Native mobile apps (iOS/Android)** | Partial | iOS `CFNetwork/Darwin` app UAs (branded installer/app names); Android via `okhttp`. A native app may strip the referrer even when in-app webviews do not. | | 5 | **Search / social crawlers** | No | `Googlebot-Image`, `facebookexternalhit`, `meta-externalads`, `bingbot`, and similar self-identifying crawler UAs. | | 6 | **Email / document image proxies** | Indirect | Gmail image proxy (`via ggpht.com GoogleImageProxy`), Outlook/Office (`ms-office; MSOffice 16`). A human may eventually see the image, but the fetch is a proxy with no referrer and no page context. | | 7 | **Monitoring / uptime bots** | No | Uptime/monitoring UAs (`...SiteMonitor`, `...-bot/`). | | — | **CDN-internal tiered-cache IP — EXCLUDE** | n/a | A single CDN-range IPv6 (the CDN’s own tiered-cache fabric) can show **one to two orders of magnitude more requests than everything else combined**. It is infra noise, flagged internal/infra. **Exclude it before any analysis** or it swamps the whole picture. See `CFINT` in the internal decoder (not published). | ### The exclusion that matters most [Section titled “The exclusion that matters most”](#the-exclusion-that-matters-most) The CDN-internal tiered-cache IP is not a traffic source — it is the cache talking to itself. If you forget to drop it, it dwarfs every real bucket and every conclusion below is wrong. Filter `likelyInternalOrInfra` first. ## Conclusion: defaulting no-referrer to clean is low-risk [Section titled “Conclusion: defaulting no-referrer to clean is low-risk”](#conclusion-defaulting-no-referrer-to-clean-is-low-risk) * Buckets 1, 2, 5, 7 (datacenter, libraries, crawlers, monitors) are non-eyeballs and the clear majority of legitimate no-referrer volume. * Bucket 3 (genuine browsers, referrer stripped) is the only conversion-relevant slice, and it is a **minority** of the cohort. * Serving the clean variant to no-referrer therefore costs \~no conversions while cutting watermark-router work on a large, mostly-robotic cohort. This de-risks the “default no-referrer → clean” path. It does **not** by itself decide watermark policy for *referrer-bearing* traffic — that is the referrer cohort’s job (see [referrer-cohort-allowlist](/runbooks/referrer-cohort-and-delivery-policy/)). ## Dev/local is referrer-BEARING — detect by referrer host, not UA [Section titled “Dev/local is referrer-BEARING — detect by referrer host, not UA”](#devlocal-is-referrer-bearing--detect-by-referrer-host-not-ua) Local/dev traffic mostly arrives **with** a `localhost` or `127.0.0.1` **referrer**, i.e. it is in the referrer-host set, not the no-referrer set. * **Detect dev by referrer host** (`localhost`, `127.0.0.1`, `*.test`, `*.local`, common preview/staging hosts). This is high-precision. * **Do NOT detect dev by user-agent.** UA-based dev detection collides head-on with bucket 2 (server/library default UAs), so it both misses real dev traffic and misclassifies bots as dev. Net: a referrer-host rule cleanly catches dev without touching the no-referrer bucket at all. ## How to reproduce the analysis [Section titled “How to reproduce the analysis”](#how-to-reproduce-the-analysis) Pull the no-referrer cohort over a recent window and bucket it. Source query sets to cross-reference: ```text noReferrerUserAgentsOverThreshold # UA taxonomy (buckets 1–7) noReferrerIpsOverThreshold # find the datacenter IPs + the CDN infra IP noReferrerIpUserAgentPairsOverThreshold # confirm "N fixed IPs, one UA" (bucket 1) xRequestedWithOverThreshold # confirm in-app webviews mostly DO send a referrer referrerHostsOverThreshold # confirm dev arrives WITH a localhost referrer ``` Steps: 1. Drop the CDN-internal/infra IP(s) (`likelyInternalOrInfra`) first. 2. Classify the top UAs into the seven buckets above. 3. Confirm bucket 1 by checking IP+UA pairs: a few fixed datacenter IPs sharing one browser UA. 4. Confirm dev is referrer-bearing by looking for `localhost`/`127.0.0.1` in the **referrer-host** results, not the no-referrer results. 5. Normalize window counts to a monthly figure (window × `30d/window-days`) only for ranking — exact magnitudes are not load-bearing. Prefer extending an existing cohort-candidate script over a one-off; see `docs/tiger/SKILL.md` for script style. ## Data caveats [Section titled “Data caveats”](#data-caveats) * **Logs are query-string-blind.** Aggregated request logs commonly strip the query string, so this analysis **cannot see cache-busting** (`?v=`, `?t=` style params). Do not claim anything about cache-busting from this data — prove it separately with a query-string-aware pull or a live probe. * **Snapshot, not live.** Figures come from a single historical window. Re-pull before acting on them. * **Query truncation.** Several over-threshold queries can hit the row cap; long tails may be cut. Treat head buckets as solid, tails as indicative. * **Threshold-gated.** Only sources above the est-requests threshold appear, so small genuine sources are invisible by construction — fine for a “where’s the volume” decision, not for a complete census. ## Open follow-ups [Section titled “Open follow-ups”](#open-follow-ups) * Online corroboration of the taxonomy (default `Referrer-Policy` behavior, HTTPS→HTTP referrer loss, native-app stripping, image-proxy fetch patterns) is a separate web-research pass. * Owner/ASN lookup on the bucket-1 datacenter IPs (one renderer vs. a scraper) decides whether they deserve a referrer-cohort entry. # Referrer cohort & delivery policy > Who gets the clean vs watermarked variant, and why each rule exists. Unlisted internal dev doc — audit before publishing publicly Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: define who gets the **clean** thumbnail vs the **watermarked** thumbnail, and why each rule exists. **Applies to:** the edge **watermark router** snippet (`apps/snippet/watermark-router.snippet.js`), the thumbnail **worker** (`apps/thumbnail/worker.ts`), the referrer **rollout cohort** (`apps/thumbnail/lib/referrer-rollout-cohort.ts`), and the paying-customer **bypass allowlists** (`apps/thumbnail/lib/customers.json`). For any concrete host, IP, customer, or video id referenced here by **role** or **code name**, decode privately in the internal decoder (not published). > Redaction note: this runbook is deliberately generic. It names system components and uses generic roles / code names only. Never paste a real host, email, IP, Stripe id, or video id into this file. *** ## 1. Two variants, one principle [Section titled “1. Two variants, one principle”](#1-two-variants-one-principle) Each thumbnail id resolves to two byte variants in R2: | Variant | R2 key shape | Who serves it | | ------------------ | -------------------------------------- | ------------------------------------- | | clean | `/thumbnails/{id}/clean.jpg` (private) | worker only; never publicly fetchable | | watermarked (`wm`) | `/thumbnails/{id}/wm.jpg` (public) | snippet 307-redirect **or** worker | **Determinism principle.** The variant a request receives is a **pure function of request signals** — referrer host, client IP, path, and (for one legacy loop) ASN/UA. The same request always yields the same variant. There is no randomness, no per-user A/B, no time-of-day flip: this keeps the **31-day referrer-blind page-rule cache** and **smart tiered cache** from poisoning one cohort with another cohort’s bytes. If the decision were non-deterministic, a cached `wm` could be served to a paying customer (or a clean leaked to a watermark cohort) on a cache-key collision. **Corollary — clean variant must stay private.** The public R2 bucket backs the redirect, so the `clean.jpg` keys must be blocked from public access (a Cloudflare rule matching `host` + `starts_with(path,"/thumbnails/")` + `ends_with(path,"/clean.jpg")`). If clean ever becomes publicly fetchable, every watermark decision is trivially bypassable. Keep that rule in sync — see the IMPORTANT block at the top of the snippet. *** ## 2. Decision order (evaluate top-down, first match wins) [Section titled “2. Decision order (evaluate top-down, first match wins)”](#2-decision-order-evaluate-top-down-first-match-wins) The router evaluates these in a fixed order. **Bypass is checked before the watermark decision**, so a paying customer is never watermarked even if their referrer would otherwise qualify. ```plaintext 1. Non-GET/HEAD ................................. pass through to worker 2. Root / favicon / malformed (`://` in path) .. pass through 3. Legacy `/.jpg` ............................... 308 -> shared error image 4. Already-public `/thumbnails/*` .............. pass through (already R2) 5. No extractable video id ..................... pass through 6. Known legacy WordPress self-loop (1 host) ... 307 -> cached wm 7. BYPASS: paying IP -> clean .................. pass through (checked FIRST) 8. BYPASS: no referrer -> clean ................. pass through 9. BYPASS: paying domain -> clean .............. pass through 10. WM cohort host + HOT id .................... 307 -> cached wm 11. Everything else ............................ pass through (worker decides) ``` Steps 7–9 are the **bypass tier**. Step 10 is the **watermark tier**. The dev/local rule (Section 4) and the rollout cohort (Section 5) layer onto this in the worker. *** ## 3. Bypass policies (who always gets clean) [Section titled “3. Bypass policies (who always gets clean)”](#3-bypass-policies-who-always-gets-clean) ### 3a. No referrer -> clean (explicit) [Section titled “3a. No referrer -> clean (explicit)”](#3a-no-referrer---clean-explicit) A request with **no `Referer` header** is served **clean**, by design — this is an explicit early return, not an accident of fall-through. Rationale (from the no-referrer traffic-source analysis): the no-referrer population is dominated by non-eyeballs — a **datacenter no-referrer source** (DC1/DC2) spoofing a browser UA from a couple of fixed IPs, plus generic SDK/library default UAs (`node`, `okhttp`, `Dart`, bare `Mozilla/5.0`). Real browsers with a stripped referrer are a minority. Serving them clean costs \~no conversion and avoids punishing privacy-stripped real users. A **CDN-internal IP** (CFINT) shows enormous tiered-cache volume and is excluded as infra noise, not real demand. ### 3b. Paying-customer bypass — by domain AND by client IP [Section titled “3b. Paying-customer bypass — by domain AND by client IP”](#3b-paying-customer-bypass--by-domain-and-by-client-ip) Paying customers are bypassed two independent ways, both evaluated **before** the watermark decision: | Key | Header / signal | Set | | --------------- | --------------------------------------------------------------- | ----------------------- | | client IP | `CF-Connecting-IP` → `True-Client-IP` → first `X-Forwarded-For` | paying-IP allowlist | | referrer domain | `Referer` host, normalized (lowercased, `www.`/`m.` stripped) | paying-domain allowlist | Either match short-circuits to clean. IP bypass covers server-side / no-referrer integrations a paying customer runs; domain bypass covers their browser embeds. The allowlists are sourced from `apps/thumbnail/lib/customers.json` (Stripe-synced: `stripeSubscriptionId`, `stripePriceId`, entitlement domains/IPs). See an internal entitlements runbook (not published) for how that file is populated. #### ⚠️ Failure mode: the unprotected paying customer [Section titled “⚠️ Failure mode: the unprotected paying customer”](#️-failure-mode-the-unprotected-paying-customer) A paying Stripe record with **empty domain/IP entitlements** receives **zero bypass** — it falls straight through to the watermark tier and gets watermarked despite paying. This is a real rollout blocker (code name **C02**, an unprotected paying customer; contrast **C01**, correctly protected). The webhook can create the subscription record while the entitlement fields stay blank. Detection / fix: * After any customer sync, assert every active subscription has at least one non-empty entitlement (domain or IP) before rollout. * Treat an active `stripeSubscriptionId` with empty domain **and** empty IP entitlements as a paging condition, not a warning. * Backfill the entitlement from the customer’s known embed host or origin IP, then redeploy the snippet/worker config. ```bash # Sketch: flag active subscriptions with no bypass entitlement. # (Run against the synced customers file; do not print PID into shared logs.) bun run scripts/cloudflare/... # entitlement audit entry point ``` *** ## 4. Dev / local -> clean (detected by REFERRER HOST, not user-agent) [Section titled “4. Dev / local -> clean (detected by REFERRER HOST, not user-agent)”](#4-dev--local---clean-detected-by-referrer-host-not-user-agent) Local and preview development is served **clean**, detected from the **referrer host**, never from the user-agent. Why referrer host, not UA: dev traffic mostly arrives **with** a `localhost`/`127.0.0.1` referrer, and UA-based dev detection collides with the large server/library no-referrer bucket (Section 3a). A referrer-host rule is therefore high-precision; a UA rule is not. Dev/local referrer-host signals (treated as clean): | Signal | Examples | | ------------------------- | ----------------------------------------------- | | loopback | `localhost`, `127.0.0.1`, `[::1]` | | dev TLDs | `*.local`, `*.test` | | private ranges (RFC-1918) | `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` | | tunnels / previews | `ngrok` hosts, preview/staging subdomains | > Hardening note: the live snippet’s `shouldWatermarkRequest` reaches dev-clean implicitly today (a dev referrer simply isn’t in any watermark cohort, so it falls through to clean). Making it **explicit** — an early dev-host return — is the safer form, because it stays clean even if a dev host is ever accidentally added to a watermark cohort. Prefer the explicit rule. *** ## 5. The referrer cohort (who is *eligible* to be watermarked) [Section titled “5. The referrer cohort (who is eligible to be watermarked)”](#5-the-referrer-cohort-who-is-eligible-to-be-watermarked) The watermark tier only fires for hosts in the **referrer cohort** — an explicit, reviewed allowlist of attributable, high-volume referrer domains plus a small prefix family. It is **opt-in by host**, never blanket. * Source of truth: `apps/thumbnail/lib/referrer-rollout-cohort.ts` (`[domain, monthlyEstimatedRequests]` pairs + a prefix list, e.g. an auction-app family prefix). * Matching: normalize the referrer host (lowercase, trim, drop trailing dots, strip `www.`), then check the domain set and the prefix list. * Inclusion threshold: a candidate must estimate **≥ 50,000 requests / 30 days** and be **attributable** (a specific host/app/family), so the watermark reads as intentional attribution rather than accidental punishment of someone prototyping. **Developer-friendly bias:** ambiguous or small direct traffic stays **clean** by default. New integrations are not watermarked until they are clearly large and attributable. This is why the no-referrer default flipped to clean (Section 3a) and why generic runtimes, crawlers, and shared optimizers are explicitly excluded from the cohort. ### Snippet HOT set gating [Section titled “Snippet HOT set gating”](#snippet-hot-set-gating) Even for an eligible cohort host, the **snippet** only 307-redirects to the cached `wm` when the id is in the **HOT/redirect set** (pre-generated `wm` bytes exist in R2; kept under the 32 KB snippet size limit). A cohort host requesting a **cold** id falls through to the worker, which makes the same deterministic decision and generates/serves `wm` on the miss path. The HOT set is purely a performance pin (skip the worker for the hottest ids), **not** a policy boundary — policy is identical hot or cold. *** ## 6. Self-referred WM\_DOMAINS (deterministically watermarked) [Section titled “6. Self-referred WM\_DOMAINS (deterministically watermarked)”](#6-self-referred-wm_domains-deterministically-watermarked) A small set of **self-referred school/health hosts** (the snippet’s `WM_DOMAINS`; code names **S01–S04**) are **always** watermarked when they request a HOT id. These are large, stable, attributable referrers whose own pages embed the thumbnails; they are a deterministic always-on subset of the cohort, separate from the gradual rollout list. They are checked **after** the paying-domain bypass, so a `WM_DOMAINS` host that later becomes a paying customer is bypassed correctly without removing it from the set. *** ## 7. Cohort roster (roles — decode internally) [Section titled “7. Cohort roster (roles — decode internally)”](#7-cohort-roster-roles--decode-internally) Use roles in conversation; reserve code names for disambiguation. Full decode lives in the internal decoder (not published). | Code | Role | | --------- | ---------------------------------------------------------------------- | | H01 | the **canary host** — first flip, watched for still-leaking-clean | | H02 | single-hero-video host, already pinned | | H03 | rotating auction-catalog host, high cache decay | | H08 | highest-volume host (\~4M/mo), rolled **last and alone** | | H09 | \~1.34M/mo host with a second subdomain | | H20–H24 | non-buyer / cost-control hosts | | S01–S04 | self-referred school/health hosts (`WM_DOMAINS`) | | K01 | gray-market network | | C01 | protected paying customer (bypass works) | | C02 | **unprotected** paying customer (empty entitlements — rollout blocker) | | DC1 / DC2 | datacenter no-referrer sources | | CFINT | excluded CDN-internal IP (tiered-cache noise) | *** ## 8. Operator checklist [Section titled “8. Operator checklist”](#8-operator-checklist) Before flipping or expanding a cohort: * [ ] Clean variant (`/thumbnails/{id}/clean.jpg`) is confirmed **private** (bypass-prevention rule live). * [ ] Every active subscription has a non-empty domain **or** IP entitlement (no C02-class records). * [ ] New cohort host is **≥ 50k/30d** and attributable; not a runtime/crawler/optimizer. * [ ] Dev/local hosts are **not** present in any watermark cohort. * [ ] HOT ids for the new host exist in R2 (or accept worker miss-path generation). * [ ] Roll the **highest-volume host alone and last** (H08), after the canary (H01) is verified clean-free. * [ ] Decode any concrete entity from the internal decoder (not published), never from this file. To undo a flip, see [reverting-deploys](/runbooks/reverting-deploys/) (snippet redeploy / cohort rollback is a config/data operation, not an emergency code change). *** ## Related [Section titled “Related”](#related) * An internal entitlements runbook (not published) — how `customers.json` entitlements get populated. * [reverting-deploys](/runbooks/reverting-deploys/) — rolling back a snippet or cohort change. * The internal decoder (not published) — private decode of every role / code name above. # Reverting deploys > Dual-lane rollback, kill switches, and residual-risk cleanup. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: revert a watermark/delivery change safely when delivery is split across an edge snippet (watermark router) and a backing worker. The core rule is **dual-lane**: a full revert needs the snippet turned off **and** the worker referrer cohort backed off. Turning off only the snippet routes traffic back to the watermarking worker — it does **not** restore clean delivery and can **increase** cost. Applies to: any host whose thumbnails are delivered through the edge snippet’s HOT/redirect set (307 → R2 `wm.jpg`) while the same host sits in the worker’s referrer rollout cohort at `rollout=1, variant=wm`. For the concrete host/customer/zone identities behind every role below, see the internal decoder (not published). Sibling runbooks: [flipping-hosts-to-watermarked](/runbooks/staged-rollouts/), [purge-cost-safety](/runbooks/cost-safe-cache-purging/), [hot-set-decay](/runbooks/hot-set-management/). *** ## Why revert is dual-lane [Section titled “Why revert is dual-lane”](#why-revert-is-dual-lane) There are two independent delivery surfaces, and they fail back differently: | Surface | What it does | Disabling it alone | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | Edge snippet (watermark router) | Matches `WM_DOMAINS` + a HOT id set, 307s to `cache./thumbnails/{id}/wm.jpg` | Routes traffic **back to the worker**, which is still in-cohort and re-watermarks | | Worker referrer cohort | Watermarks in-path and **re-pins** `wm` on the bare `/{id}.jpg` key under the 31-day referrer-blind page-rule cache | Returns the host to the clean/R2-cheap path for non-HOT ids | Consequences that make single-lane reverts wrong: 1. **Snippet disable does NOT restore clean.** While a host stays at `rollout=1`, disabling the snippet just moves its traffic from the snippet’s 307 path onto the watermarking worker. Viewers still get watermarked images. 2. **Snippet disable can INCREASE cost.** The snippet 307 path is served from R2 (cheap). The worker path bills worker invocations + KV reads (a much higher blended rate). Disabling the snippet on a high-volume cohort host swings the whole HOT-307 volume back onto the worker — the single largest spend swing in this system. Never reach for snippet-disable as a cost brake. 3. **Reverse-leak purges don’t stick while the host is in-cohort.** Any non-HOT id re-pins `wm` on its bare key on the next request after a purge. Clean is only durable once the host is **out of the cohort**. > Rule of thumb: **snippet off = “stop redirecting”; cohort off = “stop watermarking.”** You usually need both, and the order matters (see [Kill action ordering](#kill-action-ordering)). *** ## Revert tiers [Section titled “Revert tiers”](#revert-tiers) Pick the smallest tier that addresses the incident. ### Tier 0 — Global kill switch (fastest stop, one API call) [Section titled “Tier 0 — Global kill switch (fastest stop, one API call)”](#tier-0--global-kill-switch-fastest-stop-one-api-call) Disable the snippet rule. This is the fastest way to stop the snippet 307 path, but it is **not** a clean rollback and it is **cost-increasing** while hosts remain in the cohort. ```bash CONFIRM_DISABLE=YES \ CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_ID= \ bun scripts/cloudflare/snippet-rollback.ts disable ``` What it does: fetches all snippet rules, flips `enabled=false` on the watermark-router rule only (unrelated rules preserved), and `PUT`s the full list back. It refuses unless `CONFIRM_DISABLE=YES` and aborts if no matching rule is found. After a `disable` you **must** also back off the worker cohort, or every affected host keeps getting watermarked by the worker (and now bills the worker path). Disable is a stop-gap, never the whole revert. ### Tier 1 — Per-host fast revert (\~3–6 min + propagation) [Section titled “Tier 1 — Per-host fast revert (\~3–6 min + propagation)”](#tier-1--per-host-fast-revert-36-min--propagation) The correct full revert for a single host. Both lanes, same release: 1. **Snippet lane** — remove the host from `WM_DOMAINS` **and** from the HOT/redirect set in the snippet source. Remove subdomain entries separately (domain normalization strips only `www.`/`m.`, so a regional or sub-brand subdomain — e.g. `eu.`, `shop.` — is its own entry). 2. **Worker lane** — remove the host from the referrer rollout cohort (`referrer-rollout-cohort.ts`) **or** set its `rollout=0`. 3. **Gate** — run unit/contract tests + lint; verify the snippet size budget still passes; capture a fresh rollback snapshot (Tier 3). 4. **Deploy both** — redeploy the snippet **and** redeploy the worker. Neither alone is sufficient. 5. **Verify with a real id** — smoke with the host’s **actual** id (not a generic `hotIds[0]` placeholder) and assert the served variant is **clean**: * a current host id → does **not** 307, served variant **clean**; * a known non-HOT id with the host referrer → if it still serves `wm`, the worker is still watermarking → the cohort backoff didn’t land; fix and re-verify before calling it reverted. ```bash # after editing snippet source + cohort source bun --filter @repo/snippet test && bun run lint:touched && bun run lint bun --filter @repo/snippet deploy # snippet lane bun --filter @repo/thumbnail deploy # worker lane (cohort change) # verify a REAL id serves clean: WM_REFERER=https:/// HOT_REDIRECT_ID= \ bun scripts/cloudflare/smoke-watermark-snippet.ts ``` Revert highest-volume / rotating-catalog hosts first when a global signal is ambiguous — they carry the most exposure and the most decay. The canary and the fast-rotating auction-catalog host are the most likely revert targets. ### Tier 2 — Content rollback = version control [Section titled “Tier 2 — Content rollback = version control”](#tier-2--content-rollback--version-control) There is **no one-command content rollback**. The snippet source is not under Worker Version Management. Restoring routing **rules** (the snapshot `restore-rules` command) restores *which rules are enabled and their expressions* — it does **not** restore snippet **source content**. Content rollback is a git operation against a committed pre-flip revision: ```bash # 1) restore the rule list (enabled state + expressions) from the latest snapshot CONFIRM_RESTORE=YES \ CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_ID= \ bun scripts/cloudflare/snippet-rollback.ts restore-rules \ artifacts/cloudflare/snippet-rollback.latest.json # 2) restore the SOURCE from the pre-flip commit, then redeploy git checkout -- apps/snippet/watermark-router.snippet.js bun --filter @repo/snippet deploy ``` Because git is the only content rollback path, a committed pre-flip revision is a **hard deploy gate** — capture the snapshot and commit the snippet + cohort edits *before* deploying, and log the pre-flip hash. > Do **not** `biome format` / reflow the snippet during any revert. The safety parsers string-scan `const HOT = new Set([…]);`; reformatting can blind the parser and produce false-green gates. ### Tier 3 — Rollback snapshot (capture before you need it) [Section titled “Tier 3 — Rollback snapshot (capture before you need it)”](#tier-3--rollback-snapshot-capture-before-you-need-it) The snapshot captures Cloudflare state git cannot represent: the deployed snippet content hash and the full snippet rule list, plus the current git head/patch for the source. ```bash CLOUDFLARE_API_TOKEN=... CLOUDFLARE_ZONE_ID= \ bun scripts/cloudflare/snippet-rollback.ts snapshot # → artifacts/cloudflare/snippet-rollback.latest.json ``` Before any deploy that you might revert, assert the snapshot’s `generatedAt` is from **this** change (a stale snapshot restores the wrong rules). The deploy itself is non-atomic (content `PUT` then rule `PUT`, no retry) — after deploy, verify live `deployedContentSha256 == local sha` **and** the rule is enabled with the expected expression before trusting it. *** ## Kill action ordering [Section titled “Kill action ordering”](#kill-action-ordering) When stopping an in-flight incident, **order matters** because snippet-disable is cost-inverting: | Step | Action | Why this order | | ---- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 1 | **Stop the bleed** — if a purge loop is running, stop it first | Reversible and cheap | | 2 | **Back off the cohort** — remove the host from `referrer-rollout-cohort.ts` + redeploy the worker | Returns the host to the non-watermarked, R2-cheap path; this is what actually restores clean | | 3 | **Disable the snippet — LAST, only if still needed** | Snippet-disable alone neither restores clean nor reduces worker cost; doing it first routes the full HOT-307 volume back onto the worker (the largest spend swing) | A true global stop = **stop purge + cohort backoff + (only if needed) snippet disable as the final step.** Never disable the snippet first. *** ## Residual risk after revert [Section titled “Residual risk after revert”](#residual-risk-after-revert) Reverting source + rules does not instantly restore every viewer. Track these: * **Cached 307s linger until TTL.** A reverted host keeps redirecting already-cached viewers to `wm.jpg` until the cached redirect expires — **unless** the 307 carries `Cache-Control: no-store`. If the redirect does not set `no-store`, treat revert latency as the redirect cache TTL (effectively unbounded for high-volume hosts). Add `no-store` to the 307 before flipping any >1M/mo host so revert latency stays bounded. * **Never delete a still-referenced variant object.** Do not delete a warm `wm.jpg` that a still-HOT id is redirecting to — deleting it 404s the host. Remove the id from HOT (so nothing redirects to the object) *before* considering any object deletion. * **Purges are never auto-undone.** A bare-key purge is one-directional; there is no rollback of a purge. If you purge into a colo set that hasn’t picked up the revert yet, those colos re-pin from the worker. Confirm the snippet/cohort change has propagated (multi-colo quorum, not a 2-colo sample) before purging anything during a revert. * **Close-gate re-probe.** Re-probe each bare `/{id}.jpg?permcheck=` and require variant **clean** for every id; re-probe a known non-HOT id with the referrer — if it serves `wm`, cohort overlap remains. Do not declare the revert complete until both are clean. *** ## Special cases [Section titled “Special cases”](#special-cases) * **Unprotected paying customer.** A paying customer with empty entitlements (no domains/IPs populated) is **not** structurally protected by the paying-bypass check and is a rollout blocker, not a revert concern — but if one slipped through, a revert (cohort backoff + snippet removal) is the immediate mitigation. See the internal decoder (not published) for the affected customer role. * **Extensionless ids.** Hosts that embed extensionless ids bypass the `*.jpg*` page rule and re-warm at default TTL, not 31 days. Do not purge extensionless paths during a revert (there is no stale `wm` key to clear). * **Redirect target on the cache zone.** The `*.jpg*` 31-day page rule lives on the bare zone and does **not** match the cache zone. A `wm.jpg` that 404s on the cache zone is not edge-pinned and re-fetches R2 on every request — verify `wm.jpg` is 200 (R2 HEAD on the bucket binding, authoritative) before purging its bare key. *** ## Quick reference [Section titled “Quick reference”](#quick-reference) ```text Stop only → snippet disable (Tier 0) [NOT a clean revert; cost-increasing] Restore one host → snippet remove + cohort backoff + deploy both + verify real id clean (Tier 1) Restore source → restore-rules (rules) + git checkout pre-flip commit + deploy (Tier 2) Before any deploy → capture snapshot + commit pre-flip revision (Tier 3) Incident order → stop purge → cohort backoff → snippet disable LAST ``` Decode any concrete host, customer, zone, or id role in the internal decoder (not published). # Safe deploys > Pre-flight gates, non-atomic deploy handling, and post-deploy verification. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. One-line purpose: deploy the edge **watermark router snippet** (and the thumbnail **worker**) without serving 404s, leaking clean assets, breaking paying/no-referrer traffic, or losing the ability to roll back. **Applies to:** anyone deploying `apps/snippet/watermark-router.snippet.js` via the Cloudflare Snippets API, or the thumbnail worker behind the same zone. For concrete hosts, ids, zone ids, and customer identities, see the internal decoder (not published) — this doc carries none of them. Sibling runbooks: [watermark-system](/runbooks/watermark-system/) (architecture), [referrer-cohort-and-delivery-policy](/runbooks/referrer-cohort-and-delivery-policy/), [no-referrer-traffic](/runbooks/no-referrer-traffic/), [reverting-deploys](/runbooks/reverting-deploys/), an internal entitlements runbook (not published). *** ## System model (read this first) [Section titled “System model (read this first)”](#system-model-read-this-first) The edge snippet is a referrer-aware router that sits in front of the thumbnail worker on the bare-image path (`/{id}.jpg`). It holds two string-scanned data structures the safety tooling depends on: * **HOT set** — `const HOT = new Set([ "", ... ]);` — the redirect-eligible ids. * **WM\_DOMAINS** — the referrer cohort (self-referred hosts) whose traffic is eligible for the watermark redirect. On a request the snippet decides, in this order: paying-bypass (evaluated **before** WM\_DOMAINS — structurally safe) → watermark-referrer + HOT id → **307** to the public R2 object `…/thumbnails/{id}/wm.jpg`. Anything else (no referrer, non-watermark referrer, non-HOT id) **passes through to the worker**. Key invariants the gates protect: | Invariant | Why it matters | | --------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | Every HOT id has a public `wm.jpg` in R2 | A 307 into a missing object is a 404 storm. | | The bare `/{id}.jpg` key ends **clean-only**; `clean.jpg` in R2 is **not** public | Determinism + no clean-asset bypass. | | Snippet stays under the size cap and is **never reformatted** | Safety parsers string-scan the HOT block; a formatter reflow blinds them → false-green gates. | | A committed pre-flip git revision exists | Git is the **only** content rollback (see [Rollback](#rollback)). | | Deployed content-hash == local, rule enabled with expected expression | The deploy is **non-atomic** (content PUT then rule PUT, no retry). | A 31-day **referrer-blind page-rule cache** sits over the bare key. Consequence: a reverted host keeps redirecting cached viewers until the cached response (the 307, and the pinned variant) expires. The snippet is **not** under Worker Version Management — snippet rollback is a separate procedure from worker version rollback. *** ## The gate ladder [Section titled “The gate ladder”](#the-gate-ladder) Run top-to-bottom. Each gate is fail-closed: a failure blocks the deploy. Add a test/HOT case per newly-flipped host before starting. ### G1 — Unit / contract tests [Section titled “G1 — Unit / contract tests”](#g1--unit--contract-tests) ```bash bun --filter @repo/snippet test ``` Cover the router’s redirect, pass-through, paying-bypass, and no-referrer contracts. One case per host you are flipping. ### G2 — Size guard (32KB) + projected-size budget + NEVER-REFORMAT [Section titled “G2 — Size guard (32KB) + projected-size budget + NEVER-REFORMAT”](#g2--size-guard-32kb--projected-size-budget--never-reformat) The Snippets API rejects payloads over **32,768 bytes** (`MAX_SNIPPET_BYTES`); the deploy script asserts this before upload. But the cap is a hard wall, not a target — budget against it: ```bash bun --filter @repo/snippet lint # forbidden-pattern + size lint wc -c apps/snippet/watermark-router.snippet.js ``` * Fail if **projected** post-edit size > **28,672** bytes (keep a \~4KB margin under the 32KB cap for the next batch). * **NEVER run a formatter on the snippet during a deploy.** The safety parsers locate the HOT set by string-scanning the literal `const HOT = new Set([` … `]);` and pull ids with a quoted-string regex (see [Parser fragility](#parser-fragility)). A reflow that wraps, re-indents, or re-quotes that block produces gates that pass while validating the wrong content. Formatting this file is forbidden during rollout. ### G3 — Repo lint [Section titled “G3 — Repo lint”](#g3--repo-lint) ```bash bun run lint:touched && bun run lint ``` ### G4 — Fresh rollback snapshot + committed pre-flip revision [Section titled “G4 — Fresh rollback snapshot + committed pre-flip revision”](#g4--fresh-rollback-snapshot--committed-pre-flip-revision) Git is the only content rollback, so this gate is mandatory and cannot be skipped. ```bash CLOUDFLARE_API_TOKEN=… CLOUDFLARE_ZONE_ID=… \ bun scripts/cloudflare/snippet-rollback.ts snapshot ``` The snapshot captures what git cannot: deployed content sha256, the full snippet-rule list, and current git state. Then: * Assert the snapshot’s `generatedAt` is **this batch** (a stale snapshot restores stale rules). * Commit the snippet (and any worker/cohort) edits. * Log the **pre-flip commit hash** — this is your content-rollback target. ### G5 — HOT-set `wm.jpg` existence verify + parse-count cross-check [Section titled “G5 — HOT-set wm.jpg existence verify + parse-count cross-check”](#g5--hot-set-wmjpg-existence-verify--parse-count-cross-check) ```bash bun scripts/cloudflare/verify-watermark-snippet-hotset.ts verify # 0 missing ``` `verify` HEADs every HOT id’s public `wm.jpg` and exits non-zero on any missing/non-200. Use `verify` as the gate; `ensure` (write path, `CONFIRM_ENSURE=YES`) is for backfill, not gating. **Cross-check the parse count** to catch a reflow-blinded parser (G2): ```plaintext parsed HOT count == rg -c '"[^"]+"' (quoted ids in the HOT block) == pre-edit count + ids added ``` If the parser reports fewer ids than `rg` sees, the HOT block was reformatted and the verify is validating a subset. Stop and re-check G2. ### G6 — Multi-id clean-bypass block check [Section titled “G6 — Multi-id clean-bypass block check”](#g6--multi-id-clean-bypass-block-check) For **≥1 id per host**, HEAD the public R2 `clean.jpg`: ```plaintext HEAD https:///thumbnails/{id}/clean.jpg ``` Any **2xx** = clean asset is publicly reachable = bypass of the watermark = **hard ABORT**. (The verify script’s `CHECK_CLEAN_BLOCK=true` does a single-id version; the gate wants multiple ids across hosts.) ### G7 — Predeploy route guard [Section titled “G7 — Predeploy route guard”](#g7--predeploy-route-guard) Required whenever the release also edits worker source (e.g. cohort changes). Fail-closed on a stale snapshot, missing expected route, drift, collision, or normalization mismatch: ```bash bun scripts/cloudflare/predeploy-thumbnail-guard.ts ``` Confirms the worker’s declared routes match live Cloudflare routes and that the route snapshot is fresh. A stale snapshot blocks — refresh it first. *** ## Deploy (non-atomic — assert before smoke) [Section titled “Deploy (non-atomic — assert before smoke)”](#deploy-non-atomic--assert-before-smoke) The deploy writes content, then writes the rule. **Two separate PUTs, no retry, no transaction.** A failure between them leaves the snippet content updated but the rule unchanged (or vice versa). ```bash CLOUDFLARE_API_TOKEN=… CLOUDFLARE_ZONE_ID=… \ bun --filter @repo/snippet deploy ``` The deploy upserts the snippet’s rule while **preserving all other rules in the zone** (it replaces the matching `snippet_name` in place, or prepends if absent). **Post-deploy, BEFORE smoke — assert the deploy actually landed:** 1. **Content hash:** fetch live deployed content, sha256 it, assert it equals the local source sha. (`snippet-rollback.ts snapshot` computes both `localSourceSha256` and `deployedContentSha256` — they must match.) 2. **Rule state:** the matching rule is `enabled: true` with the **expected expression** (host match). If either fails → the non-atomic deploy half-applied. Targeted `disable`, redeploy, re-snapshot. Do **not** smoke a half-applied deploy. Allow a short propagation soak before smoke — a colo where the rule hasn’t taken yet still routes to the worker. *** ## Smoke (the real flip gate) [Section titled “Smoke (the real flip gate)”](#smoke-the-real-flip-gate) Smoke runs against live traffic. `health.ts` (often auto-run by deploy) does **not** assert flip success — smoke is the only gate that does. ```bash EDGE_BASE_URL=https:// R2_BASE_URL=https:// \ WM_REFERER=https:///page \ bun scripts/cloudflare/smoke-watermark-snippet.ts ``` Asserts, per the snippet contract: | Case | Expectation | | --------------------------------------------------------- | ----------------------------------------------------------------------- | | HOT id + watermark referrer | **307** → R2 `…/{id}/wm.jpg`, and that object is a live 2xx `image/*` | | HOT id + size/numeric suffix (`{id}_large`, `{id}_12345`) | 307 to the **base** id’s `wm.jpg` | | HOT id + **no referrer** | **no** redirect → continues to worker (`X-Thumbnail-*` headers present) | | HOT id + **non-watermark** referrer | no redirect, served variant `clean` | | **Non-HOT** id + no referrer | no redirect → worker | | R2 `clean.jpg` | **not** publicly accessible | > **Per-host smoke caveat:** by default the smoke picks `hotSample[0] ?? hotIds[0]` for the redirect case **regardless of `WM_REFERER`** — so a per-host smoke is non-validating unless you pin the host’s real id (e.g. a `HOT_REDIRECT_ID` override, if available; otherwise order the HOT set so the host’s id is first, or smoke that id directly). Run subdomain entries twice (each subdomain is its own WM\_DOMAINS/HOT entry). Re-confirm the paying-bypass and no-referrer controls on every batch. *** ## Rollback [Section titled “Rollback”](#rollback) Snippet rollback is **separate** from Worker Version Management and is **dual-lane** when a flipped host is also in the worker referrer cohort: disabling the snippet rule only routes traffic **back to the worker**, which may still watermark. Full rollback = snippet off **and** worker cohort backed off **and** bare keys handled. **Fastest stop (snippet rule off):** ```bash CONFIRM_DISABLE=YES CLOUDFLARE_API_TOKEN=… CLOUDFLARE_ZONE_ID=… \ bun scripts/cloudflare/snippet-rollback.ts disable ``` **Restore the full rule list from a snapshot:** ```bash CONFIRM_RESTORE=YES … \ bun scripts/cloudflare/snippet-rollback.ts restore-rules \ artifacts/cloudflare/snippet-rollback.latest.json ``` **Content rollback = git** (there is no one-command content restore): ```bash git checkout -- apps/snippet/watermark-router.snippet.js bun --filter @repo/snippet deploy ``` Residual risk: cached 307s and the pinned variant linger under the **31-day referrer-blind page-rule cache** until expiry unless the 307 carries `Cache-Control: no-store`. Treat revert latency for high-volume hosts as the redirect cache TTL. Never delete a still-HOT id’s warm `wm.jpg` (that 404s the host). Purges are not auto-undone. See [reverting-deploys](/runbooks/reverting-deploys/) for the worker-side version rollback. *** ## Parser fragility (why NEVER-REFORMAT exists) [Section titled “Parser fragility (why NEVER-REFORMAT exists)”](#parser-fragility-why-never-reformat-exists) Three independent scripts string-scan the snippet rather than parse it: * The hotset verifier and the smoke checker both locate the literal tokens `const HOT = new Set([` and `]);`, slice between them, and extract ids with a `"([^"]+)"` regex. * The size guard counts raw bytes. None of this survives a formatter pass that re-wraps, re-indents, splits, or re-quotes the HOT block. A reflow can leave a **passing** verify that validated a subset (or zero) ids — a false green. The G5 parse-count cross-check (`parsed == rg-count == expected`) exists specifically to catch this. **Do not format the snippet during a deploy.** *** ## Quick checklist [Section titled “Quick checklist”](#quick-checklist) * [ ] G1 tests · G2 size + no-reformat · G3 lint · G4 snapshot + committed pre-flip hash * [ ] G5 hotset verify (0 missing) + parse-count cross-check * [ ] G6 multi-id clean-bypass HEAD (no 2xx) · G7 predeploy route guard (if worker edited) * [ ] Deploy → assert deployed-hash == local AND rule enabled w/ expected expression → soak * [ ] Smoke green (with host’s real id) before any purge * [ ] Rollback is dual-lane; content rollback = git; concrete entities in the internal decoder (not published) # Staged rollouts > Blast-radius ordering, canary batches, soak windows, and advance gates. Unlisted internal dev doc — audit before treating as public Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. > **Purpose:** Roll a delivery-path change (e.g. clean → watermarked, or worker-path → edge-redirect) out across a cohort of third-party hosts without a broad incident or a runaway bill. > > **Applies to:** any change where one shared edge surface serves many distinct hosts at different volumes, the variant decision is made upstream of a long-lived edge cache, and a mistake is served *per request* until cache expiry. Concrete hosts, ids, domains, and account ids live in the internal decoder (not published) — this doc stays generic. See also: [reverting-deploys](/runbooks/reverting-deploys/), [edge-cache-and-purge](/runbooks/cost-safe-cache-purging/), [cost-safety-gates](/runbooks/cost-safe-cache-purging/). *** ## 1. Operating principles [Section titled “1. Operating principles”](#1-operating-principles) 1. **Order by incident exposure, not dollar cost.** Exposure = `volume × fraction-served-wrong`. A high-volume host that is mostly already correct can rank *below* a smaller host whose every request is wrong. Cost ranking and exposure ranking are different orderings — use exposure for sequencing and treat cost as a separate gate (§6). 2. **Smallest live blast radius first.** Canary the host with the smallest exposure *and* an observable change — prefer a host whose delivery actually flips (mixed/live) over one already in the target state, so the canary exercises the real code path. 3. **Highest-volume host last and alone, ramped one id at a time.** The largest host is a batch of one, flipped after every smaller host has soaked. For it, ramp by id: top id first, soak, then add the rest. 4. **Every change reversible — and reversibility may be multi-lane.** If the variant decision lives in two places (e.g. an edge snippet *and* an upstream worker cohort), turning off one lane does not restore baseline. Full rollback = both lanes backed off *plus* any pinned cache purged. Confirm which lanes exist before you start (§7). 5. **A committed pre-flip revision is a hard deploy gate.** If there is no one-command content rollback, git is the rollback. Log the pre-flip commit hash before deploying. 6. **Smoke is synthetic and single-id; it is not a volume health signal.** If the flipped path is a redirect that skips the worker, worker analytics *undercount* flipped traffic. Confirm with real-traffic proxies (object-store GET volume on the target key, coverage reports), not worker invocation counts. 7. **Don’t reformat the artifacts your safety checks parse.** If gates string-scan a hand-formatted set/list (e.g. `const HOT = new Set([…])`), a formatter run mid-rollout can blind the parser and produce false-green gates. Freeze formatting on those files during a rollout. *** ## 2. Order the cohort by exposure [Section titled “2. Order the cohort by exposure”](#2-order-the-cohort-by-exposure) Build a per-host risk table before sequencing. Decode roles/code names in the internal decoder (not published). | Column | Meaning | | ----------------- | ------------------------------------------------------------------------------------------------- | | `req/mo` | Traffic volume — one factor in exposure. | | Incident exposure | `volume × fraction-served-wrong` — the sort key. | | Catalog shape | Stable / single-hero / rotating. Drives decay risk. | | Decay | How fast the host’s live ids rotate out of the pinned set. | | Subdomain trap | Does a subdomain need its own cohort/allowlist entry? (Normalizers often strip only `www.`/`m.`.) | | Visibility | Is the asset on an operator-visible surface, and is the rendered id known? Unverified = gated. | Representative role ordering (smallest exposure first): | Batch | Role | Soak | Why this slot | | -------- | ---------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------- | | 0 canary | the canary host (smallest exposure, *live/mixed* delivery) | 48h | only host whose delivery actually flips; smallest blast radius; exercises any subdomain trap once | | 1 | a single-hero host already in the pinned set | 24h | lowest surprise — the one id is already covered | | 2 | a batch of stable low-volume hosts | 24h | each independently smoked; revert all on any one regression | | 3 | a rotating-catalog host, high decay | 48h | isolated; decay-monitored alone; blocked until the decay alarm is wired | | 4 | first >1M host (+ its second subdomain) | 72h | subdomain census; each subdomain is its own entry | | 5 | the highest-volume host, **alone, last** | 72h | highest exposure; ramp by id — top id first, soak, then add | | 6 tail | bundle of trivial-volume hosts | final | each gated on operator-visibility first | > **No per-host percentage knob.** If “staging” means editing a set and redeploying, then a deploy’s blast radius = every host added since the last deploy. Add hosts in the batches above, never in bulk. *** ## 3. Pre-flight gates [Section titled “3. Pre-flight gates”](#3-pre-flight-gates) Split gates into **global** (run once, program-blocking), **per-deploy-batch**, and **per-host**. ### Global G0 (run once — program-blocking) [Section titled “Global G0 (run once — program-blocking)”](#global-g0-run-once--program-blocking) | Gate | Check | Fail action | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | | Operator visibility / live variant | Record the *current* live variant per host (probe a non-pinned id with the host’s referrer and a cache-buster). Establishes whether the flip is a genuine baseline→target change or just a delivery-path change. | document per host | | Auth probes | Object-store / CDN API credentials work *before* any source edit. Failing auth after editing the pinned set strands you mid-flip. | fix creds first | | **No unprotected paying host** | Every paying customer’s bypass entitlements are populated. A paying host with *empty* entitlements is structurally UNPROTECTED and a hard program blocker. | ABORT until populated or waived in `docs/` | | Fail-closed dev/test scope | Any “exempt dev/test referrer” branch either exists (with tests) or is explicitly descoped. Don’t assume it exists. | implement or descope | | Decay alarm wired | A coverage-floor check exits non-zero below floor (e.g. 0.85 rotating / 0.80 stable). Report-only tooling is not an alarm. | wire it or accept written manual daily re-verify | | Policy sign-offs | Any policy that the flip depends on (e.g. no-referrer handling) has a dated approval doc in `docs/`. | fail-closed | | Cost estimate committed | If spend could shift by ≥ the org threshold, the estimate is committed to `docs/` *before* flipping. See [cost-safety-gates](/runbooks/cost-safe-cache-purging/). | commit first | | Redirect cacheability | If the flip emits a redirect, confirm it carries `Cache-Control: no-store` (or treat revert latency as the redirect’s cache TTL) before flipping any high-volume host. | add `no-store` first | ### Per-deploy-batch (G1–G7) [Section titled “Per-deploy-batch (G1–G7)”](#per-deploy-batch-g1g7) * **G1** unit/contract tests — add a case per flipped host. * **G2** artifact-size + forbidden-pattern lint — if the edge artifact has a hard byte cap, assert `projected < cap − margin`. * **G3** repo lint (`bun run lint:touched && bun run lint`). * **G4** fresh rollback snapshot + committed pre-flip revision — assert the snapshot timestamp is *this* batch; log the pre-flip hash. * **G5** target-asset existence + **parse-count cross-check** — verify every pinned id’s target object exists (0 missing) *and* that the parsed count equals an independent `rg -c` count equals `pre-edit + added`. The cross-check catches a reflow-blinded parser (see principle 7). * **G6** baseline-bypass block, multi-id — the baseline variant must NOT be publicly fetchable for a flipped id from ≥1 id per host. Any success = hard ABORT. * **G7** predeploy route guard — if this release also edits upstream (worker) source, run the route/collision guard. ### Per-host (H0–H3) [Section titled “Per-host (H0–H3)”](#per-host-h0h3) * **H0 operator visibility** (fail-closed for high-volume + tail): render the host’s key pages, capture evidence the asset lands on an operator-visible surface *and* the actual rendered id(s). For high-volume hosts, assert each rendered id is pinned with its target object 200 from ≥3 colos; block if rendered id ≠ assumed id. * **H1 allowlist↔pinned-set consistency**: host’s normalized domain in the bypass/flip allowlist AND every current live id (re-pulled same-day for rotating hosts) in the pinned set. Subdomains are separate entries. * **H2 not a paying host**: host ∉ paying domains and shares no paying IP. * **H3 top id has a live target object** (200). If 404-but-exists, purge that one URL and re-HEAD. *** ## 4. Per-batch rollout procedure [Section titled “4. Per-batch rollout procedure”](#4-per-batch-rollout-procedure) Ordering is load-bearing — each step removes a way the *next* step can land traffic on the wrong path. ```plaintext backfill → verify+count-check → remove from upstream cohort (same release) → size guard → snapshot+commit → deploy → soak+multi-colo smoke → per-host real-id smoke → purge → close gate → observe ``` 1. **Backfill the target object before anything can redirect to it.** Edit the pinned set (+ allowlist, subdomains as their own entries) and ensure each target object exists. Guard against double-processing: backfill from the *baseline* source object, not from a path that may already be transformed. 2. **Verify + parse-count cross-check (G5):** 0 missing. 3. **Remove each flipped host from the upstream cohort, in the SAME release.** Otherwise every non-pinned id re-pins the wrong variant after purge, and the single-lane “disable” rollback never reaches baseline. Accepting residual leak on non-pinned ids requires a written `docs/` waiver. 4. **Size guard (G2):** post-edit artifact under cap. 5. **Rollback snapshot + commit (G4).** 6. **Deploy.** If deploy is non-atomic (content PUT then rule PUT, no retry), gate on the *live* artifact: live content hash == local hash AND rule enabled with the expected expression, *before* smoke. Mismatch → targeted disable, redeploy, re-snapshot. 7. **Propagation soak ≥10 min, then multi-colo smoke BEFORE any purge.** A colo where the rule hasn’t propagated still routes upstream and re-pins the wrong variant right after a purge. Confirm the flipped response from a **multi-colo quorum (≥10–20 colos spanning regions)** — a 2-colo sample is not all-colo proof. 8. **Per-host smoke with the host’s REAL id** (not `ids[0]` — see the warning below). Assert: top-3 ids serve the target variant; a current host id *not* in the pinned set does not flip and its served variant matches intent; no-referrer → baseline variant; baseline asset blocked. Run subdomains twice. Re-confirm paying + no-referrer controls each batch. 9. **Purge the wrongly-pinned bare keys ONLY after** deploy-live + cohort-removed + smoke-green. Per id, just-in-time re-HEAD the target object (require 200) before purging its bare key; skip+flag 404s to avoid purging into a 404 storm. Rate-limit and retry with backoff (see [edge-cache-and-purge](/runbooks/cost-safe-cache-purging/)). Write a manifest per id. 10. **Close gate** — re-probe each bare key with a cache-buster → variant baseline for *every* id (treat a tiered-fill header as a hit). Also re-probe a known *non-pinned* id with referrer — if it serves the wrong variant, the upstream cohort still overlaps → fix the cohort, do **not** advance. 11. **Observe through soak (§5).** Re-run smoke at T+15m and T+24h (+48h for fast-rotating hosts). For >1M hosts, a human loads one real embedding page and visually confirms before advancing. > **Zone-id footgun:** if the bare key and the redirect target live in **different zones**, a wrong-zone purge silently no-ops. Pin both zone ids explicitly; a purge tool that validates the host but not zone ownership will appear to succeed while doing nothing. *** ## 5. Observability + numeric ABORT criteria [Section titled “5. Observability + numeric ABORT criteria”](#5-observability--numeric-abort-criteria) | # | Signal | ABORT threshold | Action | | - | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------- | | 1 | Scheduled per-host smoke (**primary**) | any regression, or a top-3 id doesn’t serve target, or a non-pinned host id flips | disable + cohort backoff | | 2 | Multi-colo target-object HEAD | any non-200 on a flipped pinned id from ≥3 colos | purge stale URL; if missing, disable + re-backfill | | 3 | Object-store GET volume on the target key (positive confirm the flip lands) | does NOT rise above baseline within the first soak hour | disable, investigate | | 4 | Coverage / decay | below floor (0.85 rotating / 0.80 stable) or drop >5 abs pts | hold + refresh the pinned set | | 5 | Paying control | any paying domain/IP request for a flipped id serves the wrong variant | disable immediately | | 6 | No-referrer control | any no/malformed-referrer request serves non-baseline | disable + purge bare key | | 7 | Wrong-variant share | > 0.5% of an id’s sampled requests | disable | **Advance gate (the return-to-baseline rule):** promote host N+1 only when host N is green at smoke **T+15m AND T+24h**, the object-store GET rose, coverage is above floor, and (for >1M hosts) a human visually confirmed. A one-time re-warm spike *decays*; a structural leak is a *flat plateau* — they are only distinguishable after **two consecutive return-to-baseline polls**. Do not advance on the single decaying-spike reading alone. Pin the pre-flip baseline as an *immutable captured value*, not a rolling re-pull, so a leak can’t be absorbed into “the new normal.” > **Why worker analytics lie here.** If the flipped path is a redirect, it skips the worker entirely, so worker invocation/subrequest counts *undercount* flipped traffic and may read flat while real traffic climbs. Treat them as informational only. Use object-store GET volume and coverage reports as the real-traffic proxies. > **Synthetic smoke is single-id.** A smoke tool that always probes `ids[0]` (or a fixed sample) is **non-validating per host** — it never exercises the host’s real id. Require a per-host real-id override before trusting smoke as a flip gate. *** ## 6. Cost ordering vs incident ordering [Section titled “6. Cost ordering vs incident ordering”](#6-cost-ordering-vs-incident-ordering) Exposure ordering (§1) and cost ordering are **different**. Sequence by exposure; gate by cost separately. * Cloudflare (and most CDNs) offer **no hard spend cap** on paid plans — every native control is notify-after-the-fact (\~24h lag). The real brakes are self-built. * The dangerous cost vector is usually **not volume** — it’s a non-pinned, cache-busting request (`/{id}?v=NNN`) that misses the edge on *every* request forever, or a purge that runs faster than the deploy propagates and re-warms a synchronized flood. * Because the at-risk metric (worker invocations + KV reads) often lags in billing and isn’t measured by default tooling, **rate caps + trickle-purge are the primary safety; monitoring confirms, it does not prevent.** Full cost threat model, per-wave caps, the fail-closed cost gate, and circuit-breakers live in [cost-safety-gates](/runbooks/cost-safe-cache-purging/). Do not purge until those gates exist. *** ## 7. Rollback (may be multi-lane) [Section titled “7. Rollback (may be multi-lane)”](#7-rollback-may-be-multi-lane) If the variant decision lives in two lanes (edge snippet + upstream cohort), **turning off one lane does not restore baseline** — and the “off” action can even *invert* cost (disabling the redirect can route the full flipped volume back through the expensive worker). Confirm the lane topology before you trust any kill switch. **Full rollback = edge lane OFF + upstream cohort backed off + bare keys purged.** * **(a) Per-host fast revert (\~3–6 min + propagation):** remove host from the flip allowlist + pinned set AND from the upstream cohort (or set its rollout to 0) → test/lint → deploy (+ upstream redeploy) → rollback-verify with a real-id smoke asserting the baseline variant. If it still serves the wrong variant, the upstream lane is still active — fix the cohort. Revert the highest-volume host first if a global signal is ambiguous; rotating hosts are the most likely culprits. * **(b) Global kill + restore:** disable the edge lane (fastest stop), restore rules from the snapshot, and `git checkout ` for the content rollback. If you disable the edge lane, **also back off the upstream cohort** or the hosts keep getting the wrong variant. > **Residual risk:** cached redirects linger until expiry unless `no-store` was set (G0) — for >1M hosts, treat revert latency as the redirect cache TTL. Never delete a still-pinned id’s warm target object (you’ll 404 the host). Purges are not auto-undone. See [reverting-deploys](/runbooks/reverting-deploys/) for the general revert workflow. *** ## 8. Pinned-set decay + artifact budget [Section titled “8. Pinned-set decay + artifact budget”](#8-pinned-set-decay--artifact-budget) * **Cadence by catalog shape:** rotating hosts daily; stable hosts weekly (or on coverage drop); single-hero / pinned-once hosts once. * A report-only coverage metric is **not** an alarm — wire a coverage-floor check that exits non-zero below floor (G0). * If a daily cron already syncs the *upstream* cohort but not the *edge* set, that gap is real: a rotating host’s edge pins decay silently. Close it with a refresh job that splices the live-id delta into the edge set (preserving any per-id comments the parsers depend on), re-backfills, audits, lint+tests, and **fails below floor** — deploy stays manual. * **Artifact byte budget:** if the edge artifact has a hard cap, a refresh must *prune*, not just append (cap per-host id count for high-rotation hosts), and must fail *before* writing if it would exceed the cap minus margin. *** ## 9. Quick checklist [Section titled “9. Quick checklist”](#9-quick-checklist) * [ ] Cohort ordered by **exposure**, canary is live/mixed + smallest, highest-volume host last and alone * [ ] Global G0 gates clear (no unprotected paying host; decay alarm wired; cost estimate committed) * [ ] Per batch: backfill → cohort-removed (same release) → deploy-live verified → multi-colo soak → real-id smoke → purge → close gate * [ ] Smoke uses the host’s **real id**, not a fixed sample * [ ] Advance only on **two-poll return-to-baseline** * [ ] Rollback rehearsed on both lanes; redirect carries `no-store` * [ ] Concrete hosts/ids/zones decoded only in the internal decoder (not published) # Watermark delivery system > How a thumbnail request becomes a clean or watermarked image, and how to operate delivery deterministically. Unlisted internal dev doc — audit before publishing publicly Internal engineering note, published **unlisted for review only**. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off. Purpose: explain the freemium watermark delivery pipeline end to end — how a thumbnail request becomes a clean or watermarked image, why delivery can drift, and how to operate it deterministically. Applies to: the thumbnail image edge (snippet router + page-rule cache + variant worker + R2), the referrer cohort, and anyone rolling a host into or out of watermarking. > Concrete entities (which hosts are flipped, paying-customer ids/IPs, code-name decode, the live page-rule id) live in the internal decoder (not published). This doc stays generic on purpose. ## TL;DR [Section titled “TL;DR”](#tldr) * Watermarking is **additive, not destructive**: it is a 307 redirect to a *distinct* `wm.jpg` key. The bare `/{id}.jpg` key only ever holds clean bytes. * A 2022 **referrer-blind 31-day page-rule cache** pins one variant per URL per colo for \~31 days. First requester wins → the **cache lottery**. * Two control planes can emit `wm` for the same URL: the **snippet** (deterministic, pre-cache) and the **worker** (cohort decision, on cache miss). They **overlap**, which is why a snippet-only rollback never reaches clean and a purge re-pins `wm`. * **Reverse leak**: once a URL is pinned `wm`, no-referrer / direct users get the watermark too, violating the developer-friendly policy. ## Components [Section titled “Components”](#components) | Component | Role | Where | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | Edge snippet (watermark router) | Pre-cache request router. Runs before the worker. Redirects eligible requests to a distinct `wm.jpg` key. No bindings, 5ms CPU, 32KB. | `apps/snippet/watermark-router.snippet.js` | | Referrer-blind page-rule cache | Zone page rule, `*.jpg*`, edge TTL \~31 days, **keyed without referrer**. Pins one variant per URL per colo. | zone config (snapshot in `docs/watermarks/initial-rules.json`) | | Variant worker | On cache miss, decides clean vs `wm` for the referrer cohort, builds the watermark, writes both variants to R2. | `apps/thumbnail/worker.ts`, `lib/cache.ts` | | Referrer cohort | The list of referrer hosts the worker watermarks. Hand-synced from analytics; decays. | `apps/thumbnail/lib/referrer-rollout-cohort.ts` | | Snippet HOT / redirect set | Pre-generated `wm.jpg` ids the snippet is allowed to 307 to, plus the WM referrer domains. Decays as catalogs rotate. | `HOT` + `WM_DOMAINS` in the snippet | | R2 bucket | Holds `clean.jpg` and `wm.jpg` at **distinct keys** per id. Public so redirects resolve. | `thumbnails/{id}/{clean,wm}.jpg` | | Smart tiered cache | Enabled on the zone; accelerates origin fetch on cold colos. Does not change variant logic. | zone setting | ## Variant via redirect (why it’s additive) [Section titled “Variant via redirect (why it’s additive)”](#variant-via-redirect-why-its-additive) The core mechanic: **watermarking = a 307 redirect to a separate URL**, never a mutation of the bare key. ```plaintext GET /{id}.jpg -> always clean bytes (bare key) 307 -> /thumbnails/{id}/wm.jpg -> watermarked bytes (distinct key) ``` R2 stores the two variants at distinct keys (`thumbnails/{id}/clean.jpg`, `thumbnails/{id}/wm.jpg`). Consequences: * The bare `/{id}.jpg` key is **deterministically clean** when only the snippet is in play — watermarking adds a redirect hop rather than overwriting bytes. * Payment-status changes need **no purge**: the next request simply routes to the other key. * Clean variants must stay **non-public** at the R2 layer (a WAF/host rule blocks `…/clean.jpg`), otherwise the watermark is trivially bypassed by requesting the clean key directly. ## The cache lottery [Section titled “The cache lottery”](#the-cache-lottery) The 31-day page rule caches **one response per URL per colo, keyed without referrer**. Whoever lands the cache miss first pins that variant — clean or `wm` — for *everyone* on that colo for up to a month. * Mixed-traffic hosts (majority no-referrer / non-cohort) tend to pin **clean** → cohort users never see the watermark. * Self-referred hosts (their own pages are the referrer) tend to pin **wm** → and then every no-referrer/ambiguous user of that URL also gets `wm`. * Net effect: delivery is nondeterministic per `URL × colo × month`. Any conversion read taken before delivery is deterministic is **noise in both directions**. Custom (referrer-aware) cache keys are Enterprise-only, so the supported fix on this plan is the snippet redirect — it runs *before* the cache and sends eligible requests to a distinct key. ## The reverse leak [Section titled “The reverse leak”](#the-reverse-leak) Because the cache is referrer-blind, a single self-referred (or cohort) request that pins `wm` on the bare URL causes **no-referrer / direct / dev** users to receive the watermark for the rest of the TTL. That violates the developer-friendly “ambiguous traffic stays clean” policy. Fix shape: serve no-referrer → clean explicitly, keep watermarking on the distinct `wm.jpg` key, and **purge** the bare URLs that are currently pinned `wm` for the affected hosts. ## The worker-cohort overlap (the rollback trap) [Section titled “The worker-cohort overlap (the rollback trap)”](#the-worker-cohort-overlap-the-rollback-trap) This is the subtlety that breaks naive rollouts and rollbacks. The **worker also watermarks the cohort** on cache miss. So for a host that is in the worker cohort: * A snippet 307 only fires for ids in the snippet `HOT` set. A **non-HOT id misses the snippet, hits the worker, gets watermarked, and re-pins `wm` on the bare URL.** * Therefore “non-HOT id → snippet passes through → served clean” is **false** for any cohort host. * A **snippet-only rollback does not restore clean**: the worker keeps watermarking and re-pins `wm` after any purge. * HOT-set decay does **not** fall back to clean — it erodes delivery *determinism* (some ids snippet-redirect, others get lottery’d by the worker). ### Rollout / rollback rules [Section titled “Rollout / rollback rules”](#rollout--rollback-rules) * **Flipping a host ON** (deterministic wm): add it to snippet `WM_DOMAINS` *and* add its current top ids to `HOT`. If it is also in the worker cohort, that’s belt-and-suspenders; the snippet just makes it deterministic and skips a worker invocation. * **Flipping a host OFF** (back to clean) is **dual-lane**: 1. Remove from snippet `WM_DOMAINS` (+ its `HOT` ids). 2. Remove from the worker `referrer-rollout-cohort.ts`. 3. Purge the pinned bare URLs (otherwise non-HOT ids re-pin `wm`). Doing only step 1 leaves the worker re-pinning `wm`. See [reverting-deploys](/runbooks/reverting-deploys/) for the general deploy-rollback procedure; the watermark case is the exception that needs the cohort + purge steps above, not just a redeploy. ## Request flow [Section titled “Request flow”](#request-flow) ```mermaid flowchart TD A["Client → /{id}.jpg"] --> G["Snippet guards
method · / · favicon · '://' · /.jpg→308 · /thumbnails/ · extract id"] G --> PAY{"paying IP / domain?"} PAY -->|yes| PASS["pass-through → clean"] PAY -->|no| NOREF{"has Referer?"} NOREF -->|no| PASS NOREF -->|yes| WMDOM{"referrer ∈ WM_DOMAINS?"} WMDOM -->|no| PASS WMDOM -->|yes| HOT{"id ∈ HOT?"} HOT -->|no| PASS HOT -->|yes| WMR["307 → /thumbnails/{id}/wm.jpg
(distinct key, edge-cached)"] subgraph DOWN["Downstream cache + worker"] PASS --> PR["Page-rule edge cache
*.jpg* · ~31d · referrer-blind
(the lottery)"] PR -->|HIT| SERVE["served bytes (pinned variant)"] PR -->|miss| WK["Worker: cohort variant decision
(watermarks bare URL → re-pins wm)"] WK --> R2["R2: clean.jpg / wm.jpg at distinct keys
→ origin on miss"] R2 --> SERVE end WMR --> R2W["R2 public wm.jpg (distinct cache key)"] ``` Labels are generic. The snippet’s special-case loop redirect (one malformed self-referring WordPress fetch) and the paying-customer allowlists decode in the internal decoder (not published). ## Operating the system [Section titled “Operating the system”](#operating-the-system) ### Keep the lists fresh [Section titled “Keep the lists fresh”](#keep-the-lists-fresh) Both the worker cohort and the snippet `HOT` set are **hand-synced from analytics and decay to \~0% coverage within \~8 weeks** as catalogs rotate. Rotating-catalog hosts (auction / gray-market / large dynamic libraries) decay fastest; single-hero-video hosts are stable. * Re-sync cohort + HOT from current analytics on a schedule. * Keep the snippet under its 32KB script limit when expanding `HOT`. * Verify with the snippet test + hotset verifier before deploy. ### Verify delivered bytes, not just headers [Section titled “Verify delivered bytes, not just headers”](#verify-delivered-bytes-not-just-headers) The historical miss was checking worker headers and R2 contents but **never the eyeball-delivered bytes at scale**. Add a periodic check: sample N cohort URLs, assert the *delivered* variant matches intent. The page-rule cache between worker and user is the fourth control plane and is the one that silently overrides the decision. ### Purge [Section titled “Purge”](#purge) * Single-URL purge tooling exists (purge the bare URL when fixing a reverse leak). Purge **both** the bare key and any internal `?_wm=1` variant the worker uses. * `cache.put()` is per-PoP; for a global purge use the Cloudflare purge-by-URL API, not a local delete. * Deploys do **not** clear the page-rule cache. Stale `wm.jpg` after a watermark-asset change needs an R2 delete + URL purge or a TTL wait. ### Cost guardrail [Section titled “Cost guardrail”](#cost-guardrail) Do not blanket-delete the 31-day edge TTL rule. Inverting it pushes tens of millions of edge-served requests back through the worker path (illustrative blended rate ≈ $1/1M requests moved onto the worker), which can add real monthly spend. Scope the rule (exempt hot paths or lower TTL) and **write a cost estimate in `docs/` first** per repo policy. The snippet redirect path is cheaper than status quo because it skips the worker entirely. ## Known traps [Section titled “Known traps”](#known-traps) | Trap | Why it bites | Guard | | --------------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------- | | Snippet-only rollback | Worker re-pins `wm` for cohort hosts | Dual-lane: cohort removal + purge | | Purge without cohort removal | Non-HOT id re-pins `wm` on next miss | Remove from cohort before purging | | HOT decay | Erodes determinism, not safety (never falls to clean) | Scheduled re-sync | | Header-only verification | Page-rule cache overrides the decision silently | Sample delivered bytes | | Unprotected paying customer | Empty entitlements → still watermarked (rollout blocker) | Verify entitlements before flipping a batch; see the internal decoder (not published) | | Public clean key | Watermark bypassable by requesting `…/clean.jpg` | WAF/host rule blocking clean keys | | Auditing the pricing/funnel with static fetch | JS-only surfaces read empty | Verify in a rendered browser | ## Related [Section titled “Related”](#related) * [reverting-deploys](/runbooks/reverting-deploys/) — general rollback; watermark needs the dual-lane exception above. * An internal entitlements runbook (not published) — entitlement source for paying-customer bypass. * The internal decoder (not published) — private decode of hosts, code names, IPs, ids, and the live page-rule. * Source: `apps/snippet/watermark-router.snippet.js`, `apps/thumbnail/lib/referrer-rollout-cohort.ts`, `apps/thumbnail/worker.ts`, `apps/thumbnail/lib/cache.ts`, `docs/watermarks/initial-rules.json`.