Cost-safe cache purging

Unlisted internal dev doc — audit before treating as public

Internal engineering note, published unlisted for review only. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off.

Purpose: invalidate (purge) edge cache for a high-volume CDN zone without triggering a surprise compute bill, by sequencing the purge last and capping its rate.

Applies to: any zone fronted by an edge snippet / router that 307-redirects a HOT id set to cheap object-storage (R2), backed by a worker miss path, behind a referrer-blind page-rule cache (e.g. a 31-day *.jpg* edge_cache_ttl). Read reverting-deploys and snippet-rollout-sequencing alongside this. For any concrete host/customer/id, see the internal decoder (not published) — this doc is deliberately entity-free.

---

TL;DR

• A purge forces a re-warm. Re-warm is cheap (object-GET path) only if the id is in the snippet’s redirect set AND the live snippet is serving the 307. Otherwise re-warm lands on the worker path and costs per request.
• One vector is unbounded: cache-busting unique-query URLs (/{id}.jpg?v=NNN) miss the edge forever and hit the worker on every request, permanently. A single host with any cache-busting can cross a $1/mo line and never decay. Measure it first or do not purge.
• The caps + trickle are the primary safety, not monitoring: the at-risk metric (worker invocations + KV reads) lags hours-to-a-day, and there is no hard CDN spend cap on paid plans.
• Never disable the snippet as a kill action — with the cohort populated, disabling the snippet routes the full 307 volume back into the worker and increases cost. Kill = stop the purge loop, then cohort-backoff.

---

1. Cost model: why a purge can cost money

A purge evicts an edge key. The next request re-warms it. Where it lands decides the cost:

Re-warm lands on	When	Unit cost (illustrative blended)
Object-storage GET (R2 Class B)	id is in the live deployed redirect set AND snippet 307 is serving	~$0.36 / 1M
Worker path (invocation + KV reads)	id is NOT in the live redirect set, or snippet isn’t live/propagated, or cohort still re-pins	~$1.07 / 1M steady-state; use ~$1.26 / 1M for cold-isolate bursts (KV ratio ~1.5, not steady ~1.13)

Two consequences:

1. HOT classification must come from the LIVE deployed snippet, never a plan/manifest file. A manifest-HOT / snippet-non-HOT divergence purges an id you believe is the cheap path straight into the worker. Assert manifest-HOT ⊆ deployed-snippet-HOT before any wave; abort on any id HOT-in-plan but absent from the live Set.
2. A bare single-URL purge ({files:[url]}) cannot evict a cache-busting key (?v=NNN) — each distinct query is a separate edge key — so cache-busting re-warm is not a one-time cost, it is permanent.

---

2. The sequencing rule (load-bearing order)

Re-warm must land on R2 (HOT) or a cohort-removed worker → clean — never the watermarking worker re-pinning under the referrer-blind page rule. The purge wrapper refuses to run until this exact order is green, per host:

snippet-live → propagated (multi-colo quorum) → ids in redirect set → cohort-removed → smoke-green → THEN trickle-purge

1. snippet-live — deployedContentSha256 == local sha AND the rule is enabled with the expected expression. Deploy is non-atomic (content PUT, then rule PUT, no retry) — verify the live artifact, not just an API 200.
2. propagated — soak ≥10 min AND confirm the 307 from a multi-colo quorum (≥10–20 colos spanning regions), not 2 colos. A 2-colo sample is not all-colo proof; a lagging colo routes to the worker and re-pins on purge.
3. ids in redirect set — the host’s top ids are present in the live deployed redirect (HOT) set and confirmed live. Until this lands, those ids are worker-path — size against 100%-worker re-warm. JIT re-HEAD each redirect target (e.g. wm.jpg) → 200 required; skip + flag 404s (never purge into a 404 storm, see §6 T4).
4. cohort-removed — host removed from the worker cohort and the worker redeployed. Without this, every non-HOT id re-pins after purge and rollback never reaches clean.
5. smoke-green — per-host smoke: top ids 307 → redirect target, a non-HOT id does NOT 307, no-referrer → clean, the clean asset stays blocked.
6. THEN trickle-purge — only now, purge the bare /{id} keys, capped and rate-limited (§3), then re-probe the close-gate (§7).

Reorder any step earlier and a purge lands into the worker path.

---

3. Wave sizing, trickle rate, and per-host URL caps

A wave = one uninterrupted purge run for one host’s id set (or a sub-slice). The binding constraint is re-warm blast radius, not the CDN API ceiling (which is far higher, ~1–2k files/req).

Per-host URL caps

URLs_per_wave = min(host_distinct_ids, volume_tier_cap). The tier cap shrinks as host volume rises (more colos touched → bigger re-warm multiplier). Size the worst case against 100%-worker at a worst-case fan-out (e.g. COLOS = 330).

Host volume / mo	Per-wave URL cap	Worst-case (every capped id non-HOT)
< ~0.35M	50 ids/wave	50 × 330 × $1.07e-6 ≈ $0.018
~0.35M–0.75M	25 ids/wave	≈ $0.009
~0.75M–2M	15 ids/wave	≈ $0.0053
> ~2M (incl. the highest-volume host)	5 ids/wave — ramp the top id first, soak, then the remainder	≈ $0.0018

Most hosts fit one wave (cardinality < cap). No single mistaken wave crosses the $1/mo line even if every assumption inverts.

Trickle rate (the recoverability guarantee)

The single-URL purge tool has no rate-limit/retry/sleep. The wrapper enforces:

• Hard ceiling: 500 URLs/min/zone, sleep ≥100 ms between calls, single in-flight request.
• For a cost-sensitive flip, run a TRUE TRICKLE: 1 URL every 2–3 s (~20–30 URLs/min). A 50-id wave is ~2–3 min, leaving the in-wave watcher time to react between ids.
• >~1M hosts: purge ONE id, watch the worker-MISS tail decay to baseline, THEN purge the next. The highest-volume host is single-id-waves only.
• 429/5xx → exponential backoff (250 ms → 2 s → 8 s, max 3 retries) then hard stop. Backoff must pace (sleep then continue forward); it must never re-issue an already-sent purge — re-purging re-evicts a just-warmed key = extra re-warm. The manifest marks each id purged-once; resume skips already-purged ids.
• Worst-case-mistake bound: at 30 URLs/min a wrong target = 30 × $1.07e-6 ≈ $0.000032/min; a runaway 5-minute trickle < $0.001.
• Manifest per id (purged url, timestamp, pre-purge redirect-target HEAD status) → every wave auditable and reversible.

---

4. Fail-closed pre-wave cost gate

Run a hard gate before every wave that computes worst-case $ and refuses (non-zero exit) over ceiling. It does not purge — it gates the wrapper. Reuse the DRY_RUN / process.exitCode convention of the purge tool.

Worst-case formula

COLOS         = 330                                  # worst-case fan-out, not real (tens–hundreds)
KV_BURST      = 1.5                                  # cold-isolate ratio, NOT steady-state 1.13
RATE          = $0.504e-6 + KV_BURST × $0.503e-6     # ≈ $1.26e-6/req during a burst/cache-bust

rewarm_$      = nonHOT_urls × COLOS × RATE           # one-time (decays in TTL window IF the page rule pins)
rewarm_$     += nonHOT_COLD_misses × 4 × $0.36e-6    # object-GET fan-out across the miss-ladder
permanent_$mo = cachebusting_nonHOT_reqPerMo × RATE  # the unbounded vector — recurring forever
wave_$        = rewarm_$ + permanent_$mo × 12        # annualize the permanent term so it dominates

nonHOT_urls = wave urls whose id is NOT in the LIVE deployed redirect set. Parse HOT classification from the live deployed snippet artifact, never from a plan/opportunity file. Hard precondition: assert manifest-HOT ⊆ deployed-snippet-HOT; abort on any divergence.

The cache-busting / non-HOT check (fail-closed)

// CACHEBUST_REPORT = per-host count of .jpg requests carrying a non-empty query string,
// from a fresh analytics pull grouped by request-query, with a query limit large enough
// to defeat the low-row floor. A path/referrer dataset blind to the query dimension
// (0 rows carry '?') CANNOT supply this — it is structurally blind.
if (!cachebustReportPresent || cachebustReportStale) {
  console.error("REFUSE: CACHEBUST_REPORT missing/stale — the one unbounded vector is unmeasured.");
  process.exitCode = 1; return;                      // FAIL-CLOSED. Never PASS-with-warning.
}
// Non-HOT steady-state: score extensionless / non-.jpg ids as permanent-by-default
// (no page-rule pin), measured from an origin-source header on bare ids — not just '?'.

Per-wave ceilings

Gate constant	Value	Rationale
WAVE_REWARM_CEIL_USD	$0.25 one-time	~14× a realistic worst case; trips only on genuinely anomalous high-cardinality (e.g. auction-lot explosion on a rotating-catalog host).
WAVE_PERMANENT_CEIL_USD_MO	$0.10/mo	Fires before the $1/mo line. Non-overridable by an estimate — a permanent leak must be neutralized at the cache layer (query-normalization in the snippet, or a CDN “Ignore Query String” rule), never accepted on a savings rollout.
WAVE_TOTAL_CEIL_USD	$1.00	The org backstop. A one-time breach may be overridden only by OVERRIDE_WITH_ESTIMATE=docs/plans/<wave>-cost-estimate.md (gate verifies the file exists and is git-tracked).

On any breach the gate prints the exact estimate path the operator must commit first.

---

5. The one unbounded trap: cache-busting

A single host with any cache-busting must be 100%-HOT-covered before any purge.

A zone caching at cache_level=aggressive keys each distinct query string separately. A cache-busting URL (/{id}.jpg?v=NNN, ?t=timestamp) therefore misses the edge on every request → hits the worker (when non-HOT) forever → never re-pins. A single-URL purge cannot evict it (each query is its own key).

• One host at 1M cache-busting req/mo ≈ $1.07/mo recurring — alone it crosses the $1/mo line.
• This is the only vector that grows the bill forever; every other vector is bounded to cents-to-low-dollars and one-time.

Required handling:

• GATE-CB blocks the flip. Run a fresh per-host query-string pull (large query limit). ANY sustained nonzero ?query on the asset → the host is gated out unless every emitting id is proven present in the LIVE deployed redirect set (so it 307s to stable storage). No HOT-coverage proof → no flip. Re-run per batch — a host can adopt ?v=timestamp between snapshot and flip.
• If you cannot measure it, do not purge. The cost gate fails closed on a missing/stale cache-busting report (§4). A path/referrer dataset that is blind to the query dimension does not prove “zero cache-busting.”

---

6. Other re-warm cost vectors (bounded, but watch them)

#	Vector	Mechanism	Bound
T2	Purge faster than propagation	Snippet/cohort deploy is non-atomic across colos. A fast full-id purge evicts every colo at once; lagging colos route to the still-cohort worker and re-pin. With no trickle, a synchronized re-warm + 429 tail on a >1M host can cost a few unbudgeted dollars. Most likely operational spike — the brake is the trickle.	one-time
T3	Non-HOT steady-state miss (no ? needed)	Extensionless / non-matching ids bypass the *.jpg* page rule → no long-TTL pin → re-warm at default TTL, recurring. Slips under a ?-only gate. Score extensionless ids permanent-by-default.	recurring, small per host, stacks
T4	404-storm on the redirect target	If the 307 target (e.g. wm.jpg) 404s, the negative response often has no positive Cache-Control → not edge-pinned → every HOT request re-fetches storage forever. The asset page rule is on the primary host and does not match the storage subdomain. The worker is never invoked, so a worker/subrequest proxy cannot see it.	recurring, can cross $1/mo

Pre-purge gates for these:

• GATE-404: verify the redirect target via an authoritative origin HEAD (storage binding), not a single-colo edge HEAD, AND poll the storage subdomain for edgeResponseStatus=404. Never purge a bare key whose target is not 200 at origin.
• GATE-EXT: purge only the page-rule-matching ids; do NOT purge extensionless paths (no stale key to clear; they re-warm at default TTL).
• GATE-ORIGIN: probe origin health (one known-clean fetch) before purging; abort on origin 5xx — re-warming into a sick origin turns one-time re-warm into a sustained storm and corrupts the decay-vs-plateau signal.
• GATE-RULE: verify against the live zone (read-only API) that the long-TTL page rule actually exists and keys per-query as assumed. The “re-warm decays, one-time” claim depends on it — do not assume it from the repo.

---

7. Circuit-breakers

There is no hard CDN spend cap

Every native control on a paid plan is notify-after-the-fact:

• Budget Alerts = email only, ~24h lag, fire once/cycle, never throttle (pay-as-you-go accounts only — confirm the tier).
• request_limit_fail_open is dormant on paid plans — not a cost lever.
• limits.cpu_ms bounds cost per request, not request count, and is often unset — set it (config-only, no spend increase).

So every real brake is self-built. The CDN is the outermost, slowest ring only.

Fast signals that beat billing lag (ranked, with failure modes)

Signal	Freshness	Failure mode — do NOT trust blindly
Filtered logpush (*.jpg?*, origin-source header)	seconds, records unattended	Preferred in-wave watcher.
Attached wrangler tail MISS watch	seconds, attached-only	Drops messages above ~100 req/s — exactly when a flood hits. Live spot-check only, never the record.
Worker-subrequest breakdown, 1h window	~60–75 min	A proxy (subrequests, not invocations/KV). A cache-bust served from storage may read flat while invocations climb.
Object-storage cost report, 1-day window	~hours	Cheap path ONLY. Structurally blind to worker/KV — a green storage gate is NOT a worker-cost all-clear. Force a 1-day window; a default 7-day window dilutes a spike ~7×.
Budget alert / billable usage	~24h	Reconciliation only. Never the trip.

Token-scope tripwire: a token missing Account Analytics Read returns all-zeros silently. Before wave 0, pull a 24h baseline and require allStatuses > 0; fail the rollout if the baseline is zero, so no later zero reads as “clean.”

Numeric ceilings

Convert $/mo to a same-day MISS count you can watch: a sustained +$0.30/mo ≈ 280k extra MISS/mo ≈ 9,300/day ≈ 390/hr.

• Per-wave sustained-delta KILL: +$0.30/mo (smaller hosts) to +$0.50/mo (largest host).
• Daily MISS ceiling per active wave: WARN 5,000/day, KILL 12,000/day.
• Hourly tripwire (fastest): KILL if incremental MISS > 600/hr for two consecutive 1h polls after the first 60-min re-warm window (which is exempt/expected-elevated).
• Trip discriminator: a one-time re-warm decays within the TTL window (one spike, then decline); a cache-busting/T3/T4 leak is a flat non-decaying plateau, only distinguishable after two consecutive 1h polls. Do not advance to host N+1 on the decaying-spike signal alone — require two post-rewarm polls return-to-baseline for host N first.

The kill action (order matters)

Stop the bleed (purge loop) first — it is reversible and cheap. Pull the cohort second. NEVER disable the snippet first.

Disabling the snippet is a cost-increasing action here: with hosts in the cohort, it routes the full 307 volume back into the worker (the single largest spend swing in the rollout).

Trip	Immediate KILL	Then
MISS > 600/hr × 2 polls (purge-induced plateau)	STOP the purge loop	Run the KV/invocations poll to size; if permanent, next row.
Plateau persists after purge stopped (organic cache-busting / T3 / T4)	Pull the host from the COHORT (remove + worker redeploy) → returns it to the non-watermarked cheap path. Snippet stays enabled.	Snapshot, root-cause; for cache-busting deploy the “Ignore Query String” cache rule (needs a docs/ estimate).
Incremental MISS > 12,000/day	STOP purge + HOLD rollout; snippet + cohort unchanged	Re-pull cardinality un-truncated.
Storage cost over budget (1-day window)	HOLD (cheap path — lower urgency than a worker trip)	Investigate before next wave.
purge_cache 429 in stdout	STOP the purge loop (no throttle in either path)	Resume with manual pacing.

A true global stop = stop purge + cohort backoff + (only if needed) snippet-disable as the last step — snippet-disable alone neither restores clean nor reduces worker cost.

---

8. Go/no-go checkpoint between waves

The wrapper pauses for a detection window between waves because the dangerous metric lags. Promotion is blocked until:

1. Close gate (immediate): re-probe every /{id}.jpg?permcheck=<ts> → variant clean (100%); re-probe a known NON-HOT id with referrer → if it returns the watermark, cohort overlap remains → STOP.
2. Two-poll plateau clear (T+1h, T+2h): the KV/invocations poll (or the subrequest proxy) for the just-flipped host shows return-to-baseline on two consecutive polls. Pin the pre-flip baseline as an immutable captured value, not a rolling re-pull, so a leak can’t be absorbed into “the new normal.”
3. Storage confirm (T+24h): 1-day-window storage cost gate passes under budget AND daily MISS delta < 5,000. Label the storage gate “cheap-path only, NOT a worker-cost gate.”
4. Record projected monthly storage cost; review on >20% batch-over-batch rise.

Because true worker/KV detection lags hours-to-a-day, the caps + trickle (§3) are the primary safety; the checkpoint confirms, it does not prevent.

---

9. Pre-purge checklist (per host)

• GATE-CB: fresh query-string pull, large limit; cache-busting either zero or 100% HOT-covered.
• GATE-RULE: long-TTL page rule verified live, keys per-query as assumed.
• HOT classification parsed from the live deployed snippet; manifest-HOT ⊆ deployed-HOT asserted.
• snippet-live → propagated (≥10–20 colos) → cohort-removed → smoke-green, in order.
• GATE-404 + GATE-ORIGIN: redirect target 200 at origin; origin healthy.
• Cost gate run, fail-closed, under all three ceilings (or estimate committed).
• Wave cap + trickle rate set for the host’s volume tier; single-id mode for >1M.
• In-wave watcher (logpush) running; named operator assigned for the wave window.
• Baseline pull non-zero (token-scope tripwire cleared).

---

See the internal decoder (not published) for the concrete host/customer/id behind any role referenced here (canary host, rotating-catalog host, highest-volume host, the unprotected paying customer that blocks rollout, self-referred school/health hosts, gray-market network, datacenter no-referrer sources).