No-referrer traffic

Unlisted internal dev doc — audit before treating as public

Internal engineering note, published unlisted for review only. Not yet audited for public release — do not cite, index, or treat as public-facing product docs until a human review signs off.

Purpose: explain what no-referrer image traffic actually is, so you can decide whether to default it to the clean (unwatermarked) variant without bleeding conversions.

Applies to: the edge snippet / watermark router decision for requests that arrive with an empty/missing Referer, on the thumbnail image path. For concrete entity decodes (specific hosts, IPs, customer ids), see the internal decoder (not published). Sibling runbooks: reverting-deploys, referrer-cohort-allowlist.

TL;DR

No-referrer does not mean “a real person with privacy settings.” On the thumbnail path the no-referrer cohort is dominated by machines: datacenter renderers spoofing browser UAs and generic SDK/library default UAs. Real browsers with a stripped referrer are a minority of the cohort. Therefore defaulting no-referrer to clean is low conversion-risk — the bulk will never convert. Detect dev/local separately (it arrives with a localhost referrer, not as no-referrer).

Why no-referrer is not “privacy-conscious humans”

The intuition is that a missing Referer is a real browser that stripped it (Referrer-Policy, HTTPS→HTTP downgrade, etc.). The data says the opposite: that real-browser bucket exists but is small relative to non-human sources. Most no-referrer volume is servers, libraries, and bots that fetch the image directly and never render a buy page.

Source taxonomy (ranked by volume)

Ranked high→low by share of no-referrer requests on the thumbnail path. Treat the counts as illustrative orders of magnitude from one window, not fixed truth — re-pull before acting (see Data caveats).

Rank	Source bucket	Eyeballs?	How to recognize it
1	Datacenter / server-side renderer spoofing a browser UA	No	A handful of fixed datacenter IPs all sharing one desktop-browser UA (e.g. a current Firefox/Windows string). Largest single bucket. Two IPs alone can be ~40% of it.
2	Generic SDK / HTTP-library default UA	No	Bare Mozilla/5.0 with nothing after it; node; okhttp/<v>; Dart/<v> (dart:io); other language/runtime defaults.
3	Real browsers, referrer stripped	Yes (minority)	Full, well-formed mobile/desktop UAs (iPhone Safari, Mac Safari, Windows Chrome, Android Chrome) with no referrer. The only conversion-relevant slice — and it is small.
4	Native mobile apps (iOS/Android)	Partial	iOS CFNetwork/Darwin app UAs (branded installer/app names); Android via okhttp. A native app may strip the referrer even when in-app webviews do not.
5	Search / social crawlers	No	Googlebot-Image, facebookexternalhit, meta-externalads, bingbot, and similar self-identifying crawler UAs.
6	Email / document image proxies	Indirect	Gmail image proxy (via ggpht.com GoogleImageProxy), Outlook/Office (ms-office; MSOffice 16). A human may eventually see the image, but the fetch is a proxy with no referrer and no page context.
7	Monitoring / uptime bots	No	Uptime/monitoring UAs (...SiteMonitor, ...-bot/<v>).
—	CDN-internal tiered-cache IP — EXCLUDE	n/a	A single CDN-range IPv6 (the CDN’s own tiered-cache fabric) can show one to two orders of magnitude more requests than everything else combined. It is infra noise, flagged internal/infra. Exclude it before any analysis or it swamps the whole picture. See CFINT in the internal decoder (not published).

The exclusion that matters most

The CDN-internal tiered-cache IP is not a traffic source — it is the cache talking to itself. If you forget to drop it, it dwarfs every real bucket and every conclusion below is wrong. Filter likelyInternalOrInfra first.

Conclusion: defaulting no-referrer to clean is low-risk

• Buckets 1, 2, 5, 7 (datacenter, libraries, crawlers, monitors) are non-eyeballs and the clear majority of legitimate no-referrer volume.
• Bucket 3 (genuine browsers, referrer stripped) is the only conversion-relevant slice, and it is a minority of the cohort.
• Serving the clean variant to no-referrer therefore costs ~no conversions while cutting watermark-router work on a large, mostly-robotic cohort.

This de-risks the “default no-referrer → clean” path. It does not by itself decide watermark policy for referrer-bearing traffic — that is the referrer cohort’s job (see referrer-cohort-allowlist).

Dev/local is referrer-BEARING — detect by referrer host, not UA

Local/dev traffic mostly arrives with a localhost or 127.0.0.1 referrer, i.e. it is in the referrer-host set, not the no-referrer set.

• Detect dev by referrer host (localhost, 127.0.0.1, *.test, *.local, common preview/staging hosts). This is high-precision.
• Do NOT detect dev by user-agent. UA-based dev detection collides head-on with bucket 2 (server/library default UAs), so it both misses real dev traffic and misclassifies bots as dev.

Net: a referrer-host rule cleanly catches dev without touching the no-referrer bucket at all.

How to reproduce the analysis

Pull the no-referrer cohort over a recent window and bucket it. Source query sets to cross-reference:

noReferrerUserAgentsOverThreshold        # UA taxonomy (buckets 1–7)
noReferrerIpsOverThreshold               # find the datacenter IPs + the CDN infra IP
noReferrerIpUserAgentPairsOverThreshold  # confirm "N fixed IPs, one UA" (bucket 1)
xRequestedWithOverThreshold              # confirm in-app webviews mostly DO send a referrer
referrerHostsOverThreshold               # confirm dev arrives WITH a localhost referrer

Steps:

1. Drop the CDN-internal/infra IP(s) (likelyInternalOrInfra) first.
2. Classify the top UAs into the seven buckets above.
3. Confirm bucket 1 by checking IP+UA pairs: a few fixed datacenter IPs sharing one browser UA.
4. Confirm dev is referrer-bearing by looking for localhost/127.0.0.1 in the referrer-host results, not the no-referrer results.
5. Normalize window counts to a monthly figure (window × 30d/window-days) only for ranking — exact magnitudes are not load-bearing.

Prefer extending an existing cohort-candidate script over a one-off; see docs/tiger/SKILL.md for script style.

Data caveats

• Logs are query-string-blind. Aggregated request logs commonly strip the query string, so this analysis cannot see cache-busting (?v=, ?t= style params). Do not claim anything about cache-busting from this data — prove it separately with a query-string-aware pull or a live probe.
• Snapshot, not live. Figures come from a single historical window. Re-pull before acting on them.
• Query truncation. Several over-threshold queries can hit the row cap; long tails may be cut. Treat head buckets as solid, tails as indicative.
• Threshold-gated. Only sources above the est-requests threshold appear, so small genuine sources are invisible by construction — fine for a “where’s the volume” decision, not for a complete census.

Open follow-ups

• Online corroboration of the taxonomy (default Referrer-Policy behavior, HTTPS→HTTP referrer loss, native-app stripping, image-proxy fetch patterns) is a separate web-research pass.
• Owner/ASN lookup on the bucket-1 datacenter IPs (one renderer vs. a scraper) decides whether they deserve a referrer-cohort entry.