Does your robots.txt let AI in? We ran 6 famous sites through the free checker

Short version: check robots.txt first — a citation bot blocked there will never cite the site, however good the page.

A few days ago I pasted reddit.com into our own AI crawler access checker, and every one of the fourteen citation engines came up blocked. Every single one! So I went digging through five more famous sites. GEO has a question #0 that comes before capsules, markup and llms.txt: can an AI engine read the site at all? The answer lives in robots.txt.

What the checker scores

Paste a domain and the checker reads the live robots.txt, then sorts 27 AI bots into three groups with an allowed/blocked verdict for each. Citation engines (14 UAs: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and others) — block one and the site’s pages vanish from that engine’s answers. Training crawlers (11 UAs: GPTBot, CCBot, Google-Extended…) — blocking those is legitimate and costs no visibility. Legacy UAs (anthropic-ai, cohere-ai) — rules targeting them are dead, and “legacy” is, arguably, the polite word.

We checked six famous sites

So I ran six major sites’ robots.txt through the checker (snapshot from July 3, 2026; “blocked” = the UA falls under a blanket Disallow: /). My hypothesis was the textbook one: training closed, citation open. It wasn’t. The spread shows robots.txt stopped being a technical file long ago — it’s now a business position in negotiations with AI companies.

Site	Citation bots blocked (of 14)	Training (of 11)	What it means
reddit.com	14	11	everything closed; access by contract only
nytimes.com	9	9	a hard licensing stance
cnn.com	10	10	same
theverge.com	10	9	but OpenAI is allowed — the Vox↔OpenAI deal is visible right in robots.txt
bbc.com	6	9	middle ground
medium.com	0	6	the model: training closed, citation open

The most telling row is The Verge: OAI-SearchBot and GPTBot are allowed, every other citation bot is closed. A publisher’s licensing agreement with OpenAI, readable in a plain text file — who needs a press release? And Medium is the row I’d copy: ZERO citation bots blocked.

Citation ≠ training: the #1 mistake

Media giants block citation bots deliberately — it’s leverage in licensing negotiations, and NYT and Reddit actually have that leverage. But when a regular business site copies their “protect yourself from AI” recipe, nobody is negotiating on the other end — only a silent disappearance from ChatGPT Search, Perplexity and Bing Copilot. Who is that block supposed to pressure? The right pattern for most sites is Medium’s: training closed, citation open — probably the least glamorous decision in GEO.

On ourselves, and how to fix it

Naturally I ran our own domain too; failing your own check would be a sad sight. Our robots.txt serves User-agent: * + Allow: / — all 27 rows come up green. Boring, and for most sites typically also correct: visibility first, the optional training opt-out later — and is there even a case for closing training crawlers on a small site? Maybe, but never at the cost of a citation bot.

👉 If the checker shows red, the AI-friendly robots.txt generator builds a ready file for your chosen policy, and crawler access — together with markup, capsules and speed — is covered by the paid GEO/SEO audit. Go check yours; ten seconds, then back to arguing about answer capsules!

→ Check AI-bot access to your site