Generate a robots.txt that lets search + AI answer engines crawl you (so you can be ranked AND cited) and optionally blocks AI training crawlers. The action half of the AI access checker.
Save as /robots.txt at your site root. Citation bots (OAI-SearchBot, PerplexityBot, Googlebot…) stay allowed so AI engines can cite you. Validate this robots.txt →
This trips up almost everyone: Disallow stops a crawler from fetching a page — it does not remove it from the index. A blocked URL can still appear in results, described only by the links pointing at it, because the engine was never allowed to read the noindex you put inside.
To keep a page out of results, let it be crawled and mark it noindex. Block in robots.txt only what you do not want fetched at all.
They now come in two kinds, and conflating them costs you either visibility or control:
The common mistake is blocking both with one rule and then wondering why no assistant ever cites the site.
*, not both.No. It prevents crawling, not indexing. A disallowed URL can still be listed — without a description, because the crawler was never allowed to read the page. Use noindex on a crawlable page instead.
Separate the two kinds. Blocking training crawlers (GPTBot, CCBot) costs no visibility. Blocking answer crawlers (OAI-SearchBot, PerplexityBot, Google-Extended) removes you from AI answers.
At the domain root — /robots.txt. It applies to that host and protocol only; subdomains need their own file.
A robots.txt is a plain-text file at the root of your domain that tells each crawler — search engines and AI answer engines alike — which URLs it may fetch. This generator builds one that keeps the crawlers you want (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot) fully allowed so your pages can be indexed and quoted in AI answers, while optionally blocking the AI training bots you don't want. Get the rules wrong and you can silently vanish from ChatGPT, Perplexity, and Google AI Overviews.
No. GPTBot is OpenAI's training crawler, while ChatGPT's live citations come from OAI-SearchBot. Block GPTBot and allow OAI-SearchBot to keep your citations while opting out of training.
At the root of your domain, reachable at https://yourdomain.com/robots.txt. It only governs the host it sits on, so each subdomain needs its own file.
Not reliably. Disallow stops crawling, but a page can still be indexed if other sites link to it; use a noindex meta tag or header to keep a page out of results.
Only training bots, and only if you don't want your content used for model training. Blocking answer-engine crawlers like PerplexityBot removes you from AI answers, which usually costs more visibility than it protects.
Free tools show you WHAT to fix. The full audit + the autonomous team of 20 AI agents fix it end-to-end — strategy, content, GEO, internal links, publishing and day-2 upkeep.
All free tools →cookies
The Matrix already knows everything about you — cookies are small change by comparison. The choice, as always, is yours: