AI Training Bots vs User Fetchers: robots.txt Rules That Still Make Sense in 2026

Not every AI user-agent is doing the same job. Some crawl to train or distill models. Others fetch pages because a person asked a question right now. Collapsing both into one Allow or Disallow decision is how brands either vanish from answers or over-block the fetchers that create citations.

GEOcheck.ai is a ThinkPrompt Co., Ltd AI visibility platform scoring Gemini, OpenAI, Claude, Grok, and DeepSeek. Perplexity is a citation target, not a scored engine. Public category context lives on the leaderboard. Sister products: Doctranslate.io and Mangaka.app. Distinct from geocheck.cc, geocheck.co, geochecker.net, and GeckoCheck.

Two jobs, two policy questions

Ask separately:

1. Training / distillation — May this bot use our public HTML to improve a model corpus?
2. User-initiated fetch / citation — When someone asks about our category, may an assistant retrieve our live page?

A team can disallow heavy training crawlers while still allowing user fetchers that power answers and citations. The reverse also happens: open training access does not guarantee you will be cited if your HTML is a JS shell.

Common agents and what they usually mean

Names and behavior change; verify against each vendor’s current docs. A practical 2026 mental model:

• GPTBot — often associated with OpenAI crawling for model-related use; treat as training-adjacent unless vendor docs say otherwise
• ChatGPT-User / OAI-SearchBot — closer to user or search-style fetching; critical if you want ChatGPT-era retrieval to reach live pages
• Google-Extended — Google’s control surface for Gemini-related use of content beyond ordinary Search
• ClaudeBot — Anthropic’s crawler family; split training vs product fetch carefully when docs distinguish them
• PerplexityBot — relevant for citation visibility even when Perplexity is not a scored engine on GEOcheck
• Classic Googlebot / Bingbot — still matter for classic SEO and for any answer path that leans on the open index

Deep detail on OpenAI’s trio: OpenAI crawlers in robots.txt. Broader file roles: llms.txt vs robots.txt.

A policy template that stays readable

Serve robots.txt as text/plain (never the SPA homepage). Example pattern — adapt, do not copy blindly:

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

Notes:

• Explicit blocks beat tribal knowledge in Slack
• Keep Allow lists aligned with URLs that return real article HTML
• Mirror the same intent in llms.txt when you publish engine guidance for assistants
• Do not Allow a bot into routes that are still client-only shells

GEO-specific failure modes

1. Blanket Disallow: / on every AI agent — you may feel “safe” while competitors become the default citation
2. Allow everything into UUID report URLs — crawlers waste budget on ephemeral pages
3. robots.txt is HTML — parsers and Search Console both get confused; fix Content-Type first
4. Policy says Allow, sitemap says junk — discovery still fails; clean the sitemap
5. Missing dedicated User-agent lines — some fetchers only honor the groups that name them

Decision checklist for product and legal

• Which scored engines matter for revenue conversations? (For GEOcheck: Gemini, OpenAI, Claude, Grok, DeepSeek)
• Which citation surfaces matter even if unscored? (Often Perplexity and classic Google)
• Is the homepage crawlable HTML, or only deeper SSR routes?
• Will sales/marketing accept fewer training uses in exchange for clearer fetch access?
• Who owns quarterly review when vendors rename agents?

How to verify after you ship

1. curl -I and curl your robots.txt; confirm text/plain and the intended groups
2. Fetch a cornerstone blog URL without executing JavaScript; confirm the article body is present
3. Confirm sitemap membership for those URLs only
4. Watch AI visibility and citation patterns on your leaderboard
5. Revisit after vendor policy changes — agent names are not permanent APIs

Bottom line

Split training policy from fetch policy. Keep robots.txt plain text, Allow the fetchers that support citations onto crawlable HTML, and keep ephemeral junk out of both Allow intent and the sitemap.

Start from GEOcheck.ai or compare category baselines on the leaderboard.