OpenAI Crawlers in robots.txt: GPTBot, ChatGPT-User, and OAI-SearchBot (2026)

If your robots.txt only has User-agent: * plus Allow: /, you have a default permission, not an OpenAI policy. In 2026 that distinction matters. OpenAI operates several crawlers with different jobs: training-oriented fetch, user-triggered retrieval for ChatGPT answers, and search-oriented indexing. Treating them as one anonymous agent hides intent, makes audits harder, and leaves citation-side crawlers (and competitor GEO tools) guessing what you meant.

This guide is a practical allowlist pattern for GPTBot, ChatGPT-User, and OAI-SearchBot, plus how to document sibling agents such as ClaudeBot, Google-Extended, and PerplexityBot. We also tie robots policy to crawlable HTML and sitemap hygiene—the other half of AI visibility. For the companion file that maps preferred pages for models, see llms.txt vs robots.txt. For SPA shells and empty root divs that defeat crawlers even when robots say Allow, see Crawlable HTML for GEO.

Product note: GEOcheck.ai is a ThinkPrompt Co., Ltd product (Andy Tran). Sister products: Doctranslate.io and Mangaka.app. It is distinct from geocheck.cc, geocheck.co, geochecker.net, and GeckoCheck. GEOcheck scores brand visibility across Gemini, OpenAI, Claude, Grok, and DeepSeek. Perplexity is a citation target, not a scored engine in the product.

Why Allow: / under User-agent: * is not an OpenAI policy

The Robots Exclusion Protocol is agent-aware. A catch-all User-agent: * block applies when no more specific User-agent matches. That is useful as a floor. It is a poor substitute for documenting what you want OpenAI’s named bots to do.

Problems with relying only on *:

1. Intent is invisible. A future teammate cannot tell whether GPTBot was deliberately allowed for training, accidentally inherited Allow from *, or was meant to be blocked on /private.
2. Audits fail. Security and legal reviews ask “what did we tell OpenAI?” A single * line cannot answer “training vs user fetch vs search.”
3. Diffs are noisy. When you later add Disallow: /app for GPTBot only, reviewers need an explicit GPTBot block to see the change.
4. Competitors and GEO platforms read the same file. Explicit blocks signal maturity; silence looks like omission.

Recommendation: keep a sane User-agent: * baseline, then add named blocks for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Google-Extended, and PerplexityBot—even when the rules are identical to *.

OpenAI’s three crawlers (roles, not myths)

Exact product names and behaviors evolve; treat the following as the 2026 operator checklist used by SEO/GEO teams.

GPTBot

Role: OpenAI’s primary named crawler associated with model-related fetching and training pipelines (as commonly documented in robots discussions).

robots.txt implication: If you want OpenAI’s GPTBot-class crawler to see public marketing and docs, say so under User-agent: GPTBot. If you want to exclude training-oriented fetch while still allowing user-triggered retrieval, GPTBot is usually the first lever people tighten—not ChatGPT-User.

Practical tip: Do not bury GPTBot inside *. Name it. Mirror Allow/Disallow paths you already use for public content (/, /blogs, /docs) and Disallow authenticated or ephemeral paths (/app, /api private routes, /analysis/ UUID dumps if those are not meant for training corpora).

ChatGPT-User

Role: User-triggered fetch when someone asks ChatGPT to retrieve or cite a live page. This is closer to “help the answer” than “bulk train.”

robots.txt implication: Many brands that block GPTBot still Allow ChatGPT-User so conversational answers can fetch canonical pages. Others Allow both. The point is to decide and document.

Practical tip: If ChatGPT-User is missing as a dedicated User-agent line, reviewers cannot prove you intended user-triggered fetch to succeed. Add the block even if the body is simply Allow: /.

OAI-SearchBot

Role: Search-oriented OpenAI crawler used in indexing / search surfaces tied to OpenAI’s search products (naming as used in 2026 robots discussions).

robots.txt implication: Treat OAI-SearchBot like a discovery crawler: it should see the same public HTML you want indexed in AI search contexts. Pair Allow with a truthful sitemap and crawlable HTML—not a JS shell.

Practical tip: Listing OAI-SearchBot next to GPTBot and ChatGPT-User makes the OpenAI triad complete: training-class, user-fetch, search-index.

Recommended robots.txt pattern (allowlist first)

Below is a documentation-first allowlist. Adjust Disallows to your product.

User-agent: *
Allow: /
Allow: /blogs
Allow: /blogs/*
Allow: /leaderboard
Allow: /sitemap.xml
Disallow: /app
Disallow: /api/

User-agent: GPTBot
Allow: /
Allow: /blogs
Allow: /blogs/*
Allow: /leaderboard
Disallow: /app
Disallow: /api/

User-agent: ChatGPT-User
Allow: /
Allow: /blogs
Allow: /blogs/*
Allow: /leaderboard
Disallow: /app

User-agent: OAI-SearchBot
Allow: /
Allow: /blogs
Allow: /blogs/*
Allow: /leaderboard
Disallow: /app

User-agent: ClaudeBot
Allow: /
Allow: /blogs
Allow: /blogs/*
Allow: /leaderboard

User-agent: Google-Extended
Allow: /
Allow: /blogs
Allow: /blogs/*

User-agent: PerplexityBot
Allow: /
Allow: /blogs
Allow: /blogs/*

Sitemap: https://example.com/sitemap.xml

Notes:

• Identical Allow lines are intentional. You are writing policy for humans and auditors, not saving bytes.
• PerplexityBot is included because Perplexity is a major citation surface. GEOcheck does not score Perplexity as an engine, but brands still care whether PerplexityBot may crawl public pages that become citations.
• Google-Extended is Google’s AI/extended-use agent separate from classic Googlebot. Name it if you have a position on Gemini-adjacent crawling.
• Serve robots.txt as text/plain, not your SPA homepage HTML. A 200 that returns a JS shell is not a robots file.

Worked example: GEOcheck.ai checklist (not a shame list)

Live file: https://geocheck.ai/robots.txt.

As of early September 2026, that file is live text/plain with a clear User-agent: * Allow floor and explicit named agents including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, plus Googlebot and several others. Sitemap is declared.

Checklist gap (frame as hygiene, not failure): dedicated User-agent blocks for ChatGPT-User and OAI-SearchBot are still worth adding so OpenAI’s user-fetch and search-oriented agents are documented the same way GPTBot already is. Until those lines exist, ChatGPT-User / OAI-SearchBot fall through to the * policy—which may be Allow-equivalent in practice, but is weaker for audits and for GEO playbooks that grep for named agents.

Use this as your own weekly checklist:

1. Is robots.txt text/plain and under ~few KB of real directives?
2. Are GPTBot, ChatGPT-User, and OAI-SearchBot each named?
3. Are ClaudeBot, Google-Extended, and PerplexityBot named if you care about those surfaces?
4. Does Sitemap: point at a URL that lists real HTML pages only?
5. Do Allow paths actually return crawlable HTML (not a 5KB SPA shell)?

GEOcheck’s SEO/GEO work already tracks crawlability of leaderboard and blog HTML; robots policy is the permission layer on top of that rendering layer.

Crawlable HTML + sitemap hygiene (robots alone is not enough)

An Allow line cannot invent HTML. If /blogs/your-post returns an empty root div and a JavaScript bundle, GPTBot and OAI-SearchBot may fetch bytes that contain no article. That is a GEO failure mode distinct from robots.

Pair every robots change with:

• Server-rendered or prerendered public pages for blogs, docs, and landing pages you want cited.
• A sitemap that excludes soft-404 SPA routes, private UUID analysis URLs, and anything you Disallow.
• Consistent canonicals and titles in the first HTML response—not only after hydration.
• Monitoring Content-Type and byte size of robots.txt, llms.txt, and key landing pages.

See the deeper rendering guide: Crawlable HTML for GEO: robots.txt, Sitemaps, and JavaScript.

Separating training, chat fetch, and search indexing in policy meetings

When legal, product, and SEO sit in one room, use three columns:

Concern	Primary agent to discuss	Typical question
Training / model corpora	GPTBot	May OpenAI’s GPTBot crawl public marketing?
Live answer grounding	ChatGPT-User	When a user asks ChatGPT about us, may it fetch the page?
AI search indexing	OAI-SearchBot	Should OpenAI search surfaces discover our public URLs?

You can Allow all three, Allow ChatGPT-User + OAI-SearchBot while restricting GPTBot on sensitive trees, or Disallow broadly. The GEO mistake is leaving the decision implicit under *.

Common mistakes in 2026

1. Copy-pasting only GPTBot from an old blog post and ignoring ChatGPT-User / OAI-SearchBot.
2. Blocking everything under * then wondering why citations collapse—without realizing user-fetch agents never got an explicit Allow.
3. Allowing bots to hit soft-404 SPA routes that inflate crawl waste and teach models empty shells.
4. Sitemap lies: listing URLs robots Disallow, or listing JS shells.
5. Confusing llms.txt with robots.txt. llms.txt is a publisher map; robots.txt is permission. Both matter; neither replaces the other (comparison).
6. Assuming Perplexity citations follow OpenAI policy. PerplexityBot is separate. Document it if citations matter.

Implementation checklist (ship this week)

1. Fetch your live robots.txt and confirm Content-Type is text/plain.
2. Add named blocks: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot.
3. Align Disallow paths with auth walls and ephemeral UUID pages.
4. Verify Allow paths return real HTML titles and body text without executing JS.
5. Rebuild sitemap to real-HTML URLs only; remove obsolete sitemap.txt if search consoles still fetch a legacy file.
6. Re-test with curl and with a GEO visibility pass on GEOcheck.ai / leaderboard after content and crawl fixes land.

FAQ

Does User-agent: * Allow GPTBot automatically?

If there is no more specific GPTBot block, many crawlers that honor robots will apply the * rules. That is not the same as documenting GPTBot policy. Add an explicit GPTBot section.

Should we block GPTBot but allow ChatGPT-User?

Some brands do. It is a policy choice: restrict training-class crawl while allowing user-triggered fetch. Document both agents so the choice is visible.

Is OAI-SearchBot required for ChatGPT citations?

No single robots line “creates” citations. OAI-SearchBot is about search-oriented discovery. Citations also need crawlable facts, entity clarity, and mention tracking across engines GEOcheck scores (Gemini, OpenAI, Claude, Grok, DeepSeek). Use GEOcheck.ai and the leaderboard to compare visibility after crawl fixes.

Where does PerplexityBot fit if Perplexity is not a GEOcheck scoring engine?

Perplexity is a citation surface. Naming PerplexityBot in robots.txt clarifies crawl permission for that surface. GEOcheck still treats Perplexity as citation-oriented, not as one of the five scored engines.

Can robots.txt fix a JavaScript-only blog?

No. Fix rendering first, then tighten robots and sitemaps. Otherwise you Allow crawlers into empty shells.

---

OpenAI crawlers are not one blob. Name GPTBot, ChatGPT-User, and OAI-SearchBot, keep sibling agents explicit, and pair permission with crawlable HTML. For ongoing AI visibility measurement across Gemini, OpenAI, Claude, Grok, and DeepSeek, start at GEOcheck.ai or compare surfaces on the leaderboard.