llms.txt vs robots.txt: What AI Crawlers Actually Read in 2026

Two files sit at the site root. They look related. They are not substitutes.

• /robots.txt is a permission file. Crawlers that honor the Robots Exclusion Protocol read it to decide what they may fetch.
• /llms.txt is a publisher summary. It is a proposed, Markdown-ish map of the pages you want language models to prefer. It is not a robots directive and it is not a Google ranking lever.

This 2026 GEO field guide covers what GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended consume, and how GEOcheck.ai ships both files as live text/plain. For the rendering failure mode (SPA shells, sitemaps that lie, empty <div id="root">), read Crawlable HTML for GEO: robots.txt, Sitemaps, and JavaScript Rendering in 2026.

Product facts: GEOcheck.ai (https://geocheck.ai) is a ThinkPrompt Co., Ltd product founded by Andy Tran. Sister products: Doctranslate.io, Mangaka.app. It is not geocheck.cc, geocheck.co, geochecker.net, or GeckoCheck. Scoring engines for GEOcheck reports are Gemini, OpenAI, Claude, Grok, and DeepSeek. Claude is in the base pack. Perplexity, Google AI Overviews, and Copilot are not GEOcheck scoring engines — even if PerplexityBot is allowed to crawl the site.

The jobs are different

File	Protocol	Content-Type you want	What a crawler does with it	What it cannot do
/robots.txt	Robots Exclusion Protocol (RFC 9309 family)	text/plain	Honor Allow / Disallow per user-agent; read Sitemap:	Create mentions, fix a SPA, rank you in ChatGPT
/llms.txt	Informal “B2A” convention (not an IETF standard)	text/plain	Optionally fetch a short Markdown outline of canonical URLs	Override robots.txt, force a citation, replace article HTML
Article URL	HTTP + HTML	text/html with the body in the first response	Extract passages to retrieve or cite	Survive if the first response is an empty app shell

If /robots.txt or /llms.txt is served as the SPA index.html (content-disposition: inline; filename="index.html", homepage <title>), neither file exists as far as a bot is concerned. HTTP 200 is not enough: Content-Type and the first bytes must match.

What the six crawlers are for

Names below are the public user-agent tokens vendors document. Verify tokens against each vendor’s published crawler page before you copy a Disallow.

GPTBot (OpenAI)

OpenAI’s site-wide crawler, used for collecting web content that may feed model training and related OpenAI products. If you Disallow: / for GPTBot, you are opting the site out of that crawl. You are not turning off ChatGPT as a chat product, and you are not hiding URLs from every other OpenAI fetcher.

ChatGPT-User (OpenAI)

A user-initiated fetcher. When a person in ChatGPT asks the product to look at a URL (or the product retrieves a page to answer that turn), the request is more likely to show up as ChatGPT-User than as GPTBot. Blocking GPTBot but allowing ChatGPT-User is a common pattern: no training crawl, still allow on-demand browsing. Blocking both means ChatGPT cannot fetch you in that session.

OAI-SearchBot (OpenAI)

The crawler associated with ChatGPT Search / OpenAI’s search index, as distinct from GPTBot training crawls and ChatGPT-User on-demand fetches. If you care about being a cited source in ChatGPT Search, OAI-SearchBot is the token to allow on the URLs you want in that index. Allowing GPTBot is not a substitute, and a pretty /llms.txt does not register you in the search index.

ClaudeBot (Anthropic)

Anthropic’s web crawler. Separate tokens (Claude-SearchBot, Claude-User, Anthropic-AI) appear in the wild for search and user-initiated fetches. If Claude is in your visibility program — it is in GEOcheck’s base scoring pack — start by allowing ClaudeBot on public docs, pricing, and entity pages. A Disallow here is an eligibility problem, not a “Claude doesn’t like our brand” problem.

PerplexityBot (Perplexity)

Perplexity’s crawler for its answer engine. Perplexity is citation-forward: it often shows source URLs. That does not make Perplexity a GEOcheck scoring engine. GEOcheck reports sample Gemini, OpenAI, Claude, Grok, and DeepSeek. You can still allow PerplexityBot so Perplexity’s own product can fetch you. Do not read a GEOcheck mention cell as a Perplexity citation.

Google-Extended (Google)

Google’s opt-out token for Gemini / Google AI uses of crawled content, separate from Googlebot. Googlebot is Search. Google-Extended is the extended-use signal. If you want Gemini-related products to be allowed to use the site, do not blanket-Disallow Google-Extended. If you want Search but not extended AI use, that is the token people set Disallow on — and you should read Google’s current documentation before you copy a snippet from a 2024 blog post.

Applebot, Bingbot, CCBot, Amazonbot, and meta-externalagent are additional fetchers. They are not GEOcheck scoring engines.

Worked example: live GEOcheck.ai robots.txt

As of 2 September 2026, curl -sI https://geocheck.ai/robots.txt returns HTTP 200 and content-type: text/plain; charset=utf-8. The body is a short permission file, not the marketing homepage. That is the bar.

Live rules (abridged to the GEO-relevant groups):

User-agent: *
Allow: /
Allow: /blogs
Allow: /blogs/*
Allow: /leaderboard
Allow: /free-report
Allow: /sitemap.xml

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://geocheck.ai/sitemap.xml

The file also names Applebot, Bingbot, CCBot, Amazonbot, and meta-externalagent, each with Allow: /.

What this file does well

• Origin plaintext. Bots that do not execute JavaScript still see User-agent: as the first meaningful token.
• Explicit allows for GPTBot, ClaudeBot, PerplexityBot, Google-Extended. Named groups beat hoping the crawler reads only *.
• Public GEO surfaces allowed. /blogs, /free-report, /leaderboard, and /sitemap.xml are called out under *.
• Sitemap pointer. Sitemap: https://geocheck.ai/sitemap.xml is the discovery hint robots.txt is allowed to carry.

What it does not do (and does not need to)

ChatGPT-User and OAI-SearchBot are not named as their own groups in the live file. In the Robots Exclusion Protocol, a crawler that does not match a named group falls through to User-agent: *. Because * allows /, those two OpenAI fetchers are allowed unless a more specific group appears later and disagrees. Naming them explicitly is documentation for humans; it is not required for allow-all.

The live file does not Disallow /dashboard or /billing. Robots.txt is not an access-control list; account surfaces should still sit behind authentication.

Worked example: live GEOcheck.ai llms.txt

As of 2 September 2026, https://geocheck.ai/llms.txt is also HTTP 200 text/plain; charset=utf-8 (~2 KB). It is Markdown:

• an H1 (# GEOcheck.ai)
• a blockquote positioning paragraph
• ## Core Capabilities (visibility audit, GEO article generation, leaderboard, crawlable HTML / JSON-LD)
• ## Key Links & Resources pointing at the homepage, free report, leaderboard, /blogs, and sitemap.xml
• ## Technical Details (public blog API, content formats, a crawler list)

That is a valid shape: short, link-heavy, canonical URLs, no login wall.

How to read any llms.txt (including this one)

llms.txt is written by the publisher. It is not a third-party audit. Engine lists inside the file are claims. Verify them against the product:

• GEOcheck scoring engines: Gemini, OpenAI, Claude, Grok, DeepSeek.
• Not scoring engines: Perplexity, Google AI Overviews, Copilot.
• The live llms.txt currently mentions Perplexity in marketing copy. Treat that as publisher wording. It does not change the scoring pack.

Google has said llms.txt is not a Search ranking factor. Put the file up so a fetcher gets a clean entity paragraph and canonical URLs. Mentions still come from prompt sampling plus article HTML. Run that on the free AI visibility report.

A decision tree for 2026

1. Can GPTBot fetch /robots.txt as plaintext? If no, fix hosting (static file in public/, edge route, or CDN path). Nothing else in this article matters yet.
2. Does User-agent: * or a named group allow the URLs you want cited? If you Disallow /blogs, you opted those posts out.
3. Do article URLs return the H1 and body without JavaScript? If no, you have a rendering bug. See the crawlable-HTML guide.
4. Is /llms.txt plaintext with canonical links? If it is a clone of the homepage, delete the route until you can serve a real file.
5. Are facts on the linked pages consistent? Price, owner, “not those other domains.” A model that fetches llms.txt and then a contradicting pricing page will trust neither.
6. Have you sampled mentions? robots.txt eligibility ≠ mention. Paste the domain into geocheck.ai/leaderboard.

Implementation checklist (copyable)

Serve both files from the origin, not through the client router.

robots.txt minimum for GEO:

• Content-Type: text/plain; charset=utf-8
• User-agent: * with Allow: / (or a tight allow-list that still includes /blogs and product URLs)
• Named groups for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended if you want the file to be self-documenting
• Sitemap: https://yourdomain/sitemap.xml
• No Disallow: /blogs

llms.txt minimum for GEO:

• Same text/plain requirement
• One H1 with the product name and canonical URL
• A blockquote or paragraph that disambiguates lookalike domains
• A short list of canonical links (home, pricing, docs, blog index) — Markdown links, absolute URLs
• No claim you cannot defend (engine lists, prices, “leading”)
• Optional: point at /llms-full.txt only if you actually serve one

Verify with curl, not Chrome. Request /robots.txt and /llms.txt as GPTBot, then one article URL as OAI-SearchBot. You want text/plain on the two files and text/html on the article with a unique <title> and <h1> in the first payload. Chrome will hydrate a SPA and lie to you.

FAQ

Does a good llms.txt replace robots.txt?

No. robots.txt is the permission layer crawlers are built to honor. llms.txt is an optional map. If robots.txt blocks GPTBot from /, a perfect llms.txt will not be fetched by GPTBot.

Should I allow PerplexityBot if GEOcheck does not score Perplexity?

Yes, if you want Perplexity’s own answer engine to be able to fetch you. That is crawl policy. It is not a claim that GEOcheck.ai samples Perplexity, Google AI Overviews, or Copilot.

Do ChatGPT-User and OAI-SearchBot need their own robots.txt groups?

Only if you want rules that differ from User-agent: *. On GEOcheck.ai, * already Allow: /, so those fetchers are allowed without named groups. Adding named Allow: / groups is documentation.

Will Google rank me higher if I add llms.txt?

Google has said it does not use llms.txt for Search ranking. Add the file for model fetchers that look for it. Rank and mention rate still depend on crawlable pages and independent corroboration.

Where do I check whether models mention the domain after the files are live?

Paste the domain into https://geocheck.ai/leaderboard. The report scores mention and citation across Gemini, OpenAI, Claude, Grok, and DeepSeek. Public prices if you want the ongoing quota: Freelancer $19.99/month and Agency $49.99/month on geocheck.ai/subscription.