Skip to main content Scroll Top

Robots.txt Checker for AI Crawlers: What Your File Actually Does

Updated September 2026 · Written and maintained by the Progression Agency strategy team

A robots.txt checker built for AI crawlers rather than for search engines. Paste your file and see which tokens can reach a given path, which are blocked, and which you have never addressed.

On this page · 11 sections
  1. One vendor, several crawlers, three different jobs
  2. What the checker does, and what it deliberately does not do
  3. The four things it flags for you
  4. What robots.txt is, and what it is not
  5. The order of operations when access is the suspected problem
  6. Using this on client sites and in a build process
  7. What we would not claim for this tool
  8. Using this as a robots.txt checker
  9. The reference tables, in one place
  10. Everything else we have written on search, AI and getting found
  11. GPTBot: what it is, and what blocking it actually costs

The short answerMajor operators run more than one crawler and state a different job for each: one that builds training corpora, one that retrieves pages to answer questions, and one that fetches on a person’s direct request. robots.txt addresses each token separately, so a blanket block aimed at training opt-out usually closes the citation path as well. This checker parses your file the way a crawler does, reports each token by the job its operator states, and flags the four configurations that are almost always accidental.

robots.txt is a voluntary convention, not an enforcement mechanism. Crawler tokens change; each row links to the operator’s own documentation, which is the source of truth rather than this page.

AI crawler access checker

Paste your robots.txt and see which AI crawlers can reach a given path, which are blocked, and which you have never addressed at all. It also flags the two combinations people most often get backwards.

Results appear here.

Runs entirely in your browser. Nothing you paste is uploaded, logged or stored — which is also why it takes pasted text rather than a URL: a browser cannot read another site’s files without that site’s permission.

One vendor, several crawlers, three different jobs

Every practical mistake in a robots.txt comes from treating one AI vendor as one crawler, when the vendor publishes separate tokens for separate purposes.

The mistake almost everyone makes once

A site owner decides they do not want their content used for model training, opens robots.txt, blocks everything that looks like an AI company, and assumes the job is done. What usually happens instead is that the retrieval crawlers get blocked too — the ones that fetch a page in order to answer a question and cite the source. The training opt-out was achieved and the citation path was closed along with it.

The tokens are separate because the jobs are separate

Major operators publish more than one user agent and state a different purpose for each: a crawl that builds training corpora, a crawl that maintains an index used to answer questions, and a fetch triggered directly by a person in a session. robots.txt addresses each token independently, which means you can allow one and refuse another. The checker groups them by the job the operator states so the two decisions stay apart.

Which is the decision worth making deliberately

Training opt-out is a legitimate position and it costs nothing in visibility. Blocking retrieval is a different choice with a direct cost: it removes the mechanism by which your page could be quoted and linked. Making both at once, by accident, is the outcome this tool exists to prevent.

Why one vendor has several crawlersWhy one vendor has several crawlers
Every practical mistake in this file comes from treating one vendor as one crawler. The checker separates them by the job each operator states for that token.

What the checker does, and what it deliberately does not do

It parses the file the way a crawler does and answers one question: which tokens can reach the path you give it.

It parses the file the way a crawler does

Groups are formed by consecutive User-agent lines, comments are stripped, and path rules are resolved by specificity: the longest matching pattern wins, and where an Allow and a Disallow match with equal specificity the Allow takes precedence. Wildcards and end-of-path anchors are handled. A token with no matching group falls back to the catch-all group, and a token with no catch-all group either is reported as unaddressed rather than as allowed by omission.

It takes pasted text rather than a URL

A browser cannot read another origin’s files without that origin’s permission, so any tool claiming to check your robots.txt from a URL is either proxying it through a server or not doing what it says. We would rather ask you to paste twenty lines than route your site through our infrastructure to save you a copy and paste.

It does not tell you what to block

That is a commercial and editorial decision that depends on what your content is worth to you, and it is not the same answer for a news publisher and a plumbing company. What the tool gives you is an accurate picture of what your current file actually does, which is usually different from what the person who wrote it believed.

Tokens are added, renamed and split over time. Every crawler in the table links to the operator’s own page where one exists, because that page is the source of truth and this one is not.

Training — One job. Opt out costs no visibility.
Retrieval — Another job. Blocking costs citations.
User fetch — A third job. Triggered by a person.
Search — The foundation. Never block by accident.
Unaddressed — The default. Permitted unless stated.
Voluntary — The whole file. A request, not a fence.

The four things it flags for you

Four configurations are surfaced automatically because they are almost always accidental, and one of them outranks every other finding on the page.

A search crawler blocked on the path you tested

This outranks every other finding. If search crawling is blocked, the page is out of the search index, and anything assembled from that index — including AI Overviews — has nothing to draw on. We surface it first because it is both the most damaging and the easiest to cause by accident during a staging-to-production move.

A training token blocked while retrieval stays open

Flagged as informational rather than as a problem, because it is usually exactly what the site owner intended and rarely what their file actually says. Seeing it confirmed is the point.

Retrieval blocked while training stays open

The reverse combination, and almost never deliberate. It gives away the thing you were presumably trying to protect while closing the thing you presumably wanted.

No catch-all group at all

Without a User-agent: * group, every crawler you have not named is unrestricted. That may be fine. It should be a decision rather than an oversight.

What blocking each token actually doesWhat blocking each token actually does
Four of these eight are believed the wrong way round often enough that we built the checker around them. robots.txt is a request, honoured by well-behaved crawlers and ignored by others.

What robots.txt is, and what it is not

It is a request honoured by well-behaved crawlers, not a control, and it governs future crawling rather than what is already known.

It is a request, not a control

The file is a convention. Well-behaved crawlers read it and comply; others do not, and nothing in the file prevents a fetch. If content genuinely must not be retrieved, the mechanism is authentication or server-side blocking, not a text file asking politely.

It governs crawling, not what is already known

Blocking a crawler today does not remove what was crawled previously, and it does not affect material about you published elsewhere. Directory entries, reviews, press and third-party pages describing your business are all outside your robots.txt entirely.

It is edited by people who are not you

Developers, plugins, migrations and hosting panels all write to this file. In our audit work it is the single most common source of a visibility change nobody can explain, because nobody remembers touching it. Re-checking after any deployment is cheap insurance.

Where this sits in the wider picture

Access is the first of several gates. How AI search works sets out the rest, and our audit covers the ones a text file cannot reach.

Parse — Groups and rules. Comments stripped.
Match — Longest path wins. Allow ties beat Disallow.
Group — Most specific agent. Falls back to *.
Flag — Known mix-ups. Four of them.
Report — Per token. With the operator named.
Link — Vendor docs. Tokens change.

The order of operations when access is the suspected problem

Start with the file because it is free and fast, then confirm what actually happened, then look past access entirely.

Start with the file, because it is free and fast

If visibility has changed and nobody can explain it, robots.txt is the cheapest thing to rule out. It takes a minute, it needs no tooling beyond this page, and in our audits it is the single most common cause of a change nobody remembers making.

Then confirm what actually happened, not what was requested

The file records what you asked crawlers to do. Search Console records what one crawler actually did. Those are different questions and a mismatch between them is itself a finding — a permissive file with pages excluded in Search Console points at something other than access.

Then look past the file entirely

Access is one gate. A page that returns the wrong status code, renders its content only after JavaScript runs, canonicalises to a different URL or is missing from the sitemap will be absent from answers with a perfectly permissive robots.txt. Those are audit findings rather than checker findings, and we say so rather than implying the tool covers them.

What each misconfiguration costsWhat each misconfiguration costs
The expensive mistakes are all over-blocking. Under-blocking costs very little in visibility terms, which is why we would rather a client opened a door than closed the wrong one.

Want this done for your site?We build and maintain the search, content and paid programmes described on this page.

Get a free proposal

Using this on client sites and in a build process

There is no licence and no attribution requirement, and the highest-value use is a deployment check rather than an annual one.

Nothing here restricts commercial use

There is no licence to agree to, no attribution requirement and no sign-up. If it is useful in client work, use it in client work. A link back is welcome and not expected.

Make it a deployment check rather than a memory test

The reason this file drifts is that it is written by build processes, plugins and hosting panels rather than by the person responsible for visibility. Putting a robots.txt diff into your release checklist catches the problem at the moment it is introduced rather than a quarter later.

Keep a copy of the file you intended

A saved reference version makes every future check a comparison rather than a fresh judgement, and it makes an unexplained change immediately visible instead of requiring someone to re-derive what the file was supposed to say.

OpenAI — Three tokens. Training, retrieval, user.
Anthropic — Three tokens. Training, retrieval, user.
Perplexity — Two tokens. Retrieval and user.
Google — Two relevant. Search and a training token.
Apple — Two tokens. Search and a training token.
Others — Dataset crawlers. Common Crawl and peers.

What we would not claim for this tool

Three things a clean result here does not tell you, stated plainly so a green row does not imply an outcome.

It does not predict whether you will be cited

Access is necessary and nowhere near sufficient. A perfectly permissive file on a page nobody would quote produces no citations at all, and we would rather say that plainly than let a green row imply a result.

It does not know about tokens that did not exist when it was written

New crawlers appear. The table covers the tokens we see in real files today, each linked to its operator’s documentation, and a token we have never heard of is simply not evaluated.

It does not detect crawlers that ignore the file

Nothing reading robots.txt can. Identifying traffic from crawlers that do not honour it is a server log exercise, not a text-parsing one.

Training tokens against retrieval tokensTraining tokens against retrieval tokens
If the goal is to stay quotable while opting out of training, these are the two rows to keep apart. Most blanket blocks collapse them.

Using this as a robots.txt checker

A general robots.txt check confirms the file parses and that search crawlers are not blocked. This one answers the AI-specific question that a general checker does not ask.

What a general robots.txt check covers

Most robots.txt checkers were written for one job: confirm the syntax is valid and that Googlebot is not blocked from anything important. That is worth doing and several free tools do it well. What none of them do is tell you which AI crawler tokens your file admits, because those tokens did not exist when the tooling was written.

What this adds

The same parse, then a per-token verdict for twenty-one AI crawlers grouped by the job the operator states for each. A file can be perfectly valid, pass every syntax check, and still close the retrieval path you meant to keep open — and no conventional robots.txt check reports that.

Use both

Run a syntax check to confirm the file is well-formed, then run this to see what it does to the crawlers that decide whether you can be quoted. They answer different questions and neither replaces the other.

Blanket block — Usually wrong. Closes the citation path.
Blocking search — Severe. Removes the foundation.
No * group — Risky by default. Unnamed bots unrestricted.
Assuming enforcement — Wrong. It is voluntary.
Copy-pasted files — Stale. Tokens change.
Set and forget — Fails quietly. Migrations rewrite it.

The reference tables, in one place

Four tables follow: the tokens the checker knows about, how rule precedence is resolved, what blocking each class of token actually achieves, and the re-check triggers. They summarise the sections above and are meant to be usable on their own.

Tokens the checker evaluates
OperatorTokenStated jobCost of blocking it
OpenAIGPTBotTrainingNone in visibility terms
OpenAIOAI-SearchBotRetrievalHigh — closes the citation path
OpenAIChatGPT-UserUser-triggered fetchModerate — a person asked for it
AnthropicClaudeBotTrainingNone in visibility terms
AnthropicClaude-UserUser-triggered fetchModerate — a person asked for it
AnthropicClaude-SearchBotRetrievalHigh — closes the citation path
PerplexityPerplexityBotRetrievalHigh — closes the citation path
PerplexityPerplexity-UserUser-triggered fetchModerate — a person asked for it
GoogleGoogle-ExtendedTraining opt-out tokenNone in visibility terms
GoogleGooglebotSearch indexHigh — removes you from search
MicrosoftBingbotSearch indexHigh — removes you from search
AppleApplebotSearch indexHigh — removes you from search
AppleApplebot-ExtendedTraining opt-out tokenNone in visibility terms
AmazonAmazonbotCrawlNone in visibility terms
Metameta-externalagentCrawlNone in visibility terms
Common CrawlCCBotOpen dataset crawlNone in visibility terms
ByteDanceBytespiderCrawlNone in visibility terms
Coherecohere-aiCrawlNone in visibility terms
DiffbotDiffbotCrawlNone in visibility terms
TimpiTimpibotCrawlNone in visibility terms
Webz.ioomgilibotCrawlNone in visibility terms
How the checker resolves a rule
SituationResolutionWhy
Several groups name the same agentThe most specific agent match is usedCrawlers use the most specific group
No group names the agentThe catch-all group appliesStandard fallback behaviour
No catch-all group eitherReported as unaddressedDefault is permitted, but say so explicitly
Allow and Disallow both matchThe longer pattern winsSpecificity decides
Equal-length Allow and DisallowAllow winsThe documented tie-break
No rule matches the pathPermittedAbsence is permission
A wildcard or end anchor appearsMatched literally around the wildcardsStandard pattern handling
What blocking each class actually achieves
ClassAchievesDoes not achieveVisibility cost
Search crawlerRemoval from search over timeNothing goodSevere
Retrieval crawlerFewer live fetches for answersTraining opt-outHigh
User-triggered fetcherBlocks fetches a person asked forTraining opt-outModerate
Training tokenA stated training opt-outRemoval from answersNone
Open dataset crawlerExclusion from that dataset going forwardRemoval of past copiesLow
Everything at onceAll of the aboveAnything selectiveSevere
NothingNothingAny opt-outNone
When to run this again
TriggerWhyFrequency
After any deploymentBuild processes rewrite robots.txtEvery time
After a migrationStaging files carry Disallow: / into productionEvery time
After installing an SEO pluginSeveral manage this fileEvery time
After a hosting changePanels can inject their own rulesEvery time
When a vendor publishes a new tokenThe file cannot cover what did not existAs announced
When visibility changes unexplainablyThis is the first file to checkImmediately
RoutinelyDrift is silentQuarterly
Crawler tokens by what they governCrawler tokens by what they govern
The top-right is where an accidental block does real damage. The bottom-left is where a deliberate block costs you nothing you were going to get anyway.
How to use the resultHow to use the result
The re-check step matters more than it sounds. In our audits, robots.txt is the file most often changed by a developer, a plugin or a migration without anybody telling the marketing side.
What the checker looks atWhat the checker looks at
The whole thing runs in your browser. We cannot see what you paste and we do not want to.
Paste — Your robots.txt. Nothing leaves the page.
Set — A path to test. Defaults to root.
Run — The check. Per token.
Read — The flags. Before the table.
Decide — Training vs retrieval. Separately.
Re-check — After every edit. It changes without you.
Separate — The two decisions. Training is not retrieval.
Protect — Search access. Above all else.
Name — What you mean. Defaults are permissive.
Verify — Against vendor docs. Linked per token.
Date — Your last check. The file drifts.
Never — Assume enforcement. Voluntary means voluntary.

Everything else we have written on search, AI and getting found

GPTBot: what it is, and what blocking it actually costs

GPTBot is OpenAI’s training crawler. It is the token most people mean when they say they want to block AI from their site, and it is also the one most often blocked by mistake.

Blocking GPTBot opts you out of training. It does not remove you from ChatGPT’s answers, because answering a question uses a different user agent entirely. Conflating the two is the single most expensive misunderstanding in this file.

The three OpenAI tokens, and what each one does

  • GPTBot: crawls to build training corpora. Blocking it is a data-licensing decision.
  • OAI-SearchBot: crawls so ChatGPT can surface and cite pages. Blocking it removes you from citations.
  • ChatGPT-User: fetches a page because a person asked for it in a session. Blocking it breaks a user’s direct request.
  • They are separate tokens and robots.txt treats them separately, so a blanket block hits all three.

What blocking GPTBot does and does not do

  • Does: signal that your content should not be used to train future models.
  • Does not: remove you from answers ChatGPT gives today.
  • Does not: affect Google, Bing, or their AI features, which use their own tokens.
  • Does not: enforce anything, because robots.txt is a voluntary convention rather than a fence.

The configuration most people actually want

If the goal is to stay out of training while remaining citable, block GPTBot and allow OAI-SearchBot and ChatGPT-User. If the goal is maximum visibility, allow all three. What almost nobody wants, and what a blanket disallow produces, is being invisible in answers while your content was already trained on years ago.

OpenAI user agents, by the job OpenAI states for each
TokenStated jobBlocking it meansAffects citations?
GPTBotTraining data collectionOpting out of future trainingNo
OAI-SearchBotRetrieval for search and citationRemoving yourself from ChatGPT citationsYes
ChatGPT-UserFetching a page a user asked forBreaking a direct user requestYes
User-agent: *Everything unnamedAll of the above at onceYes

AI, AEO and what is changing

Websites and design

Choosing and working with an agency

Social, content and brand

By industry and by situation

Want the rest of the access picture checked

The file is the first gate, not the only one. We run the full access audit — rendering, status codes, canonicalisation, sitemap accuracy and the entity records outside your site.

/contact

Not sure which of these applies to you?Tell us the situation and we will say plainly what we would do first, and what we would not.

Talk it through

Getting found in search

AI, AEO and what is changing

Paid media and lead generation

Websites and design

Choosing and working with an agency

Social, content and brand

By industry and by situation

Frequently asked questions

Is this a robots.txt checker?
Yes, with an AI-specific focus. It parses the file the way a crawler does and then reports a per-token verdict for twenty-one AI crawlers. A conventional robots.txt check confirms syntax and search-crawler access; this reports what your file does to the crawlers that decide whether you can be quoted.
How do I check robots.txt for AI crawlers specifically?
Paste the file above and set the path you care about. Every token is reported separately, grouped by the job its operator states, because blocking a training crawler and blocking a retrieval crawler have completely different consequences.
Do I need a separate robots.txt check as well?
Run one, yes. A syntax validator confirms the file is well-formed and that search crawlers are unblocked. This tool assumes the file parses and answers the AI-token question that syntax checks do not ask.
What is GPTBot?
OpenAI’s training crawler. It collects content used to build training corpora, and it is a different user agent from the ones that retrieve pages to answer a question.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is the training crawler. Answering a question uses OAI-SearchBot and ChatGPT-User, which are separate tokens. Blocking GPTBot alone opts you out of training while leaving you citable.
How do I block GPTBot in robots.txt?
Add a User-agent: GPTBot group with Disallow: / beneath it. Be deliberate about whether you also want to block OAI-SearchBot and ChatGPT-User, because those are what affect whether you appear in answers.
Does GPTBot respect robots.txt?
OpenAI states that it does, and publishes the token for that purpose. robots.txt is a voluntary convention rather than an enforcement mechanism, so it is a request that well-behaved crawlers honour rather than a technical barrier.
Should I block GPTBot?
It is a licensing decision, not an SEO one. Blocking it signals your content should not train future models and costs you nothing in visibility. Blocking the retrieval tokens alongside it is what costs visibility, and that is usually done by accident.
Is this tool free?
Yes. The analysis runs entirely in your browser and nothing you paste is uploaded. The summary is free to read; unlocking the full findings asks for your name, business email and phone so we can send you a copy.
Why does it ask me to paste the file instead of taking my URL?
Because a browser cannot read another site’s files without that site’s permission. A tool that claims to fetch your robots.txt from a URL is routing it through a server, and we would rather ask for a copy and paste than route your site through our infrastructure.
Does blocking Google-Extended remove me from AI Overviews?
No. Per Google’s own documentation it is a training opt-out token and does not affect inclusion in Google Search, and AI Overviews are built from the search index that Googlebot crawls. This is the single most common misunderstanding in this file.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI states different jobs for them — one is described as a training crawl, the other as search retrieval. Blocking the first is a training opt-out with no visibility cost; blocking the second closes the path by which you could be fetched and cited.
Should I block AI crawlers at all?
That depends on what your content is worth to you, and the honest answer is different for a news publisher and a local service business. What the tool gives you is an accurate picture of what your file currently does, which is usually not what the person who wrote it intended.
Is robots.txt actually enforced?
No. It is a convention that well-behaved crawlers honour and others ignore. Nothing in the file prevents a fetch. If content genuinely must not be retrieved, the mechanism is authentication or server-side blocking.
Does blocking a crawler remove content that was already crawled?
No. It governs future crawling. It also has no effect on material about you published elsewhere — directories, reviews, press and third-party pages are outside your robots.txt entirely.
What does ‘no rule at all’ mean in the results?
That the token is not named anywhere in your file and there is no catch-all group covering it either. The default is permitted, but we report it separately so it is a decision rather than an oversight.
Why does it test a specific path?
Because rules are path-based. A crawler can be allowed on your homepage and blocked on a subdirectory, and the answer for ‘/’ tells you nothing about ‘/blog/’ if a rule targets it.
How does it decide between a conflicting Allow and Disallow?
The longer matching pattern wins, and where both match with equal specificity the Allow takes precedence. That is the documented tie-break and it is what the checker implements.
Does it handle wildcards?
Yes, both the asterisk wildcard and the end-of-path anchor are handled when matching patterns to the path you test.
Why link to each operator’s documentation?
Because tokens are added, renamed and split over time, and the operator’s own page is the source of truth. This page is not, and we would rather send you to the authority than have you rely on our snapshot.
My file has no User-agent: * group. Is that bad?
It means every crawler you have not named is unrestricted. That may be exactly what you want. The tool flags it so it is deliberate.
Can this tell me why I am not appearing in AI answers?
Only whether access is part of it. Access is the first gate, not the only one — content that is not extractable, an inconsistent entity record or a question owned by a publisher will all keep you out of an answer with a perfectly permissive robots.txt.
How often should I re-run it?
After every deployment, migration, plugin install or hosting change, and quarterly otherwise. In our audit work this is the file most often changed without anybody on the marketing side being told.
Does the tool store or log what I paste?
No. There is no request to any server. You can confirm that by opening your browser’s network panel while you run it.
Can I use this for a client site?
Yes. There is no restriction and no attribution requirement, though a link back is always welcome.
What if a crawler I care about is not listed?
The table covers the tokens we see most often. A token that is not listed is not evaluated, and the general rules the checker applies would still resolve it the same way — you can add a temporary User-agent line for it and re-run to see how your file treats it.
Does blocking Bytespider or CCBot matter?
Those are crawl and open-dataset tokens rather than retrieval paths, so blocking them carries little visibility cost. Whether it achieves what you want is a separate question, and past copies are not withdrawn.
Is there a version that checks a whole site?
Not as a free tool. Checking every path, every subdomain and the rendering behind them is audit work rather than a paste-in widget, and we would rather be honest about which is which.
What should I do first if the results look wrong?
Confirm the search crawlers are allowed. That finding outranks everything else on the page, and it is the one most often caused accidentally by a staging file reaching production.
Does this replace Search Console?
No. Search Console reports what Google actually did with your site; this reports what your file asks every crawler to do. They answer different questions and you want both.

Want this done for your site?We build and maintain the search, content and paid programmes described on this page.

Get a free proposal

Get a free marketing proposal

Tell us what you are trying to grow and we will come back with a plan, not a pitch deck. Same-day reply on weekdays.

Our website uses cookies from third party services to improve your browsing experience. Read more about this and how you can control cookies by clicking "Privacy Preferences".
Privacy Preferences
When you visit our website, it may store information through your browser from specific services, usually in form of cookies. Here you can change your privacy preferences. Please note that blocking some types of cookies may impact your experience on our website and the services we offer.
Privacy Policy
You have read and agreed to our privacy policy
Required
Contact Us