Skip to main content Scroll Top

Robots txt Generator: Create a robots.txt File With Search and AI Crawler Rules

Updated November 2026 · Written and maintained by the Progression Agency strategy team

This robots txt generator turns a short form into a robots.txt file you can copy or download, written to the rules in RFC 9309, the IETF standard for the file. Choose which crawlers to name, list the paths to keep crawlers out of and the exceptions inside them, add your sitemap, and read the hints before you upload. Search crawlers and AI crawler tokens appear with the job each operator gives them in its own documentation, so a training opt out never takes a search crawler with it by accident. Everything happens in your browser, and Progression’s technical team in New York reviews crawl rules for clients across the US.

On this page · 15 sections
  1. Illustrative example: one slash copied from a staging site
  2. What the robots txt generator writes, line by line
  3. How a crawler picks the one set of rules that applies to it
  4. When an Allow and a Disallow both match one URL
  5. Search crawler tokens in the generator
  6. AI crawler tokens, grouped by what each operator says they do
  7. Crawl-delay: three operators, three answers
  8. Where the file lives and which addresses it covers
  9. Blocking a crawl is not the same as removing a page
  10. What crawlers do when the file is missing, moved or failing
  11. Robots txt create, test, upload: the checklist
  12. Starting files for five common situations
  13. The validation hints, explained
  14. What a crawl access review costs
  15. Where this fits with the other free tools

Already have a robots.txt file? To test a file you already publish against the AI crawler tokens, paste it into our AI crawler access checker. It reads an existing file; this page writes a new one.

Examples marked “Illustrative example.” are hypothetical; no real business, site or file is shown.

Robots txt generator

Pick a starting point or fill in the fields, then generate. The file follows RFC 9309, and every crawler note comes from that operator’s own documentation. Copy it or download it as robots.txt.

Your file appears here.

Runs entirely in your browser. Nothing you type is uploaded, logged or stored, and the tool makes no network request: the file is built on this page and the download is created on your device.

Every field in the robots txt generator is optional. Leave the form blank and the file still comes out valid: one line that names every crawler and one line that allows everything, which is what crawlers assume when there is no file at all.

Illustrative example: one slash copied from a staging site

Picture a hypothetical online shop with a staging host for testing. The staging file asks every crawler to stay away, which is correct. During a release the whole folder is copied to the live server, robots.txt included.

Two drawn boxes with the same two lines, User-agent star and Disallow slash. The first box is labeled as the staging host's file, the second as the same file copied to a hypothetical live shop, with a note that compliant crawlers are now asked to stay off every product page.Two drawn boxes with the same two lines, User-agent star and Disallow slash. The first box is labeled as the staging host's file, the second as the same file copied to a hypothetical live shop, with a note that compliant crawlers are now asked to stay off every product page.
Illustrative example. The same two lines on a staging host and on a live shop.

Nothing breaks that a customer can see. The pages load and orders go through. What changed is the request the shop now makes to every crawler that honors the file: stay off every URL. Google’s own documentation notes that a page blocked this way can still appear in results, but without a description, so the first sign is often a set of listings that look bare.

Two columns. Before: a star section with Disallow slash, no sitemap line, copied with the release. After: a star section that disallows only the cart and checkout folders, a sitemap line with a full URL, and a file kept per host in version control.Two columns. Before: a star section with Disallow slash, no sitemap line, copied with the release. After: a star section that disallows only the cart and checkout folders, a sitemap line with a full URL, and a file kept per host in version control.
Illustrative example. The shop’s file before and after the fix.
  • Keep a separate robots.txt for each host, and never copy the staging file in a release.
  • Check the live file in a private window after every deployment.
  • Add a comment on the first line naming the host the file belongs to.
  • Keep the Sitemap line as a full URL so it survives a move between hosts.
  • Store the intended file in version control beside the site’s code.

What the robots txt generator writes, line by line

A robots.txt file has only a handful of line types, and the generator writes each one in the form the standard expects. The table sets out what each line does and the rule behind it.

The lines a robots.txt file can hold
LineWhat it doesWhat RFC 9309 saysWhat Google adds
User-agent: *Starts a set of rules for every crawler the file does not nameCrawlers with no matching token obey the * rulesAdsBot crawlers ignore *; they must be named
User-agent: GPTBotStarts rules for one named crawlerTokens hold letters, underscores and hyphens; matched without regard to caseBoth the field name and its value are case-insensitive
Disallow: /cart/Asks the named crawlers not to fetch paths that start with /cart/Matching starts at the first character of the pathPaths start with /; a folder ends with /
Allow: /cart/help/Opens a path inside a disallowed folderThe most specific match winsIn a tie, Google uses the least restrictive rule
Crawl-delay: 2Asks for a pause between requestsNot defined by the standardNot supported by Google
Sitemap: https://...Points to a sitemapOther records may be read and must not end a set of rulesA full URL; any number of lines; any host
# commentA note for peopleEverything after # is ignoredComments are ignored during processing

User-agent lines name the crawler

A User-agent line names the crawler the rules below it are for, using the product token the operator publishes. RFC 9309 limits a token to letters, underscores and hyphens and tells crawlers to match it without regard to case. Several User-agent lines in a row share the rules that follow them, which is how the generator lists every crawler you block in one place.

Disallow and Allow take a path that starts with a slash

Each rule holds a path relative to the root of the site. Google’s guide to writing the file asks for paths that start with a slash, and for a folder to end with one. Allow exists to reopen something inside a disallowed folder, such as a help page inside an account area.

The Sitemap line takes a full URL

Google reads Sitemap lines as complete URLs with the protocol and host, accepts any number of them, and lets the sitemap live on another host. The line belongs to no crawler in particular, so the generator writes it at the end, after every set of rules.

  • One User-agent line per crawler token.
  • Rules directly under the User-agent lines they apply to.
  • A blank line between sets of rules, for people reading the file.
  • Sitemap lines at the end, each a full URL.
  • Comments on their own lines, starting with #.

How a crawler picks the one set of rules that applies to it

A crawler never reads the whole file as one list. It looks for the rules written for its own token and follows only those, and the standard calls each set of rules a group.

A four step flow: read the file at the root, look for the crawler's own token, fall back to the star rules, then compare the URL path where the longest rule wins.A four step flow: read the file at the root, look for the crawler's own token, fall back to the star rules, then compare the URL path where the longest rule wins.
How a crawler picks the rules that apply to it, step by step.

RFC 9309 says that two sets of rules naming the same token are combined into one. Google adds that the section with the most specific matching user agent wins and the rest are ignored, and that a named section and the * section are never combined. In practice that means a crawler you name gets only the rules you wrote for it. If the * section disallows /cart/ and a named section allows the whole site, the named crawler may fetch /cart/.

What happens to a crawler the file never names?

It follows the * rules. When the file has no * section either, RFC 9309 says no rules apply, so the crawler may fetch every path. The generator always writes a * section so every crawler has a defined set of rules.

Why AdsBot needs its own line

Google’s special-case crawlers ignore the * section. AdsBot-Google checks the quality of ad landing pages and reads only rules written for its own token, so a Disallow in the * section never reaches it. The generator points this out whenever the * section holds Disallow lines.

  • Name a crawler only when it needs different rules from everyone else.
  • Remember that a named crawler drops the * rules completely.
  • Repeat any * Disallow lines inside a named section if they should still apply.
  • Name AdsBot-Google if ad landing checks must stay out of a folder.

When an Allow and a Disallow both match one URL

Rules inside one section can overlap, and the standard settles the overlap by length. Google’s reading of the specification uses the most specific rule, measured by the length of its path, and in a true conflict picks the least restrictive one.

A drawn checklist of six sample URLs from Google's documentation with their Allow and Disallow pairs, ticked where the URL may be crawled and crossed where it may not.A drawn checklist of six sample URLs from Google's documentation with their Allow and Disallow pairs, ticked where the URL may be crawled and crossed where it may not.
Six of Google’s sample situations for conflicting Allow and Disallow rules.
Google’s sample situations for conflicting rules
URLRulesRule Google applies
https://example.com/pageallow: /p, disallow: /allow: /p, the longer path
https://example.com/folder/pageallow: /folder, disallow: /folderallow: /folder, the less restrictive
https://example.com/page.htmallow: /page, disallow: /*.htmdisallow: /*.htm, which matches more characters
https://example.com/allow: /$, disallow: /allow: /$, the root only
https://example.com/page.htmallow: /$, disallow: /disallow: /, since /$ covers only the root

The asterisk and the dollar sign

Two special characters work inside paths. An asterisk stands for any run of characters, and a dollar sign marks the end of the URL. Google’s examples show that /fish* behaves exactly like /fish, because a trailing asterisk is ignored, while /*.php$ stops at URLs that end in .php.

Is a robots.txt path case sensitive?

Yes. RFC 9309 says matching should be case sensitive and Google confirms it: a rule for /file.asp does not cover /FILE.asp. Field names are the opposite, so Disallow and disallow are read the same way.

  • /fish matches /fish.html, /fishheads and /fish.php?id=anything.
  • /fish does not match /Fish.asp, /catfish or /desert/fish.
  • /fish/ matches only what sits inside the /fish/ folder.
  • /*.php matches any path that contains .php.
  • /*.php$ matches only paths that end in .php.

Search crawler tokens in the generator

The search tokens in the form come from each operator’s own crawler page. Google lists its tokens on its common crawlers page, and Apple and Amazon publish theirs on About Applebot and the Amazonbot page.

Search crawler tokens and what a rule for each one affects
TokenOperatorWhat a rule for it affects, per the operator
GooglebotGoogleGoogle Search, Images, Video mode, News and Discover
Googlebot-ImageGoogleGoogle Images and image features in Search
Googlebot-VideoGoogleVideo features in Google Search
Googlebot-NewsGoogleGoogle News and the News app
Storebot-GoogleGoogleGoogle Shopping surfaces
AdsBot-GoogleGoogleGoogle Ads landing page quality checks; ignores the * rules
ApplebotAppleSpotlight, Siri and Safari search
Amzn-SearchBotAmazonAmazon search experiences such as Alexa
500 KiB: Google's size limit. Content after it is ignored.
24 hours: Usual cache. Google generally caches the file for up to a day.
5 hops: Redirects followed. Then Google treats the file as a 404.
12 hours: Server errors. Google stops crawling the site while it retries.
30 days: Last good copy. Used while a 5xx error persists.
50,000: URLs per sitemap. Or 50MB uncompressed, per Google.

Google says its common crawlers always obey robots.txt for automatic crawls. Apple adds a detail worth knowing: when a file does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Amazon says a file that does not mention Amzn-SearchBot but allows other search bots lets it crawl under the rules given to them.

Do image and news crawlers need their own lines?

Only if they need different rules. Googlebot-Image, Googlebot-Video and Googlebot-News also answer to the Googlebot token, so a Googlebot section covers them unless you write a separate one for the narrower token.

Rewriting the file for a live site?Send the current robots.txt and the folders you are unsure about. You get a free proposal that covers the crawl rules, the sitemap and the pages they affect.

Get a free proposalTell us about a project you’d like help with.

AI crawler tokens, grouped by what each operator says they do

AI operators publish several tokens each, one per job, and the generator sorts them the same way: AI search and answers, AI training and datasets, and fetches a user asks for. Each description below follows the operator’s own page.

AI crawler tokens and the operator’s own description
TokenOperatorWhat the operator says it doesrobots.txt note from the operator
OAI-SearchBotOpenAISurfaces sites in ChatGPT search answersOpted out sites can still appear as navigational links
GPTBotOpenAICrawls content that may be used to train its modelsDisallowing it signals content should not be used in training
ChatGPT-UserOpenAIVisits pages for a user’s request in ChatGPTrobots.txt rules may not apply
Claude-SearchBotAnthropicImproves search results for Claude usersAnthropic’s bots honor robots.txt
ClaudeBotAnthropicCollects content that could contribute to trainingA rule naming it signals future content should be excluded
Claude-UserAnthropicFetches pages when people ask Claude questionsDisabling it stops retrieval for user queries
PerplexityBotPerplexitySurfaces and links sites in Perplexity searchNot used to crawl content for foundation models
Perplexity-UserPerplexityFetches pages for a user’s questionGenerally ignores robots.txt
Google-ExtendedGoogleGemini training and groundingNo effect on Google Search inclusion or ranking
Applebot-ExtendedAppleTraining of Apple foundation modelsDisallowing it opts content out of that use
AmazonbotAmazonAmazon products and services; may train Amazon AI modelsEach Amazon token is set separately
Amzn-SearchBotAmazonSearch experiences such as AlexaNot used for generative AI training
Amzn-UserAmazonLive answers on a user’s behalfMay not follow all robots.txt directives
meta-externalagentMetaUses such as training foundation AI modelsBlocked with its own Disallow line
Meta-WebIndexerMetaMeta AI search, citing and linking contentAllowing it helps Meta AI cite the site
Meta-ExternalFetcherMetaFetches links at a user’s requestMay bypass robots.txt
CCBotCommon CrawlBuilds the open Common Crawl datasetBlocked with User-agent: CCBot and Disallow: /

The descriptions come from OpenAI’s bots page, Anthropic’s crawler article, Perplexity’s crawler page, Meta’s web crawler page and Common Crawl’s CCBot page. Perplexity and Amazon note that a changed rule can take about 24 hours to reach their systems, and Meta asks for up to 24 hours because its crawlers cache the file.

Fetchers that act on a user’s request

Several operators separate automatic crawling from a fetch a person asks for. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores the file, Amazon says Amzn-User may not follow every directive, and Meta says Meta-ExternalFetcher may bypass it. Anthropic describes Claude-User differently: disabling it stops Claude from retrieving your content for user queries. The generator warns when you block a fetcher whose operator says the file may not apply.

Training opt outs that leave search alone

Several operators split training from search. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and that it intends to support a shared training preference written as user-agent: __usage_ai-training in early 2027. Apple separates Applebot-Extended from Applebot, and OpenAI separates GPTBot from OAI-SearchBot. The preset called Keep search, opt out of AI training tokens blocks only the training tokens.

Once the file is live, the AI crawler access checker reads it back and shows, token by token, which AI crawlers it lets in.

Crawl-delay: three operators, three answers

Crawl-delay asks a crawler to wait between requests. RFC 9309 does not define it, and the operators that mention it disagree, so the generator treats it as optional and writes a hint whenever you use it.

A drawn checklist: Anthropic ticked as supporting Crawl-delay, Google, Apple and RFC 9309 crossed.A drawn checklist: Anthropic ticked as supporting Crawl-delay, Google, Apple and RFC 9309 crossed.
Crawl-delay support as Anthropic, Google, Apple and RFC 9309 describe it.
Crawl-delay, as each source states it
WhoWhat they say about Crawl-delay
RFC 9309Not a defined record; crawlers may read records outside the standard
GoogleCrawl-delay is not a supported field
AppleApplebot does not follow crawl-delay
AnthropicSupports the non-standard Crawl-delay extension, for example Crawl-delay: 1 under User-agent: ClaudeBot

Should a robots.txt file include Crawl-delay?

Only when a crawler that reads it is causing real load. Google ignores the line, so it does nothing for Googlebot, and Apple says Applebot does not follow it. Anthropic supports it for its own bots. When a server is struggling with Google’s crawling, Google’s documentation points to reducing the crawl rate through its own guidance rather than through this line.

  • Leave Crawl-delay out unless a specific crawler is causing load.
  • Write it inside the section for that crawler, not only in the * section.
  • Use whole seconds; the generator drops anything that is not a number.
  • Never return 401 or 403 to slow Google down: Google says those codes do not affect crawl rate.

Where the file lives and which addresses it covers

A robots.txt file only works at one address: the root of the host it governs. Google’s documentation sets out which URLs a given file covers, and the table follows its examples.

Which URLs a robots.txt file covers, by location
File locationCoversDoes not cover
https://example.com/robots.txthttps://example.com/ and every folder under ithttps://other.example.com/, http://example.com/, https://example.com:8181/
https://www.example.com/robots.txthttps://www.example.com/https://example.com/, https://shop.www.example.com/
https://example.com/folder/robots.txtNothing: crawlers do not look in subfoldersAny URL
https://example.com:8181/robots.txtContent served on port 8181https://example.com/

One file per host, protocol and port

Rules in https://example.com/robots.txt apply only to that host, protocol and port. A shop at shop.example.com, a blog at blog.example.com and the http version of the site each read their own file. Anthropic adds that its opt out has to be written for every subdomain you want covered.

Encoding and the 500 KiB limit

RFC 9309 asks for UTF-8 plain text, served as text/plain. Google ignores a byte order mark at the start of the file, ignores invalid lines, and stops reading at 500 KiB. Google also warns against word processors, which can add curly quotes that crawlers cannot parse.

  • Name the file robots.txt, all lowercase.
  • Place it at the root of every host and subdomain.
  • Save it as UTF-8 plain text from a plain text editor.
  • Keep it well under 500 KiB by grouping excluded material in folders.

Blocking a crawl is not the same as removing a page

robots.txt manages crawler traffic. Google’s introduction to robots.txt says it is not a mechanism for keeping a page out of Google, and RFC 9309 says its rules are not a form of access authorization.

Why can a disallowed URL still show in Google?

Because Google can index a URL it never fetched when other pages link to it. The result can show the address and anchor text but no description. To keep a page out of results, Google points to noindex or a password instead, and its noindex guide adds that the page must stay crawlable, or the crawler never sees the noindex rule.

Private folders need a password, not a Disallow line

Every path in the file is public, since anyone can open /robots.txt. RFC 9309 notes that listing paths makes them discoverable and recommends a real security measure such as HTTP authentication. A Disallow line for an admin folder tells the world where the folder is.

  • Use noindex, not Disallow, for pages that must leave search results.
  • Use a password or server rules for anything private.
  • Leave CSS and JavaScript crawlable if pages need them to render.
  • Expect some crawlers to ignore the file entirely.

What crawlers do when the file is missing, moved or failing

The status code your server returns for /robots.txt changes how crawlers treat the whole site. Google’s handling is summarized below, and RFC 9309 sets the same direction: a missing file means no restrictions and an unreachable one means complete disallow.

How Google treats each robots.txt status code
What the server returns for robots.txtWhat Google does
2xx successReads and follows the file as served
3xx redirectFollows at least five hops, then treats the file as a 404
4xx, except 429Acts as if no file existed: no crawl restrictions
5xx server errorStops crawling for 12 hours, then uses the last good copy for up to 30 days
DNS or network failureTreated as a server error
A timeline of Google's handling of a server error on robots.txt: crawling stops for 12 hours, the last good version is used for up to 30 days, then the outcome depends on whether the site is generally available.A timeline of Google's handling of a server error on robots.txt: crawling stops for 12 hours, the last good version is used for up to 30 days, then the outcome depends on whether the site is generally available.
Google’s timeline for a robots.txt file that returns a server error.

How long until crawlers read a changed file?

Usually within a day. RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, and Google generally caches it for up to 24 hours. Google’s guide to updating the file explains how to ask for a faster refresh after an urgent fix.

Robots txt create, test, upload: the checklist

Google lists four steps for a new file: create it, add rules, upload it to the root, and test it. The generator covers the first two; the last two happen on your server and in Search Console.

A four step flow: create the file with the form, save it as UTF-8 plain text named robots.txt, upload it to the root of each host, then test it in a private window and in Search Console.A four step flow: create the file with the form, save it as UTF-8 plain text named robots.txt, upload it to the root of each host, then test it in a private window and in Search Console.
Create, save, upload and test, in that order.
  1. Robots txt create: fill in the form above, generate, and read every hint.
  2. Download the file, or copy it into a plain text editor and save it as UTF-8.
  3. Upload it to the root of the host, replacing any older copy.
  4. Open the live URL in a private browsing window and compare it with your download.
  5. Check the robots.txt report in Search Console once the file is reachable.
  6. Repeat for every subdomain that needs its own rules.

Testing before the file goes live

Google offers two routes: the robots.txt report in Search Console for files already on the site, and its open source robots.txt library, which developers can build to test files on their own machine. For a live file, curl https://example.com/robots.txt prints exactly what crawlers receive.

Search traffic dropped after a release?Share the release date and the file that went live. The free proposal sets out what to check first and what to fix.

Get a free proposalTell us about a project you’d like help with.

Starting files for five common situations

The presets in the robots txt generator cover the situations most sites start from. Each row below is a hypothetical example, not a file to copy without checking your own folders.

Five starting files, each an illustrative example
SituationLines in the * sectionNamed crawlersSitemap
Illustrative example. A brochure site with nothing to hideAllow: /NoneOne line
Illustrative example. A shop with cart and account pagesDisallow: /cart/, /checkout/, /account/; Allow: /account/help/NoneOne line
Illustrative example. A staging hostDisallow: /NoneNone
Illustrative example. Search yes, AI training noAllow: /GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot: Disallow: /One line
Illustrative example. A site with internal search pagesDisallow: /searchNoneOne line

A brochure site with nothing to hide

One * section with Allow: / and a Sitemap line. Google notes that allowing everything is also what crawlers assume when no rule says otherwise.

A shop with cart and account pages

Cart, checkout and account pages hold nothing a searcher needs, and keeping crawlers off them saves crawl effort for product pages. An Allow line can reopen a help folder inside the account area.

A staging host

Disallow: / for every crawler, plus a password on the host itself, because the file only asks and Google says other crawlers may not obey. Never let this file travel to the live site.

Search yes, AI training no

The * section allows everything, and one named section blocks the training tokens: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot and CCBot. Search crawlers, including the AI search tokens, keep reading the site.

A site with internal search pages

Google’s introduction suggests using the file to avoid crawling unimportant or similar pages. Internal search results are the usual example, and a Disallow line on the search path keeps crawlers on the pages that matter.

The validation hints, explained

After each run the robots txt generator lists hints under the file. Some fix the input and say so; others flag a choice worth a second look.

A drawn card with eight rows: path without a slash, space or hash in a path, Disallow slash for everyone, the same path allowed and blocked, a path with no closing slash, capital letters, an invalid token and a sitemap without a full URL.A drawn card with eight rows: path without a slash, space or hash in a path, Disallow slash for everyone, the same path allowed and blocked, a path with no closing slash, capital letters, an invalid token and a sitemap without a full URL.
A drawn card listing eight checks the generator runs, from paths without a slash to sitemaps without a full URL.
  • Path without a leading slash: the slash is added, as Google asks.
  • Space in a path: written as %20, so the line parses.
  • A # in a path: written as %23, since # starts a comment.
  • Disallow: / in the * section: flagged as a block on the whole site.
  • The same path in Allow and Disallow: RFC 9309 says Allow is used.
  • A path with no closing slash: it also matches longer paths that start the same way.
  • Capital letters in a path: matching is case sensitive.
  • A token with dots or spaces: left out, since tokens hold only letters, underscores and hyphens.
  • A Sitemap that is not a full URL: left out until it has a protocol and host.
  • Crawl-delay present: support differs by operator.
  • Googlebot blocked: flagged, since Google Search depends on it.
  • A file over 500 KiB: Google ignores the rest.

What a crawl access review costs

Crawl rules are usually reviewed as part of a technical SEO or AEO audit rather than bought alone. The bands below run from the typical low to the published high in the US market.

Audit and consulting work: published US price bands (market ranges only)
WorkTypical low to published highUnit
Technical SEO audit, one off$1,000 to $10,000Per project
AEO audit, one off$5,000 to $50,000Per project
SEO consulting$100 to $350Per hour
Horizontal range bars for a one off technical SEO audit and a one off AEO audit, each from typical low to published high.Horizontal range bars for a one off technical SEO audit and a one off AEO audit, each from typical low to published high.
Published US price bands (market ranges only) for one off audits that include crawl rules.

Progression’s price is set in the written proposal, after scope, and is subject to change. Our SEO pricing guide explains how SEO work is priced more broadly.

What moves the price of a review?

The number of hosts and subdomains, whether staging and live servers share a release process, how many crawlers need their own rules, and whether the review extends to sitemaps, noindex tags and server logs.

Pricing depends on scope and is confirmed in a written proposal. Published prices are subject to change.

Where this fits with the other free tools

The generator writes the access file. The other tools cover what crawlers find once they are in.

When the rules need more than a form, our technical SEO agency team reviews crawl access with sitemaps, redirects and indexing, and our answer engine optimization agency page covers how AI answers find and cite a site. Sites with several languages should also read the hreflang guide.

Crawl access is where SEO and AEO meet: the same file decides what search engines index and what AI answer engines can read and cite. Progression’s SEO and AEO work on crawl rules draws on the Progression Method, which is proprietary and reserved for clients. Reach out to learn more.

Frequently asked questions

What does a robots txt generator do?
It writes a robots.txt file from a form. You choose crawlers, paths and sitemaps, and the tool produces the lines in the format RFC 9309 and Google expect, ready to copy or download.
Where does the robots.txt file go after I download it?
It goes in the root of the host, so it loads at an address such as https://www.example.com/robots.txt. Google says crawlers do not look for the file in subfolders.
Will a robots.txt on the main domain apply to blog or shop subdomains?
Each subdomain reads its own file instead. Google says a file applies only to the protocol, host and port where it is posted, so blog.example.com reads blog.example.com/robots.txt.
Does a Disallow line remove a page from Google?
It stops Google from crawling the page, which is different from removing it. Google can still index the URL from links elsewhere and show it without a description, so use noindex or a password to keep a page out of results.
Which rule wins when Allow and Disallow match the same URL?
The rule with the longer path wins. When the two are equally specific, RFC 9309 says the Allow rule should be used and Google applies the least restrictive rule.
Are robots.txt paths case sensitive?
Yes. A rule for /file.asp does not cover /FILE.asp, although field names such as Disallow are read without regard to case.
Does Google read the Crawl-delay line?
Google says crawl-delay is not a supported field, so Googlebot ignores it. Apple says Applebot does not follow it either, while Anthropic says its bots support it.
Which AI crawler tokens can the generator add?
It lists the tokens operators document for AI search, AI training and user requested fetches, including GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent and CCBot. Any other token can be typed in.
Will a Google-Extended rule change how my site appears in Google Search?
Google says it will not. Google-Extended manages use for Gemini training and grounding, and Google states it does not affect inclusion in Google Search and is not a ranking signal.
Do ChatGPT-User, Claude-User and Perplexity-User follow robots.txt?
Their operators describe them differently. OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores the file, while Anthropic says disabling Claude-User stops retrieval for user queries.
Does the Sitemap line need a full URL?
Yes. Google asks for a fully qualified URL with the protocol and host, and the sitemap may sit on another host.
How many Sitemap lines can one file hold?
As many as you need. Google says there is no limit on the number of sitemap fields, and each sitemap holds up to 50,000 URLs or 50MB uncompressed.
How does Google treat a robots.txt URL that answers 404 Not Found?
As a site with no file at all. Google treats a 404, and every 4xx code except 429, as if no file existed, so it crawls without restrictions.
If the server fails while serving robots.txt, does Google keep crawling?
It pauses. Google stops crawling the site for the first 12 hours while it retries, then uses the last good version for up to 30 days.
How quickly do crawlers pick up an edited file?
Usually within a day. Google generally caches robots.txt for up to 24 hours, and RFC 9309 asks crawlers not to keep a cached copy longer than that unless the file is unreachable.
How large can a robots.txt file be?
Google reads up to 500 KiB and ignores anything after that, and RFC 9309 asks crawlers to parse at least 500 KiB.
Can robots.txt keep a private folder secret?
It cannot, because the file is public and every path in it is visible. RFC 9309 recommends a real security measure such as HTTP authentication instead.
Does a site need a robots.txt file at all?
A site runs fine without one. When the file is missing, crawlers treat the whole site as open, so you need a file only when something should be kept out or a sitemap listed.
Does anything I type into the generator leave my browser?
Nothing leaves it. The robots txt generator builds the file on the page, the download is created on your device, and the tool makes no network request.
Can I opt out of AI training and still appear in AI search answers?
Yes, where the operator separates the two. Block GPTBot, ClaudeBot, Google-Extended and the other training tokens, and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot open.
Can Progression review our crawl rules?
Yes. Progression’s technical team in New York reviews robots.txt, sitemaps and indexing rules as part of technical SEO and AEO work, with on-site sessions available on request.

Rewriting the file for a live site?Send the current robots.txt and the folders you are unsure about. You get a free proposal that covers the crawl rules, the sitemap and the pages they affect.

Get a free proposalTell us about a project you’d like help with.

Get a free marketing proposal

Tell us what you are trying to grow and we will come back with a plan, not a pitch deck. Same-day reply on weekdays.

Privacy Preferences
When you visit our website, it may store information through your browser from specific services, usually in form of cookies. Here you can change your privacy preferences. Please note that blocking some types of cookies may impact your experience on our website and the services we offer.
Contact Us