Updated November 2026 · Written and maintained by the Progression Agency strategy team
This robots txt generator turns a short form into a robots.txt file you can copy or download, written to the rules in RFC 9309, the IETF standard for the file. Choose which crawlers to name, list the paths to keep crawlers out of and the exceptions inside them, add your sitemap, and read the hints before you upload. Search crawlers and AI crawler tokens appear with the job each operator gives them in its own documentation, so a training opt out never takes a search crawler with it by accident. Everything happens in your browser, and Progression’s technical team in New York reviews crawl rules for clients across the US.
On this page · 15 sections
- Illustrative example: one slash copied from a staging site
- What the robots txt generator writes, line by line
- How a crawler picks the one set of rules that applies to it
- When an Allow and a Disallow both match one URL
- Search crawler tokens in the generator
- AI crawler tokens, grouped by what each operator says they do
- Crawl-delay: three operators, three answers
- Where the file lives and which addresses it covers
- Blocking a crawl is not the same as removing a page
- What crawlers do when the file is missing, moved or failing
- Robots txt create, test, upload: the checklist
- Starting files for five common situations
- The validation hints, explained
- What a crawl access review costs
- Where this fits with the other free tools
Already have a robots.txt file? To test a file you already publish against the AI crawler tokens, paste it into our AI crawler access checker. It reads an existing file; this page writes a new one.
Examples marked “Illustrative example.” are hypothetical; no real business, site or file is shown.
Robots txt generator
Pick a starting point or fill in the fields, then generate. The file follows RFC 9309, and every crawler note comes from that operator’s own documentation. Copy it or download it as robots.txt.
Your file appears here.
Runs entirely in your browser. Nothing you type is uploaded, logged or stored, and the tool makes no network request: the file is built on this page and the download is created on your device.
Every field in the robots txt generator is optional. Leave the form blank and the file still comes out valid: one line that names every crawler and one line that allows everything, which is what crawlers assume when there is no file at all.
Illustrative example: one slash copied from a staging site
Picture a hypothetical online shop with a staging host for testing. The staging file asks every crawler to stay away, which is correct. During a release the whole folder is copied to the live server, robots.txt included.
Nothing breaks that a customer can see. The pages load and orders go through. What changed is the request the shop now makes to every crawler that honors the file: stay off every URL. Google’s own documentation notes that a page blocked this way can still appear in results, but without a description, so the first sign is often a set of listings that look bare.
- Keep a separate robots.txt for each host, and never copy the staging file in a release.
- Check the live file in a private window after every deployment.
- Add a comment on the first line naming the host the file belongs to.
- Keep the Sitemap line as a full URL so it survives a move between hosts.
- Store the intended file in version control beside the site’s code.
What the robots txt generator writes, line by line
A robots.txt file has only a handful of line types, and the generator writes each one in the form the standard expects. The table sets out what each line does and the rule behind it.
| Line | What it does | What RFC 9309 says | What Google adds |
|---|---|---|---|
User-agent: * | Starts a set of rules for every crawler the file does not name | Crawlers with no matching token obey the * rules | AdsBot crawlers ignore *; they must be named |
User-agent: GPTBot | Starts rules for one named crawler | Tokens hold letters, underscores and hyphens; matched without regard to case | Both the field name and its value are case-insensitive |
Disallow: /cart/ | Asks the named crawlers not to fetch paths that start with /cart/ | Matching starts at the first character of the path | Paths start with /; a folder ends with / |
Allow: /cart/help/ | Opens a path inside a disallowed folder | The most specific match wins | In a tie, Google uses the least restrictive rule |
Crawl-delay: 2 | Asks for a pause between requests | Not defined by the standard | Not supported by Google |
Sitemap: https://... | Points to a sitemap | Other records may be read and must not end a set of rules | A full URL; any number of lines; any host |
# comment | A note for people | Everything after # is ignored | Comments are ignored during processing |
User-agent lines name the crawler
A User-agent line names the crawler the rules below it are for, using the product token the operator publishes. RFC 9309 limits a token to letters, underscores and hyphens and tells crawlers to match it without regard to case. Several User-agent lines in a row share the rules that follow them, which is how the generator lists every crawler you block in one place.
Disallow and Allow take a path that starts with a slash
Each rule holds a path relative to the root of the site. Google’s guide to writing the file asks for paths that start with a slash, and for a folder to end with one. Allow exists to reopen something inside a disallowed folder, such as a help page inside an account area.
The Sitemap line takes a full URL
Google reads Sitemap lines as complete URLs with the protocol and host, accepts any number of them, and lets the sitemap live on another host. The line belongs to no crawler in particular, so the generator writes it at the end, after every set of rules.
- One User-agent line per crawler token.
- Rules directly under the User-agent lines they apply to.
- A blank line between sets of rules, for people reading the file.
- Sitemap lines at the end, each a full URL.
- Comments on their own lines, starting with #.
How a crawler picks the one set of rules that applies to it
A crawler never reads the whole file as one list. It looks for the rules written for its own token and follows only those, and the standard calls each set of rules a group.
RFC 9309 says that two sets of rules naming the same token are combined into one. Google adds that the section with the most specific matching user agent wins and the rest are ignored, and that a named section and the * section are never combined. In practice that means a crawler you name gets only the rules you wrote for it. If the * section disallows /cart/ and a named section allows the whole site, the named crawler may fetch /cart/.
What happens to a crawler the file never names?
It follows the * rules. When the file has no * section either, RFC 9309 says no rules apply, so the crawler may fetch every path. The generator always writes a * section so every crawler has a defined set of rules.
Why AdsBot needs its own line
Google’s special-case crawlers ignore the * section. AdsBot-Google checks the quality of ad landing pages and reads only rules written for its own token, so a Disallow in the * section never reaches it. The generator points this out whenever the * section holds Disallow lines.
- Name a crawler only when it needs different rules from everyone else.
- Remember that a named crawler drops the * rules completely.
- Repeat any * Disallow lines inside a named section if they should still apply.
- Name AdsBot-Google if ad landing checks must stay out of a folder.
When an Allow and a Disallow both match one URL
Rules inside one section can overlap, and the standard settles the overlap by length. Google’s reading of the specification uses the most specific rule, measured by the length of its path, and in a true conflict picks the least restrictive one.
| URL | Rules | Rule Google applies |
|---|---|---|
| https://example.com/page | allow: /p, disallow: / | allow: /p, the longer path |
| https://example.com/folder/page | allow: /folder, disallow: /folder | allow: /folder, the less restrictive |
| https://example.com/page.htm | allow: /page, disallow: /*.htm | disallow: /*.htm, which matches more characters |
| https://example.com/ | allow: /$, disallow: / | allow: /$, the root only |
| https://example.com/page.htm | allow: /$, disallow: / | disallow: /, since /$ covers only the root |
The asterisk and the dollar sign
Two special characters work inside paths. An asterisk stands for any run of characters, and a dollar sign marks the end of the URL. Google’s examples show that /fish* behaves exactly like /fish, because a trailing asterisk is ignored, while /*.php$ stops at URLs that end in .php.
Is a robots.txt path case sensitive?
Yes. RFC 9309 says matching should be case sensitive and Google confirms it: a rule for /file.asp does not cover /FILE.asp. Field names are the opposite, so Disallow and disallow are read the same way.
- /fish matches /fish.html, /fishheads and /fish.php?id=anything.
- /fish does not match /Fish.asp, /catfish or /desert/fish.
- /fish/ matches only what sits inside the /fish/ folder.
- /*.php matches any path that contains .php.
- /*.php$ matches only paths that end in .php.
Search crawler tokens in the generator
The search tokens in the form come from each operator’s own crawler page. Google lists its tokens on its common crawlers page, and Apple and Amazon publish theirs on About Applebot and the Amazonbot page.
| Token | Operator | What a rule for it affects, per the operator |
|---|---|---|
| Googlebot | Google Search, Images, Video mode, News and Discover | |
| Googlebot-Image | Google Images and image features in Search | |
| Googlebot-Video | Video features in Google Search | |
| Googlebot-News | Google News and the News app | |
| Storebot-Google | Google Shopping surfaces | |
| AdsBot-Google | Google Ads landing page quality checks; ignores the * rules | |
| Applebot | Apple | Spotlight, Siri and Safari search |
| Amzn-SearchBot | Amazon | Amazon search experiences such as Alexa |
Google says its common crawlers always obey robots.txt for automatic crawls. Apple adds a detail worth knowing: when a file does not mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Amazon says a file that does not mention Amzn-SearchBot but allows other search bots lets it crawl under the rules given to them.
Do image and news crawlers need their own lines?
Only if they need different rules. Googlebot-Image, Googlebot-Video and Googlebot-News also answer to the Googlebot token, so a Googlebot section covers them unless you write a separate one for the narrower token.
Rewriting the file for a live site?Send the current robots.txt and the folders you are unsure about. You get a free proposal that covers the crawl rules, the sitemap and the pages they affect.
Get a free proposalTell us about a project you’d like help with.
AI crawler tokens, grouped by what each operator says they do
AI operators publish several tokens each, one per job, and the generator sorts them the same way: AI search and answers, AI training and datasets, and fetches a user asks for. Each description below follows the operator’s own page.
| Token | Operator | What the operator says it does | robots.txt note from the operator |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search answers | Opted out sites can still appear as navigational links |
| GPTBot | OpenAI | Crawls content that may be used to train its models | Disallowing it signals content should not be used in training |
| ChatGPT-User | OpenAI | Visits pages for a user’s request in ChatGPT | robots.txt rules may not apply |
| Claude-SearchBot | Anthropic | Improves search results for Claude users | Anthropic’s bots honor robots.txt |
| ClaudeBot | Anthropic | Collects content that could contribute to training | A rule naming it signals future content should be excluded |
| Claude-User | Anthropic | Fetches pages when people ask Claude questions | Disabling it stops retrieval for user queries |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search | Not used to crawl content for foundation models |
| Perplexity-User | Perplexity | Fetches pages for a user’s question | Generally ignores robots.txt |
| Google-Extended | Gemini training and grounding | No effect on Google Search inclusion or ranking | |
| Applebot-Extended | Apple | Training of Apple foundation models | Disallowing it opts content out of that use |
| Amazonbot | Amazon | Amazon products and services; may train Amazon AI models | Each Amazon token is set separately |
| Amzn-SearchBot | Amazon | Search experiences such as Alexa | Not used for generative AI training |
| Amzn-User | Amazon | Live answers on a user’s behalf | May not follow all robots.txt directives |
| meta-externalagent | Meta | Uses such as training foundation AI models | Blocked with its own Disallow line |
| Meta-WebIndexer | Meta | Meta AI search, citing and linking content | Allowing it helps Meta AI cite the site |
| Meta-ExternalFetcher | Meta | Fetches links at a user’s request | May bypass robots.txt |
| CCBot | Common Crawl | Builds the open Common Crawl dataset | Blocked with User-agent: CCBot and Disallow: / |
The descriptions come from OpenAI’s bots page, Anthropic’s crawler article, Perplexity’s crawler page, Meta’s web crawler page and Common Crawl’s CCBot page. Perplexity and Amazon note that a changed rule can take about 24 hours to reach their systems, and Meta asks for up to 24 hours because its crawlers cache the file.
Fetchers that act on a user’s request
Several operators separate automatic crawling from a fetch a person asks for. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores the file, Amazon says Amzn-User may not follow every directive, and Meta says Meta-ExternalFetcher may bypass it. Anthropic describes Claude-User differently: disabling it stops Claude from retrieving your content for user queries. The generator warns when you block a fetcher whose operator says the file may not apply.
Training opt outs that leave search alone
Several operators split training from search. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and that it intends to support a shared training preference written as user-agent: __usage_ai-training in early 2027. Apple separates Applebot-Extended from Applebot, and OpenAI separates GPTBot from OAI-SearchBot. The preset called Keep search, opt out of AI training tokens blocks only the training tokens.
Once the file is live, the AI crawler access checker reads it back and shows, token by token, which AI crawlers it lets in.
Crawl-delay: three operators, three answers
Crawl-delay asks a crawler to wait between requests. RFC 9309 does not define it, and the operators that mention it disagree, so the generator treats it as optional and writes a hint whenever you use it.
| Who | What they say about Crawl-delay |
|---|---|
| RFC 9309 | Not a defined record; crawlers may read records outside the standard |
| Crawl-delay is not a supported field | |
| Apple | Applebot does not follow crawl-delay |
| Anthropic | Supports the non-standard Crawl-delay extension, for example Crawl-delay: 1 under User-agent: ClaudeBot |
Should a robots.txt file include Crawl-delay?
Only when a crawler that reads it is causing real load. Google ignores the line, so it does nothing for Googlebot, and Apple says Applebot does not follow it. Anthropic supports it for its own bots. When a server is struggling with Google’s crawling, Google’s documentation points to reducing the crawl rate through its own guidance rather than through this line.
- Leave Crawl-delay out unless a specific crawler is causing load.
- Write it inside the section for that crawler, not only in the * section.
- Use whole seconds; the generator drops anything that is not a number.
- Never return 401 or 403 to slow Google down: Google says those codes do not affect crawl rate.
Where the file lives and which addresses it covers
A robots.txt file only works at one address: the root of the host it governs. Google’s documentation sets out which URLs a given file covers, and the table follows its examples.
| File location | Covers | Does not cover |
|---|---|---|
| https://example.com/robots.txt | https://example.com/ and every folder under it | https://other.example.com/, http://example.com/, https://example.com:8181/ |
| https://www.example.com/robots.txt | https://www.example.com/ | https://example.com/, https://shop.www.example.com/ |
| https://example.com/folder/robots.txt | Nothing: crawlers do not look in subfolders | Any URL |
| https://example.com:8181/robots.txt | Content served on port 8181 | https://example.com/ |
One file per host, protocol and port
Rules in https://example.com/robots.txt apply only to that host, protocol and port. A shop at shop.example.com, a blog at blog.example.com and the http version of the site each read their own file. Anthropic adds that its opt out has to be written for every subdomain you want covered.
Encoding and the 500 KiB limit
RFC 9309 asks for UTF-8 plain text, served as text/plain. Google ignores a byte order mark at the start of the file, ignores invalid lines, and stops reading at 500 KiB. Google also warns against word processors, which can add curly quotes that crawlers cannot parse.
- Name the file robots.txt, all lowercase.
- Place it at the root of every host and subdomain.
- Save it as UTF-8 plain text from a plain text editor.
- Keep it well under 500 KiB by grouping excluded material in folders.
Blocking a crawl is not the same as removing a page
robots.txt manages crawler traffic. Google’s introduction to robots.txt says it is not a mechanism for keeping a page out of Google, and RFC 9309 says its rules are not a form of access authorization.
Why can a disallowed URL still show in Google?
Because Google can index a URL it never fetched when other pages link to it. The result can show the address and anchor text but no description. To keep a page out of results, Google points to noindex or a password instead, and its noindex guide adds that the page must stay crawlable, or the crawler never sees the noindex rule.
Private folders need a password, not a Disallow line
Every path in the file is public, since anyone can open /robots.txt. RFC 9309 notes that listing paths makes them discoverable and recommends a real security measure such as HTTP authentication. A Disallow line for an admin folder tells the world where the folder is.
- Use noindex, not Disallow, for pages that must leave search results.
- Use a password or server rules for anything private.
- Leave CSS and JavaScript crawlable if pages need them to render.
- Expect some crawlers to ignore the file entirely.
What crawlers do when the file is missing, moved or failing
The status code your server returns for /robots.txt changes how crawlers treat the whole site. Google’s handling is summarized below, and RFC 9309 sets the same direction: a missing file means no restrictions and an unreachable one means complete disallow.
| What the server returns for robots.txt | What Google does |
|---|---|
| 2xx success | Reads and follows the file as served |
| 3xx redirect | Follows at least five hops, then treats the file as a 404 |
| 4xx, except 429 | Acts as if no file existed: no crawl restrictions |
| 5xx server error | Stops crawling for 12 hours, then uses the last good copy for up to 30 days |
| DNS or network failure | Treated as a server error |
How long until crawlers read a changed file?
Usually within a day. RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, and Google generally caches it for up to 24 hours. Google’s guide to updating the file explains how to ask for a faster refresh after an urgent fix.
Robots txt create, test, upload: the checklist
Google lists four steps for a new file: create it, add rules, upload it to the root, and test it. The generator covers the first two; the last two happen on your server and in Search Console.
- Robots txt create: fill in the form above, generate, and read every hint.
- Download the file, or copy it into a plain text editor and save it as UTF-8.
- Upload it to the root of the host, replacing any older copy.
- Open the live URL in a private browsing window and compare it with your download.
- Check the robots.txt report in Search Console once the file is reachable.
- Repeat for every subdomain that needs its own rules.
Testing before the file goes live
Google offers two routes: the robots.txt report in Search Console for files already on the site, and its open source robots.txt library, which developers can build to test files on their own machine. For a live file, curl https://example.com/robots.txt prints exactly what crawlers receive.
Search traffic dropped after a release?Share the release date and the file that went live. The free proposal sets out what to check first and what to fix.
Get a free proposalTell us about a project you’d like help with.
Starting files for five common situations
The presets in the robots txt generator cover the situations most sites start from. Each row below is a hypothetical example, not a file to copy without checking your own folders.
| Situation | Lines in the * section | Named crawlers | Sitemap |
|---|---|---|---|
| Illustrative example. A brochure site with nothing to hide | Allow: / | None | One line |
| Illustrative example. A shop with cart and account pages | Disallow: /cart/, /checkout/, /account/; Allow: /account/help/ | None | One line |
| Illustrative example. A staging host | Disallow: / | None | None |
| Illustrative example. Search yes, AI training no | Allow: / | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot: Disallow: / | One line |
| Illustrative example. A site with internal search pages | Disallow: /search | None | One line |
A brochure site with nothing to hide
One * section with Allow: / and a Sitemap line. Google notes that allowing everything is also what crawlers assume when no rule says otherwise.
A shop with cart and account pages
Cart, checkout and account pages hold nothing a searcher needs, and keeping crawlers off them saves crawl effort for product pages. An Allow line can reopen a help folder inside the account area.
A staging host
Disallow: / for every crawler, plus a password on the host itself, because the file only asks and Google says other crawlers may not obey. Never let this file travel to the live site.
Search yes, AI training no
The * section allows everything, and one named section blocks the training tokens: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot and CCBot. Search crawlers, including the AI search tokens, keep reading the site.
A site with internal search pages
Google’s introduction suggests using the file to avoid crawling unimportant or similar pages. Internal search results are the usual example, and a Disallow line on the search path keeps crawlers on the pages that matter.
The validation hints, explained
After each run the robots txt generator lists hints under the file. Some fix the input and say so; others flag a choice worth a second look.
- Path without a leading slash: the slash is added, as Google asks.
- Space in a path: written as %20, so the line parses.
- A # in a path: written as %23, since # starts a comment.
- Disallow: / in the * section: flagged as a block on the whole site.
- The same path in Allow and Disallow: RFC 9309 says Allow is used.
- A path with no closing slash: it also matches longer paths that start the same way.
- Capital letters in a path: matching is case sensitive.
- A token with dots or spaces: left out, since tokens hold only letters, underscores and hyphens.
- A Sitemap that is not a full URL: left out until it has a protocol and host.
- Crawl-delay present: support differs by operator.
- Googlebot blocked: flagged, since Google Search depends on it.
- A file over 500 KiB: Google ignores the rest.
What a crawl access review costs
Crawl rules are usually reviewed as part of a technical SEO or AEO audit rather than bought alone. The bands below run from the typical low to the published high in the US market.
| Work | Typical low to published high | Unit |
|---|---|---|
| Technical SEO audit, one off | $1,000 to $10,000 | Per project |
| AEO audit, one off | $5,000 to $50,000 | Per project |
| SEO consulting | $100 to $350 | Per hour |
Progression’s price is set in the written proposal, after scope, and is subject to change. Our SEO pricing guide explains how SEO work is priced more broadly.
What moves the price of a review?
The number of hosts and subdomains, whether staging and live servers share a release process, how many crawlers need their own rules, and whether the review extends to sitemaps, noindex tags and server logs.
Pricing depends on scope and is confirmed in a written proposal. Published prices are subject to change.
Where this fits with the other free tools
The generator writes the access file. The other tools cover what crawlers find once they are in.
- The AI crawler access checker reads an existing robots.txt against AI crawler tokens.
- The llms.txt generator builds the separate llms.txt file.
- The schema markup generator writes FAQ structured data.
- The free marketing tools page lists every tool in one place.
When the rules need more than a form, our technical SEO agency team reviews crawl access with sitemaps, redirects and indexing, and our answer engine optimization agency page covers how AI answers find and cite a site. Sites with several languages should also read the hreflang guide.
Crawl access is where SEO and AEO meet: the same file decides what search engines index and what AI answer engines can read and cite. Progression’s SEO and AEO work on crawl rules draws on the Progression Method, which is proprietary and reserved for clients. Reach out to learn more.
Getting found in search
AI, AEO and what is changing
Paid media and lead generation
Websites and design
Choosing and working with an agency
Software and app development
Website design by industry and type
Web development, platforms and hosting
Social, content and brand
By industry and by situation
Frequently asked questions
What does a robots txt generator do?
Where does the robots.txt file go after I download it?
Will a robots.txt on the main domain apply to blog or shop subdomains?
Does a Disallow line remove a page from Google?
Which rule wins when Allow and Disallow match the same URL?
Are robots.txt paths case sensitive?
Does Google read the Crawl-delay line?
Which AI crawler tokens can the generator add?
Will a Google-Extended rule change how my site appears in Google Search?
Do ChatGPT-User, Claude-User and Perplexity-User follow robots.txt?
Does the Sitemap line need a full URL?
How many Sitemap lines can one file hold?
How does Google treat a robots.txt URL that answers 404 Not Found?
If the server fails while serving robots.txt, does Google keep crawling?
How quickly do crawlers pick up an edited file?
How large can a robots.txt file be?
Can robots.txt keep a private folder secret?
Does a site need a robots.txt file at all?
Does anything I type into the generator leave my browser?
Can I opt out of AI training and still appear in AI search answers?
Can Progression review our crawl rules?
Rewriting the file for a live site?Send the current robots.txt and the folders you are unsure about. You get a free proposal that covers the crawl rules, the sitemap and the pages they affect.
Get a free proposalTell us about a project you’d like help with.
Get a free marketing proposal
Tell us what you are trying to grow and we will come back with a plan, not a pitch deck. Same-day reply on weekdays.
