Skip to main content Scroll Top

Robots.txt Checker for AI Crawlers: What Your File Actually Does

Updated September 2026 · Written and maintained by the Progression Agency strategy team

A robots.txt checker built for AI crawlers rather than for search engines. Paste your file and see which tokens can reach a given path, which are blocked, and which you have never addressed.

On this page · 12 sections
  1. One vendor, several crawlers, three different jobs
  2. What the checker does, and what it deliberately does not do
  3. The four things it flags for you
  4. What robots.txt is, and what it is not
  5. The order of operations when access is the suspected problem
  6. Using this on client sites and in a build process
  7. What we would not claim for this tool
  8. Using this as a robots.txt checker
  9. The reference tables, in one place
  10. Everything else we have written on search, AI and getting found
  11. GPTBot: what it is, and what blocking it actually costs
  12. Video: crawling, indexing and AI access

The short answerMajor operators run more than one crawler and state a different job for each: one that builds training corpora, one that retrieves pages to answer questions, and one that fetches on a person’s direct request. robots.txt addresses each token separately, so a blanket block aimed at training opt-out usually closes the citation path as well. This checker parses your file the way a crawler does, reports each token by the job its operator states, and flags the four configurations that are almost always accidental.

robots.txt is a voluntary convention, not an enforcement mechanism. Crawler tokens change; each row links to the operator’s own documentation, which is the source of truth rather than this page.

AI crawler access checker

Paste your robots.txt and see which AI crawlers can reach a given path, which are blocked, and which you have never addressed at all. It also flags the two combinations people most often get backwards.

Results appear here.

Runs entirely in your browser. Nothing you paste is uploaded, logged or stored — which is also why it takes pasted text rather than a URL: a browser cannot read another site’s files without that site’s permission.

One vendor, several crawlers, three different jobs

Every practical mistake in a robots.txt comes from treating one AI vendor as one crawler, when the vendor publishes separate tokens for separate purposes.

The mistake almost everyone makes once

A site owner decides they do not want their content used for model training, opens robots.txt, blocks everything that looks like an AI company, and assumes the job is done. What usually happens instead is that the retrieval crawlers get blocked too — the ones that fetch a page in order to answer a question and cite the source. The training opt-out was achieved and the citation path was closed along with it.

The tokens are separate because the jobs are separate

Major operators publish more than one user agent and state a different purpose for each: a crawl that builds training corpora, a crawl that maintains an index used to answer questions, and a fetch triggered directly by a person in a session. robots.txt addresses each token independently, which means you can allow one and refuse another. The checker groups them by the job the operator states so the two decisions stay apart.

Which is the decision worth making deliberately

Training opt-out is a legitimate position and it costs nothing in visibility. Blocking retrieval is a different choice with a direct cost: it removes the mechanism by which your page could be quoted and linked. Making both at once, by accident, is the outcome this tool exists to prevent.

Why one vendor has several crawlers
Every practical mistake in this file comes from treating one vendor as one crawler. The checker separates them by the job each operator states for that token.

What the checker does, and what it deliberately does not do

It parses the file the way a crawler does and answers one question: which tokens can reach the path you give it.

It parses the file the way a crawler does

Groups are formed by consecutive User-agent lines, comments are stripped, and path rules are resolved by specificity: the longest matching pattern wins, and where an Allow and a Disallow match with equal specificity the Allow takes precedence. Wildcards and end-of-path anchors are handled. A token with no matching group falls back to the catch-all group, and a token with no catch-all group either is reported as unaddressed rather than as allowed by omission.

It takes pasted text rather than a URL

A browser cannot read another origin’s files without that origin’s permission, so any tool claiming to check your robots.txt from a URL is either proxying it through a server or not doing what it says. We would rather ask you to paste twenty lines than route your site through our infrastructure to save you a copy and paste.

It does not tell you what to block

That is a commercial and editorial decision that depends on what your content is worth to you, and it is not the same answer for a news publisher and a plumbing company. What the tool gives you is an accurate picture of what your current file actually does, which is usually different from what the person who wrote it believed.

Tokens are added, renamed and split over time. Every crawler in the table links to the operator’s own page where one exists, because that page is the source of truth and this one is not.

Training — One job. Opt out costs no visibility.
Retrieval — Another job. Blocking costs citations.
User fetch — A third job. Triggered by a person.
Search — The foundation. Never block by accident.
Unaddressed — The default. Permitted unless stated.
Voluntary — The whole file. A request, not a fence.

The four things it flags for you

Four configurations are surfaced automatically because they are almost always accidental, and one of them outranks every other finding on the page.

A search crawler blocked on the path you tested

This outranks every other finding. If search crawling is blocked, the page is out of the search index, and anything assembled from that index — including AI Overviews — has nothing to draw on. We surface it first because it is both the most damaging and the easiest to cause by accident during a staging-to-production move.

A training token blocked while retrieval stays open

Flagged as informational rather than as a problem, because it is usually exactly what the site owner intended and rarely what their file actually says. Seeing it confirmed is the point.

Retrieval blocked while training stays open

The reverse combination, and almost never deliberate. It gives away the thing you were presumably trying to protect while closing the thing you presumably wanted.

No catch-all group at all

Without a User-agent: * group, every crawler you have not named is unrestricted. That may be fine. It should be a decision rather than an oversight.

What blocking each token actually does
Four of these eight are believed the wrong way round often enough that we built the checker around them. robots.txt is a request, honoured by well-behaved crawlers and ignored by others.

What robots.txt is, and what it is not

It is a request honoured by well-behaved crawlers, not a control, and it governs future crawling rather than what is already known.

It is a request, not a control

The file is a convention. Well-behaved crawlers read it and comply; others do not, and nothing in the file prevents a fetch. If content genuinely must not be retrieved, the mechanism is authentication or server-side blocking, not a text file asking politely.

It governs crawling, not what is already known

Blocking a crawler today does not remove what was crawled previously, and it does not affect material about you published elsewhere. Directory entries, reviews, press and third-party pages describing your business are all outside your robots.txt entirely.

It is edited by people who are not you

Developers, plugins, migrations and hosting panels all write to this file. In our audit work it is the single most common source of a visibility change nobody can explain, because nobody remembers touching it. Re-checking after any deployment is cheap insurance.

Where this sits in the wider picture

Access is the first of several gates. How AI search works sets out the rest, and our audit covers the ones a text file cannot reach.

Parse — Groups and rules. Comments stripped.
Match — Longest path wins. Allow ties beat Disallow.
Group — Most specific agent. Falls back to *.
Flag — Known mix-ups. Four of them.
Report — Per token. With the operator named.
Link — Vendor docs. Tokens change.

The order of operations when access is the suspected problem

Start with the file because it is free and fast, then confirm what actually happened, then look past access entirely.

Start with the file, because it is free and fast

If visibility has changed and nobody can explain it, robots.txt is the cheapest thing to rule out. It takes a minute, it needs no tooling beyond this page, and in our audits it is the single most common cause of a change nobody remembers making.

Then confirm what actually happened, not what was requested

The file records what you asked crawlers to do. Search Console records what one crawler actually did. Those are different questions and a mismatch between them is itself a finding — a permissive file with pages excluded in Search Console points at something other than access.

Then look past the file entirely

Access is one gate. A page that returns the wrong status code, renders its content only after JavaScript runs, canonicalises to a different URL or is missing from the sitemap will be absent from answers with a perfectly permissive robots.txt. Those are audit findings rather than checker findings, and we say so rather than implying the tool covers them.

What each misconfiguration costs
The expensive mistakes are all over-blocking. Under-blocking costs very little in visibility terms, which is why we would rather a client opened a door than closed the wrong one.

Want this done for your site?We build and maintain the search, content and paid programmes described on this page.

Get a free proposal

Using this on client sites and in a build process

There is no licence and no attribution requirement, and the highest-value use is a deployment check rather than an annual one.

Nothing here restricts commercial use

There is no licence to agree to, no attribution requirement and no sign-up. If it is useful in client work, use it in client work. A link back is welcome and not expected.

Make it a deployment check rather than a memory test

The reason this file drifts is that it is written by build processes, plugins and hosting panels rather than by the person responsible for visibility. Putting a robots.txt diff into your release checklist catches the problem at the moment it is introduced rather than a quarter later.

Keep a copy of the file you intended

A saved reference version makes every future check a comparison rather than a fresh judgement, and it makes an unexplained change immediately visible instead of requiring someone to re-derive what the file was supposed to say.

OpenAI — Three tokens. Training, retrieval, user.
Anthropic — Three tokens. Training, retrieval, user.
Perplexity — Two tokens. Retrieval and user.
Google — Two relevant. Search and a training token.
Apple — Two tokens. Search and a training token.
Others — Dataset crawlers. Common Crawl and peers.

What we would not claim for this tool

Three things a clean result here does not tell you, stated plainly so a green row does not imply an outcome.

It does not predict whether you will be cited

Access is necessary and nowhere near sufficient. A perfectly permissive file on a page nobody would quote produces no citations at all, and we would rather say that plainly than let a green row imply a result.

It does not know about tokens that did not exist when it was written

New crawlers appear. The table covers the tokens we see in real files today, each linked to its operator’s documentation, and a token we have never heard of is simply not evaluated.

It does not detect crawlers that ignore the file

Nothing reading robots.txt can. Identifying traffic from crawlers that do not honour it is a server log exercise, not a text-parsing one.

Training tokens against retrieval tokens
If the goal is to stay quotable while opting out of training, these are the two rows to keep apart. Most blanket blocks collapse them.

Using this as a robots.txt checker

A general robots.txt check confirms the file parses and that search crawlers are not blocked. This one answers the AI-specific question that a general checker does not ask.

What a general robots.txt check covers

Most robots.txt checkers were written for one job: confirm the syntax is valid and that Googlebot is not blocked from anything important. That is worth doing and several free tools do it well. What none of them do is tell you which AI crawler tokens your file admits, because those tokens did not exist when the tooling was written.

What this adds

The same parse, then a per-token verdict for twenty-one AI crawlers grouped by the job the operator states for each. A file can be perfectly valid, pass every syntax check, and still close the retrieval path you meant to keep open — and no conventional robots.txt check reports that.

Use both

Run a syntax check to confirm the file is well-formed, then run this to see what it does to the crawlers that decide whether you can be quoted. They answer different questions and neither replaces the other.

Blanket block — Usually wrong. Closes the citation path.
Blocking search — Severe. Removes the foundation.
No * group — Risky by default. Unnamed bots unrestricted.
Assuming enforcement — Wrong. It is voluntary.
Copy-pasted files — Stale. Tokens change.
Set and forget — Fails quietly. Migrations rewrite it.

The reference tables, in one place

Four tables follow: the tokens the checker knows about, how rule precedence is resolved, what blocking each class of token actually achieves, and the re-check triggers. They summarise the sections above and are meant to be usable on their own.

Tokens the checker evaluates
OperatorTokenStated jobCost of blocking it
OpenAIGPTBotTrainingNone in visibility terms
OpenAIOAI-SearchBotRetrievalHigh — closes the citation path
OpenAIChatGPT-UserUser-triggered fetchModerate — a person asked for it
AnthropicClaudeBotTrainingNone in visibility terms
AnthropicClaude-UserUser-triggered fetchModerate — a person asked for it
AnthropicClaude-SearchBotRetrievalHigh — closes the citation path
PerplexityPerplexityBotRetrievalHigh — closes the citation path
PerplexityPerplexity-UserUser-triggered fetchModerate — a person asked for it
GoogleGoogle-ExtendedTraining opt-out tokenNone in visibility terms
GoogleGooglebotSearch indexHigh — removes you from search
MicrosoftBingbotSearch indexHigh — removes you from search
AppleApplebotSearch indexHigh — removes you from search
AppleApplebot-ExtendedTraining opt-out tokenNone in visibility terms
AmazonAmazonbotCrawlNone in visibility terms
Metameta-externalagentCrawlNone in visibility terms
Common CrawlCCBotOpen dataset crawlNone in visibility terms
ByteDanceBytespiderCrawlNone in visibility terms
Coherecohere-aiCrawlNone in visibility terms
DiffbotDiffbotCrawlNone in visibility terms
TimpiTimpibotCrawlNone in visibility terms
Webz.ioomgilibotCrawlNone in visibility terms
How the checker resolves a rule
SituationResolutionWhy
Several groups name the same agentThe most specific agent match is usedCrawlers use the most specific group
No group names the agentThe catch-all group appliesStandard fallback behaviour
No catch-all group eitherReported as unaddressedDefault is permitted, but say so explicitly
Allow and Disallow both matchThe longer pattern winsSpecificity decides
Equal-length Allow and DisallowAllow winsThe documented tie-break
No rule matches the pathPermittedAbsence is permission
A wildcard or end anchor appearsMatched literally around the wildcardsStandard pattern handling
What blocking each class actually achieves
ClassAchievesDoes not achieveVisibility cost
Search crawlerRemoval from search over timeNothing goodSevere
Retrieval crawlerFewer live fetches for answersTraining opt-outHigh
User-triggered fetcherBlocks fetches a person asked forTraining opt-outModerate
Training tokenA stated training opt-outRemoval from answersNone
Open dataset crawlerExclusion from that dataset going forwardRemoval of past copiesLow
Everything at onceAll of the aboveAnything selectiveSevere
NothingNothingAny opt-outNone
When to run this again
TriggerWhyFrequency
After any deploymentBuild processes rewrite robots.txtEvery time
After a migrationStaging files carry Disallow: / into productionEvery time
After installing an SEO pluginSeveral manage this fileEvery time
After a hosting changePanels can inject their own rulesEvery time
When a vendor publishes a new tokenThe file cannot cover what did not existAs announced
When visibility changes unexplainablyThis is the first file to checkImmediately
RoutinelyDrift is silentQuarterly
Crawler tokens by what they govern
The top-right is where an accidental block does real damage. The bottom-left is where a deliberate block costs you nothing you were going to get anyway.
How to use the result
The re-check step matters more than it sounds. In our audits, robots.txt is the file most often changed by a developer, a plugin or a migration without anybody telling the marketing side.
What the checker looks at
The whole thing runs in your browser. We cannot see what you paste and we do not want to.
Paste — Your robots.txt. Nothing leaves the page.
Set — A path to test. Defaults to root.
Run — The check. Per token.
Read — The flags. Before the table.
Decide — Training vs retrieval. Separately.
Re-check — After every edit. It changes without you.
Separate — The two decisions. Training is not retrieval.
Protect — Search access. Above all else.
Name — What you mean. Defaults are permissive.
Verify — Against vendor docs. Linked per token.
Date — Your last check. The file drifts.
Never — Assume enforcement. Voluntary means voluntary.

Everything else we have written on search, AI and getting found

GPTBot: what it is, and what blocking it actually costs

GPTBot is OpenAI’s training crawler. It is the token most people mean when they say they want to block AI from their site, and it is also the one most often blocked by mistake.

Blocking GPTBot opts you out of training. It does not remove you from ChatGPT’s answers, because answering a question uses a different user agent entirely. Conflating the two is the single most expensive misunderstanding in this file.

The three OpenAI tokens, and what each one does

  • GPTBot: crawls to build training corpora. Blocking it is a data-licensing decision.
  • OAI-SearchBot: crawls so ChatGPT can surface and cite pages. Blocking it removes you from citations.
  • ChatGPT-User: fetches a page because a person asked for it in a session. Blocking it breaks a user’s direct request.
  • They are separate tokens and robots.txt treats them separately, so a blanket block hits all three.

What blocking GPTBot does and does not do

  • Does: signal that your content should not be used to train future models.
  • Does not: remove you from answers ChatGPT gives today.
  • Does not: affect Google, Bing, or their AI features, which use their own tokens.
  • Does not: enforce anything, because robots.txt is a voluntary convention rather than a fence.

The configuration most people actually want

If the goal is to stay out of training while remaining citable, block GPTBot and allow OAI-SearchBot and ChatGPT-User. If the goal is maximum visibility, allow all three. What almost nobody wants, and what a blanket disallow produces, is being invisible in answers while your content was already trained on years ago.

OpenAI user agents, by the job OpenAI states for each
TokenStated jobBlocking it meansAffects citations?
GPTBotTraining data collectionOpting out of future trainingNo
OAI-SearchBotRetrieval for search and citationRemoving yourself from ChatGPT citationsYes
ChatGPT-UserFetching a page a user asked forBreaking a direct user requestYes
User-agent: *Everything unnamedAll of the above at onceYes

AI, AEO and what is changing

Websites and design

Choosing and working with an agency

Social, content and brand

By industry and by situation

Want the rest of the access picture checked

The file is the first gate, not the only one. We run the full access audit — rendering, status codes, canonicalisation, sitemap accuracy and the entity records outside your site.

/contact

Not sure which of these applies to you?Tell us the situation and we will say plainly what we would do first, and what we would not.

Talk it through

Video: crawling, indexing and AI access

Background viewing only. The token model and the flag logic above are written out in full and are not drawn from these.

Getting found in search

AI, AEO and what is changing

Paid media and lead generation

Websites and design

Choosing and working with an agency

Social, content and brand

By industry and by situation

Frequently asked questions

Is this a robots.txt checker?
Yes, with an AI-specific focus. It parses the file the way a crawler does and then reports a per-token verdict for twenty-one AI crawlers. A conventional robots.txt check confirms syntax and search-crawler access; this reports what your file does to the crawlers that decide whether you can be quoted.
How do I check robots.txt for AI crawlers specifically?
Paste the file above and set the path you care about. Every token is reported separately, grouped by the job its operator states, because blocking a training crawler and blocking a retrieval crawler have completely different consequences.
Do I need a separate robots.txt check as well?
Run one, yes. A syntax validator confirms the file is well-formed and that search crawlers are unblocked. This tool assumes the file parses and answers the AI-token question that syntax checks do not ask.
What is GPTBot?
OpenAI’s training crawler. It collects content used to build training corpora, and it is a different user agent from the ones that retrieve pages to answer a question.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is the training crawler. Answering a question uses OAI-SearchBot and ChatGPT-User, which are separate tokens. Blocking GPTBot alone opts you out of training while leaving you citable.
How do I block GPTBot in robots.txt?
Add a User-agent: GPTBot group with Disallow: / beneath it. Be deliberate about whether you also want to block OAI-SearchBot and ChatGPT-User, because those are what affect whether you appear in answers.
Does GPTBot respect robots.txt?
OpenAI states that it does, and publishes the token for that purpose. robots.txt is a voluntary convention rather than an enforcement mechanism, so it is a request that well-behaved crawlers honour rather than a technical barrier.
Should I block GPTBot?
It is a licensing decision, not an SEO one. Blocking it signals your content should not train future models and costs you nothing in visibility. Blocking the retrieval tokens alongside it is what costs visibility, and that is usually done by accident.
Is this tool free?
Yes. The analysis runs entirely in your browser and nothing you paste is uploaded. The summary is free to read; unlocking the full findings asks for your name, business email and phone so we can send you a copy.
Why does it ask me to paste the file instead of taking my URL?
Because a browser cannot read another site’s files without that site’s permission. A tool that claims to fetch your robots.txt from a URL is routing it through a server, and we would rather ask for a copy and paste than route your site through our infrastructure.
Does blocking Google-Extended remove me from AI Overviews?
No. Per Google’s own documentation it is a training opt-out token and does not affect inclusion in Google Search, and AI Overviews are built from the search index that Googlebot crawls. This is the single most common misunderstanding in this file.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI states different jobs for them — one is described as a training crawl, the other as search retrieval. Blocking the first is a training opt-out with no visibility cost; blocking the second closes the path by which you could be fetched and cited.
Should I block AI crawlers at all?
That depends on what your content is worth to you, and the honest answer is different for a news publisher and a local service business. What the tool gives you is an accurate picture of what your file currently does, which is usually not what the person who wrote it intended.
Is robots.txt actually enforced?
No. It is a convention that well-behaved crawlers honour and others ignore. Nothing in the file prevents a fetch. If content genuinely must not be retrieved, the mechanism is authentication or server-side blocking.
Does blocking a crawler remove content that was already crawled?
No. It governs future crawling. It also has no effect on material about you published elsewhere — directories, reviews, press and third-party pages are outside your robots.txt entirely.
What does ‘no rule at all’ mean in the results?
That the token is not named anywhere in your file and there is no catch-all group covering it either. The default is permitted, but we report it separately so it is a decision rather than an oversight.
Why does it test a specific path?
Because rules are path-based. A crawler can be allowed on your homepage and blocked on a subdirectory, and the answer for ‘/’ tells you nothing about ‘/blog/’ if a rule targets it.
How does it decide between a conflicting Allow and Disallow?
The longer matching pattern wins, and where both match with equal specificity the Allow takes precedence. That is the documented tie-break and it is what the checker implements.
Does it handle wildcards?
Yes, both the asterisk wildcard and the end-of-path anchor are handled when matching patterns to the path you test.
Why link to each operator’s documentation?
Because tokens are added, renamed and split over time, and the operator’s own page is the source of truth. This page is not, and we would rather send you to the authority than have you rely on our snapshot.
My file has no User-agent: * group. Is that bad?
It means every crawler you have not named is unrestricted. That may be exactly what you want. The tool flags it so it is deliberate.
Can this tell me why I am not appearing in AI answers?
Only whether access is part of it. Access is the first gate, not the only one — content that is not extractable, an inconsistent entity record or a question owned by a publisher will all keep you out of an answer with a perfectly permissive robots.txt.
How often should I re-run it?
After every deployment, migration, plugin install or hosting change, and quarterly otherwise. In our audit work this is the file most often changed without anybody on the marketing side being told.
Does the tool store or log what I paste?
No. There is no request to any server. You can confirm that by opening your browser’s network panel while you run it.
Can I use this for a client site?
Yes. There is no restriction and no attribution requirement, though a link back is always welcome.
What if a crawler I care about is not listed?
The table covers the tokens we see most often. A token that is not listed is not evaluated, and the general rules the checker applies would still resolve it the same way — you can add a temporary User-agent line for it and re-run to see how your file treats it.
Does blocking Bytespider or CCBot matter?
Those are crawl and open-dataset tokens rather than retrieval paths, so blocking them carries little visibility cost. Whether it achieves what you want is a separate question, and past copies are not withdrawn.
Is there a version that checks a whole site?
Not as a free tool. Checking every path, every subdomain and the rendering behind them is audit work rather than a paste-in widget, and we would rather be honest about which is which.
What should I do first if the results look wrong?
Confirm the search crawlers are allowed. That finding outranks everything else on the page, and it is the one most often caused accidentally by a staging file reaching production.
Does this replace Search Console?
No. Search Console reports what Google actually did with your site; this reports what your file asks every crawler to do. They answer different questions and you want both.

Sources and further reading

  1. Google Search Essentials — SEO starter guide
  2. Google: creating helpful, reliable, people-first content
  3. Google: intro to structured data
  4. Google: LocalBusiness structured data
  5. Google: FAQPage structured data
  6. Google: Article structured data
  7. Google: Product structured data
  8. Google: title links in search results
  9. Google: control your snippets
  10. Google: robots.txt introduction
  11. Google: sitemaps overview
  12. Google: consolidate duplicate URLs
  13. Google: redirects and Search
  14. Google: JavaScript SEO basics
  15. Google: multi-regional and multilingual sites
  16. Google Search Central Blog
  17. Google: get started with Search Console
  18. Google: how local search results are determined
  19. Google Business Profile: prohibited and restricted content
  20. Google Business Profile: address and service area guidelines
  21. Google Business Profile: review policy
  22. Google Business Profile: add or edit categories
  23. Google Ads: location targeting settings
  24. Google Ads: about negative keywords
  25. Google Ads: about Quality Score
  26. Google Ads: importing offline conversions
  27. Google Ads: about Smart Bidding
  28. Google Ads: about Performance Max
  29. Google Local Services Ads: eligibility and screening
  30. Google Ads: keyword match types
  31. Google Analytics 4: about conversions
  32. Google Analytics 4: attribution models
  33. US Census Bureau QuickFacts: New Jersey
  34. US Census Bureau: American Community Survey
  35. US Census: Statistics of US Businesses
  36. Bureau of Labor Statistics: New Jersey data
  37. BLS: Occupational Employment and Wage Statistics
  38. NJ Department of Labor: labor market information
  39. New Jersey Business Action Center
  40. US Small Business Administration: New Jersey district
  41. USA.gov: business resources
  42. web.dev: Core Web Vitals explained
  43. web.dev: Largest Contentful Paint
  44. web.dev: Cumulative Layout Shift
  45. web.dev: Interaction to Next Paint
  46. Google PageSpeed Insights
  47. Google Rich Results Test
  48. Google Search Console
  49. W3C Markup Validation Service
  50. Schema.org: LocalBusiness type
  51. Schema.org: Service type
  52. Schema.org: FAQPage type
  53. Schema.org: HowTo type
  54. W3C: WCAG 2.2 quick reference
  55. FTC: CAN-SPAM Act compliance guide
  56. FCC: telemarketing and robocall rules (TCPA)
  57. FTC endorsement guides — reviews and testimonials
  58. FTC: rule on consumer reviews and testimonials
  59. HHS: HIPAA guidance on online tracking technologies
  60. New Jersey Courts: attorney advertising guidelines
  61. New Jersey DCA: construction codes and permits
  62. New Jersey Home Improvement Contractor registration
  63. New Jersey Division of Consumer Affairs
  64. TikTok for Business
  65. TikTok Creative Center
  66. TikTok Ads Help Center
  67. TikTok Community Guidelines
  68. TikTok Terms of Service
  69. TikTok Privacy Policy
  70. TikTok Safety Center
  71. TikTok Transparency Center
  72. TikTok Creator Portal
  73. TikTok Newsroom
  74. TikTok for Developers
  75. TikTok advertising solutions
  76. TikTok Creator Marketplace
  77. TikTok Business Center
  78. TikTok for Business blog
  79. TikTok Creative Center: top ads
  80. TikTok Branded Content policy
  81. TikTok Shop for sellers
  82. Instagram for Business
  83. Instagram for Creators
  84. Instagram Help Center
  85. About Instagram
  86. Meta Business Suite
  87. Meta Business Help Center
  88. Meta Transparency Center
  89. About Meta
  90. Meta: Instagram platform docs
  91. YouTube Creators
  92. YouTube Official Blog
  93. YouTube Shorts help
  94. How YouTube Works
  95. YouTube Studio
  96. LinkedIn Marketing Solutions
  97. LinkedIn Help
  98. Pinterest Business
  99. Pinterest Business Help
  100. Snapchat for Business
  101. X for Business
  102. Reddit communities
  103. Reddit for Business Help
  104. ASCAP
  105. BMI
  106. SESAC
  107. Global Music Rights
  108. PRS for Music (UK)
  109. PPL (UK)
  110. SOCAN (Canada)
  111. APRA AMCOS (Australia)
  112. GEMA (Germany)
  113. SACEM (France)
  114. SIAE (Italy)
  115. JASRAC (Japan)
  116. IFPI
  117. RIAA
  118. National Music Publishers Association
  119. Harry Fox Agency
  120. SoundExchange
  121. Music Reports
  122. Epidemic Sound
  123. Artlist
  124. Soundstripe
  125. PremiumBeat
  126. AudioJungle
  127. Free Music Archive
  128. Creative Commons
  129. Incompetech
  130. FTC: advertising and marketing
  131. FTC: disclosures 101
  132. FTC: endorsement guides
  133. FTC: consumer reviews rule
  134. FTC: advertising FAQs
  135. US Copyright Office
  136. US Copyright Office: DMCA
  137. US Copyright Office: music FAQ
  138. US Copyright Office: fair use FAQ
  139. USPTO: trademarks
  140. UK Advertising Standards Authority
  141. ACCC (Australia)
  142. Competition Bureau Canada
  143. GDPR overview
  144. California Consumer Privacy Act
  145. COPPA
  146. FTC: children’s privacy
  147. W3C Web Accessibility Initiative
  148. W3C: WCAG
  149. W3C: captions
  150. W3C: making audio and video accessible
  151. ADA.gov
  152. WebAIM
  153. Epilepsy Foundation
  154. Pew Research: internet and technology
  155. DataReportal
  156. US Census Bureau
  157. US Bureau of Labor Statistics
  158. Interactive Advertising Bureau
  159. Think with Google
  160. Google Trends
  161. Nielsen insights
  162. Schema.org: VideoObject
  163. Schema.org: SocialMediaPosting
  164. Schema.org: MusicRecording
  165. Schema.org: HowTo
  166. Schema.org: FAQPage
  167. Schema.org: Organization
  168. Google: video best practices
  169. Google: video structured data
  170. CapCut
  171. Adobe Premiere Rush
  172. DaVinci Resolve
  173. Canva
  174. Descript
  175. VEED
  176. Kapwing
  177. Otter.ai
  178. Later
  179. Buffer
  180. Hootsuite
  181. Sprout Social
  182. Google Analytics
  183. Google Search Console
  184. Google Analytics developer docs
  185. GA4: events and conversions
  186. Matomo
  187. Plausible Analytics
  188. Similarweb
  189. UK Information Commissioner’s Office
  190. Office of the Privacy Commissioner of Canada
  191. Australian OAIC
  192. European Data Protection Board
  193. EU data protection
  194. EU Digital Services Act
  195. Ofcom
  196. FCC
  197. AIGA
  198. Nielsen Norman Group
  199. Smashing Magazine
  200. web.dev
  201. MDN: web media
  202. MDN: the video element
  203. ISO 21001 (reference)
  204. Buma/Stemra (Netherlands)
  205. STIM (Sweden)
  206. Teosto (Finland)
  207. Koda (Denmark)
  208. TONO (Norway)
  209. IMRO (Ireland)
  210. SGAE (Spain)
  211. ZAiKS (Poland)
  212. KOMCA (South Korea)
  213. MCSC (China)
  214. CISAC
  215. World Intellectual Property Organization
  216. TikTok: creating videos
  217. TikTok: exploring videos
  218. TikTok: privacy settings
  219. TikTok: growing your audience
  220. TikTok Creator Academy
  221. TikTok Effect House
  222. TikTok for small business
  223. Instagram: Reels help
  224. YouTube: Shorts best practice
  225. How YouTube recommends
  226. Pinterest Predicts
  227. Snapchat for Business
  228. Hootsuite blog
  229. Social Media Examiner
  230. Marketing Week
  231. Adweek
  232. Google Search Essentials — SEO starter guide
  233. Google: creating helpful, reliable, people-first content
  234. Google: intro to structured data
  235. Google: LocalBusiness structured data
  236. Google: FAQPage structured data
  237. Google: Article structured data
  238. Google: Product structured data
  239. Google: title links in search results
  240. Google: control your snippets
  241. Google: robots.txt introduction
  242. Google: sitemaps overview
  243. Google: consolidate duplicate URLs
  244. Google: redirects and Search
  245. Google: JavaScript SEO basics
  246. Google: multi-regional and multilingual sites
  247. Google Search Central Blog
  248. Google: get started with Search Console
  249. Google: how local search results are determined
  250. Google Business Profile: prohibited and restricted content
  251. Google Business Profile: address and service area guidelines
  252. Google Business Profile: review policy
  253. Google Business Profile: add or edit categories
  254. web.dev: Core Web Vitals explained
  255. web.dev: Largest Contentful Paint
  256. web.dev: Cumulative Layout Shift
  257. web.dev: Interaction to Next Paint
  258. Google PageSpeed Insights
  259. Google Rich Results Test
  260. Google Search Console
  261. W3C Markup Validation Service
  262. Schema.org: LocalBusiness type
  263. Schema.org: Service type
  264. Schema.org: FAQPage type
  265. Schema.org: HowTo type
  266. W3C: WCAG 2.2 quick reference
  267. US Census Bureau QuickFacts: New Jersey
  268. US Census Bureau: American Community Survey
  269. US Census: Statistics of US Businesses
  270. Bureau of Labor Statistics: New Jersey data
  271. BLS: Occupational Employment and Wage Statistics
  272. NJ Department of Labor: labor market information
  273. New Jersey Business Action Center
  274. US Small Business Administration: New Jersey district
  275. USA.gov: business resources
  276. OpenAI – GPTBot documentation
  277. OpenAI – OAI-SearchBot documentation
  278. OpenAI – ChatGPT-User documentation
  279. Anthropic – ClaudeBot documentation
  280. Anthropic – Claude-User documentation
  281. Anthropic – Claude-SearchBot documentation
  282. Perplexity – PerplexityBot documentation
  283. Perplexity – Perplexity-User documentation
  284. Google – Google-Extended documentation
  285. Google – Googlebot documentation
  286. Microsoft – Bingbot documentation
  287. Apple – Applebot documentation
  288. Apple – Applebot-Extended documentation
  289. Amazon – Amazonbot documentation
  290. Meta – meta-externalagent documentation
  291. Common Crawl – CCBot documentation

Want this done for your site?We build and maintain the search, content and paid programmes described on this page.

Get a free proposal

Get a free marketing proposal

Tell us what you are trying to grow and we will come back with a plan, not a pitch deck. Same-day reply on weekdays.

Our website uses cookies from third party services to improve your browsing experience. Read more about this and how you can control cookies by clicking "Privacy Preferences".
Privacy Preferences
When you visit our website, it may store information through your browser from specific services, usually in form of cookies. Here you can change your privacy preferences. Please note that blocking some types of cookies may impact your experience on our website and the services we offer.
Privacy Policy
You have read and agreed to our privacy policy
Required
Contact Us
0