Skip to main content Scroll Top

Why ChatGPT Recommends Your Competitor Instead of You

Updated September 2026 · Written and maintained by the Progression Agency strategy team

Six causes, ordered so the free ones get ruled out first — and the three checks that explain most cases in about an hour.

On this page · 12 sections
  1. Why ChatGPT recommends your competitor instead of you
  2. Which symptom do you actually have?
  3. Run these checks before you spend anything
  4. Explanations that are usually wrong
  5. Why your competitor actually wins
  6. What fixes each cause, and what it costs
  7. Why the answer differs by platform
  8. Everything else we have written on search, AI and getting found
  9. Symptom to cause: the diagnostic table
  10. What each platform needs from you
  11. The five-minute version
  12. What this page cannot tell you

The short answerUsually it is not that the model preferred them. It is that you were never eligible. Rule out six causes in order: a blocked AI crawler, content that only exists after JavaScript runs, not being in the index, no clean passage to quote, sources that disagree about your business, and no third-party corroboration. The first three are binary, free to test and account for most cases. And if you checked once and saw nothing, that proves little — answers vary between runs, so ask it five times first.

Nobody can tell you why a model specifically chose a competitor; no provider exposes that reasoning. What is knowable is whether you were eligible at all, and usually you were not.

Work backwards from the symptomWork backwards from the symptom
Causes one to three are binary and cheap to test. Four to six are gradual and are where competitors usually win rather than where you are blocked.

Why ChatGPT recommends your competitor instead of you

In almost every case we have looked at, the answer is not that the model prefers them. It is that you were never eligible. Something between your server and the model stopped your content arriving at all, and your competitor simply did not have that problem.

There are six possible causes and they should be ruled out in order, because the first three are binary, free to test, and account for most cases. The last three are gradual, and they are where a competitor genuinely wins rather than where you are simply absent.

Before any of that, one correction worth making: if you checked once and concluded you are invisible, you have not established anything. Answers vary between runs of the identical prompt. Ask it five times before drawing a conclusion.

Blocked crawler — Cause. Free to fix. Check first, always.
JavaScript-only content — Cause. Expensive. The big budget decision.
Not indexed — Cause. Ordinary SEO is the prerequisite.
Buried answer — Cause. Cheap to fix, affects every page.
Entity confusion — Cause. Slow, and it causes wrong answers not just absence.
No corroboration — Cause. Slowest. Usually why a competitor wins.

Cause one: the crawler never reached you

If robots.txt disallows GPTBot, or your server or CDN returns a 403 to it, your content has never entered the system. This is free to check and free to fix, and it turns up far more often than anyone expects — usually as a security plugin default nobody remembers enabling.

Cause two: the page needs JavaScript

Many AI crawlers do not execute it. If a scriptless fetch of your page returns a navigation bar, a footer and nothing else, the model has never seen a word of your actual content. It looks perfect to you because you check it in a browser.

Cause three: you are not in the index

Google’s AI surfaces retrieve from the Google index. A page that was never indexed cannot be cited, which is why answer engine work sits on top of ordinary search work rather than replacing it.

Cause four: there is no clean passage to lift

Retrieval selects a section, not a page. If the answer to the question lives in paragraph four after three paragraphs of context, there is nothing tidy to quote, and a competitor whose first sentence answers the heading gets chosen.

Cause five: your sources disagree about you

When the descriptions of your business differ across your site, your listings and third-party profiles, a model has conflicting information. Conflict produces caution, and caution reads as omission.

Cause six: nobody else says it about you

Your competitor is probably mentioned by third parties who are not them. Independent corroboration is what makes a claim safe to repeat, and it is the slowest thing on this list to build.

Which symptom do you actually have?

The pattern of your absence narrows the cause considerably. Absent on every platform points at access or rendering. Absent on one points at that platform’s specific crawler. Present but always behind competitors points at structure and corroboration rather than at anything broken.

Being named but described inaccurately is a different problem again, and a more urgent one — a wrong claim about your pricing or services repeated in an answer does more damage than silence.

Which symptom do you actually have?Which symptom do you actually have?
The last row is the commonest error. Answers vary between runs, so a single check cannot distinguish absence from variance.
Locating your causeLocating your cause
Absent on every platform points at access or rendering. Absent on one points at that platform’s specific crawler. Behind competitors everywhere points at structure and corroboration.
Absent everywhere — Symptom. Points at access or rendering.
Absent on one platform — Symptom. Points at that platform's crawler.
Always ranked behind — Symptom. Points at structure and corroboration.
Described wrongly — Symptom. Points at entity inconsistency.
Was there, now gone — Symptom. Points at a site change or a model update.
Ranks in Google, absent in AI — Symptom. Points at rendering or passage structure.

Run these checks before you spend anything

Six tests, four of them free, and together they take about an hour. Most sites find their cause inside them, which is why we publish the method rather than making it the paid part.

Do them in this order. If test one fails, the others are moot until it is fixed.

What to do this weekWhat to do this week
Four of these six cost nothing and take under an hour in total. Most sites find their cause inside them.
Read robots.txt — Test. Five crawlers by name. Two minutes.
Scriptless fetch — Test. curl, or dev tools with JS off.
User-agent request — Test. Confirm 200, not 403 or soft 404.
Five runs of the prompt — Test. Absence needs repetition to mean anything.
Log grep — Test. Did the bots actually arrive?.
Read your own section — Test. Is the answer in sentence one?.

Read your robots.txt

Search for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended. Any Disallow under one of those is a complete explanation for that platform.

Fetch a page without JavaScript

Disable it in your browser’s developer tools and reload, or fetch with curl. If very little remains, this is your answer and it is an engineering problem.

Request the page as an AI crawler

Set the user agent string. A 403 or a soft 404 served only to bots is invisible in ordinary browsing and completely disqualifying.

Ask the failing prompt five times

Store every answer verbatim. If you appear in two of five, you are marginally retrievable rather than absent, which is a different problem.

Check your server logs

Filter for the AI user agents over the last thirty days. This is the only evidence that access is real rather than theoretical.

Read your own section

Find the page that should answer the question and read the section that promises it. Is the answer in the first sentence, or the fourth paragraph?

Explanations that are usually wrong

Four keep coming up and none of them is the cause. They did not pay for placement, because no ad inventory exists inside AI answers. The model is not biased against you. Publishing more articles will not fix it. And llms.txt is not the missing piece.

Domain authority is the interesting one. It matters here, but far less than in a search result, which is precisely why small sites do get cited alongside large publishers. If someone tells you your authority is the whole explanation, they have not run the technical checks.

Explanations that are usually wrongExplanations that are usually wrong
Domain authority is ‘part’ deliberately: it matters far less here than in a search result, which is why small sites can be cited beside large publishers. It is rarely the whole explanation.
Not the cause — They paid. No ad inventory exists in AI answers.
Not the cause — Model bias. Retrieval, not preference.
Not the cause — Too few articles. Volume is not the lever.
Not the cause — Missing llms.txt. Low adoption, not a ranking factor.
Not the cause — Wrong keywords. There is no keyword to target.
Rarely the cause — Low domain authority. Matters far less here than in search.

‘They paid to be there’

There is no ad inventory, no submission process and no paid tier inside AI answers. Nobody bought that position.

‘The model is biased’

Retrieval selects on relevance and apparent reliability, not preference. Absence is nearly always an eligibility problem rather than a judgement.

‘We need more content’

Volume is not the lever. One page whose section answers the question cleanly beats twenty that circle it.

‘We need llms.txt’

Adoption is low and it is not a ranking factor. Adding it is fine; expecting it to change anything is not.

‘Our authority is too low’

It matters less than in search. A DA-18 site currently holds position one for a $45-a-click commercial term in this very category.

Why your competitor actually wins

Rarely because they are outstanding on one dimension. Usually because they fail none of them: reachable, renders, clean passage, clear entity, corroborated by people who are not them.

That is an encouraging finding rather than a discouraging one, because four of those five are things you control directly and three of them are cheap.

Why a model picks one source over anotherWhy a model picks one source over another
Weights are indicative, not measured — no provider publishes them. The shape is the point: a competitor rarely wins on one factor, they win by not failing any of them.

They are reachable

Their robots.txt permits the crawlers and their server returns 200s.

Their page renders

Content is in the HTML rather than assembled after load.

Their answer is liftable

The section that promises the answer delivers it immediately.

Their identity is unambiguous

Every source describes the business the same way.

Other people vouch for them

Independent mentions, reviews and coverage, none of it theirs.

None of this is proprietary

There is no trick here you cannot replicate this quarter.

What fixes each cause, and what it costs

Matched directly to the diagnosis. Unblocking a crawler costs minutes. Fixing rendering is an engineering project. Moving an answer into the first sentence of a section is cheap and affects every page. Corroboration is slow and cannot be rushed.

The inversion is the important part: the cheapest fix on the list is frequently the one that was silently costing you everything.

Once you know the cause, what fixes itOnce you know the cause, what fixes it
The first fix is free and the second is expensive. Establishing which one you need is the entire value of diagnosing before acting.
What each fix costs, relativeWhat each fix costs, relative
Note the inversion against impact: the cheapest fix is often the one that was silently costing you everything.
Free fixes first — Act. Crawler access and a clean answer sentence.
Then decide on rendering — Act. The largest single cost.
Then structure — Act. Per template, not per page.
Then entity — Act. One description everywhere.
Then corroboration — Act. Slow, durable, hardest to fake.
Measure throughout — Act. Fixed prompts, stored answers, monthly.

Why the answer differs by platform

Each surface uses its own crawler and its own retrieval. Perplexity fetches live at question time and is generally kindest to smaller sites. Google’s AI surfaces need ordinary indexation. Copilot leans on the Bing index. ChatGPT uses both a crawled index and live retrieval.

So being absent on one platform and present on others is genuinely informative — it usually points at that platform’s specific crawler rather than at anything wrong with the page.

ChatGPT — Platform. GPTBot and OAI-SearchBot; index plus live retrieval.
Perplexity — Platform. PerplexityBot; fetches live, kindest to small sites.
Google AI Overviews — Platform. Google index; needs ordinary indexation.
Copilot — Platform. Leans on the Bing index.
Claude — Platform. ClaudeBot; web search when enabled.
Gemini — Platform. Google-Extended governs grounding use.
Numbers that frame the problemNumbers that frame the problem
The second zero matters: nobody, including your competitor’s agency, can tell you why a model chose them. What can be established is whether you were even eligible.

Want the diagnosis rather than the checklist?

The tests above are free and complete. An AI visibility audit runs them properly, reads your server logs, and tells you which of the six causes is actually yours — with a baseline recorded so you can prove the fix worked.

/ai-visibility-audit

Want this done for your site?We build and maintain the search, content and paid programmes described on this page.

Get a free proposal

Everything else we have written on search, AI and getting found

AI, AEO and what is changing

Websites and design

Choosing and working with an agency

Social, content and brand

By industry and by situation

Not sure which of these applies to you?Tell us the situation and we will say plainly what we would do first, and what we would not.

Talk it through

Symptom to cause: the diagnostic table

Work from what you are seeing to the cause that produces it, rather than guessing at fixes.

Start with what you observe, not with what you assume
What you observeMost likely causeHow to confirm itCost to fix
Absent on every platformBlocked crawler or JS-only contentrobots.txt read, then a scriptless fetchFree, or an engineering project
Absent on one platform onlyThat platform’s crawler is blocked or never visitedrobots.txt for that agent, then a log grepUsually free
Present but always behind competitorsWeak passage structure or no corroborationRead your own section; search for third-party mentionsCheap, then slow
Named but described wronglySources disagree about your businessList every source describing you and compareWeeks of tidying
Named on ‘what is X’, absent on ‘who provides X’No provider-intent content, or no corroborationCheck whether a page answers the provider question at allModerate
Was cited last month, gone nowA site change, or a model updateDiff your robots.txt and templates; check the dateDepends
Ranks well in Google, absent in AIRendering or passage structureScriptless fetch; read the sectionVaries
Appears in two runs out of fiveMarginal retrievability, not absenceRepeat runs and store the answersCheap: strengthen the passage

What each platform needs from you

The requirements differ by platform, and a fix that satisfies one may do nothing for another.

Different crawlers, different retrieval, different failure modes
PlatformCrawlerRetrieval styleMost common reason you are absent
ChatGPTGPTBot, OAI-SearchBotCrawled index plus live retrievalCrawler disallowed in robots.txt
PerplexityPerplexityBotLive fetch at question timeServer returns 403 to the bot
Google AI OverviewsGooglebot, Google-ExtendedGoogle indexPage not indexed, or thin passage
Google AI ModeGooglebot, Google-ExtendedGoogle indexSame as above
CopilotBingbotBing indexNot indexed in Bing specifically
ClaudeClaudeBotWeb search when enabledClaudeBot disallowed
GeminiGoogle-ExtendedGoogle groundingGoogle-Extended disallowed

The five-minute version

If you only have five minutes, these are the checks that most often find the cause.

If you only do three things
Do thisWhereWhat a failure looks like
Read robots.txtyoursite.com/robots.txtDisallow under GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot or Google-Extended
Load a page with JavaScript offBrowser dev tools, or curlNavigation and footer only; no body content
Ask the prompt five timesChatGPT and PerplexityNamed zero times out of five, consistently

What this page cannot tell you

Why the model chose them, specifically. No provider exposes model reasoning, so nobody can answer that — not us, not your competitor’s agency, and not any tool on the market. What can be established with certainty is whether you were eligible to be chosen at all, and for most sites asking this question, the honest answer turns out to be no.

Knowable versus unknowable
QuestionCan it be answered?How
Did the crawler reach my page?Yes, definitivelyServer logs
Does my page render for a machine?Yes, definitivelyScriptless fetch
Am I in the index?YesIndex coverage checks
Is there a clean passage?Yes, by reading itHuman judgement
Do my sources agree?YesList and compare them
Why did the model prefer them?NoNot exposed by any provider

Getting found in search

AI, AEO and what is changing

Paid media and lead generation

Websites and design

Choosing and working with an agency

Social, content and brand

By industry and by situation

Frequently asked questions

Why does ChatGPT recommend my competitor instead of me?
Usually because you were never eligible rather than because the model preferred them. Six causes, in order of how cheaply they can be ruled out: a blocked crawler, JavaScript-only content, not being indexed, no clean passage to quote, sources that disagree about your business, and no third-party corroboration.
Did my competitor pay to be recommended?
No. There is no ad inventory, no submission process and no paid tier inside AI answers. Nobody purchased that position.
Is the model biased against my brand?
No. Retrieval selects on relevance and apparent reliability. Absence is nearly always an eligibility problem — the content never arrived — rather than a judgement about you.
How do I check if ChatGPT can even see my site?
Read your robots.txt for GPTBot and OAI-SearchBot, request a page with those user agent strings and confirm a 200 rather than a 403, then grep your server logs for those agents over the last thirty days.
I checked once and I was not mentioned. Am I invisible?
Not established. Answers vary between runs of the identical prompt, so a single check cannot distinguish absence from ordinary variance. Ask it five times and store every answer before concluding anything.
Why am I absent on one platform but present on others?
That points at the specific crawler for the missing platform rather than at anything wrong with the page. Each surface uses its own crawler and its own retrieval.
My site ranks well in Google but never appears in AI answers. Why?
Most often rendering. Google usually executes JavaScript and indexes the result; many AI crawlers do not, so a page assembled in the browser returns almost nothing to them.
Is my domain authority too low to be recommended?
It matters far less here than in a search result, which is why small sites are cited alongside large publishers. A site with authority 18 currently holds position one for a $45-a-click term in this category.
Will publishing more articles fix it?
No. Volume is not the lever. One page whose section answers the question in its first sentence beats twenty that circle the topic.
Will adding llms.txt fix it?
No. Adoption is low and it is not a ranking factor. Add it if you like, but do not expect it to change anything.
Why is my competitor cited when their content is worse?
Because citation is not a quality judgement. They are reachable, their page renders, their answer is liftable, their identity is unambiguous and third parties vouch for them. Any of those can be true of a worse page.
What is the cheapest thing that might fix this?
Removing a Disallow on an AI crawler in robots.txt. It costs nothing, takes minutes, and it is a complete explanation for absence on a meaningful minority of sites.
What is the most expensive thing that might fix this?
Rendering. If your commercial pages only produce content after JavaScript runs, correcting that is an engineering project and it dominates every other decision.
AI describes my business incorrectly. Is that the same problem?
No, and it is more urgent. Inaccurate description points at sources disagreeing about you. A wrong claim about your pricing repeated in an answer does more damage than silence.
I was cited last month and now I am not. What changed?
Either something on your site — a robots.txt edit, a template change, a new security rule — or a model update outside your control. Diff your configuration first, then check the date against known platform changes.
How many times should I test a prompt?
Three to five, in one sitting, with every answer stored. Appearing in two runs out of five means you are marginally retrievable, which is a different and more fixable problem than being absent.
Can a tool tell me why the model chose my competitor?
No. No provider exposes model reasoning, so every explanation offered by a tool is inference. What tools can establish is whether you appear; only a site diagnosis explains why not.
Which platform is easiest to get cited by?
Perplexity, generally, because it fetches live at question time and depends less on deep indexation. It is often where a smaller site sees movement first.
How long until this is fixed?
Crawler access shows in logs within days. Rendering takes a week or two. Passage structure improves extraction over weeks. Corroboration takes months and cannot be rushed.
Should I hire someone or fix it myself?
Run the free checks first. They cost an hour and find the cause on most sites. Hire when the answer is rendering, or when you want a baseline recorded properly so the fix can be proven.
What if all six checks pass and I am still absent?
Then you are likely competing on corroboration against a better-established competitor, which is the slow problem rather than the broken one. That is a digital PR and original-data question, not a technical one.
Does being named without a link still count?
Yes, and it should be counted separately. A mention builds familiarity even without the click, and mentions often precede citations as corroboration accumulates.

Want this done for your site?We build and maintain the search, content and paid programmes described on this page.

Get a free proposal

Get a free marketing proposal

Tell us what you are trying to grow and we will come back with a plan, not a pitch deck. Same-day reply on weekdays.

Our website uses cookies from third party services to improve your browsing experience. Read more about this and how you can control cookies by clicking "Privacy Preferences".
Privacy Preferences
When you visit our website, it may store information through your browser from specific services, usually in form of cookies. Here you can change your privacy preferences. Please note that blocking some types of cookies may impact your experience on our website and the services we offer.
Privacy Policy
You have read and agreed to our privacy policy
Required
Contact Us