LLM SEO: The Technical Work That Decides Whether AI Can Quote You

Updated September 2026 · Written and maintained by the Progression Agency strategy team

Most sites fail before any content question arises — the crawler is blocked, or the page is empty without JavaScript. This is the technical half of AI visibility, written for the person who has to implement it.

What is LLM SEO?

LLM SEO is the technical and structural work that makes a website retrievable and quotable by large language models — so an assistant can reach your page, parse it, select it, and reuse your wording rather than a competitor’s.

It is the engineering half of what is also sold as answer engine optimization and generative engine optimization. The three terms describe substantially the same practice; LLM SEO is simply the name that gets used when the conversation is with a developer rather than a marketer. We explain that overlap properly on our terminology page.

The reason it deserves its own treatment is that the technical failures are both the most common and the least discussed. Agencies sell content programmes; the sites we audit are usually failing before any content question arises.

What has to be true before an LLM can quote you?

Four things, in order: the crawler can reach the page, the content parses without JavaScript, the passage is selected as a candidate, and the wording is clear enough to reuse.

Fail the first and nothing downstream matters. Most of the money spent on this discipline is spent on the fourth while the first is quietly broken.

Reach

The crawler has to be permitted and the server has to answer. Accidental blanket disallows, bot-management rules and aggressive rate limiting are the three commonest causes, and all three are usually invisible from the browser.

Parse

The content has to exist in the HTML. Many AI crawlers execute little or no JavaScript, so a page that assembles its text client-side can return effectively empty to the systems you care about while looking perfect to you.

Retrieve

Your passage competes against every other candidate for the question. Relevance to the question as asked matters more than keyword presence, because matching happens on meaning.

Reuse

The passage has to be quotable. Hedged, meandering writing gets paraphrased into something generic and uncredited; a crisp declarative sentence gets lifted intact.

What does a technical LLM SEO audit actually check?

Twelve things, split between access (can a bot get the content) and structure (can a model use it once it has).

Every one of these is verifiable. None of them requires trusting a proprietary score.

robots.txt

Which AI crawlers are permitted. We find accidental blocks constantly, usually inherited from a template or added during a scraping scare and never reviewed.

Server logs

The only unarguable evidence of which crawlers arrived and what they fetched. This is the first artefact we ask for and the one most clients have never looked at.

JavaScript rendering

What each key page contains with scripts disabled. The gap between that and what you see in a browser is the gap the crawler experiences.

Response time

Whether fetches complete before the crawler gives up. Slow pages are not merely ranked lower here; they are sometimes not retrieved at all.

Canonical consistency

Whether duplicate URLs split the signal across variants, which dilutes everything downstream.

Status codes for bots

Whether key pages return 200 to crawlers as well as to browsers. Bot management sometimes serves a challenge page instead, which reads as empty content.

Schema accuracy

Structured data that agrees with the visible page. Markup asserting something the copy contradicts reduces machine confidence in the rest of the page.

Heading structure

Whether each section answers one question. Headings phrased as questions, with the answer immediately beneath, are measurably easier to extract.

Answer position

Whether the answer precedes the explanation. This single change moves more than any other editorial intervention we make.

Entity consistency

Whether the same description of the organisation appears everywhere, without variation. Inconsistency is why assistants describe companies wrongly.

Internal linking

Whether related pages reinforce each other, which affects both retrieval and the model’s sense of what you are authoritative about.

Freshness signals

Whether update dates are genuine and visible. Fabricated freshness is detectable and counterproductive.

Which AI crawlers should you allow?

At minimum GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, ClaudeBot and Bingbot — unless you have a deliberate business reason to exclude one.

This is a commercial decision dressed as a technical one. Blocking protects your content from uncredited use; it also guarantees you are not cited. Most businesses selling something want to be found, and block by accident rather than by choice.

Which levers move which stage?

Crawler rules move reach. Server rendering moves parsing. Answer structure moves reuse. Entity clarity moves retrieval. Nothing moves all four.

This is why single-tactic programmes stall. A schema-only engagement improves one column; an content-only engagement improves another; the site with a blocked crawler improves nothing at all until that is fixed.

What are the most common technical mistakes?

Blocking everything, rendering client-side only, schema that contradicts the page, burying the answer, treating llms.txt as a strategy, and publishing volume instead of clarity.

Five of these six are free to fix. The sixth is free to stop doing.

Blocking everything

Usually accidental. A rule added during a scraping scare, or inherited from a starter template, and never revisited.

Client-side rendering only

The most expensive silent failure. The page looks complete in a browser and returns nothing useful to a crawler that does not run scripts.

Schema that contradicts the page

Worse than no schema. It undermines machine confidence in everything else the page asserts.

Burying the answer

The commonest editorial failure. The answer exists, four hundred words down, and a competitor’s one-sentence version gets quoted instead.

Treating llms.txt as a strategy

Adoption is limited and it is not a ranking factor. Add it if you like; do not mistake it for the work.

Publishing volume

Thin pages dilute. One page that is genuinely the best answer outperforms fifty that restate the consensus.

What fixes actually work?

Answer first, render server-side, allow the right bots, write one entity description, match the markup to the copy, and measure against a fixed prompt set.

Six fixes. Four of them can be done in a week by a developer who already has access.

What order should the work happen in?

Access, then rendering, then structure, then schema, then entity, then measurement. Each step depends on the one before it.

Restructuring content before confirming the crawler can reach it is the most common way to waste a quarter, and it happens because content work is easier to sell than log analysis.

1. Pull server logs and confirm which AI crawlers arrived in the last ninety days.
2. Check robots.txt and any bot-management rules for accidental blocks.
3. Render your twenty most important pages with JavaScript disabled and read what remains.
4. Fix anything that returns effectively empty.
5. Restructure those pages so each section answers its heading immediately.
6. Reconcile schema with the visible copy on the same pages.
7. Write one canonical entity description and apply it everywhere.
8. Build a fixed prompt set and take a baseline measurement.
9. Re-measure monthly against the same set.
10. Re-check crawler behaviour quarterly, because it changes.

What should you ask a provider about the technical side?

Ask whether they want your server logs. Everything else in an LLM SEO pitch is assertion; the logs are the one artefact that settles the question.

Four follow-ups separate the people doing this work from the people describing it.

Will you render our templates without JavaScript, and show us the output?

This is the check that finds the expensive failure. A provider who does not run it is guessing about the single most common reason a site is invisible, and a screenshot of the rendered text costs them ten minutes.

Which crawler user agents will you test, by name?

A vague answer about ‘AI bots’ means no one has looked at a log. The names are public, the list is short, and each one behaves differently enough that testing one proves nothing about another.

What will you measure before you start, and how often afterwards?

Without a baseline taken against a fixed prompt set, any later claim of improvement is unfalsifiable. The prompt set should be written down and handed to you, not kept in-house.

What will you not do?

A provider who claims to influence every AI surface is overselling. Some surfaces are index-led and respond to conventional work; some are live-fetching and respond to technical work; none of them accept payment for placement, and the honest answer names the limits.

What do you deliver?

Six artefacts: crawler access report, render audit, schema reconciliation, answer-first rewrites, entity file, and a prompt set with a baseline.

All six are documents you keep, not a dashboard you rent.

How is technical LLM SEO measured?

Crawler hits in server logs, render completeness, schema validity, and then the same prompt-set measures used across all of this work.

The technical measures are binary and fast; the visibility measures are slow and probabilistic. Reporting that mixes them without saying which is which is reporting designed to look better than it is.

Does LLM SEO replace ordinary SEO?

No. Assistants that lean on a search index cannot retrieve a page that is not indexed, so conventional SEO remains the foundation this work sits on.

The honest framing is that the technical bar has risen. Everything that mattered for search still matters, and three things now matter that did not: whether AI crawlers specifically are allowed, whether content survives without JavaScript, and whether your answer is extractable.

What carries over

Crawlability, indexation, site speed, internal linking, canonical hygiene and authority. All of it still applies.

What is new

AI-specific crawler permissions, render-without-JS as a hard requirement, and answer-first structure as an editorial standard.

What changes in priority

Page speed matters more, because retrieval has tighter timeouts than indexing does.

What to stop doing

Optimising thin informational pages for traffic that assistants now absorb before the click.

Frequently asked questions