Insight · Analysis
How AI Answer Engines Choose Sources
The short answer
AI answer engines follow a retrieval-then-generation pipeline: they crawl and index pages, match a query to candidate content, score candidates for relevance and authority, and select passages to cite or paraphrase. Which signals drive selection inside each system is not fully public; providers document some requirements and leave others unspecified. What is documented: indexability and snippet eligibility are prerequisites. What is practitioner inference: how entity corroboration and passage quotability affect ranking.
71-word direct answer
Key takeaways
- The pipeline has two stages: retrieval (finding candidate pages) and generation (synthesizing an answer). These stages have different requirements, and most “AI visibility” advice conflates them.
- What providers actually document: crawl access requirements, the distinction between search and training crawlers, indexability as a prerequisite for Google AI Overviews, and snippet eligibility.
- What is practitioner inference, not documented fact: the weight given to entity corroboration, the specific advantage of quotable-unit structure, and how freshness affects citation selection inside LLM-based systems.
- What is genuinely unknown: the full ranking signals inside ChatGPT Search, Perplexity, and Google AI Overviews. No outside party has access to these systems' retrieval scoring weights.
- The honest implication: optimize for what is documented and measurable. Do not pay for optimization of signals that no one outside the company can verify.
The retrieval-then-generation pipeline
AI answer engines do not generate answers from nothing. They retrieve candidate content first, then synthesize a response. Understanding the sequence clarifies what can be influenced and what cannot.
Crawl
A crawler requests the page (Google's Googlebot for AI Overviews, OpenAI's OAI-SearchBot for ChatGPT Search, Anthropic's Claude-SearchBot for Claude search). If the page is blocked in robots.txt, loads too slowly to render, or presents content only inside JavaScript the crawler cannot execute, the subsequent steps do not occur. Crawl access is the first gate and the most auditable one.
Index
The crawled content is processed and added to an index. For Google AI Overviews, Google states explicitly that a page must be indexed and eligible to appear with a snippet. A page carrying a noindex or nosnippet directive is excluded from AI Overviews. This is documented; the precise indexing criteria for other AI search systems are less fully disclosed.
Retrieval
When a query arrives, the system retrieves candidate pages from the index using a combination of keyword matching and semantic (vector) similarity. Pages that directly and specifically address the query topic surface as candidates. Pages that cover the topic in passing, or that bury the relevant content deep in general background prose, are weaker candidates at this stage.
Scoring and ranking
Candidate pages are scored and ranked against each other. What goes into this score varies by system and is not fully disclosed by any provider. Documented factors for Google include the same quality and relevance signals as Google Search broadly: authority, accuracy, helpfulness. The specific weights and the role of AI-specific factors inside the ranking are not public.
Passage selection and citation
From the top-ranked pages, the system selects specific passages to cite, quote, or paraphrase in generating a response. Passages that can stand alone accurately (a definition, a numbered step, a direct answer) are structurally easier to extract and attribute than passages embedded in running argument. This is the basis for the “quotable unit” content advice, though the precise selection mechanism is not publicly documented.
Third-party evidence
What providers actually document
Google Search Central states that AI Overviews use the same foundational SEO requirements as Google Search and that no special AI schema, llms.txt file, or additional markup is required. Google further states that a page must be indexed and eligible to appear with a snippet to be considered for AI Overviews. Pages carrying a noindex or nosnippet directive are excluded from consideration.
OpenAI documents two distinct crawlers: OAI-SearchBot (the search crawler used for real-time retrieval in ChatGPT Search) and GPTBot (the training data crawler). These have separate user-agent strings and should be configured separately in robots.txt. Blocking GPTBot does not block OAI-SearchBot; a site owner who wants search visibility but not training inclusion must allow OAI-SearchBot explicitly.
Anthropic similarly documents two crawlers: Claude-SearchBot (search) and ClaudeBot (training). The separation mirrors OpenAI's architecture. Anthropic's support documentation describes how to block each independently.
IndexNow, supported by Bing and other search engines, provides a real-time URL submission protocol that can accelerate indexing of new or updated content. Google does not currently participate in IndexNow; new Google indexing is triggered through sitemap submission and organic crawling.
None of the major providers publicly document the specific ranking signals used to order retrieval candidates within their AI search systems. What is above is documented; what follows is inference.
Interpretation
What practitioners infer, and why it should be labeled inference
Much of the practitioner literature on AI citation selection presents inference as documented fact. The claims below are reasonable working hypotheses supported by the logic of how retrieval systems work and by observation of outcomes. They are not confirmed by provider documentation and should inform strategy but not be treated as established mechanics.
Entity corroboration: practitioners observe that pages making claims about a person, organization, or product that are also supported by independent credible sources tend to appear as citations more reliably than pages where the same claim appears only on the subject's own site. This is consistent with how retrieval systems would logically value confidence in entity claims. It is not documented as a specific ranking factor by any major provider.
Passage quotability: practitioners observe that content structured as direct answers, definitions, numbered steps, and labeled principles tends to appear in AI-generated answers more often than equivalent content expressed as continuous argument. This is structurally plausible, because isolated passages are easier to extract and attribute accurately. The specific advantage this confers in scoring is not documented.
Freshness: practitioners observe that content with recent publication or review dates sometimes surfaces more reliably for time-sensitive queries. This is consistent with how conventional search handles freshness for query types where recency matters. Whether freshness affects AI retrieval ranking specifically, and how, is not documented independently of conventional search.
The honest implication of this uncertainty: invest in the documented prerequisites first. Then apply practitioner inference where the cost is low and the downside is also low. Do not optimize heavily for signals that cannot be verified from the outside.
What is genuinely unknown, including to practitioners
The full ranking criteria used to score retrieval candidates inside ChatGPT Search, Perplexity, Google AI Overviews, and similar systems are not public. The companies that operate these systems do not publish retrieval scoring weights. Practitioners who claim specific knowledge of these weights are either misrepresenting their access or conflating inference with fact.
Whether citation history affects future citation likelihood (whether a page that has been cited accurately in past queries gets preferential treatment in future queries) is frequently asserted in practitioner content. It is consistent with the logic of these systems and with how recommendation algorithms generally work. It is not documented.
How AI search systems handle contradictory information across sources is not publicly described: whether they defer to the source with higher conventional authority, average across sources, or select the most recent. The behavior appears to vary by query type and by system.
The degree to which adding structured data (JSON-LD schema markup) specifically improves AI retrieval ranking, beyond its established effect on conventional search rich results, is not documented by any provider. Google states structured data is not required for AI features. Whether it confers any AI-specific advantage is unknown.
Martin's observation of his own site's behavior in AI queries is a sample of one, for a specific type of content, in a specific domain. It is not a generalizable finding. This article reflects that limit.
Why crawlability and snippet eligibility come before everything else
The easiest mistake in AI visibility work is optimizing content and entity signals before confirming the page can be found. A page with excellent content, clear entities, and good external corroboration that is blocked in robots.txt or carries a nosnippet directive is invisible to AI Overviews regardless of its content quality.
The prerequisite audit is short and entirely auditable: (1) confirm the relevant AI search crawlers are allowed in robots.txt, by user-agent string rather than blanket rule; (2) verify the priority pages are indexed in Google Search Console; (3) confirm no priority page carries a noindex or nosnippet directive; (4) verify the sitemap is current and submitted.
This work is unglamorous and does not require specialized AI visibility tools. It is also the only part of AI citation optimization that is fully verifiable from outside the provider's system. Everything after this is optimization, useful but secondary to a prerequisite that must be confirmed first.
Entity confidence and corroboration
Retrieval systems analyze indexed content to identify entities (people, organizations, places, concepts) and the relationships between them. The consistency of entity claims across independent sources affects how confidently the system can characterize the entity, and likely (by inference) how reliably it cites claims about it.
A person whose name, professional role, organizational affiliation, and location appear identically across their own site, their LinkedIn profile, relevant publisher author pages, and independent external mentions is a higher-confidence entity than a person whose profiles contradict one another or whose claims appear only on their own site.
Circular sourcing, where the only evidence for a claim is other pages controlled by the same entity, does not build external confidence. A press release quoting only the subject, a guest post linking only back to the author's site, or a Wikipedia article written by the subject adds limited corroboration value. The useful corroboration is independent: a byline on a publication that editorially selected the work, a speaker profile at an event that accepted the speaker on their record, a mention in a third-party article that cites them as a source.
Principle
A claim that appears on a single page with no independent corroboration is a weak citation candidate. Not because the claim is wrong, but because a retrieval system cannot cross-reference it. The system has no way to distinguish a correct isolated claim from an incorrect one. Corroboration is the mechanism by which a system develops confidence.
What makes a passage quotable in isolation
A passage is quotable in isolation when it is accurate, complete, and attributable without its surrounding context. These are the structural properties that support extraction.
It answers a specific question directly
A passage that begins with the answer, rather than with background, can be extracted and remain accurate. A passage that begins with context required to understand the answer cannot be quoted without that context.
It names its entities explicitly
Pronouns and phrases like 'the program' or 'this approach' cannot be cited without the surrounding sentences that identify the referent. Named entities ('the TLE Foundation training program,' 'Google AI Overviews') survive extraction.
It states its scope
A definition that says 'AEO, as used in this article, refers to...' can be cited accurately even if the term is used differently elsewhere. A definition without scope can be misapplied by extraction.
It can be wrong without the rest of the page being wrong
If the passage makes a claim that depends on qualifications ten paragraphs earlier, it cannot be extracted safely. Each passage should be independently assessable for accuracy.
It identifies its epistemic basis
A passage that says 'Google states...' or 'Practitioners observe, though this is inference...' is safer to cite than one that presents every claim with equal confidence. The citation system benefits from the distinction.
Limitations of this article
This article describes the pipeline as it can be understood from provider documentation, practitioner inference, and structural reasoning about how retrieval systems work. It does not represent insider knowledge of any AI system's ranking mechanics. No such access exists outside the companies building these systems.
The pipeline description reflects how retrieval-augmented generation systems broadly work as of July 2026. Specific implementations at Google, OpenAI, Anthropic, and others change without public announcement. Claims about Google AI Overviews should be cross-referenced against current Google Search Central documentation; claims about ChatGPT Search against current OpenAI documentation.
The sections labeled “practitioner inference” represent a synthesis of the practitioner literature as of the publication date, assessed against available documentation. They are not empirical findings from controlled studies. Where a reader encounters a study that tests these inferences, that study should take precedence over the synthesis here.
This article does not cover Perplexity, Gemini, or other AI search systems in technical detail; provider documentation for each should be consulted directly. The principles described (crawlability, indexation, specificity, corroboration) apply broadly, but the specific mechanics differ.
For what to do in response to these mechanics, see the AEO/GEO and AI search visibility pillar. For help evaluating vendor claims, see AEO, GEO and AI search visibility: what the terms actually mean.
Sources
Primary provider documentation cited in this article. Practitioner inference sections are labeled as such and do not correspond to a single citable source.
- AI features and your website: Google Search CentralGoogle
- AI Overviews: Google Search documentationGoogle
- Optimizing your website for generative AI features on Google SearchGoogle
- How Google Search works: crawling, indexing, and servingGoogle
- Snippet controls: nosnippet and related meta tagsGoogle
- Overview of OpenAI crawlers (OAI-SearchBot and GPTBot)OpenAI
- Does Anthropic crawl data from the web? (ClaudeBot and Claude-SearchBot documentation)Anthropic
- IndexNow protocol documentationIndexNow.org (Microsoft / Bing)
- Robots.txt introduction: Google Search CentralGoogle
Frequently asked questions
Does Google use the same signals for AI Overviews as for regular search?
Yes, according to Google. Google Search Central states that AI Overviews use the same foundational SEO requirements as conventional Google Search (indexability, quality signals, relevance). Google also states that no special AI schema, llms.txt file, or additional markup is required for AI Overviews.
This does not mean the ranking is identical. AI Overviews involve a generation step that conventional results do not. But the prerequisite layer, being crawlable, indexed, snippet-eligible, and quality-rated, is documented as shared.
Can I block AI training without blocking AI search?
Yes, for the systems that have separated their crawlers. OpenAI has two separate crawlers: OAI-SearchBot (search retrieval) and GPTBot (training data). Anthropic has Claude-SearchBot (search) and ClaudeBot (training). You can allow the search crawlers and disallow the training crawlers separately in robots.txt using their distinct user-agent strings.
Google does not currently operate a separate AI training crawler in the same way. Googlebot is used for indexing that supports both Search and AI features. Blocking Googlebot removes the page from all Google features, including AI Overviews.
Check CDN and edge-layer rules as well as robots.txt. Services like Cloudflare can apply bot-blocking rules that affect crawlers before they reach your origin, and misconfigured rules can block all bot traffic including beneficial search crawlers.
Why is a single source a weak citation candidate?
Because retrieval systems cannot distinguish a correct isolated claim from an incorrect one without cross-referencing. If a claim appears only on the subject's own page with no independent corroboration, the system has no way to validate it. Corroboration from independent sources (a credible publication citing the same fact, a third-party profile confirming the same credentials) gives the system a basis for confidence.
This is the same logic as academic peer review or journalistic corroboration: a claim from a single source is less reliable than the same claim confirmed independently. AI retrieval systems, designed to produce accurate answers, apply a similar standard.
What is the single most important thing I can verify about my AI search visibility?
Confirm that the AI search crawlers you care about can actually reach and index your priority pages. This is the documented prerequisite for everything else. Check robots.txt for OAI-SearchBot and Claude-SearchBot specifically, not just Googlebot. Check server logs to confirm crawl activity. Verify in Google Search Console that priority pages are indexed and carry no nosnippet directive.
Everything after that (content specificity, entity consistency, off-site corroboration) is optimization on top of a working foundation. Optimizing content for a page that cannot be crawled is effort spent on the wrong problem.