Skip to main content

Why isn’t my website cited by ChatGPT or other AI search engines?

A stage-by-stage diagnosis — fetch, index, retrieve, select, cite — of why an AI assistant does not cite a page, with what you can check yourself and what no scan can tell you.

By Akash Yadav · Published 26 October 2026 · Reviewed 26 October 2026

The short answer

An AI assistant can only cite a page it was able to fetch, has indexed or retrieved for the question being asked, and then chose over the other candidate sources. A missing citation is a failure at one of those stages — and only the first two are mostly technical. If an assistant's search crawler is blocked, or the page is noindexed or snippet-restricted, it cannot be cited at all. If it is reachable and indexed, the reason is usually that another page answered the question more directly, or came from a source the engine trusts more.

So check the technical stages first, because they are binary and you control them. Then judge the content against the specific question, because that is where most sites actually lose.

The five stages between a page and a citation

  1. Fetch.The engine's crawler requests the URL and gets a readable response. Blocked in robots.txt, refused by a firewall, served an error, or served an empty JavaScript shell — and nothing downstream happens.
  2. Index.The page is stored in the engine's (or its search partner's) index and is allowed to be shown. A noindex directive removes it; nosnippet or a very small max-snippet can keep it from being quoted.
  3. Retrieve. For a given question, the engine runs one or more searches and pulls a shortlist of candidate pages. Your page has to match what was searched — which is often a rewritten version of what the user typed.
  4. Select. From the shortlist, the model uses the passages that answer the question best. Specific, self-contained, well-structured passages win over general product copy.
  5. Cite. The answer links to some of the sources it used. A brand can be mentioned without a link, and a page can be used without being the one linked.

Diagnose it in order

Work down this list and stop at the first “no” — fixing a later stage is wasted while an earlier one fails.

  1. Can the engine's search crawler fetch the page? Check robots.txt for the crawler that powers citations — OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google's AI features, bingbot for Copilot. Blocking a training crawler such as GPTBot or ClaudeBot does not, per the vendors, remove you from their search answers. Then check that your CDN or firewall is not challenging those crawlers. See how to verify AI crawler access.
  2. Is the URL indexed and allowed to be quoted? Look for noindex in the HTML and in the X-Robots-Tag header, and for nosnippet or max-snippet. Google states there are no extra technical requirements for AI Overviews beyond being indexed and eligible to show with a snippet. Confirm indexing in Google Search Console and Bing Webmaster Tools — a 200 response does not mean a page is indexed.
  3. Is the substance in the served HTML? Many crawlers do not run JavaScript. Fetch the page with curl and confirm the main text, headings and links are in the response, not injected later.
  4. Does the page answer a specific question? Write down the questions you expect to be cited for. If the page never states the answer in a sentence that would make sense quoted on its own, it gives a model nothing to cite.
  5. Are there better sources for that question? Ask the assistant the question yourself and read what it cites. If it cites independent reviews, documentation or comparison pages, that is the bar — and usually the gap is evidence and independent corroboration, not markup.

What a single-URL scan can and cannot check

A deterministic audit like AI Search Audit covers stages one and two from the outside: which AI crawlers your robots.txt allows, what a small user-agent probe of your homepage observed, whether the page is noindexed or snippet-restricted, whether canonicals and the title are usable, and whether content is present without JavaScript. It also reports structured data, llms.txt and sitemap findings as advice.

It cannot tell you:

  • whether a vendor's crawler, from its own IP addresses, actually reached you;
  • whether the page is in an engine's index (Search Console and Bing Webmaster Tools can);
  • which searches an engine runs for a question, or how it ranks the shortlist;
  • whether your page is the best available answer — that is a content judgment.

The report says the same thing in one line: “A 100 does not measure access from AI vendors’ own IP addresses, content that only appears after JavaScript runs, citations, rankings or traffic.”

A worked example

A common pattern: a site adds User-agent: * / Disallow: / on a staging deploy, and the rule reaches production. Every crawler is refused at stage one. The check is reproducible:

curl -s https://example.com/robots.txt
# User-agent: *
# Disallow: /        <- every crawler, including OAI-SearchBot, is refused

Removing the rule is necessary, not sufficient. After the fix, crawlers have to revisit (OpenAI notes robots.txt changes can take about 24 hours to apply to search), the page has to be indexed, and it still has to be the best answer to the question asked.

What this does not prove

  • Passing every technical check is a precondition for being cited, not a prediction of it.
  • One manual question to one assistant is an anecdote. Answers vary between runs; measure a fixed set of questions repeatedly before drawing conclusions.
  • A brand mention is not a citation, and a citation is not a visit.

Primary sources

Related guides

Spotted something wrong or out of date? Vendor crawler documentation changes often. Email support@aisearchaudit.pro and the guide will be corrected and its review date updated.