The short answer

Verifying AI crawler access means confirming six things, in order: robots.txt allows the search crawler for each assistant you care about; the URL returns 200 over HTTPS without a redirect loop; there is no noindex or restrictive snippet directive in the HTML or headers; the main content is in the served HTML; your CDN or firewall does not challenge the crawler; and your server logs show verified requests from it. The first four you can test from your own machine. The last two need access to your edge configuration and logs.

Which crawlers matter

Vendors separate crawlers by job. Blocking a training crawler is an editorial choice that, per OpenAI and Anthropic, does not remove you from their search answers. Blocking a search crawler does. User-triggered fetchers run when a person asks about a link, and OpenAI and Perplexity document that they may not follow robots.txt.

Token	Vendor	Purpose
GPTBot	OpenAI	Model training
OAI-SearchBot	OpenAI	Search / citations
ChatGPT-User	OpenAI	User-triggered fetch
ClaudeBot	Anthropic	Model training
Claude-SearchBot	Anthropic	Search / citations
Claude-User	Anthropic	User-triggered fetch
anthropic-ai	Anthropic	Model training
Google-Extended	Google	Model training
Googlebot	Google	Search / citations
PerplexityBot	Perplexity	Search / citations
Perplexity-User	Perplexity	User-triggered fetch
bingbot	Microsoft	Search / citations
Applebot	Apple	Search / citations
Applebot-Extended	Apple	Model training
CCBot	Common Crawl	Model training
meta-externalagent	Meta	Model training
Bytespider	ByteDance	Model training

This is the same registry the AI Search Audit scanner reports on. User-agent strings can be spoofed, so a request that merely claims to be one of these is not proof of anything.

1. Read robots.txt the way a crawler does

curl -sS -D - https://example.com/robots.txt

Expect 200 and Content-Type: text/plain. A 5xx makes many crawlers treat the whole site as disallowed.
A crawler obeys only the most specific group that names it. If User-agent: OAI-SearchBot has its own group, the * rules no longer apply to it — so repeat any disallows you still want.
Check each subdomain separately; robots.txt is per host.

To allow search crawlers while opting out of training, for example:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

2. Check status codes and redirects

curl -sS -o /dev/null -L -w "%{http_code} %{num_redirects} %{url_effective}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
  https://example.com/your-page

Expect a final 200, one redirect at most, and a final URL that matches the page's canonical. Compare against a browser user agent: if only the bot UA gets a 403 or a challenge page, your edge is filtering by user agent.

3. Check noindex and snippet controls — in both places

curl -sS -D - -o page.html https://example.com/your-page | grep -i x-robots-tag
grep -io '<meta name="robots"[^>]*>' page.html

noindex keeps a page out of the index; nosnippet and small max-snippet values limit what can be quoted. Google states these same preview controls apply to its AI features. A header directive is easy to miss because it is invisible in the page source.

4. Confirm the content is in the served HTML

Open page.html from the previous step. The main heading, the paragraphs you want quoted and your internal links should be there as text and <a href> anchors. If the body is a single empty <div id="root">, crawlers that do not execute JavaScript see nothing — render on the server or prerender.

5. Check your CDN and firewall

Bot-management and WAF rules frequently block AI crawlers by default. Look at your provider's bot settings and firewall event log for the tokens above. Prefer allowing crawlers by their published IP ranges or the provider's verified-bot list rather than by user agent alone: OpenAI, Anthropic and Perplexity all publish IP lists for their crawlers.

6. Confirm real requests in server logs

The only proof a vendor's crawler reached you is a request in your logs from that vendor's published IP range. Filter access logs by user agent, then check the source IP against the vendor list (Google and Bing also document reverse-DNS verification). Export only the fields you need; logs contain visitor IPs and should not be shared wholesale.

Common failures and their fixes

Symptom	Likely cause
Disallow: / under User-agent: *	A staging robots.txt shipped to production.
403 or challenge page for bot UAs only	CDN bot management or WAF rule.
200 but page missing from Search Console	Not indexed yet, noindex, or a canonical pointing elsewhere.
Empty body in curl output	Client-side rendering with no server-rendered HTML.
robots.txt returns 5xx	Origin or edge error; crawlers may back off the whole site.

What this does not prove

A request sent with a crawler's user agent from your machine is not that crawler — vendors fetch from their own IPs.
Access is necessary for a citation, never sufficient. See why AI search may still not cite you.

Run a free audit to check steps 1–4 for your homepage and up to four more pages in one pass.

Primary sources

Spotted something wrong or out of date? Vendor crawler documentation changes often. Email support@aisearchaudit.pro and the guide will be corrected and its review date updated.