Crawler access probe
What the extra requests in your server logs mean.
September 2026
Why we send these requests
During an audit, we may request the homepage using a few published answer-engine user-agent strings. This can reveal a firewall or CDN rule that refuses that user agent even though robots.txt allows it. The requests carry our own transparent identifier, AISearchAudit-probe/1.0.
What we observe
We compare each response with the ordinary AI Search Audit homepage response. A refusal or challenge is reported for that request. A normal response means only that this request was not refused. We do not send requests from the vendors' IP addresses, so we cannot verify what a real vendor crawler receives.
Limits and control
We probe only the homepage, obey relevant robots.txt disallow rules, and use the same HTTPS, address-safety and request-budget limits as the audit. We do not impersonate Googlebot. You can recognize or filter the AISearchAudit-probe/1.0 token in logs or firewall rules. Results never promise indexing, citations or ranking.