User Agent
A user agent is the identifier that a browser or crawler sends with every request, declaring what it is. Googlebot, Bingbot and the AI crawlers each have their own. Servers can respond differently by user agent, which is where problems begin.
Why it matters for your rankings
User agent handling is where accidental cloaking and accidental blocking both originate. Bot protection, firewalls and CDN rules routinely block crawlers that were never meant to be blocked.
The symptom is distinctive and easy to misdiagnose: the site works perfectly in a browser while Search Console reports crawl failures, because the server is rejecting Googlebot specifically. The same mechanism now matters for AI visibility, since GPTBot, ClaudeBot, PerplexityBot and others can each be allowed or refused independently. Deciding which AI crawlers may access your content is a real strategic choice, and many sites have made it accidentally through a default firewall rule nobody reviewed.
How to check it on your site
Test as Googlebot
Crawl your site with the user agent set to Googlebot Smartphone and compare the response against a normal browser.
Check firewall and CDN rules
Review bot protection settings in Cloudflare or your WAF for rules blocking legitimate crawlers.
Review robots.txt per agent
Confirm which AI crawlers you allow or disallow, and that this reflects a deliberate decision.
Watch for crawl errors in Search Console
Sudden crawl failures with a healthy site usually mean a server or firewall change blocked the bot.