Can the bots get in?
For each bot (Google, Bing, ChatGPT, Claude, Perplexity, Ahrefs…), we read your robots.txt, visit the page under its identity and read its tags. You see who gets in, who is blocked, and by which rule.
Bot by bot
| Bot | robots.txt | Server response | Robots tags | Verdict |
|---|
Click a row to see the details. Bots blocked by robots.txt are not visited.
robots.txt
We apply RFC 9309, like Google: the group naming the bot wins over “*”, then the most specific rule.
The server
We visit the page under each bot’s identity and compare with a browser: a gap reveals a filtering firewall or CDN.
The tags
We read the meta robots tag, the one naming the bot (googlebot, bingbot…) and the X-Robots-Tag header: a noindex prevents indexing.
Common questions
How do I know whether Googlebot can reach my page?
Three conditions: your robots.txt must not block it, your server must answer “200 OK” (with no firewall blocking it) and the page must not carry a noindex tag. The tool checks all three, for each bot.
Why is a bot “refused by the server”?
Your robots.txt allows it, but the server or a firewall (Cloudflare, Sucuri, Wordfence…) answers 403, 429 or a challenge page, while a browser gets the page. This is common with firewall “block AI bots” settings.
Is the result exactly what the real Googlebot gets?
Almost always. The tool presents each bot’s identity (user-agent) from our server. A site that checks the IP address of real bots may answer them differently; for Google, confirm with the URL Inspection tool in Search Console.
What are Google-Extended and Applebot-Extended?
They are not bots that visit pages but names read in robots.txt. Blocking them stops Google and Apple from using your content for their AI (Gemini, Apple Intelligence), without changing your search ranking.