Write your robots.txt, then test it.
Start from a template, pick the bots to block, then check whether a URL is allowed or blocked, using Google’s rules.
Starting point
Rules for every bot
AI crawlers
A checked box blocks that bot site-wide. Google-Extended and Applebot-Extended block AI training without affecting search ranking.
SEO bots and foreign engines
robots.txt
You can also edit the text by hand: the test below always reads the version shown.
Test a URL
Common questions
What is robots.txt for?
Placed at the site root, it tells crawlers which parts they may visit. It protects nothing: a blocked page can still be indexed if other sites link to it. To remove a page from results, use the “noindex” tag.
How do I block AI crawlers?
Add a “User-agent” group for each bot (GPTBot, ClaudeBot, Google-Extended, CCBot…) followed by “Disallow: /”. Major vendors’ bots respect these rules; Google-Extended blocks AI use without affecting Google Search.
Which rule applies when Allow and Disallow conflict?
Under RFC 9309, followed by Google, the longest (most specific) rule wins; at equal length, Allow wins. The tester applies exactly this logic, with the * and $ wildcards.