RobotslabBot

The robots.txt tools’ crawler

RobotslabBot is Robotslab’s crawler. It never crawls the web on its own: it only visits a site when a user asks to analyse its public files (robots.txt, sitemaps, llms.txt…) or to check which bots can reach it.

Identification

Good behaviour

• It reads your robots.txt before crawling and always obeys rules that name it.
• It visits a single URL per check, with at most 8 requests at a time.

Bot-by-bot check

To find out whether your server lets a given bot through, the tool visits the requested page presenting the identity (user-agent) of that bot: Googlebot, GPTBot, ClaudeBot… These visits always come from our server, never from Google or OpenAI, and carry the X-Robotslab-Check header so you can recognise them. A bot your robots.txt blocks is not visited.

• It submits no forms, logs into no accounts and keeps no pages.

Blocking RobotslabBot

Add these lines to your robots.txt:

Any question: Contact page.