Every public file on a site, at a glance.
robots.txt, sitemaps, llms.txt, ads.txt, security.txt, .well-known files: the bot checks 34 locations and opens every sitemap it finds.
Files
Sitemaps
robots.txt
Common questions
What is llms.txt?
A Markdown file at the site root that introduces the site to AI models: summary, key pages, documentation. It is a recent, still optional convention adopted by many documentation sites.
What are ads.txt and security.txt for?
ads.txt lists the ad networks allowed to sell your ad space, against fraud. security.txt (in /.well-known/) explains how to report a security flaw; it must contain at least “Contact:” and “Expires:”.
What does “soft 404” mean?
The server answers “200 OK” but returns an HTML page (often the homepage or an error page) instead of the requested file. The file therefore does not really exist, and crawlers may be misled.