Free Robots.txt Viewer & Tester
Enter a domain and view its robots.txt instantly: user-agent rules, disallow counts and declared sitemaps, without digging through DevTools.
Frequently asked questions
What does robots.txt do?
It tells well-behaved crawlers which paths they may fetch. It is advisory — sensitive pages need real access control, not a Disallow line.
Does OrbiCrawl respect robots.txt?
Yes — crawl jobs respect robots.txt by default (you can see exactly what a site allows before crawling).