Audit & Analysis · Free tool

Robots.txt Tester — Validate Any Site’s Robots.txt

Free robots.txt tester: fetch any site’s live robots.txt, validate its rules, see user-agents, Disallow and Allow directives and sitemap lines.

Step by step

How to use the Robots.txt Tester

01Enter the domain or url
02Click "Run Robots.txt Tester" — it's free and instant
03Review your results on screen and download the PDF/Excel report

What is the Robots.txt Tester?

robots.txt is a plain-text file at the root of a domain that tells crawlers which paths they may or may not request. This tester fetches the live file for any site, parses its user-agent groups, Disallow and Allow rules and Sitemap lines, and flags problems such as a site-wide Disallow: /.

Why it matters for SEO

A single wrong line in robots.txt can stop Google from crawling an entire site — one of the most damaging SEO mistakes, and one that often slips through after a redesign. robots.txt is also where you point crawlers, including AI crawlers, to your sitemaps and decide what they may access.

How to read your Robots.txt Tester results

Check that important sections are not disallowed and that every sitemap is listed with an absolute URL. Remember that Disallow stops crawling, not indexing — to keep a page out of search results use a noindex tag and let it be crawled.

Expert tips

Never use Disallow: / on a live site unless you mean it.
List all sitemaps with Sitemap: https://… lines.
Do not block CSS and JavaScript files — Google needs them to render pages.
Generate a clean file with the Robots.txt Generator.

Limitations

Checks only the single URL you enter, as our server fetches it. It does not crawl the whole site, run JavaScript, or see pages behind a login.

Why use our Robots.txt Tester?

Completely free — use Robots.txt Tester as many times as you want
No login or registration needed to see results
Download the report as PDF or Excel with a free account
Works on any device — desktop, tablet or mobile

Guides that use this tool

Frequently asked questions

Always at the root of the host, for example https://example.com/robots.txt. Files in subfolders are ignored by crawlers.

No. It stops crawling, but a blocked URL can still be indexed from links. Use noindex to remove pages from results.

Yes. Crawlers such as GPTBot, ClaudeBot and Google-Extended have their own user-agent names you can allow or disallow.

Google generally caches robots.txt for up to 24 hours.

Yes. Robots.txt Tester is 100% free with unlimited usage. You can run it as often as you like without paying anything.

No login is required to use Robots.txt Tester and see your results. You only need a free account if you want to download reports as PDF/Excel or save your history.