Robots.txt Tester

Will Googlebot see this URL? Test any robots.txt against any path.

URL to test

User agent

Robots.txt rules paste or leave blank to fetch live

FAQ

Where should robots.txt live?

At the root of your origin — https://example.com/robots.txt. Subdirectory robots.txt files are ignored by search engines.

How does rule priority work?

Google applies the longest matching rule. If both "Allow: /blog" and "Disallow: /" match, Allow wins because it's longer and more specific.

Does blocking a URL remove it from Google?

No — it only stops crawling. Already-indexed URLs can still appear in results (with a "no description" snippet). Use a noindex meta tag for actual removal.

What user agents should I test?

Googlebot is the most important. Also test Bingbot, Twitterbot, and any social crawlers your content depends on.

Ready to publish?

Share your writing on WriteUpCafe

Free account. Thousands of daily readers. Built-in SEO. Nothing to install.

Create your free account