Enter a domain to fetch its robots.txt, then test whether a
specific path is crawlable for a given user-agent. Wildcards
(*) and end-anchors ($) are evaluated the same
way Googlebot handles them.
The robots.txt file tells search engine crawlers which parts of your site they may and may not access. A single misplaced "Disallow" line can quietly deindex your entire website, tank your search rankings, and cut off organic traffic overnight. This robots.txt tester fetches the live robots.txt from any domain, parses it into its user-agent groups, and tells you exactly whether a specific URL path is allowed or blocked for a specific crawler. It applies the same longest-match rule Google uses, so the verdict reflects how Googlebot actually interprets your file rather than a naive top-to-bottom read.
Enter a domain or full URL in the first field. In the second field, enter the path you want to test, such as /blog/ or /wp-admin/ (defaults to /). In the third field, enter the crawler user-agent token to test as, such as Googlebot or Bingbot (defaults to * for the catch-all group). Click "Test robots.txt." The tool fetches the site's robots.txt over HTTPS (falling back to HTTP), shows a clear Allowed or Blocked verdict, names the exact directive that decided it, lists every declared sitemap, and prints the full raw robots.txt so you can confirm what the server actually served.
Remember that robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it, so use a noindex meta tag or header to truly keep a page out of the index. The most specific matching rule wins, not the first one, so a long "Allow" can override a shorter "Disallow" for the same path. An empty "Disallow:" line means allow everything, while "Disallow: /" blocks your whole site. Always serve robots.txt as text/plain from the domain root; if it is served as HTML or with the wrong content type, crawlers may ignore your rules entirely. Wildcards (*) and end-anchors ($) are supported by Google and Bing but not by every crawler.
Need reliable hosting? These free tools are brought to you by Ultra Web Hosting. Fast, secure shared and reseller hosting with 24/7 expert support. View hosting plans →