A single line in your robots.txt file can silently block Google from crawling your entire website. Most site owners set it up once and never look at it again — which is exactly when problems creep in. This guide explains what robots.txt is, why it matters for SEO, common mistakes to avoid, and how to check yours in seconds with a free tool.

What Is robots.txt?

robots.txt is a plain text file that sits at the root of your website (e.g., https://yourdomain.com/robots.txt). It follows the Robots Exclusion Protocol — instructions that tell search engine crawlers which parts of your site they are allowed or not allowed to visit.

Every major search engine — Google, Bing, DuckDuckGo — checks this file before crawling your site. It is one of the first things Googlebot reads when it visits your domain.

How robots.txt Works

The file is made up of directives:

  • User-agent — specifies which crawler the rule applies to. User-agent: * applies to all bots.
  • Disallow — tells the crawler which paths not to visit.
  • Allow — explicitly permits a path, even within a disallowed directory.
  • Sitemap — tells crawlers where to find your XML sitemap.
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Why robots.txt Matters for SEO

If you accidentally block the wrong paths, Google cannot crawl those pages — and pages Google cannot crawl cannot rank. Common damage scenarios: blocking Disallow: / (your entire site), blocking CSS and JavaScript files (Google needs these to render pages), blocking image directories, and missing a Sitemap declaration.

Common robots.txt Mistakes

Blocking the entire site after a staging migration. Developers often add Disallow: / to staging environments and forget to change it before going live. This is one of the most catastrophic and common SEO accidents.

Blocking /wp-content/ on WordPress. This also blocks CSS and JS files that Google uses to render pages correctly, hurting your rankings.

Not declaring the sitemap. Your robots.txt should always include a Sitemap: line. Without it, crawlers discover your content more slowly.

Overly broad wildcards. Patterns like Disallow: /*? can accidentally block important pages that use query strings for content.

Check Your robots.txt for Issues — Free

Manually reading a robots.txt file is error-prone, especially on large sites. CachePlex Robots.txt Checker fetches and analyses any site's robots.txt instantly and gives you a health score, parsed directives, detected sitemaps, flagged issues with fixes, and the raw file with syntax highlighting — all in seconds, no sign-up required.

Check Any robots.txt File Free →

Instant health score · No sign-up required