"Blocked by robots.txt" is one of the most common statuses in Search Console's Page indexing report, and one of the most misread. Sometimes it is exactly what you want: your cart, your admin area and your internal search results should be blocked. Sometimes it means a single careless line is keeping your best pages out of Google. The status looks identical either way, so the job is to work out which situation you are in and, if it is the bad one, find the rule.

The short answer: open the affected URL in Search Console's URL Inspection tool, confirm it says crawling is not allowed, then read your robots.txt with the URL's path in mind. The rule responsible is the most specific Disallow that matches it in the group that applies to Googlebot. Change or remove that rule, ask Google to re-fetch robots.txt, and request indexing.

First, decide whether it is actually a problem

Before changing anything, look at which URLs are listed. Click the status in the Page indexing report and scan the examples. They usually fall into one of two groups.

The first group is URLs you never wanted in search: checkout and cart pages, account pages, internal search results, filtered or sorted category URLs with parameters, admin and API paths. If that is what you see, the report is simply confirming that your robots.txt works. You can leave it alone. Google lists these so you can check, not because they are errors.

The second group is pages you do want found: product pages, articles, the blog, sometimes the whole site. That is the case worth fixing today, because a page Google cannot crawl cannot rank on its content.

Confirm it with URL Inspection

Paste one affected URL into the URL Inspection bar at the top of Search Console. Under the coverage details, "Crawl allowed?" will say No: blocked by robots.txt. That confirms Google, not some other tool, is reading your file that way. If it says crawling is allowed, the report is out of date and the next crawl will clear it.

Search Console also has a robots.txt report, under Settings. It shows the robots.txt files Google has fetched for your site, when it last fetched them, and any parse errors. Check the fetched version matches what you think is live. A CDN serving an old copy, or a different file on the www and non-www hostnames, explains a surprising number of these cases.

How Google decides which rule applies

Most people read robots.txt top to bottom and assume the first matching line wins. Google does not work that way, and that misunderstanding is behind most of the confusion.

  • Only one group applies. Googlebot follows the group with the most specific User-agent line that matches it. If your file has a User-agent: Googlebot group, Google ignores the User-agent: * group entirely, including any Allow lines you put there for everyone.
  • Within the group, the longest matching path wins. Allow: /blog/ beats Disallow: / for a blog URL because it is more specific, wherever it appears in the file.
  • On a tie, the less restrictive rule wins. If an Allow and a Disallow match with paths of equal length, Google uses the Allow.
  • Wildcards count. * matches any sequence of characters and $ marks the end of the URL. Disallow: /*? blocks every URL with a query string, which is often far more than intended.
  • Paths are case-sensitive. Disallow: /Blog/ does not block /blog/.

Our Robots.txt Checker parses a live file into its groups and rules, which makes the "which group applies" question much easier to answer than reading raw text. If the file needs rewriting from scratch, the Robots.txt Generator builds a clean one.

The usual culprits

After a lot of audits, the same handful of causes come up again and again:

  • A staging file copied to production. Staging sites are rightly blocked with Disallow: /. When the site is launched by copying files across, the blocking robots.txt comes with it. If your whole site suddenly shows as blocked after a launch or migration, check this first.
  • A parameter rule that is too broad. Disallow: /*? was meant to block faceted navigation and ends up blocking paginated archives, tracking-tagged landing pages and anything else with a query string.
  • A folder rule that catches more than its name suggests. Disallow: /p blocks /products/, /pricing and /press/, because rules match prefixes. Use /p/ if you mean the folder.
  • A separate Googlebot group. Someone added User-agent: Googlebot with one rule, not realising Google now ignores every rule written for *.
  • A plugin or platform setting. SEO plugins and some site builders generate robots.txt for you, and a setting changed in one place silently rewrites the file. On older WordPress versions, the "Discourage search engines" option added a blocking rule to the virtual robots.txt; newer versions use a noindex tag instead, which shows up in Search Console as a different status.

Fixing it

  1. Edit the rule, not the symptom. Narrow the Disallow so it only matches what you meant, or add a more specific Allow for the section that should be crawled. Keep the change minimal; robots.txt mistakes tend to come from big rewrites.
  2. Check the result before you rely on it. Run the live file through the Robots.txt Checker, and test a couple of URLs from the report with URL Inspection's live test.
  3. Ask Google to fetch the new file. Google caches robots.txt for up to a day. The robots.txt report in Search Console lets you request a recrawl so you do not have to wait.
  4. Request indexing for the important pages through URL Inspection, and use "Validate fix" on the status in the Page indexing report so you get an email when Google has rechecked the list.
  5. Make sure the pages are in your sitemap. If they were blocked for a long time, the sitemap is how Google rediscovers them quickly. The Sitemap Validator confirms it is readable.

A worked example

Here is a file of the kind I see on real shops, and a product URL Search Console reports as blocked: /products/blue-backpack?colour=navy.

User-agent: *
Disallow: /cart
Disallow: /checkout
Allow: /products/

User-agent: Googlebot
Disallow: /*?
Disallow: /account

Reading it top to bottom, you would expect Allow: /products/ to let the product through. But Google first picks a group, and because a User-agent: Googlebot group exists, it ignores the * group completely, including that Allow. In the Googlebot group, Disallow: /*? matches any URL containing a question mark, so the colour variant is blocked. The plain /products/blue-backpack is not blocked, which is why only some product URLs appear in the report.

Whether that is a problem depends on intent. If colour variants are meant to be separate, indexable pages, the rule is wrong; replace it with narrower rules for the parameters you actually want to keep out, such as Disallow: /*?sort=. If variants should consolidate into the main product page, the block is doing its job, though a canonical tag pointing at the clean URL is the better tool for that, because it lets Google crawl the variant and pass its signals to the main page.

The other lesson from this example: the person who added the Googlebot group almost certainly did not realise it switched off the Allow line above. When you add a group for a specific crawler, copy every rule it still needs into that group.

The mistake to avoid: using robots.txt to remove pages

robots.txt controls crawling, not indexing. If you block a page that is already in Google's index, Google can no longer see it, which also means it can no longer see a noindex tag on it. The page can stay in results, usually with no description, and Search Console will report it as Indexed, though blocked by robots.txt.

To get a page out of Google, do the opposite: allow crawling, add noindex, and let Google recrawl it. Only once it has dropped out should you consider blocking it again, and in most cases you do not need to.

Frequently asked questions

How long does it take for the status to clear after I fix robots.txt?

Google re-reads robots.txt within about a day, sooner if you request a recrawl. The pages themselves clear as they are recrawled, which can take days to a few weeks for a large site. "Validate fix" tracks the progress for you.

Is "Blocked by robots.txt" bad for my rankings?

Not in itself. It only hurts if the blocked pages are ones you want to rank. Blocking low-value URLs is normal and can help Google spend its crawling on the pages that matter.

My robots.txt looks fine but Search Console still says blocked. Why?

Check the robots.txt report for the version Google actually fetched. Different files on http and https, or on www and the bare domain, a CDN caching an old copy, or a server returning an error for robots.txt are the usual explanations.

Can I block a page from Google but keep it for other search engines?

Yes, with a User-agent: Googlebot group. Remember that once that group exists, Googlebot ignores the general * group, so repeat any rules it still needs.

Short version

Decide whether the blocked URLs are ones you want in Google. If they are, confirm with URL Inspection, find the most specific matching Disallow in the group Googlebot uses, narrow it, request a robots.txt recrawl and validate the fix. Never use robots.txt to remove a page from the index; use noindex and let Google crawl it. For the bigger picture of what robots.txt can and cannot do, see what robots.txt is and how to check it.