"Indexed, though blocked by robots.txt" sounds contradictory, and the first time most site owners see it in Search Console they assume something is broken. Nothing is broken. It is Google telling you it has put a URL in its index even though your robots.txt stopped it from reading the page, and that the result is probably not what either of you wants.

The short answer: decide whether the page should be in Google. If it should, remove the robots.txt rule blocking it so Google can crawl it properly. If it should not, remove the block and add a noindex tag, so Google can crawl the page, see the tag and drop it. Keeping the block in place is the one option that never fixes anything.

How a page gets indexed without being crawled

robots.txt stops Google from fetching a URL. It does not stop Google from knowing the URL exists. If other pages link to it, from your own menus, from another site or from an old sitemap, Google can decide the URL is worth showing in results based on those links alone, without ever seeing the content.

When that happens, the search result usually looks poor. The title is guessed from link text, and instead of a description you get a line saying no information is available for the page. That is the giveaway: Google is showing a page it has never read.

When it is harmless

Open the status in the Page indexing report and look at the example URLs. If they are pages nobody searches for, such as an obscure parameter URL or an old login path, and they are not appearing for real queries, this is a low priority. The warning is accurate but the cost is close to zero.

It matters in two situations: when the URLs are pages you want to rank properly, and when they are pages you definitely do not want public, like staging copies, internal tools or thank-you pages that give away a discount.

If the page should be in Google

This is the easy case. The block is a mistake, so remove it.

  1. Find the rule. The page is blocked by the most specific Disallow that matches its path, in the robots.txt group Googlebot follows. Our guide to fixing "Blocked by robots.txt" explains how Google picks the rule, which is not top to bottom.
  2. Narrow or remove that rule and check the result with the Robots.txt Checker.
  3. Request a robots.txt recrawl in Search Console, then request indexing for the page with URL Inspection.

Once Google can read the page, the result picks up a real title and description, and the page can rank on its content rather than on guesswork.

If the page should not be in Google

This is the case people get wrong, because the instinct is to block harder. Blocking is what caused the problem. Google cannot see a noindex tag on a page it is not allowed to fetch, so the URL stays in the index indefinitely.

The fix is counter-intuitive but reliable:

  1. Remove the robots.txt block for those URLs.
  2. Add noindex, either as <meta name="robots" content="noindex"> in the page head or as an X-Robots-Tag: noindex HTTP header for files and non-HTML resources.
  3. Let Google recrawl. It sees the tag and removes the URL. Request indexing to speed this up for important cases.
  4. Use Search Console's Removals tool if it is urgent. It hides a URL from results for about six months, which covers the gap while the noindex takes effect. On its own it is temporary.

After the URLs have dropped out, you can leave them crawlable with noindex, which is the most robust setup. Re-blocking them is possible, but if links keep pointing at them, the same warning can come back.

For anything genuinely private, neither robots.txt nor noindex is the right tool. robots.txt is a public file that lists the paths you are hiding, and anyone can read it. Put private pages behind a login.

Check before you change anything

Before acting on the whole list, check whether these URLs are actually being shown to anyone. In Search Console's Performance report, filter by page and paste in a few of the affected URLs. If they have no impressions, they are sitting in the index without appearing for real searches, and you can fix them calmly as part of routine cleanup.

If some do have impressions, look at the queries. A blocked page ranking for your brand name, or for a query you care about, is where the poor "no information available" result is costing you clicks. Fix those first.

It is also worth checking where the links to these URLs come from. URL Inspection shows the referring sitemap and sometimes a referring page. If the URLs are linked from your own navigation or templates, removing those links stops the problem recurring; if they come from other sites, you cannot remove the links, which is one more reason to handle the URLs with noindex or a redirect rather than a block.

Where these URLs usually come from

  • Parameter and filter URLs blocked with a rule like Disallow: /*? but linked from category pages. Canonical tags pointing at the clean URL usually handle these better than robots.txt.
  • Internal search result pages, linked from somewhere public or shared by users.
  • Old sections that were blocked rather than redirected when they were retired. A 301 redirect to the replacement page is almost always the better choice.
  • Staging or development copies that someone linked to. These should be password protected, not just blocked.
  • URLs listed in your own sitemap that robots.txt blocks, which sends Google two opposite instructions. The sitemap and robots.txt conflict has its own guide.

Frequently asked questions

Does this warning hurt the rest of my site?

Not directly. It affects only the listed URLs. The indirect cost is that a poor result with no description can appear for your brand, and that a page you wanted to rank is ranking on guesswork.

Why doesn't Google just respect robots.txt and leave the page out?

Because robots.txt only governs crawling. Google respects it by not fetching the page. Deciding to show a URL that many pages link to is a separate, indexing decision, which is what noindex controls.

Can I use the X-Robots-Tag header for PDFs?

Yes. For files where you cannot add a meta tag, such as PDFs, send X-Robots-Tag: noindex as an HTTP header, and make sure robots.txt does not block the file so Google can see the header.

Short version

Google indexed a URL it was not allowed to read, from links alone. If you want it in search, unblock it and request indexing. If you do not, unblock it and add noindex, then let Google crawl it. Leaving the block in place keeps the warning, and the poor search result, exactly where they are. For the wider set of indexing problems, our guide to running an SEO audit yourself covers the order to check things in.