A sitemap is a list of pages you want search engines to crawl. robots.txt is a list of places they are not allowed to go. When a URL appears in both, you are giving the same search engine two opposite instructions, and the warning you see is it pointing that out.

The exact wording depends on the tool. Older versions of Search Console, Bing Webmaster Tools and most SEO crawlers say something close to "Sitemap contains URLs which are blocked by robots.txt". Current Search Console shows the same thing as "Blocked by robots.txt" in the Page indexing report when you filter it to submitted pages. The cause and the fix are the same.

The short answer: list the conflicting URLs, decide for each group whether it should be crawled, then fix whichever file is wrong. Usually it is the sitemap: a plugin is listing pages that were deliberately blocked, and the fix is to exclude them from the sitemap.

Why it matters

On its own, a handful of conflicts does little harm. Google follows robots.txt and skips the blocked URLs. The problems are what the conflict tells you and what it costs:

  • It is often the symptom of a bigger mistake. If important pages are in the sitemap and blocked, the sitemap is right and robots.txt is hiding pages you need. That is worth finding quickly.
  • It weakens your sitemap as a signal. Search engines use the sitemap to decide what matters on your site. A sitemap full of URLs they cannot fetch is a less trustworthy guide.
  • It clutters your reports. Real indexing problems are harder to spot among hundreds of expected warnings.

Step 1: find the conflicting URLs

In Search Console, open Pages under Indexing, change the filter at the top from "All known pages" to "All submitted pages", and open the "Blocked by robots.txt" row. Those are URLs your own sitemap submitted that robots.txt blocks. Export the list.

If you do not have Search Console data yet, or want to check before submitting, run the sitemap through the Sitemap Validator to see what it actually lists, and the live robots.txt through the Robots.txt Checker to see its rules side by side.

Step 2: group them and decide

Sort the exported URLs by pattern rather than looking at them one by one. You will usually find two or three groups, such as tag archives, attachment pages, parameter URLs or a whole section. For each group, ask one question: do I want these pages in search?

  • No, they should stay blocked. The sitemap is wrong. Remove them from it.
  • Yes, they should be found. robots.txt is wrong. Narrow the rule that blocks them.
  • They should not exist. Old URLs that redirect or return errors should not be in a sitemap at all, blocked or not.

Step 3a: fixing the sitemap

Most sitemaps are generated by a CMS or SEO plugin, so fix the generator rather than the file, or your edit will be overwritten.

  • Exclude whole content types you block, such as tag archives, author pages, media attachment pages or internal search. SEO plugins have per-type settings for this.
  • Exclude individual pages that are blocked on purpose, like cart, checkout and account pages, if the generator includes them.
  • Do not list parameter variants. A sitemap should contain clean, canonical URLs only.

A good rule of thumb: every URL in a sitemap should return 200, be the canonical version of itself, be indexable and be allowed by robots.txt. Anything that fails one of those four does not belong there.

Step 3b: fixing robots.txt

If the blocked pages should be crawled, the rule blocking them is too broad. The common offenders are prefix rules that match more than their name suggests, like Disallow: /p catching /products/, and query-string rules like Disallow: /*?. Google applies the most specific matching rule within the group that applies to it, so a targeted Allow often fixes this without loosening anything else. Our guide to fixing "Blocked by robots.txt" walks through how Google chooses the rule.

A typical case: a blog with tag archives

The single most common version of this warning I see looks like this. A blog's robots.txt contains Disallow: /tag/, added at some point to stop Google crawling hundreds of thin tag archive pages. Separately, the SEO plugin's sitemap includes a tag-sitemap.xml, because listing taxonomies is its default. Search Console now reports every tag archive as submitted but blocked.

Neither file is wrong on its own. They simply disagree, and the fix is to pick a side. If the tag pages are thin, keep the block and switch off the tag sitemap in the plugin settings. If some tag pages are genuinely useful hubs, the better long-term setup is usually to remove the robots.txt block, keep the useful tags indexable, and set the thin ones to noindex, so Google can crawl all of them but only indexes the ones worth showing.

Either way, the number of conflicts in the report drops to zero once the sitemap is rebuilt and resubmitted, and the remaining warnings in your Page indexing report become ones worth reading.

Step 4: resubmit and confirm

  1. Check the sitemap again with the Sitemap Validator once the generator has rebuilt it.
  2. Resubmit it in Search Console and Bing Webmaster Tools.
  3. If you changed robots.txt, request a recrawl from Search Console's robots.txt report.
  4. Use "Validate fix" on the Page indexing status so Google rechecks the list and emails you when it is done.

If Search Console reports that it couldn't fetch your sitemap at all, that is a separate problem with its own causes, and worth fixing first.

Frequently asked questions

Will Google crawl a URL because it is in my sitemap, even if robots.txt blocks it?

No. robots.txt takes precedence. The sitemap entry is simply ignored for crawling, and the URL shows up in this warning.

How many conflicts are acceptable?

Ideally none, because a clean sitemap is easy to keep. In practice a few leftovers from a plugin are harmless. Hundreds, or any important pages among them, are worth fixing.

Should I block my sitemap file in robots.txt?

No. Search engines need to fetch the sitemap. Reference it with a Sitemap: line in robots.txt instead, which also helps search engines that you have not submitted it to.

Short version

A URL in your sitemap and blocked by robots.txt is a contradiction. Find the URLs, group them, decide which file is wrong for each group and fix that file at its source. Most of the time the sitemap is listing pages that were blocked on purpose, and the fix is one plugin setting.