Robots.txt Generator — Build a Perfect robots.txt File — Robots.txt Generator

How to Use This Generator

Build, preview, and download a properly formatted robots.txt in three easy steps.

1. Pick a Template

Start with SEO Friendly to block admin areas and common SEO crawlers, or choose Allow All for a simple open policy. Customise from there.

2. Add Your Rules

Add Disallow rules for paths you want blocked and Allow rules for exceptions. Add multiple user-agent groups to target specific bots.

3. Download & Deploy

Copy or download your robots.txt and upload it to the root of your website at yourdomain.com/robots.txt. Verify it with our Robots.txt Checker.

Common robots.txt Rules Explained

*

Wildcard User-agent

User-agent: * applies rules to all crawlers not explicitly named. Always include this as your default rule group.

Disallow

Disallow: /admin/ tells bots not to crawl that path. An empty value Disallow: explicitly allows everything.

Allow

Allow: /wp-admin/admin-ajax.php overrides a broader Disallow for a specific path. Useful for AJAX endpoints.

Sitemap

Sitemap: https://example.com/sitemap.xml helps all search engines discover your sitemap automatically.

robots.txt explained

What Is a robots.txt File and Why Does It Matter for SEO?

A robots.txt file is a plain-text file placed at the root of your website that tells search engine crawlers which pages or sections of your site they are — and aren't — allowed to access. It's one of the first files Googlebot, Bingbot, and other crawlers fetch when they visit your site.

While robots.txt doesn't directly affect rankings, it controls how your crawl budget is spent. If crawlers waste time on low-value pages like admin panels, login pages, or duplicate parameter URLs, they may crawl fewer of your important pages — which can slow down indexation.

A well-configured robots.txt file keeps bots focused on the pages that matter: your blog posts, product pages, and landing pages.

Quick facts

Location

Must live at the exact path yourdomain.com/robots.txt — no subdirectory.

Case sensitive

Directives like Disallow must be capitalized correctly. The file itself is case-insensitive on most servers.

Not a security tool

robots.txt is a polite request, not a lock. Malicious bots ignore it. Use server-level authentication for truly private pages.

Max 500 KB

Google only reads up to 500 KB of your robots.txt file. Keep it concise.

step by step

How to Create a robots.txt File

Follow these steps to build, test, and deploy a robots.txt file that improves your site's crawl efficiency.

Step 1

Audit your URL structure

Before writing a single rule, decide which sections of your site bots should skip. Common candidates: /admin/, /login, /cart, /checkout, duplicate parameter URLs like ?sort= or ?page=, and staging subdirectories.

Step 2

Choose a template

Use our SEO Friendly template as a starting point — it blocks common admin paths and disruptive SEO crawlers. If you run WordPress, it already covers /wp-admin/ while keeping admin-ajax.php accessible for AJAX functionality.

Step 3

Set your user-agent rules

The User-agent: * group applies to all bots. Add specific groups for crawlers like AhrefsBot or SemrushBot if you want to block third-party SEO tools from consuming bandwidth. Order matters: more specific rules take precedence.

Step 4

Add your sitemap URL

Include a Sitemap: line pointing to your sitemap.xml. This helps Google and Bing discover it automatically without relying on Search Console submissions. Use the full absolute URL including https://.

Step 5

Download and deploy

Download your robots.txt and upload it to the root of your web server — the same directory as your homepage. On most hosts this is /public_html/, /www/, or /htdocs/. The final URL must be https://yourdomain.com/robots.txt.

Step 6

Verify with our checker

After deploying, use our Robots.txt Checker to confirm your file is live, correctly formatted, and not accidentally blocking important pages. Re-test after any edits.

ready-made examples

robots.txt Examples for Popular Platforms

Copy-paste starting points tailored for the most common website platforms.

WordPress

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php

User-agent: AhrefsBot
Disallow: /

Sitemap: https://example.com/sitemap_index.xml

Shopify

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts
Disallow: /checkout
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /blogs/*+*

Sitemap: https://example.com/sitemap.xml

Next.js / SPA

User-agent: *
Disallow: /api/
Disallow: /_next/
Disallow: /dashboard/
Disallow: /auth/
Allow: /

Sitemap: https://example.com/sitemap.xml

Wix

User-agent: *
Disallow: /_api/
Disallow: /_partials/
Disallow: /pro-gallery-webapp/
Disallow: /corvid-bi/
Disallow: /wix-warmup-data/
Allow: /

Sitemap: https://example.com/sitemap.xml

avoid these

7 Common robots.txt Mistakes That Hurt SEO

These errors regularly cause sites to lose traffic — most can be fixed in minutes.

1. Disallow: / on the wildcard agent

The most damaging single line in SEO history. This one rule blocks every search engine from indexing your entire site. Always double-check before deploying.

2. Blocking CSS and JavaScript files

If bots can't load your stylesheets and scripts, Google can't render your pages properly. Poor rendering leads to lower perceived quality and weaker rankings.

3. Missing User-agent: * group

Without a wildcard group, any unnamed crawler has no instructions at all. Always include a catch-all rule group as your default policy.

4. No Sitemap declaration

While Google can find your sitemap via Search Console, declaring it in robots.txt ensures every crawler — including Bing, DuckDuckGo, and others — can locate it automatically.

5. Blocking pages then expecting them indexed

robots.txt prevents crawling. But Google may still index a Disallowed URL if another page links to it. To remove a URL from search results you need noindex combined with allowing the crawl.

6. Overly high Crawl-delay

A Crawl-delay of 10+ seconds tells Google to wait 10 seconds between each request. For large sites this means it takes weeks to re-crawl all your pages. Only use it if your server genuinely struggles under crawl load.

7. Using robots.txt as a security measure

robots.txt is publicly visible. Anyone can read it to map your private URLs. Use proper authentication to protect sensitive pages — robots.txt is a crawl guide, not a lock.

FAQ

Frequently Asked Questions

1 Does robots.txt affect Google rankings directly?

No — robots.txt controls crawl access, not ranking signals. However, it indirectly affects SEO by directing your crawl budget toward the pages that matter. A misconfigured robots.txt can prevent important pages from being crawled at all, which stops them from being indexed and ranked.

2 What happens if I don't have a robots.txt file?

Without a robots.txt file, search engine bots will crawl your entire site by default. This is fine for most small sites. The main downside is that bots will waste time on admin pages, session URLs, and other non-indexable content. For larger sites, missing a robots.txt can mean poor crawl budget allocation.

3 Can I block specific bots like AhrefsBot or SemrushBot?

Yes. Add a new user-agent group with their bot name and Disallow: /. Reputable SEO tools like Ahrefs and Semrush honour robots.txt. Note this has no effect on Googlebot — you'll need to handle Googlebot separately if needed.

4 Does robots.txt block a page from appearing in search results?

Disallowing a URL stops Google from crawling it but does not remove it from search results if it's already indexed, or if other sites link to it. To remove a page from search results, use a noindex meta tag on the page itself (and allow crawling so Google can read that tag).

5 How often does Google re-read my robots.txt?

Google typically re-crawls robots.txt at least once a day for active sites, and caches it for up to 24 hours. After you make changes, you can request a faster re-crawl via Google Search Console under the URL Inspection tool.

6 Do I need separate robots.txt files for subdomains?

Yes. Each subdomain needs its own robots.txt. The file at blog.example.com/robots.txt applies only to that subdomain. The root domain's robots.txt does not apply to subdomains.

Robots.txtGenerator