robots.txt Explained: Syntax, Crawl Control and the Mistakes That Hide Pages

What robots.txt Actually Does — and What It Does Not

robots.txt controls crawling. It does not control indexing. This distinction is responsible for more SEO confusion than any other single topic.

If you Disallow a URL, compliant crawlers will not fetch it. But Google can still index that URL if other pages link to it — it will appear in search results with no snippet and the note "No information is available for this page." You blocked the fetch, not the listing.

The corollary is the mistake that costs the most traffic: if you want a page out of the index, you must let Google crawl it so it can see your noindex tag. Blocking a page in robots.txt and adding noindex to it means Google never reads the noindex. The page stays indexed indefinitely.

Goal                          Correct mechanism
Keep out of the index         <meta name="robots" content="noindex"> (crawlable)
Reduce crawl load             robots.txt Disallow
Remove from public access     Server-side auth (401/403)
Consolidate duplicates        rel=canonical or 301

File Location and Format Rules

• It must live at the protocol-and-host root: https://example.com/robots.txt. A file at /blog/robots.txt is ignored entirely.
• Each subdomain needs its own file. shop.example.com does not inherit from example.com.
• Serve it as text/plain with a 200 status. Google caps its read at 500 KiB.
• Paths are case-sensitive. /Admin and /admin are different rules.
• A 404 means "crawl everything". A persistent 5xx is treated by Google as "crawl nothing" for around 30 days — so a broken server can silently pull your site out of the index.

The Directives That Matter

User-agent

Opens a group of rules. * matches any crawler that does not have its own group.

Disallow and Allow

Path prefixes, with two wildcards supported by Google and Bing:

• * matches any sequence of characters.
• $ anchors the end of the URL.

Disallow: /search          # blocks /search, /search?q=x, /searchresults
Disallow: /*?sort=         # blocks any URL containing ?sort=
Disallow: /*.pdf$          # blocks URLs ending in .pdf
Allow: /wp-admin/admin-ajax.php

When Allow and Disallow both match, the more specific rule (longer path) wins. On an exact tie, Allow wins.

Sitemap

Absolute URL, and it is group-independent — it applies globally regardless of where you put it. You can list several.

Sitemap: https://example.com/sitemap.xml

The Group-Matching Rule Almost Everyone Gets Wrong

This is the bug worth checking your own file for right now. A crawler obeys exactly one group — the most specific one that names it — and completely ignores every other group, including *.

So this file does not do what its author intended:

User-agent: *
Disallow: /admin
Disallow: /api/

User-agent: Googlebot
Allow: /

Googlebot matches its own group, so it ignores the * group entirely. It is now explicitly permitted to crawl /admin and /api/. The rules you cared about apply to every crawler except the one you most wanted them to apply to.

The fix is to repeat the disallows in every named group:

User-agent: *
Disallow: /admin
Disallow: /api/

User-agent: Googlebot
Allow: /
Disallow: /admin
Disallow: /api/

If you have per-bot groups for GPTBot, PerplexityBot, ClaudeBot and friends, each one needs the full rule set. There is no inheritance.

Five Mistakes That Cost Traffic

1. Blocking CSS and JS. Google renders pages. Block /assets/ and it sees an unstyled, non-interactive page, which harms both mobile-friendliness and content extraction.
2. Disallow: / left over from staging. One character, whole site gone. Check this first whenever traffic falls off a cliff.
3. Using robots.txt to hide sensitive URLs. The file is public and is itself the roadmap. Anyone can read your disallow list. Use authentication.
4. Blocking a page you also want de-indexed. Covered above — this is the classic own-goal.
5. Assuming Crawl-delay works on Google. It does not; Google ignores it. Bing and Yandex honour it. Control Google's rate in Search Console instead.

Test Before You Ship

Generate a correct file with the robots.txt Generator, pair it with the XML Sitemap Generator, then validate the live file in Google Search Console's robots.txt report — it shows you the exact rule that matched for any URL you paste in.

Next, make sure the pages you do allow are worth crawling: see the Technical SEO Checklist and XML Sitemaps in 2026.