Skip to content
Tutorials

robots.txt and XML Sitemaps: Google's Documented Rules, AI Crawler Tokens and Common Mistakes

AIDToolStack EditorialNovember 7, 20268 min read
robots.txtSitemapsSEOCrawlersGoogle Search
Share:Twitter

The classic robots.txt accident goes like this: staging has Disallow: / so it stays out of search, someone copies the config to production, and a week later traffic falls off a cliff. The second most common one is quieter. A page you want gone from Google gets a Disallow rule, and months later it's still in the results, now with no description, because Google can no longer fetch the page to see your noindex.

Both come from treating robots.txt as an indexing switch. It isn't. This post walks through what Google and the robots.txt RFC actually specify, how the matching rules work, what Google does when the file is missing or broken, the AI crawler tokens worth knowing, and the sitemap rules Google still uses. Sources are Google Search Central, RFC 9309 and sitemaps.org, checked in November 2026.

robots.txt controls crawling, not indexing

Google's robots.txt introduction says it directly: robots.txt is "not a mechanism for keeping a web page out of Google." Its main job is managing crawler traffic. A disallowed URL can still be indexed if other pages link to it; it just shows up without a description.

To keep a page out of results you need noindex, as a meta tag or an HTTP header, and Google's block-indexing page adds the catch: "For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file." Block it and Google never sees the instruction.

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

The header form is the one to use for PDFs and other files where you can't add a meta tag. And for anything genuinely private, neither helps. RFC 9309 states that robots.txt rules "are not a form of access authorization." Anyone can read your robots.txt, which makes a list of secret admin paths in it a small gift to whoever's looking.

Decision tree: to keep a page out of Google results allow crawling and add noindex; to cut crawl load disallow paths in robots.txt; to keep content private require authentication
Decision tree: to keep a page out of Google results allow crawling and add noindex; to cut crawl load disallow paths in robots.txt; to keep content private require authentication

The format, as Google parses it

Google's robots.txt specification and RFC 9309 agree on the basics:

  • The file lives at the root: https://example.com/robots.txt. It applies only to that protocol, host and port. https://blog.example.com needs its own, and so does http://example.com if you serve it.
  • It must be UTF-8 plain text. Google reads the first 500 KiB and ignores the rest.
  • Google supports four fields: user-agent, allow, disallow and sitemap. Everything else, including crawl-delay, is ignored by Google. Other crawlers may read it; Anthropic's crawler page, for example, shows Crawl-delay for ClaudeBot.
  • Field names are case-insensitive. Path values are case-sensitive, so Disallow: /Admin does not block /admin.

Here's a realistic file:

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Disallow: /*?sessionid=
Disallow: /*.json$

User-agent: Googlebot
Disallow: /experiments/

Sitemap: https://example.com/sitemap.xml

Three rules explain almost every surprise:

A crawler follows one group. Google picks the group with the most specific matching user agent and ignores the rest. In the file above, Googlebot obeys only Disallow: /experiments/. It is not blocked from /cart/, because specific groups and the * group aren't merged. Repeat the shared rules in every named group, or the named crawler gets fewer restrictions than everyone else.

The longest matching path wins. For /search/help, both Disallow: /search and Allow: /search/help match, and the longer one wins, so it's allowed. On a tie in length, Google applies the least restrictive rule. RFC 9309 says the same: an equivalent allow beats disallow.

* and $ are the only wildcards. * matches zero or more characters, $ anchors the end of the URL. Disallow: /*.json$ blocks /data/feed.json but not /data/feed.json?v=2. There's no regex beyond that.

The Sitemap line can appear anywhere, takes a full URL and doesn't belong to a group.

What happens when robots.txt fails

This part catches people during outages and migrations. Google treats the HTTP status of /robots.txt as an instruction in itself:

How Google handles robots.txt responses: 2xx applies rules, 3xx follows five hops then 404, 4xx means no restrictions, 5xx stops crawling for 12 hours then uses a cached copy for up to 30 days
How Google handles robots.txt responses: 2xx applies rules, 3xx follows five hops then 404, 4xx means no restrictions, 5xx stops crawling for 12 hours then uses a cached copy for up to 30 days

Two consequences:

  • A 404 means "crawl everything". If your new framework stops serving the file, nothing is blocked any more. Google also says not to use 401 or 403 to slow crawling, since those count as 4xx too.
  • A 5xx stops Google crawling the site. If your robots.txt route depends on a database or an app server that's down, the whole site's crawling pauses with it. Serve the file statically.

Google generally caches robots.txt for up to 24 hours, and a Cache-Control: max-age header can change that (there's more on cache headers in our HTTP caching guide). So a fix to a bad file isn't instant.

AI crawlers: which token does what

Most AI companies now separate training crawlers from search and user-triggered fetchers, and blocking the wrong one costs you visibility for nothing. As of November 2026:

TokenOperatorPurpose, per the operator
GPTBotOpenAICrawls content that may be used to train foundation models
OAI-SearchBotOpenAISurfaces sites in ChatGPT search
ChatGPT-UserOpenAIUser-initiated fetches; robots.txt rules may not apply
ClaudeBotAnthropicCollects content that may be used for training
Claude-SearchBotAnthropicIndexing for search results
Claude-UserAnthropicFetches pages when a person asks Claude something
Google-ExtendedGoogleControls use of crawled content for Gemini training and grounding

Sources: OpenAI's crawler page, Anthropic's crawler page, Google's common crawlers list. Google states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." It isn't a separate crawler either; Google crawls with its usual user agents and the token only governs use.

A file that opts out of training but stays in search:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

One change is on the way. On 30 October 2026 Google announced it plans to support a shared AI-training preference signal from the IETF AIPREF working group, so one rule can express a training opt-out across participating providers. Its crawler page says Google intends to support it with user-agent: __usage_ai-training in early 2027. Until it ships, keep the per-company groups; Google says existing controls will keep working alongside it.

Our Robots.txt Generator is handy for laying out groups, allow and disallow rules and sitemap lines. Its "Block AI Training" preset disallows only the tokens vendors document as training controls (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent) and leaves search crawlers and user-initiated fetchers like ChatGPT-User alone. Crawl-delay and Host are opt-in, with a note that Google ignores both.

XML sitemaps: what Google still reads

A sitemap is a list of canonical URLs you'd like crawled. It doesn't block anything and it doesn't force indexing. The minimal valid file:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/blog/rotating-api-keys</loc>
    <lastmod>2026-11-02</lastmod>
  </url>
  <url>
    <loc>https://example.com/docs/search?lang=en&amp;page=2</loc>
  </url>
</urlset>

The limits and rules, from Google's sitemap guide and sitemaps.org:

  • At most 50,000 URLs or 50 MB uncompressed per file. Beyond that, split it and list the parts in a sitemap index, which has the same limits. Gzip is allowed.
  • UTF-8, fully qualified absolute URLs, and <loc> under 2,048 characters. Escape & as &amp;, as in the second entry above.
  • A sitemap only covers URLs at or below its own directory unless you submit it through Search Console. Put it at the root.
  • Google ignores <priority> and <changefreq>. It uses <lastmod> only when it's "consistently and verifiably accurate", meaning the last significant change to the main content, structured data or links, not a footer tweak.
  • The ping endpoint is gone. Google deprecated it in June 2023 and pings now return 404. Submit in Search Console, through its API, or with a Sitemap: line in robots.txt.
  • RSS and Atom feeds are accepted as sitemaps too, though they only carry recent URLs.

That lastmod rule is where generated sitemaps go wrong. Stamping every URL with today's date on each build makes the field unverifiable, and Google only uses it when it can trust it. Our Sitemap Generator turns a pasted URL list into valid XML with escaping, and it only writes lastmod when you put a real date after a URL. It leaves out changefreq and priority by default, since Google ignores both. For a build-time sitemap, take lastmod from your content's real updated timestamp. If you're on Next.js, sitemap.ts does this cleanly, as shown in our Next.js SEO guide.

Mistakes worth checking for today

  1. Staging rules in production. Diff your production robots.txt in CI, or generate it from an environment variable with a production default that allows crawling.
  2. Blocking CSS and JS. Google warns against blocking resources it needs to understand the page.
  3. Blocking a page to deindex it. Remove the Disallow, add noindex, wait for a recrawl.
  4. A named group that forgets the shared rules. Googlebot reads only its own group.
  5. Wrong case in paths. /Blog/ and /blog/ are different rules.
  6. robots.txt served by the app. If it can return 5xx, it will, at the worst moment.
  7. Sitemaps listing non-canonical URLs. Redirects, noindex pages and parameter duplicates send mixed signals. List only the URLs you want in results.
  8. Fake lastmod dates. Use real ones or leave the tag out.

Before shipping a change, fetch the live file with curl -i https://example.com/robots.txt and confirm a 200, a text/plain content type and the rules you expect. Then open the Search Console robots.txt report, which shows the version Google last fetched, any parse errors, and lets you request a recrawl after a fix.

AE

Written by AIDToolStack Editorial

Researched from official documentation, specifications and vendor pages, listed under Sources below. Last checked November 2026.

Sources

  1. developers.google.com/search/docs/crawling-indexing/robots/intro
  2. developers.google.com/search/docs/crawling-indexing/robots/robots_txt
  3. developers.google.com/search/docs/crawling-indexing/block-indexing
  4. www.rfc-editor.org/rfc/rfc9309.html
  5. developers.google.com/search/docs/crawling-indexing/google-common-crawlers
  6. developers.google.com/search/blog/2026/10/ai-training-control
  7. developers.openai.com/api/docs/bots
  8. support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
  9. developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
  10. www.sitemaps.org/protocol.html
  11. developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping
  12. support.google.com/webmasters/answer/6062598

Related Posts

Explore on AIDToolStack

Discover tools and servers featured in our directory.

MCP Servers

Trigger.dev Jobs MCP ServerManage Trigger.dev background jobs from your AI coding assistant. Monitor job runs, trigger tasks manually, inspect execution logs, manage schedules, and debug failed runs. The server provides operational visibility into your background job infrastructure — see which tasks are running, queued, or failed, and take action. Your AI can help diagnose job failures from execution logs, retry failed tasks, and optimize concurrency configurations for better throughput.Tailscale Network MCP ServerManage your Tailscale mesh VPN network from your AI coding assistant. View connected devices, manage ACLs, configure exit nodes, check device status, and manage DNS settings. The server provides visibility into your Tailscale network — see which machines are online, troubleshoot connectivity issues, manage network access policies, and configure routes. Useful for developers managing distributed infrastructure where Tailscale provides secure networking between services.Portainer Docker MCP ServerManage Docker environments through Portainer's API from your AI coding assistant. Control containers, images, networks, and volumes across multiple Docker hosts and Kubernetes clusters. The server provides tools for container lifecycle management, image pulling and building, stack deployment, and resource monitoring. Portainer's multi-environment management makes this server particularly valuable for teams managing containers across development, staging, and production environments.

Reviews

Sign in to leave a review.

No reviews yet. Be the first!

Comments

Sign in to leave a comment.

No comments yet. Be the first!

Sponsor Our Newsletter

Reach thousands of developers who are actively evaluating AI tools, MCP servers, and dev infrastructure. Our weekly newsletter goes to engaged technical decision-makers.

Growing subscriber baseHigh open rates

All sponsored content is clearly labeled per our editorial policy.