Robots.txt Was Never One Rule, and AI Assistants Are Proving It

Robots.txt has spent three decades as a polite request, not a lock. A publisher lists what an automated system should leave alone, and a compliant crawler reads the list and honors it. Independent web researchers are now testing what that convention even means when a human, not a bot, is the one asking, and they are not getting the same answer twice.

The clearest small-scale version came from Reddit user u/SurfacedBy, who put a page on a test server carrying a unique code, blocked it in robots.txt for every AI user agent they could name, and kept a second unblocked page as a control. Eight assistants were then asked to open the blocked page and report the code.

Three Behaviors, Not One

ChatGPT, Claude, and Meta AI refused. Claude was the clearest case, returning an explicit ROBOTS_DISALLOWED error rather than a generic failure, which matters because a silent failure proves nothing about intent.

Gemini, Grok, and Manus retrieved the blocked content anyway. Google's own documentation gives Gemini's behavior a plausible explanation: a user-triggered fetch is treated as the person's request, not autonomous crawling, so the traditional publisher-to-crawler contract does not obviously apply. Grok went further. Across three rounds, it reportedly never requested robots.txt at all, a different failure mode entirely from reading the rule and deciding it does not apply.

Copilot and Perplexity refused to even attempt the check, which u/SurfacedBy correctly separated from compliance. A refusal to try and a refusal after trying are not the same data point.

Not an Isolated Result

This is not the only independent test to land on the same inconsistency. Search Engine World ran a separate investigation across twelve AI assistants, verifying retrieval through server logs and unique canary strings rather than self-reported answers, and found the same pattern of uneven enforcement. DataDome's threat research team, led by Jerome Segura, reported separately that ChatGPT does not reliably check robots.txt at all unless explicitly reminded. Three independent groups, three different methodologies, one shared conclusion: no AI assistant treats robots.txt as a settled, uniform boundary.

Where the Small Test Falls Short

u/SurfacedBy's test is useful precisely because it is small enough to see clearly, and that size is also its limit. Named-agent blocking alone cannot rule out an unlisted fetcher doing the retrieving. A rigorous version needs a wildcard condition, User-agent: * Disallow: /, alongside the named-agent block, because a successful retrieval under a named block could still originate from an undocumented crawler the researcher simply did not list. It also needs unique, high-entropy canary strings unlikely to exist anywhere else online, so a correct answer can be tied to an actual fetch rather than a lucky guess or cached knowledge from elsewhere.

That last distinction is the sharpest one raised across all three investigations: a live fetch, an indirect retrieval through a search index or cache, and prior knowledge baked into training are three different operations that can produce the identical-looking correct answer. Robots.txt only ever governs the first one. An assistant that refuses to fetch a blocked page live has said nothing about whether it already knows what is on it.

What This Actually Means

The old mental model was one file, one meaning: crawl or do not. AI assistants now route requests through training crawlers, indexing systems, retrieval infrastructure, and user-triggered fetchers, and nothing requires those pathways to honor the same instruction the same way. A publisher who blocks a name-listed agent may still get fetched by an unnamed one, or by a system that never checks the file at all.

That is the finding worth sitting with: robots.txt compliance is looking less like a property of a company and more like a property of a specific pathway inside that company's stack. Which agent, in which mode, for which purpose, honors which instruction, is a harder and more useful question than whether an AI company respects robots.txt in general.