OpenAI, Google, Anthropic and Bing back a single robots.txt signal for AI training preferences

For years, telling AI companies not to train on your website has meant keeping a list: one rule for each company's crawler, updated every time a new lab appears or an old one renames its bot. On October 30, the industry took a step toward ending that chore. OpenAI said it will honor a single shared signal in robots.txt, the humble text file that already lets sites state how their content may be used — and Google, Anthropic and Bing announced the same day that they will support it too.

One line instead of a list#

The mechanism lives in an Internet-Draft published October 30, written by Kevin Kelley of Anthropic, under the Internet Engineering Task Force's AI Preferences working group. It introduces a cross-vendor token, __usage_ai-training, used in the familiar User-Agent line of a robots.txt group. A site that wants to opt out of training writes a group for that token and disallows the paths it wants to protect; it can also carve out exceptions, for example by allowing a press-kit directory while disallowing everything else.

Nothing about the file format changes. The draft adds no new directives and no new grammar: it reuses the path language, longest-match precedence and group structure of the robots.txt standard, RFC 9309. Parsers that do not know the token simply ignore the group, as they already ignore groups naming crawlers they do not recognize. Two details matter. A wildcard group, addressed to every user agent, does not count as expressing a training preference — the token must appear explicitly. And if no group names the token, no preference has been expressed at all.

Note also what the token does not touch: it is a training preference, not a crawling instruction. Blocking it says nothing about whether a bot may visit a page.

Who signed on, and when it takes effect#

OpenAI framed its announcement as a voluntary effort and said it aims to implement the signal in the coming months, with details to follow. Its existing robots.txt controls — which distinguish its training crawler from its search and user-fetch bots — will continue to apply alongside the new signal, not be replaced by it.

Google said it will support the shared control along with other providers in 2027. When a site uses the shared token to opt out and has not explicitly specified Google-Extended, Google plans to read that as an opt-out of Google-Extended too. Sites can still set company-specific rules on top of the shared one.

The draft itself is an individual Internet-Draft, which means it carries no formal endorsement from the IETF and no standing in its standards process yet. The token's meaning tracks a separate working-group vocabulary draft that defines AI training as using content to change the learned parameters of a generative model — itself still a work in progress that cautions it does not yet reflect working-group consensus, and states plainly that following preferences remains the recipient's choice.

Why publishers asked for this#

The current system punishes diligence. A publisher that wants to opt out must identify each provider's training bot by name and maintain a rule for each one — and the list never stops growing, because every new model launch can bring a new crawler. The shared token flips the effort: write the preference once, and any provider that has adopted the signal honors it.

That is also why the announcement landed as a joint move rather than a solo one. Google pointed to feedback from publishers, civil society and regulators calling for an easier, industry-wide way to express training preferences, and noted that the proposal is consistent with the European Union's Code of Practice for general-purpose AI models, which recognizes robots.txt as a primary way for publishers to state machine-readable preferences.

What it does not do#

Enthusiasm should be measured against four limits. First, the signal is voluntary. Nothing in the draft compels a provider to honor it, and the vocabulary draft is explicit that compliance is left to the recipient. A shared preference is only as strong as the set of companies that adopt it — and for now, that is the companies that chose to announce on day one.

Second, private agreements can override it; the draft says so directly. Third, the signal is forward-looking: it covers future training, not models already trained, and not content obtained through other channels. Fourth, it is narrow by design — training of generative models only, not search, not retrieval for answers, and not whether a site appears in AI overviews or chatbots.

None of that makes the move empty. A standard opt-out that is cheap to write is more likely to actually be written, especially by the small publishers and independent sites that never had time to maintain a per-bot list. The realistic reading is a lower barrier to opting out, not a guarantee.

What to watch#

The next tests are concrete. Watch for OpenAI's implementation details in the coming months and for Google's rollout during 2027, including whether other large providers join the shared token or keep their own controls. Watch whether the draft advances toward a working-group document with real consensus, and whether hosting platforms add a one-click version of the token — that is how a standard like this reaches sites that never hand-edit a robots.txt file. And watch the opt-out rate itself: if a meaningful share of the web adopts the shared signal, training-data sourcing will have to adapt.

Sources: OpenAI's announcement, openai.com; Google's announcement as reported by Search Engine Land; and Unite.AI. Images are AI-generated editorial illustrations.