OpenAI will honour a shared robots.txt token for opting out of AI training

Serguey Asael Shinder31 October 2026, 10:242 min read

Website operators who want to keep their pages out of generative AI training currently write a separate robots.txt block for every crawler they know about. OpenAI said on 30 October that it will also respect one common signal meant to be read by several AI providers at once.

The company's post is short on mechanics and links to an Internet-Draft submitted to the IETF the same day, "Machine-Readable Training Preference Signal" (draft-kelley-aipref-training-preference-signal-00). Its author is listed with Anthropic. The draft defines a policy token, __usage_ai-training, that goes in the User-Agent line of an ordinary robots.txt group. A group with that token and Disallow: / states a preference against training on the whole site; an Allow line can carve out paths, for example a press kit, where the operator is happy for content to be used.

The design deliberately adds nothing new to the robots.txt grammar. Path matching and longest-match precedence stay as RFC 9309 defines them, so existing parsers simply see a group for a user agent they do not recognise and ignore it. Two rules matter in practice: a wildcard group (User-Agent: *) does not count as a preference about training, and the token is independent of crawl rules, so allowing a crawler to fetch pages does not imply consent to training on them. The definition of "AI training" is taken from the IETF AI Preferences working group's vocabulary draft: changing the learned parameters of a generative model, excluding models used only for classification, ranking or scoring.

OpenAI says it aims to implement the signal "in the coming months", will publish implementation details then, and that its existing per-crawler controls remain in force.

What it means

Adding the group costs an operator four lines and breaks nothing for crawlers that ignore it. What it buys depends on who reads it: today that is one company's stated intention, with no date, plus a draft that has no standing in the IETF process yet. Keep the per-crawler rules you already have, and treat the shared token as an addition rather than a replacement until providers say they actually check it.