OpenAI backs a shared robots.txt signal for AI training

What OpenAI committed to support

OpenAI's October 30 announcement [1] says the company will honor a common robots.txt signal for foundational generative AI training. OpenAI expects to implement it in the coming months and plans to publish implementation details later.

OpenAI describes the effort as voluntary and asks other providers to honor the same signal. Its existing crawler-specific controls will remain in place.[1] Website operators therefore gain a possible shared preference, but they do not lose the controls they already use for individual OpenAI crawlers.

The announcement is a commitment to future support rather than confirmation that OpenAI systems process the token today.[1] Operators should wait for provider implementation details and test observed behavior before removing established rules.

How the policy token works

The October 30 token draft [2] proposes this ordinary robots.txt group: User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/

Under the proposal, Disallow expresses a preference against generative AI training for the matching path, while Allow expresses a preference in favor. Normal robots.txt path matching and longest-match rules apply.[2] In this example, the site opposes training for every path except /press-kit/.

A wildcard User-Agent: * group does not count as this training preference. If the dedicated token is absent, the common preference remains unknown.[2] The double underscore reduces accidental matches with ordinary crawler names.

What the signal does not control

The token does not tell a crawler whether it may fetch a page.[2] A site that wants to limit retrieval still needs rules for the crawler's own product token. OpenAI currently documents GPTBot for possible training use and OAI-SearchBot for ChatGPT search.[4]

The IETF working-group vocabulary draft [3] says that expressing a preference does not ensure that recipients follow it and does not decide any legal, contractual or technical obligation. It also labels its text as work in progress without working-group consensus.

The common token covers using an asset to modify the learned parameters of a generative AI model.[2][3] It does not cover every possible AI use, non-generative classification, ranking or scoring. Private agreements and more specific provider rules may also take precedence.[2]

How it differs from OpenAI's current controls

OpenAI's crawler documentation [4] treats GPTBot, OAI-SearchBot and ChatGPT-User as different services. Disallowing GPTBot indicates opposition to training, while allowing OAI-SearchBot supports appearance in ChatGPT search.

The proposed token expresses one purpose across participating providers. A GPTBot rule addresses one OpenAI crawler. These rules can coexist because the draft says content-use preferences and crawler access are independent.[2]

ChatGPT-User handles some user-requested visits, and OpenAI says robots.txt rules may not apply to those actions.[4] The shared training token does not change that documented behavior.

A cautious rollout checklist

• Inventory current crawler rules, contracts, content licenses and platform-level controls before editing robots.txt.
• Decide separately whether you want search discovery, automatic crawling, model training and user-requested retrieval.
• Add the common token only after each provider confirms support. Keep established provider-specific rules until observed behavior and documentation justify a change.
• Use explicit paths and test longest-match results for allowed exceptions such as a press kit. Avoid copying the example without checking your own URL structure.
• Fetch the live robots.txt from outside your publishing system and verify that caches, CDNs and deployment templates serve the intended file.
• Monitor provider documentation, the token draft and the IETF vocabulary because each is still subject to change.
• Keep a dated record of the preference, the person who approved it and the pages covered at the time content was retrieved.
• Use contracts, access controls and legal review where the preference carries material business or rights consequences.

Limits and unresolved questions

• OpenAI has not provided a production date, migration procedure or conflict-resolution table for its implementation.[1]
• The policy-token document is an individual Internet-Draft with no formal standing, no IETF endorsement and no working-group consensus.[2]
• The related vocabulary is also a draft and says its contents do not yet reflect working-group consensus.[3]
• A robots.txt preference is public and unauthenticated. The vocabulary draft warns that a recipient may not know whose preference the server represents.[3]
• The specifications communicate a preference, but they do not provide enforcement, audit evidence or a remedy when a recipient ignores it.[3]
• Existing copies and models may have been created under an earlier robots.txt file. The token draft ties the preference to the file in effect when an asset is retrieved.[2]

Frequently asked questions

• Can I use __usage_ai-training today? You can publish it, but OpenAI says its support will arrive in the coming months. Check every intended recipient before relying on it.[1]
• Does Disallow stop crawling? No. In this group it expresses a training preference. Use each crawler's product token to control retrieval.[2]
• Can I allow search and oppose training? OpenAI's current rules separate OAI-SearchBot from GPTBot, and the common token does not replace those controls.[1][4]
• Does User-Agent: * express the training preference? No. The token draft says wildcard groups must not match the policy token.[2]
• Is this an IETF standard? No. The token is an individual Internet-Draft without IETF endorsement or formal standing.[2]
• Does the signal create a legal right? The drafts do not decide that question. Applicable law, contracts and provider commitments still matter.[3]