On October 30, 2026, four of the biggest names in AI said they plan to support a single shared signal that lets a website say no to generative-AI training. If you run a blog, a news site or a store, the promise is simple: one entry in your robots.txt file instead of a separate rule for every AI company and every crawler. The catch is in the timetable and in what the signal does not do.
This guide explains what was announced, who has committed to what date, and what a site owner should (and should not) expect. It matters because robots.txt is the oldest and most widely understood way for a website to tell automated visitors what they may do, and until now AI companies have each invented their own crawler names and their own opt-out rules.
Quick summary: OpenAI, Google, Microsoft and Anthropic said on October 30, 2026 that they plan to support a shared robots.txt policy token for opting out of AI training. Anthropic has reportedly started; OpenAI says in the coming months; Microsoft targets February 1, 2027 (per AI Weekly); Google says 2027. It governs training, not fetching, and does not remove data already collected.
What was announced
According to SiliconANGLE, Google, Microsoft, Anthropic and OpenAI announced plans to support a new AI crawling opt-out standard, and Meta expressed support for the idea without committing to implement it. Site operators could block all scrapers that collect training data for generative AI models with a single robots.txt entry. Today they must add a separate entry for each AI provider and each scraper. (Source: SiliconANGLE)
AI Weekly reports the signal is a robots.txt policy token written as __usage_ai-training. A site adds a User-Agent line with that token and Disallow: / to opt out across all four companies' crawlers in one entry, and can add Allow: lines for exceptions. (Source: AI Weekly)
The proposal is still a draft. SiliconANGLE says the companies submitted it to the Internet Engineering Task Force, which maintains the Robots Exclusion Protocol. AI Weekly describes it as an individual IETF Internet-Draft titled 'Machine-Readable Training Preference Signal', dated October 30, 2026, and notes the IETF says individual drafts have no formal standing. It attributes the draft to Anthropic's Kevin Kelley; that authorship detail comes from AI Weekly alone, so treat it as single-sourced.
Who is doing what, and when
The four companies are not moving at the same speed. The dates below are as reported by the two sources above.
| Company | Reported timeline | Notes |
|---|---|---|
| Anthropic | Already started | Per AI Weekly |
| OpenAI | In the coming months | Describes it as a coordination mechanism, not a security control |
| Microsoft | February 1, 2027 | Also plans a separate 'AI grounding' setting |
| Sometime in 2027 | Reportedly treats a token opt-out that does not name Google-Extended as a Google-Extended disallow | |
| Meta | No commitment | Signaled support for the idea only |
What it does not do
This is where the headline needs care. Per AI Weekly, the signal governs training, not fetching, so bots can still access pages. It is not a security control, and a scraper that ignores robots.txt can ignore this too. Updating robots.txt also does not remove material that has already been collected into existing datasets or trained models. (Source: AI Weekly)
SiliconANGLE adds that the standard would not apply to models that perform exclusively non-generative tasks, and that Microsoft plans an 'AI Grounding' setting. Grounding is when an AI model reads a page to answer a user's question rather than to collect training data. Microsoft's setting would let sites block that use as well, and would extend to third-party AI tools using Microsoft technology. OpenAI plans similar controls. (Source: SiliconANGLE)
Training versus answering: the line to watch
The practical question for publishers is whether blocking training also blocks appearing in AI answers. Under what has been announced, the answer is no: the token is about training data. Grounding, the live retrieval that powers chat answers and AI search, is being handled as a separate setting, at least at Microsoft. AI Weekly flags that it is still unclear where the line between training and grounding will fall, and whether the other companies will adopt Microsoft's split.
- Training opt-out: the new shared token, covering generative-AI training crawlers of the four companies once each implements it.
- Grounding opt-out: a separate control Microsoft plans, with OpenAI planning similar ones.
- Existing controls such as Google-Extended still exist; Google reportedly will treat a token-only opt-out as a Google-Extended disallow.
What a site owner can do today
Nothing is mandatory and nothing is urgent, but there is a sensible order of operations. First decide your policy: do you want to be in training data at all, and do you want to appear in AI answers? Those are different decisions. Second, keep your current per-company rules in place until each company actually implements the token, because a rule nobody reads yet protects nothing. Third, review your robots.txt once the rollouts land, starting with Microsoft's February 1, 2027 date as the first firm calendar marker.
Do not delete your existing AI-crawler rules just because the shared token exists. Only Anthropic is reported to have started support; the others are months away, and a non-compliant scraper will ignore both.
Why it matters beyond publishers
For years, AI training consent has been a patchwork of company-specific crawler names. A shared token is a small step toward one switch. It is also a reminder that opt-outs are forward-looking: they affect future crawls, and the open question of data already in past training sets remains. For everyday users of chatbots, the story is background, but it shapes which sites will keep allowing AI access and, indirectly, what these systems can learn from.
01What is the new robots.txt token?
It is a policy token written __usage_ai-training. Adding it with Disallow: / signals that a site does not want its content used for AI training, across the four companies that said they plan to support it (per AI Weekly).
02Which companies have committed to it?
OpenAI, Google, Microsoft and Anthropic said on October 30, 2026 that they plan to support it. Meta signaled support for the idea but has not committed (per SiliconANGLE).
03When will it work?
Anthropic has reportedly started. OpenAI says in the coming months, Microsoft targets February 1, 2027 (per AI Weekly), and Google says 2027.
04Does it remove my content from existing AI models?
No. Per AI Weekly, updating robots.txt does not remove material already collected into datasets or trained models, and it does not stop bots from fetching pages.
05Will blocking training hide my site from AI answers?
Not under what was announced. The token covers training; grounding for live answers is being handled as a separate setting, at least at Microsoft.
Want to see how these claims hold up? LumiChats lets you run one prompt through 40+ premium AI models in a single subscription and compare the answers yourself.
