Website owners who want to keep their pages out of AI training may soon need just one robots.txt group instead of a growing list of crawler names. Google, Microsoft and OpenAI said on October 30, 2026 that they plan to honor a shared policy token, __usage_ai-training, and an Anthropic engineer published the Internet-Draft that defines it the same day. A robots.txt AI training opt-out written this way would apply to every provider that adopts it.
There is a catch for each company, though. None of them is reading the line yet, the dates run from “the coming months” to February 2027, and the signal means something slightly different at Google than it does at Microsoft.
- The line:
User-Agent: __usage_ai-trainingfollowed byDisallow: /asks participating AI developers not to train generative models on your site. - Who says they will honor it: Microsoft from February 1, 2027; Google in “early 2027”; OpenAI “in the coming months”; Anthropic says it has started implementing it.
- What it doesn’t do: it isn’t a crawl block, it doesn’t touch search rankings, and it doesn’t set up licensing or payment.
- The Google catch: Google will read it as a Google-Extended disallow, which also covers grounding in Gemini Apps.
What the robots.txt AI training opt-out says
The new signal piggybacks on the oldest instruction file on the web. In robots.txt, a site normally addresses a crawler by name, such as Googlebot or GPTBot. The shared token is not the name of any crawler. It is a reserved “policy token” that every participating company agrees to read as a statement about how your content may be used.
The Internet-Draft, draft-kelley-aipref-training-preference-signal-00, was posted on October 30, 2026 by Kevin Kelley of Anthropic. Its example blocks training for a whole site but leaves one folder open:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Microsoft’s implementation guide shows the simplest cases: Disallow: / under the token for the whole site, or Disallow: /example-path/ for a single section. Allowing training needs no action at all, because Allow is the default when a site publishes nothing.
The draft adds a few rules that matter in practice. A wildcard group (User-Agent: *) must not be treated as a match for the token, so an existing blanket rule doesn’t silently opt you out. If a site has no __usage_ai-training group, it has expressed no preference. And the token is not a crawl directive: it says nothing about whether a bot may fetch a page, only about what may be done with the content afterward.
“Training” has a narrow meaning here. The draft borrows the definition from the IETF AI Preferences (AIPREF) working group’s vocabulary draft: using an asset to modify the learned parameters of a generative AI model. Training models that only do non-generative work, such as classification or ranking, falls outside it.
Who has committed, and when it starts
The four companies are not moving on the same clock. Here is what each one has put in writing.
- Microsoft: the most concrete. A Bing blog post by Krishna Madhavan, Principal Product Manager, says Microsoft “will honor the new preferences as it crawls” starting February 1, 2027.
- Google: a Search Central blog post by John Mueller says, “We aim to support this new control along with other AI providers in 2027.” Google’s crawler documentation now says it intends to support the signal “in early 2027.”
- OpenAI: its announcement says the company aims to implement the signal “in the coming months” and that it “will not replace the controls we already provide through robots.txt,” meaning GPTBot rules keep working.
- Anthropic: Kelley wrote on the IETF ai-control mailing list that “Anthropic has already started implementing support for this draft,” adding that it will take time to reach all of the company’s services and that it “complements our existing crawler controls rather than replacing them.”
Meta, Apple and Amazon are missing. Meta and Amazon staff are among the people the draft thanks for taking part in the discussion, but none of the posts names either company, or Apple, as one that will read the token, as PPC Land pointed out in its October 31 analysis. A site that wants to reach those companies still has to use their own crawler names.
The paperwork also deserves a plain reading. The draft is an individual submission. Its own header says it is not endorsed by the IETF and has no formal standing, and it expires on May 3, 2027 unless it is revised. Google describes the proposal as consistent with the EU’s General-Purpose AI Code of Practice, which treats robots.txt as a main way for publishers to state machine-readable preferences.
The Google catch: training and Gemini grounding move together
Google will not read the shared token as a new, separate switch. Its blog post says that when a site uses the token to opt out of training “and Google-Extended is not explicitly specified,” Google “plans to interpret that as if the publisher had disallowed Google-Extended.”
That matters because Google-Extended covers more than training. Google’s crawler documentation says the token controls whether crawled content may be used to train future Gemini models for Gemini Apps and the Vertex AI API, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Grounding is the step where a model pulls live web content into an answer at prompt time. So a site that writes only the shared opt-out also steps out of Gemini’s grounded answers.
Google spells out the trade in one line: “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” A site can still opt out with the shared line and add a separate group that allows Google-Extended, because Google treats the named token as the more specific rule. But that allow covers both uses. Google offers no separate grounding control, so there is no Google setting that keeps a page citable in Gemini Apps while keeping it out of Gemini training.
Google’s documentation also says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Neither the token nor Google-Extended is described as a control for AI features inside Search.
The Microsoft version: training and grounding split
Microsoft took the opposite approach. It uses the shared token only for training and adds a separate grounding preference that is set page by page, not in robots.txt. A page can carry:
<meta name="Bingbot" content="ai-training:disallow">or the headerX-Robots-Tag: Bingbot: ai-training:disallow, which overrides the robots.txt training setting for that page;ai-grounding:disallow, which keeps the page out of generated answers;ai-grounding:disallowextended, which allows grounding only in Microsoft-operated experiences, such as Copilot, Microsoft 365 and Bing’s AI Mode and AI Overview, but not in services Microsoft sells to third parties, such as Web IQ and the grounding API in Foundry.
Microsoft says the training and grounding settings are independent, that neither changes eligibility for regular Bing results, and that they “do not establish licensing or payment arrangements.” Its post adds: “Neither choice guarantees inclusion, traffic, or payment.”
Sites that rely on the older NoArchive and NoCache tags have dates to watch. Bing Webmaster Tools opens a temporary grounding preference on December 15, 2026, for those sites only. New preferences take effect from February 1, 2027 as pages are recrawled. After March 1, 2027, Microsoft will no longer read NoArchive or NoCache as a training or grounding preference, and the temporary preferences expire on September 30, 2027. If Microsoft finds any of the new preferences on a site, it ignores leftover NoArchive and NoCache tags.
What site owners can do now
Adding the group today costs nothing, and the draft says a site without it has expressed no preference. One exception needs care: if you rely on NoArchive or NoCache to keep pages out of Copilot answers, Microsoft says it will disregard those tags once it detects any of the new preferences on a site. Adding only the training line could therefore drop your grounding restriction at Bing, so publish the page-level grounding setting at the same time. Just don’t remove the crawler rules you already have. OpenAI and Anthropic both say their existing controls stay in place, and Google keeps Google-Extended. Until each company confirms it is reading the token, those named rules are the ones doing the work.
Three cautions apply. First, the token is a usage preference, not a lock: Microsoft notes that an Allow does not override a rule that blocks Bingbot from crawling, and nothing in the draft stops a company that hasn’t signed on. Second, settings take effect only after a recrawl; IndexNow can prompt Bing to revisit a page but doesn’t guarantee when. Third, decide on grounding separately. A training opt-out is cheap. A grounding opt-out can cost visibility, and Microsoft warns that Disallow or DisallowExtended “will significantly reduce” a site’s presence in AI answers.
What to watch next
The first test comes in February, when Microsoft says it starts honoring the preferences. Watch for OpenAI’s and Anthropic’s documentation to name the token, for Google to publish exact syntax for combining it with Google-Extended, and for a revised draft or working-group adoption at the IETF. The other open question is whether Meta, Apple and Amazon join. Until then, the shared line is a useful extra, not a replacement for the crawler rules a site already has.
Frequently asked questions
What is __usage_ai-training in robots.txt?
It is a shared policy token defined in an IETF Internet-Draft published on October 30, 2026. Writing User-Agent: __usage_ai-training followed by Disallow: / asks participating AI developers not to use a site’s content to train generative AI models. It is not the name of a crawler and does not block crawling.
When will Google, Microsoft and OpenAI honor the new opt-out?
Microsoft says it will honor the preference from February 1, 2027, as it crawls. Google says it intends to support it in early 2027. OpenAI says it aims to implement it in the coming months. Anthropic’s draft author says the company has started implementing it but will need time to cover all of its services.
Does the robots.txt AI training opt-out affect search rankings?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal, and Microsoft says its training and grounding preferences do not change eligibility for traditional Bing search results. Grounding opt-outs can still reduce how often a site appears in AI answers and chatbot citations.
Will opting out of AI training remove my site from Gemini answers?
It can. Google plans to read the shared opt-out as a Google-Extended disallow when Google-Extended is not named separately, and Google-Extended also covers grounding in Gemini Apps. Google says sites that specifically allow Google-Extended remain eligible to appear in Gemini Apps responses through grounding.
Should I keep my GPTBot and Google-Extended rules?
Yes. OpenAI says the shared signal will not replace its existing robots.txt controls, Anthropic says it complements its crawler controls, and Google says company-specific controls will keep working alongside it. Since no company is reading the token yet, the named rules are still the ones that take effect today.
Sources
- Google Search Central Blog, John Mueller, “Supporting a shared AI training control for websites” (October 30, 2026): developers.google.com
- Google, “Google’s common crawlers” (Google-Extended scope; early 2027 note): developers.google.com
- Microsoft Bing Blog, Krishna Madhavan, “Your content, your choice: clearer preferences for training and grounding” (October 30, 2026): blogs.bing.com
- OpenAI, “Making it easier for websites to express AI training preferences” (October 30, 2026): openai.com
- IETF, K. Kelley, draft-kelley-aipref-training-preference-signal-00 (October 30, 2026): datatracker.ietf.org
- IETF ai-control mailing list, Kevin Kelley, “New draft: draft-kelley-aipref-training-preference-signal-00” (October 30, 2026): mailarchive.ietf.org
- PPC Land, Luis Rijo, “Google, Microsoft, OpenAI and Anthropic back one AI training opt-out” (October 31, 2026): ppc.land
- Google Search Central, “What can you actually put in a robots.txt?” (YouTube, October 21, 2026): youtube.com
All sources accessed November 1, 2026.
Last updated: November 1, 2026. To report an error, see our corrections page. Articles are drafted with AI assistance and reviewed and edited by an editor; see our editorial policy.
