OpenAI, Google, Microsoft, and Anthropic just shipped a unified robots.txt signal for AI training opt-out: `User-Agent: __usage_ai-training` with `Disallow: /` blocks foundational model training across all four providers.
The mechanism is dead simple—standard robots.txt syntax, zero new directives. Parsers that don't recognize the token just ignore it. You can carve out exceptions with Allow rules (e.g., `/press-kit/` stays crawlable).
Backed by an IETF individual draft (Kevin Kelley, Anthropic) that defines AI training as "using an asset to modify learned parameters of a generative model." Non-generative inference doesn't count, even if the model can generate.
Company-specific tokens (GPTBot, Google-Extended, ClaudeBot) still work. If you set the shared token to Disallow but don't mention Google-Extended, Google treats it as if you disallowed Google-Extended. Explicit Allow for Google-Extended keeps you eligible for Gemini grounding.
Microsoft is adding page-level AI Grounding preferences: Allow, Disallow, or DisallowExtended (blocks third-party grounding, allows Copilot/M365). Page-level overrides robots.txt. Old NoArchive/NoCache tags stop working March 1, 2027—you must migrate to the new preferences by then.
Rollout timeline:
• Anthropic: already live
• OpenAI: coming months
• Google: 2027
• Microsoft: February 1, 2027 (hard cutover March 1)
• Meta: vague "looking forward to working on standards" non-commitment
Critical nuance: this is a *usage preference*, not a crawl block. It doesn't stop fetching—just tells the provider not to use fetched content for training. Traditional search indexing is unaffected (Bing results, Google Search ranking stay the same). The preference only applies to content crawled *after* the robots.txt is updated—doesn't retroactively remove data from existing training sets.
Voluntary compliance. No enforcement mechanism. Other AI labs haven't signed on. If you're already blocking named crawlers, keep those rules—this just adds a broadcast channel to hit four providers at once.
The mechanism is dead simple—standard robots.txt syntax, zero new directives. Parsers that don't recognize the token just ignore it. You can carve out exceptions with Allow rules (e.g., `/press-kit/` stays crawlable).
Backed by an IETF individual draft (Kevin Kelley, Anthropic) that defines AI training as "using an asset to modify learned parameters of a generative model." Non-generative inference doesn't count, even if the model can generate.
Company-specific tokens (GPTBot, Google-Extended, ClaudeBot) still work. If you set the shared token to Disallow but don't mention Google-Extended, Google treats it as if you disallowed Google-Extended. Explicit Allow for Google-Extended keeps you eligible for Gemini grounding.
Microsoft is adding page-level AI Grounding preferences: Allow, Disallow, or DisallowExtended (blocks third-party grounding, allows Copilot/M365). Page-level overrides robots.txt. Old NoArchive/NoCache tags stop working March 1, 2027—you must migrate to the new preferences by then.
Rollout timeline:
• Anthropic: already live
• OpenAI: coming months
• Google: 2027
• Microsoft: February 1, 2027 (hard cutover March 1)
• Meta: vague "looking forward to working on standards" non-commitment
Critical nuance: this is a *usage preference*, not a crawl block. It doesn't stop fetching—just tells the provider not to use fetched content for training. Traditional search indexing is unaffected (Bing results, Google Search ranking stay the same). The preference only applies to content crawled *after* the robots.txt is updated—doesn't retroactively remove data from existing training sets.
Voluntary compliance. No enforcement mechanism. Other AI labs haven't signed on. If you're already blocking named crawlers, keep those rules—this just adds a broadcast channel to hit four providers at once.