Four AI Giants Back a Shared robots.txt Training Opt-Out — Why Indie Bloggers Should Drop the Anti-Scraping Obsession

On October 30, 2026, Kevin Kelley, Head of Standards at Anthropic, submitted an individual Internet-Draft titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) to the Internet Engineering Task Force (IETF), defining a cross-provider AI training preference token in standard robots.txt files via User-Agent: __usage_ai-training. That same day, Google, Microsoft, OpenAI, and Anthropic took the rare step of simultaneously pledging support. On the surface, the three-year game of whack-a-mole — forcing webmasters to blacklist well over a hundred crawlers vendor by vendor — finally looks ready to end with a single configuration block.

Put the draft specification, the four providers' implementation rules, and third-party server telemetry side by side, however, and this compromise turns out to be far less generous than advertised. It governs only pre-training and post-training runs that modify model weights, leaving live retrieval-augmented generation (RAG and grounding) — the very mechanism siphoning away search clicks — completely untouched. Google even bundles the opt-out with Gemini answer citations in an all-or-nothing tradeoff. And against open robots.txt defiance rates of 15% to 30%, alongside commercial scraping brokers rotating residential proxy IPs, a plaintext robots.txt file barely qualifies as a screen door.

For independent developers and solo site owners, that reality points to a deeper strategic reckoning. The era of traditional desktop website traffic is largely over. Unless you operate a community like linux.do built on dense, real-time human conversation, guarding a handful of public static articles neither stops free-riding nor generates a cent in return. Rather than burning energy maintaining hundred-line crawler blocklists, indie builders are better off making their content radically open — friendly to search engines, friendly to AI crawlers, and friendly to human readers — while building real moats inside products that actually monetize. And if your site was never meant to make money in the first place, let it go without losing sleep over it.

One Line for Four Giants — What the New IETF Draft Actually Establishes

To understand why four rival AI labs suddenly closed ranks at the end of October 2026, look at how dysfunctional per-vendor robots.txt blocklists had already become.

Just one day before the joint announcement (October 29, 2026), independent researcher Kuldeep Paul published an empirical study auditing 5,701 valid robots.txt files across the Tranco Top 10,000 high-traffic domains against 197 known AI crawlers (45 officially documented plus 152 community-tracked). Combined with Suff Digital's web-wide HTTP Archive scan of 15.47 million websites — which found that 9.9% of all sites and 28.9% of the CrUX Top 1,000 block at least one AI crawler — three years of company-by-company blocking had suffered a systemic breakdown:

1. Half of all blockers run leaky blocklists. Among the 810 top domains that explicitly block OpenAI's GPTBot, a full 50.0% (405 sites) missed at least one other major LLM training crawler — 79.5% failed to block cohere-training-data-crawler, 75.3% missed MistralAI-Training, 37.0% missed Amazonbot, 35.1% missed Applebot-Extended, 34.1% missed Meta-ExternalAgent, 27.3% missed Google-Extended, 23.3% missed Bytespider, and 18.8% missed ClaudeBot. Across the entire sample, only 6.0% (344 sites) completely blocked all eight major training crawlers.
2. Two-thirds still carry deprecated zombie tokens. Out of 1,037 sites blocking three or more AI crawlers, 66.2% (687 sites) still list Anthropic's long-retired anthropic-ai or Claude-Web tokens. Worse, 51 large websites — including espn.com and abcnews.com — blocked only those dead names while leaving the active ClaudeBot wide open, rendering their Anthropic training opt-out completely ineffective.
3. Rule bloat and standards gridlock. Desperate to plug every hole, operators resorted to sprawling copy-paste lists peaking at 192 AI crawlers on a single site (brand-catcher.com). Meanwhile, after the IETF chartered the aipref working group in January 2025, its attachment draft draft-ietf-aipref-attach-05 stalled out. Because that proposal tried introducing a brand-new Content-Usage: train-ai=n directive line inside robots.txt — requiring grammar changes to existing RFC 9309 parsers worldwide while deadlocking over real-time RAG permissions — progress ground to a halt.

Against that working-group impasse, Anthropic's Kevin Kelley sidestepped the new-grammar debate altogether, submitting an individual draft valid through May 3, 2027 with synchronized backing from all four providers:

"This document describes how a service operator binds usage preferences (e.g., 'no AI training') to paths by writing ordinary robots.txt groups keyed by a per-purpose policy token in the User-Agent line. It introduces no new directive names and no new grammar: the path language, longest-match precedence, and group structure of robots.txt are not changed."

Under the specification, if a site operator wants to prohibit generative AI model training across the entire site while exempting /press-kit/ media assets, three lines in robots.txt suffice:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Behind those straightforward lines sit four hard technical boundaries every operator must understand:

• Double-underscore (__) collision avoidance. As Japanese SEO specialist Kenichi Suzuki highlighted in his technical breakdown, __usage_ai-training conforms strictly to the RFC 9309 character set, and its leading double underscore __ prevents overly permissive legacy parsers — such as implementations performing loose substring matching — from accidentally colliding with real crawler names. Unaware search crawlers simply treat the block as an unknown bot and skip it, guaranteeing zero impact on traditional search indexing.
• Zero inheritance from wildcard User-Agent: *. Section 4.2 of the draft establishes an explicit prohibition: a User-Agent: * rule group MUST NOT be interpreted as matching __usage_ai-training. Even if a site declares a blanket User-Agent: * with Disallow: /, omitting an explicit __usage_ai-training group leaves its training preference in the unknown state under this mechanism.
• Scoped strictly to generative parameter updates, with non-generative tasks exempt. Importing its definition directly from vocabulary draft draft-ietf-aipref-vocab-08, the token governs only modifying the learned parameters of generative AI models. Training models used solely for classification, ranking, or spam scoring remains completely exempt — even if those models generate text rationales alongside their scores.
• No crawl blocking, no retroactivity, and commercial contracts take precedence. __usage_ai-training never appears in HTTP request headers and does not stop crawlers from fetching pages locally. It applies only at the moment of retrieval, requiring neither deletion of previously ingested datasets nor rollbacks of already-trained model weights, and private commercial agreements between platforms and publishers override robots.txt declarations outright.

Four Giants with Divergent Agendas — Transition Configs and Bundling Traps

The immediate force pushing all four giants onto the same stage on the exact same day is regulatory pressure looming across Europe and the UK.

On one front, Article 53(1)(c) of the EU AI Act and its accompanying General-Purpose AI (GPAI) Code of Practice require model providers to honor cross-industry machine-readable robots.txt controls standardized through bodies like the IETF — alignment Google cited directly in its announcement. On the other front, the UK Government's March 18, 2026 parliamentary Report on Copyright and Artificial Intelligence disclosed that across more than 11,500 public consultation responses, 88% demanded mandatory prior licensing (opt-in) while a mere 3% backed machine-readable opt-outs, forcing the UK Government to abandon its proposed statutory TDM opt-out exception. Determined to show regulators that voluntary industry standards can fix fragmented opt-outs without harsher opt-in mandates, the four labs rolled out this shared token.

Read each company's fine print, however, and the exact same User-Agent: __usage_ai-training rule triggers sharply different consequences behind the scenes:

Nowhere is the divergence starker than in the opposite product incentives chosen by Microsoft and Google.

On the official Bing blog, Principal Product Manager Krishna Madhavan pledged that "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work" — splitting AI Training and page-level AI Grounding into two completely independent controls while introducing a targeted DisallowExtended tier. That middle tier lets a publisher's pages appear in first-party Bing AI, Copilot, and Microsoft 365 answers for citation exposure while barring Microsoft from reselling that content to third-party commercial customers through Web IQ or the Azure Foundry Grounding API. The sole operational catch applies to legacy sites using NoArchive or NoCache tags to block AI: they must migrate via the temporary transition panel opening in Bing Webmaster Tools on December 15, 2026 (valid through September 30, 2027) or update their tags before the March 1, 2027 deadline.

Google Search Advocate John Mueller, by contrast, unveiled a fallback mapping rule that functions as a calculated two-for-one bundle:

"When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended... Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."

As Search Engine Journal Senior News Editor Matt G. Southern criticized, because Google-Extended ties "training future Gemini model generations" and "live grounding citations in Gemini Apps" to the exact same lever, opting out with __usage_ai-training automatically forfeits answer citations in Gemini Apps. Yet if a publisher writes Google-Extended: Allow to preserve Gemini referral clicks, Google's specific-token precedence overrides __usage_ai-training and puts that content right back on the Gemini training menu. Meanwhile, Google Search's far higher-volume AI Overviews (2.5 billion monthly users) and AI Mode (over 1 billion monthly users) remain welded to the primary Googlebot crawler — and Kuldeep Paul's opt-out asymmetry study confirmed that among 667 top sites blocking Google-Extended, 99.85% (666 sites) dare not touch Googlebot.

Add in Meta — whose communications chief praised a common signal on social media as an "important step" while declining to commit to honoring it (having just attempted in October 2026 to revise its commercial terms to force media outlets using Facebook and Instagram business tools into authorizing external website scraping, postponing enforcement to November 20 only after joint protests from WAN-IFRA and global publisher associations) — plus holdouts including Apple, ByteDance, Perplexity, and Chinese LLM providers. If a site owner genuinely wants to block model training at the rule layer while preserving AI search citations today, legacy bot tokens cannot be removed during the 2026–2027 transition. The baseline configuration must stack the new shared token alongside existing Big Four training tokens and non-participating crawlers:

# 1. Cross-provider unified AI training opt-out signal (rolling out late 2026 to early 2027)
User-Agent: __usage_ai-training
Disallow: /

# 2. Retain individual Big Four training tokens during transition (leaving OAI-SearchBot / Claude-SearchBot open for search citations)
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: Google-Extended
Disallow: /

# 3. Other major AI training and data-collection crawlers not yet committed to the shared signal
User-Agent: CCBot
User-Agent: Bytespider
User-Agent: Meta-ExternalAgent
User-Agent: Applebot-Extended
User-Agent: Amazonbot
User-Agent: cohere-training-data-crawler
User-Agent: MistralAI-Training
Disallow: /

Running the Traffic and Security Math — Why a Handshake Deal Cannot Lock Down Content

Once you deploy that meticulous robots.txt file to production, is your original work actually protected from uncompensated extraction? Run the hard numbers on referral traffic and adversarial scraping, and the answer turns out to be unforgiving.

Start with the asymmetric traffic ledger — the exact imbalance that makes creators viscerally resent AI free-riding.

Across 12 months of global network telemetry from Cloudflare Radar and its crawler purpose breakdown, 80% of all AI bot requests worldwide feed model training, whereas just 18% support AI search indexing and 2% to 3.2% serve user-triggered fetches. To send back 1 real human click to an origin website, Anthropic crawls an average of 147,754.7 pages (hitting an astonishing peak of 286,930.1:1 in January 2025), OpenAI crawls 1,437.8 pages, Perplexity crawls 172.4 pages, Microsoft crawls 42.3 pages, and Google crawls 11.8 pages.

First-party publisher server logs analyzed by TollBit underscore that lopsided exchange even more starkly: the ratio of AI bot visits to human visits surged from 1:200 in Q1 2025 to 1:31 by Q4 2025 (reaching 1:33 across European publishers in H1 2026), yet all AI platforms combined deliver just 0.04% to 0.16% of total external referral traffic. Average AI referral click-through rate (CTR) plunged inside six months from 0.80% to 0.27% — nearly 96% below traditional Google organic search's 8.6% average CTR (and even Google AI Overviews register roughly 0.9% CTR, per Search Engine Roundtable).

Crucially, as BBC representative Chris Needham pointed out in IETF working group Issue #172, what actively drains clicks from original publishers is real-time RAG and answer synthesis — which __usage_ai-training leaves completely untouched. Engineers from Adobe and Mozilla similarly warned in Issue #246 and Issue #254 that slicing web pages into cached vector embeddings or fetching pages via AI agents under "user-triggered" exemptions creates "a black hole for preferences" that sidelines training opt-outs. And unlike standards built around compensation — such as RSL 1.0 with native payment tags, Europe's W3C TDMRep, or Cloudflare's HTTP 402 Pay Per Crawl and Content Signals Policy spanning over 3.8 million domains (blocking training crawlers at the network edge by default since September 15, 2026 for newly onboarded ad-supported sites) — __usage_ai-training is an unpaid binary switch that generates zero copyright revenue.

Now run the adversarial security ledger, where gentleman's agreements bind only polite actors:

• Up to 30% open non-compliance. TollBit's server log audits reveal that declared AI bots blatantly ignoring robots.txt Disallow directives climbed from 3.3% at the end of 2024 to 30% by late 2025 (and remained at 15% across European publishers in H1 2026).
• Stealth proxies and commercial corpus resellers. Back in August 2025, Cloudflare's security team forensically proved that when websites blocked PerplexityBot via robots.txt and firewall rules, the service pivoted to undeclared stealth crawlers spoofing macOS Chrome browsers and rotating residential proxy IPs to bypass blocks. TollBit's H1 2026 canary tests further exposed 14 to 15 commercial scraping platforms reselling text corpora to AI labs: by rotating 32 to 43 residential proxy IPs within a single request while spoofing Googlebot, they breached hard paywalls at 27 of 30 (90.0%) tested US and UK mainstream news outlets, reselling the harvested pages to model builders for up to USD 22.50 per 1,000 pages.
• Zero observability into backend training. Security and AI governance analyst P.K. Sharma laid bare the final blind spot in a briefing:

"A 'signal' that a provider will 'honor' sounds like a lock. The documents say otherwise. The vocabulary draft states that preferences are 'not a security mechanism'... A training preference attached to content that was fetched for another purpose leaves no such trace in a site's logs. Whether the training use of a page happened under one crawl or another is something only the operator can say."

Once Googlebot, Bingbot, or OAI-SearchBot legitimately fetches your page for search indexing, an independent site owner inspecting Nginx access logs has zero visibility into whether that HTML subsequently flowed into an internal post-training or synthetic-data pipeline.

The Post-PC Moat Divide — Who Needs an Iron Wall and Who Should Let Go

Having walked through the protocol clauses and telemetry data, I want to step outside spec sheets and share how I view this as an independent developer and blog operator.

From a creator's seat, refusing to let trillion-dollar companies free-ride on your work is an entirely justified instinct. When you spend days debugging an obscure failure, running real benchmarks, and writing a thorough technical post — only for frontier labs to feed it into commercial models without asking, paying, or returning even a handful of clicks — anyone would feel shortchanged. That frustration explains why every new anti-scraping proposal triggers a fresh round of robots.txt copy-pasting across webmaster forums.

Step back and look at the board dispassionately, though: even if we configure robots.txt without a single crack, blocking all AI training and even all AI citations, what does that actually change?

We have to face a structural reality that is already behind us: the era of the traditional desktop website is fundamentally over.

For two decades, independent webmasters built around a predictable loop: write posts, wait for search engines to rank them on page one, bring readers into desktop web pages, and monetize through AdSense display units or affiliate links. Regardless of whether you block AI crawlers today, that loop has shrunk irreversibly. User discovery has already migrated from opening ten blue links side by side to querying conversational interfaces, mobile feeds, and agentic workflows. Welding your blog's doors shut will not resurrect desktop search traffic from a bygone era — it will only erase you from the new information graph.

As I see it, whether you should bother fighting crawlers today comes down entirely to what kind of website you run:

The tiny fraction of sites that genuinely need ironclad anti-scraping defenses are forums and communities like linux.do or Reddit that concentrate high-density, real-time human interaction. In an era awash with synthetic AI slop, thousands of real developers posting daily, helping one another debug in real time, trading jokes, and sharing firsthand war stories form a living social fabric — the scarcest digital asset on the internet and those platforms' true competitive moat. Leave a community like that wide open to high-concurrency scraper sweeps, and bots will not only hollow out its core asset overnight, they will crush its origin database and bandwidth bills. Notice, too, that communities like these never naively rely on a polite line in robots.txt; they deploy aggressive Cloudflare interstitial challenges, behavioral WAF rules, authenticated sessions, and trust-level read gates — because they are defending a live human ecosystem with real economic value.

For the overwhelming majority of independent blogs, personal docs sites, and long-tail web properties, by contrast, a few dozen public static posts are simply not a moat you can preserve by hiding them from AI. When locking things down fails technically (stopping compliant lab bots while letting residential proxy brokers and single-crawl reuse slip right through) and delivers zero commercial upside, what needs updating is not robots.txt — it is our mindset.

Make Content Radically Open — Perfect the Experience and Monetize Real Products

Once you recognize that bot blocking is both porous and unprofitable, the smarter play for independent site owners is the exact opposite — let go of the anti-scraping obsession, make your content radically open, and build real products that are far easier to monetize.

Making content radically open does not mean neglecting craft. It means taking every ounce of energy previously burned maintaining crawler blocklists and channeling it into three forms of friendliness:

1. Search-engine friendly (SEO). Nail semantic HTML markup, clean URL hierarchies, accurate title and summary metadata, and fast initial page loads so traditional search engines can crawl and understand your pages without friction.
2. AI-crawler and answer-engine friendly (GEO). Tear down awkward bot blocks and write thesis-first, evidence-rich, cleanly structured articles with reproducible code blocks. When fellow engineers ask ChatGPT, Claude, Copilot, or Gemini to solve a real problem, your blog becomes the authoritative primary source models trust and cite — turning AI systems into a free distribution amplifier that filters for readers on your wavelength.
3. Human-reader friendly (UX). This is the core pillar too many site owners overlook. Strip away intrusive popups and cluttered ad banners, refine typography weights and contrast ratios, and offer smooth multilingual reading. Even if only one in ten people clicks through from search or an AI citation, a crisp, respectful reading experience makes them remember your name and taste immediately.

Once that clicks, the strategic fork ahead becomes refreshingly clear:

If your site carries commercial ambitions and you want to build real income in the AI era, stop hoping to collect tolls by fencing off static articles. Build products and services that monetize cleanly and cannot be replicated just by scraping a few web pages — SaaS utilities solving sharp engineering pain points, managed cloud offerings around open-source tools, deep workflow plugins, or high-value consulting. Even if you sit down someday to negotiate commercial partnerships or licensing terms, you need strong user experience, domain reputation, and reach before you have any leverage at the table. The more openly your public writing circulates, the wider and stronger the trust funnel feeding your commercial product becomes.

And if your independent blog was never meant to be a monetized product — just a personal corner of the web for logging engineering notes and working through ideas after hours — there is even less reason to wrestle with crawlers. Let it be. Whether your words end up indexed by a search engine or absorbed into a frontier model's parameters and retrieval store, as long as they eventually help a stranger solve a concrete problem, the post has already done its job.

References and Further Reading

1. Kevin Kelley (Anthropic) / IETF — Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00) and Official IETF Datatracker Status Page (2026-10-30)
2. John Mueller / Google Search Central Blog — Supporting a shared AI training control for websites and Google Common Crawlers Documentation (2026-10-30)
3. Krishna Madhavan / Microsoft Bing Blog — Your content, your choice: clearer preferences for training and grounding and Bing Webmaster Tools New AI Preferences Help (2026-10-30)
4. OpenAI Official Announcement — Making it easier for websites to express AI training preferences (2026-10-30)
5. Kevin Kelley / IETF ai-control Mailing List — Announcement of draft-kelley-aipref-training-preference-signal-00 (2026-10-30)
6. Kuldeep Paul — Who blocks AI crawlers in 2026: a robots.txt study of 10,000 top domains and The AI search opt-out asymmetry (2026-10-29)
7. Matt Suffoletto / Suff Digital — What Percentage of Websites Block AI Crawlers? (HTTP Archive 15.47M Sites) (2026-09-24)
8. João Tomé / Cloudflare Radar — The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals and Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives (2025-08)
9. TollBit — State of the Bots Report Series (Q4 2024 – H1 2026: The Bad Bots) (2026-07-15)
10. Matt G. Southern / Search Engine Journal — Google, Microsoft & OpenAI Plan To Honor Shared AI Training Opt-Out (2026-11-01) and Kenichi Suzuki — Batch Opt-Out for AI Training, as Google, OpenAI, and Other Major AI Providers Move to Support a Shared Signal (2026-11-02)
11. Parminder Kumar Sharma (P.K. Sharma) — OpenAI to Honor Common robots.txt Signal for AI Training Preferences: What the Post Says and Does Not Establish (2026-10-30)
12. UK Government — Report on Copyright and Artificial Intelligence (2026-03-18) and European Commission — EU AI Act Art. 53(1)(c) & GPAI Code of Practice