Reflections
2026.11.03·22 min read
Four AI Giants Back a Shared robots.txt Training Opt-Out — Why Indie Bloggers Should Drop the Anti-Scraping Obsession
On October 30, 2026, OpenAI, Google, Microsoft, and Anthropic jointly backed __usage_ai-training, a shared robots.txt token for opting out of generative AI model training — yet the draft leaves live RAG traffic siphoning untouched and cannot stop 30% rogue bot defiance or residential proxy scrapers. Now that the desktop search traffic era has faded, unless you run a high-density human community like linux.do, indie site owners are far better off dropping sprawling crawler blocklists, opening content to search, AI engines, and human readers alike, and building defensible moats in monetizable products.
AFeng