The desire to block scraping, especially as AI models increasingly rely on media owner's content, is understandable. However, “Stop the Scrape” isn’t a silver bullet approach, and more often than not does more harm than good.
Blocking doesn’t stop the biggest players. Both Google and Cloudflare have publicly stated that their scraping restrictions don’t apply to Google. And with the DOJ’s remedy requiring Google to open-source its search index, that content will remain accessible for research and model training.
AI is now built into the browser. With major Answer Engines embedding AI directly into web browsers, nearly all of which are Chromium-based, it’s nearly impossible to distinguish between a human visitor and an automated agent. Trying to block only agents could inadvertently block real users.
Blocking doesn’t restore traffic. Many AI models rely on non-copyrighted sources like Wikipedia or public domain books. Even if your site isn’t scraped, users will still get an answer, just one that cannot be linked to your additional insights or expertise. The result? Continued traffic decline, not recovery.
Exclusion will backfire. As search behavior shifts toward AI-powered Answers, visibility becomes critical. To be discoverable, content must be accessible to, and interpreted by, these engines. Blocking scraping risks invisibility.
It’s not about toggles. Cloudflare is just one player in a fragmented CDN market. The real issue isn’t whether there’s a toggle, it’s whether media owners are properly compensated when their content powers AI-generated answers.
The Atlantic: AI and the Collapse of the Open Web
New York Times Lawsuit Against OpenAI & Microsoft
OpenAI and the Risk of Copyright Infringement
Google’s AI Overviews and Traffic Implications for Publishers