Google is confronting a growing revolt from publishers, regulators, and infrastructure providers over its practice of using a single web crawler to serve both traditional search and its AI products. Critics argue this approach gives the company an unfair advantage while draining the economic life from the open web.
The line between content collected for search and content used to generate AI answers has blurred, with the same bot now feeding both Google Search and its Gemini AI model. Google's crawler sees 3.2 times more of the web than OpenAI's and 4.8 times more than Microsoft Corp.'s, according to Cloudflare CEO Matthew Prince, whose firm acts as a traffic gatekeeper for roughly a fifth of the web.
The bundling creates a dilemma for publishers. Blocking AI training means losing the search visibility they depend on for revenue. Cloudflare CEO Matthew Prince noted that humans will be a rounding error on the internet versus what their agents and their bots are doing. AI agents now generate more than 57 percent of web traffic in 2026 compared to about 42 percent for humans, according to Cloudflare data, marking the first time machine activity has surpassed human activity online.
Publishers Feel the Squeeze
The consequences are already visible in declining referral traffic. Media group Reach has reported a 50 percent drop in Google referral traffic as AI Overviews reduce the need for users to click through to original sources. Broader data supports a similar trend, with less than one-third of Google searches now resulting in a click to an external website, according to SparkToro analysis published in June. Small publishers have lost as much as 60 percent of search referral traffic over two years, per Chartbeat data.
These shifts have placed significant pressure on the open web ecosystem, where publishers rely on consistent search visibility to sustain their operations.
Resistance Builds
Two major forces are now pressing Google to adjust its approach. In June, the UK's Competition and Markets Authority ordered Google to let publishers opt out of AI features while remaining in search results. The BBC described this as a world first in regulatory action. Google confirmed it is testing the opt-out globally once UK trials conclude.
At the same time, Prince's Cloudflare issued an ultimatum. Starting September 15, the firm will block mixed-purpose crawlers by default on ad-supported pages, potentially cutting Google off from millions of websites. Internal Google slides from 2024, disclosed in a US antitrust case, revealed the company had considered offering an opt-out before launching AI Overviews but chose not to because it was evolving into a space for monetisation.
Google still controls roughly 90 percent of the search market. Whether the company will fully separate its crawlers rather than simply asking publishers to trust a toggle remains the central unresolved question.