Publishers Get a New Default Setting Starting September 15

Cloudflare just gave the AI industry a hard deadline. Beginning September 15, 2026, the company's default settings will automatically block "mixed-use" web crawlers from any page that runs ads. Mixed-use crawlers are the ones that blend three different jobs into a single bot: powering traditional search results, feeding AI agents, and pulling training data for AI models.

Under the new defaults, any crawler doing all three gets shut out of ad-supported pages unless the site owner manually changes the setting. Cloudflare says the change rolls out across new customer accounts, any new site launched by an existing customer, and every free-tier account already on the platform.

The practical effect is significant. AI companies that want to keep crawling for legitimate search purposes will need to separate that function from the crawlers they use for agentic tools and model training. Blurring the two together is no longer the safe default.

Why Cloudflare Is Targeting "Mixed-Use" Bots Specifically

Cloudflare's reasoning centers on fairness. The company argues most website owners are fine with being found through search and even AI tools, but they don't want their content harvested for free in the process. Mixed-use crawlers make that distinction impossible to enforce, since a single bot can be indexing a page for search while simultaneously feeding that same content into a training pipeline.

That's the gap this policy is designed to close: give site owners a way to allow search discovery without also handing over free training data by default.

The Google-Sized Elephant in the Room

Cloudflare didn't name Google directly, but the reference is hard to miss. The company points to what it calls the "world's largest search engine" as having roughly twice the content access of other AI players, because staying visible in that search engine's results effectively requires also being available to its AI systems.

Google has disputed that framing before. The company points to Google-Extended, a bot that lets site owners opt out of having their content used in Gemini Apps and the Vertex API without losing their spot in Search. That said, Google's core crawler, Googlebot, still crawls for Search itself, and that includes AI-driven features like AI Overviews and AI Mode. So the opt-out only goes so far.

From Charging Per Crawl to Charging Per Use

This isn't Cloudflare's first move to give publishers leverage against AI scraping. The company previously rolled out tools aimed at blocking unwanted AI bots outright, followed by a marketplace called Pay Per Crawl that let sites charge AI companies simply for accessing their pages.

That marketplace is now evolving into something broader: Pay Per Use. Instead of publishers getting paid every time a bot fetches a page, they'd get paid when their content actually generates value for the AI company using it — a meaningfully different, and potentially more lucrative, model.

The Bandwidth Waste Behind the Shift

Part of the motivation is practical. Cloudflare's own data shows that more than half of all AI crawler traffic is spent re-fetching pages that haven't even changed. That's wasted bandwidth and compute on both sides, and it's part of the case Cloudflare is making for why the crawling model needs to change.

Who's Already Testing Pay Per Use

Cloudflare is launching Pay Per Use with two initial partners: Ceramic.ai and You.com. Publishers who opt in get paid when their content shows up in Ceramic's AI search results, or when You.com pulls from their premium content. Cloudflare says other AI companies will be able to adapt the model to fit how they operate.

The Bigger Picture

Cloudflare's CEO and co-founder, Matthew Prince, framed the move as a response to a shift already underway: non-human traffic now makes up the majority of activity on the internet, a milestone that arrived faster than expected — it wasn't projected to happen until next year. Prince said Cloudflare wants to "go further and act faster" so the ecosystem around AI and web content can stay sustainable.

The stated goal isn't to lock AI companies out. It's to push mixed-use bots toward transparency — separating search, agent, and training functions — while giving publishers real visibility and, increasingly, real compensation for how their content gets used.