The news: Cloudflare unveiled new controls for publishers to help them defend their websites against mixed-use crawling, where one crawler scrapes content for both a search index and AI training, per a press release.

Three independent controls replace a single “Block AI Bots” switch, each corresponding to a different type of crawling: search, AI training, and AI agents.

“We have to set defaults on what makes sense across the broader ecosystem, and [these settings are] what we have found surveying tens of thousands of content creators that are ad- or subscription-supported,” Matthew Prince, co-founder and CEO of Cloudflare, told EMARKETER.

The content delivery network (CDN) provider also introduced an “Accountable” designation for crawler operators that meet or have committed to meet certain criteria for transparent crawling. Google, Microsoft, and Apple now carry the label, joining OpenAI, Anthropic, and other operators.

When asked how Cloudflare would respond if any Accountable players went against their commitments, Prince said the company would simply block them. Cloudflare previously threatened to block Google over mixed-use crawling.

Why this matters: Publishers have mostly had to accept crawlers scraping their sites for AI training purposes if they wanted that content to appear in search results. Mixed-use crawlers make up the largest category of verified crawler traffic (36.6%) on Cloudflare’s network, per the company.

With the ability to control which crawlers have access to their sites, publishers gain leverage to force licensing deals with operators that want to train AI models on their content. Google recently launched a pilot program to compensate publishers when their content is used in its AI platforms, such as AI Overviews and AI Mode.

Recommendations for publishers: Update crawler controls as soon as possible to set the stage for licensing deals.

• Enable search crawling and disable AI training and agent crawling—these settings are optimal for protecting against AI training while preserving search visibility.
• Refer to Cloudflare’s bot accountability list to determine which crawler operators are the best fit for potential licensing deals.

Stay tuned for an update from Cloudflare around AI summaries given how much they are accelerating the reduction in click-through rates (CTRs) to publisher websites; AI Overviews cut CTRs for the top-ranking search result page by 58%, per Ahrefs.

“[The mixed-use update] is just one stage in what is a larger war,” Prince said.

This content is part of EMARKETER’s subscription Briefings, where we pair daily updates with data and analysis from forecasts and research reports. Our Briefings prepare you to start your day informed, to provide critical insights in an important meeting, and to understand the context of what’s happening in your industry. Non-clients can click here to get a demo of our full platform and coverage.