Cloudflare pressures Google over AI crawling

The news: Publishers are considering opting out of Google Search as AI Overviews eat into traffic, Adweek reports. Content delivery network Cloudfare, which hosts approximately 20% of global websites, gave Google an ultimatum last week: Pay licensing fees or it will block Google’s AI scrapers and remove itself from Google Search.

  • Starting Sept. 15, Cloudfare’s free-tier users and new sign ups will automatically block “multi-purpose crawlers” from ad-supported pages unless the site owner opts out.
  • In practice, bots that gather content for both search indexing and AI training will be blocked by default—a change that could hit Google especially hard since it uses a single crawler for both purposes.
  • Cloudflare chief strategy officer Stephanie Cohen stated that the company “want[s] a technical solution that allows you to be discoverable without having to give your content away for free.”

Zooming out: The rise of Google’s AI Overviews and related offerings has ignited various critiques from publishers who claim the features have disrupted site performance.

  • Penske Media became the first major US publisher to sue Google over its AI Overviews feature in September 2025, alleging that Google used journalism without permission to train and display AI summaries—reducing referral traffic.
  • Media publishers have noted a drop in referral traffic—nearly 50% decline in some cases—attributed to Google’s AI tools. Similarweb data noted a 27% drop in organic referrals from search between July 2024 and May 2025. AI Overviews have also led to some publishers experiencing declines in referral traffic as high as 25%, per Digital Content Next.
  • MailOnline saw a 56% decline in clickthrough rates linked to AI Overviews, according to Search Engine Land.
  • Pew Research found that only 8% of Google users with searches triggering an AI Overview clicked on a link.
  • Ahrefs found that AI Overviews decreased CTRs by 34.5% in 2025.

Implications for publishers and marketers: Cloudflare’s move is a bet that the rise of AI search has weakened Google’s grip on publishers. As AI Overviews rely on publisher content while sending less traffic back, Cloudflare is testing whether publishers can use the threat of blocking mixed-use crawlers to negotiate from a stronger position.

But that leverage is far from guaranteed; opting out of Google Search remains a daunting prospect for publishers who still depend heavily on search visibility. Still, Cloudflare’s move gives publishers a stronger technical lever in negotiations over AI crawling. By locking out mixed-use crawlers, Cloudflare could pressure Google to sign licensing deals or separate search indexing from AI training and give publishers more granular control over how their content is used.

For marketers who place ads on publisher sites, stronger crawler controls could help preserve the value of those environments by limiting uncompensated AI scraping, but they may also alter how audiences find publisher content through search if Google gets blocked by default.

You've read 0 of 2 free articles this month.

Get more articles - create your free account today!