The news: Cloudflare, which serves 20% of the web and 35% of the Fortune 500, launched beta tests of Pay per Crawl, a private marketplace that lets websites charge AI companies for scraping content. Conde Nast, TIME, AP, Adweek, and Fortune are among the marketplace’s early adopters, per TechCrunch.
It’s a strategy other content delivery networks (CDNs) will likely follow that could signal the end of AI’s unchecked scraping—while offering web publishers a new revenue stream beyond dwindling search referrals.
Free-for-all AI scraping is ending: New Cloudflare websites will block all AI crawlers by default, and publishers will be able to control who can crawl their websites. Publishers can block AI bots, give them free access, or set microtransaction rates per crawl.
Based on Cloudflare's crawler data, those fees could add up. OpenAI scrapes a website 1,700 times per referral. Anthropic scrapes websites a staggering 73,000 per referral. Google, by contrast, scrapes websites 14 times per referral.
Publishers set their pricing in Cloudflare’s marketplace, which will serve as the intermediary for transactions. Cloudflare is considering issuing its own stablecoin to facilitate the microtransactions.
Caveats: Pay per Crawl relies on AI companies choosing to participate to access websites for a fee—many may bypass or dispute the model. Enforcement outside Cloudflare’s network of websites could depend on how effectively other CDNs follow suit with similar services.
Key takeaway: Though still in beta, Pay per Crawl could give websites a new layer of protection—and a path to profit—if AI companies agree to pay for content they've long used without compensation. If AI wants to keep reading the internet, it may finally have to pay the bill.
This content is part of EMARKETER’s subscription Briefings, where we pair daily updates with data and analysis from forecasts and research reports. Our Briefings prepare you to start your day informed, to provide critical insights in an important meeting, and to understand the context of what’s happening in your industry. Non-clients can click here to get a demo of our full platform and coverage.