Meta whiplashes brands on AI scraping, fueling urgency to establish crawler protections

The news: Meta quietly unveiled—and then retracted—a privacy policy update that would have allowed it to scrape off-platform content associated with any Meta business account to train its AI system, per PCMag.

  • The initial scraping change permitted its crawlers to visit any webpage that a Meta business account linked to, notably including content external to Meta’s ecosystem, per Nieman Lab. There was no opt-out mechanism.
  • As the media world reacted negatively to the change, Meta retracted the update, specifying that crawlers would only scrape information to power advertising and business products. There is still no opt-out mechanism, and legal experts voiced concerns that this ambiguity could include training purposes, per Nieman Lab.

Why it matters: Dependence on tech platforms for public visibility makes brands vulnerable to the whims of how those platforms use their content.

There is no standard for how tech platforms scrape content. Google lets brands opt out of AI training, but this is a recent decision that is only legally enforceable in the UK. And while firms like Cloudflare offer some level of control for website owners, these solutions rely on tech platforms’ compliance.

It’s largely up to brands to implement their own defenses against AI crawlers, yet many remain unprepared.

Recommendations for brands: Defend against invasive crawlers by fortifying websites, and stay vigilant for privacy policy updates.

  • Treat robots.txt files as the first line of defense. Work with engineering teams to develop clear instructions for crawlers, outlining both what they can and cannot scrape.
  • Build user experiences that AI search platforms cannot provide, such as immersive product detail pages or engaging news feeds. These will give consumers more reasons to visit owned web properties.
  • Deploy AI agents to monitor privacy policies for all tech platforms and deliver weekly reports outlining updates to how they access, scrape, and use brand content.

This content is part of EMARKETER’s subscription Briefings, where we pair daily updates with data and analysis from forecasts and research reports. Our Briefings prepare you to start your day informed, to provide critical insights in an important meeting, and to understand the context of what’s happening in your industry. Non-clients can click here to get a demo of our full platform and coverage.

You've read 0 of 2 free articles this month.

Get more articles - create your free account today!