The data: 68% of top US brands don’t provide specific instructions for AI crawlers in their websites’ robots.txt files, per Clicks Dynasty. Brands that neglect robots.txt files in this way are vulnerable to leading AI search crawlers astray and surrendering proprietary information to large-language-model (LLM) training.

Clicks Dynasty's research looked at robots.txt files—which outline web access for visiting bots—for 56 major brands across seven industries: banking and payments, news and media, health and insurance, retail, restaurants, tech and telecom, and travel.

• No brands in the banking and payments and restaurants categories provide specific instructions for AI crawlers.
• 6 out of 8 brands in travel, tech and telecom, and health and insurance lack specific instructions, as do 5 out of 8 in retail.
• In contrast, 7 out 8 brands in the news and media category do provide specific instructions.

Why it matters: AI crawlers use robots.txt files as a directory for website access. While most non-media brands want search-focused crawlers to scrape their information, they’re providing no instructions for which content to prioritize and which to ignore. Without clear directions, AI search crawlers may scrape low-value content and surface that information in search results.

Publishers are more on top of their robots.txt files compared with non-media brands because AI crawlers pose a direct threat to their business models. But non-media brands are also affected by the degree of access that crawlers have to their content.

Non-media brands have reason to block AI crawlers, specifically those that scrape information for training purposes. Like publishers, commercial brands may offer content or AI experiences of their own that are meant to attract consumers to their websites, where they can acquire first-party data. But when AI crawlers can freely absorb this information, those efforts go to waste, without any compensation for the brand.

Recommendations for marketers: Work hand-in-hand with engineering teams to make robots.txt files a core piece of AI search strategies.

• Analyze web traffic to determine which search crawlers are most commonly visiting websites and should therefore be directed by name in robots.txt files.
• Clean up website content by identifying and removing redundant or out-of-date information to prevent AI crawlers from scraping it.
• Strengthen robots.txt instructions by ensuring that the content it directs crawlers to is AI-readable and supplemented by customer reviews and other user-generated content to boost AI search appearances.

This content is part of EMARKETER’s subscription Briefings, where we pair daily updates with data and analysis from forecasts and research reports. Our Briefings prepare you to start your day informed, to provide critical insights in an important meeting, and to understand the context of what’s happening in your industry. Non-clients can click here to get a demo of our full platform and coverage.