Cloudflare to Block Mixed AI Crawlers from Ad-Supported Pages Starting September 2026

Cloudflare announced that starting September 15, 2026, its default settings will block mixed-use web crawlers from any pages that host ads. The deadline requires AI companies to separate web crawlers used for search from those used for AI training and agents.

The new default blocking settings apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers.

TechCrunch reported that the move targets crawlers that blend search, agent use, and training. Cloudflare co-founder and CEO Matthew Prince stated that the majority of traffic on the Internet is now non-human. He said the company must act faster so that a sustainable ecosystem can emerge.

Prince added that Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities. He expressed hope that the default changes will encourage mixed-use crawlers to separate search from agent use and training. Cloudflare data showed that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.

The company launched the Pay Per Crawl marketplace that lets websites charge AI bots for scraping and is evolving it into Pay Per Use. Pay Per Use will allow publishers to charge AI companies when their content creates value. Cloudflare is initially working with partners Ceramic.ai and You.com on the model.

Bots surpassed human traffic online for the first time, earlier than the previously expected timeline of next year.