technology
Cloudflare to Block Mixed-Use Web Crawlers Serving AI Companies
Cloudflare will automatically block web crawlers that both index sites for search engines and train AI models. The change gives website owners more control over how their content is used by AI systems.
EngadgetAudio version
Tap play to generate a narrated version.
Developing·Limited corroboration so far. This page will refresh as more sources emerge.
Cloudflare announced it will automatically block mixed-use web crawlers that index websites for search engines while also serving as AI agents and trainers. The company previously gave customers an optional tool to stop AI scraping. It is now making the block the default setting.
"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Cloudflare's CEO and co-founder said in a statement.
Mixed-use crawlers that do not let site owners choose whether their content trains AI models will be blocked on ad-supported pages by default. Free-account users will receive the new defaults unless they opt out before the September 15 deadline. Cloudflare is also updating its Pay Per Crawl feature, now called Pay Per Use.
Site owners will be paid when their content appears in AI chatbot answers rather than when a page is crawled. com for the updated system.
Google's main crawler, Googlebot, both indexes sites for search and collects data to train Gemini and power AI features. Google offers a separate crawler called Google-Extended for traditional search results, but publishers cannot opt into AI Mode results without also allowing model training. Cloudflare's policy aims to require companies with mixed-use crawlers to separate those functions.
