Cloudflare Data Reveals AI Companies' Web Crawling and Referral Ratios

Cloudflare, which supports about 20% of internet traffic, released data in early April 2026 on how AI bots interact with websites. The data measures the 'crawl-to-refer' ratio, which compares the number of times bots access webpages to the number of user referrals sent back to those sites. This metric indicates the balance of value extracted from and returned to the web.

The data shows varying ratios among AI companies. Anthropic had a ratio of 8,800 to 1, meaning its bots crawled pages 8,800 times for every referral. OpenAI's ratio was 993 to 1. Microsoft, Google, and DuckDuckGo had more balanced ratios.

Historically, websites permitted search engines to crawl and index content for free in exchange for referral traffic that supported monetization.

Generative AI chatbots often provide direct answers to users, which reduces clicks to original sources. This change decreases traffic to websites while AI companies access content for model training. Heavy bot activity can also increase operational costs for website owners due to server demands.

Cloudflare's data provides a snapshot of these interactions based on its network observations. The company tracks bot crawls and referral patterns across its infrastructure.

Anthropic has previously stated that Cloudflare's methodology may not fully account for referral traffic from its features.

Anthropic did not respond to a request for comment on the latest data. Cloudflare announced a new marketplace for web content to facilitate compensated access. The data raises questions about incentives for sharing information online if referral traffic declines.

Website operators rely on such traffic for revenue. Future developments may include more tools to manage bot access and ensure fair exchanges.