Why we-re rethinking cache for the AI era (opens in new tab)
AI traffic is fundamentally changing how CDNs should think about caching. Unlike human visitors, AI crawlers make broad, high-volume, often sequential requests for long-tail content, creating low reuse and substantial cache churn. Cloudflare argues that traditional LRU-based caching and techniques such as prefetching are increasingly poorly suited to this traffic, forcing operators to rethink cache design if they want to support AI access without harming human performance. ## Why AI Traffic Is Different - Automated traffic accounts for 32% of Cloudflare’s network traffic, including crawlers, scrapers, and AI assistants. - AI agents often: - Send many requests in parallel. - Scan large portions of a website sequentially. - Request rarely visited or loosely related pages. - Fetch documentation, images, and articles from many sources. - AI crawlers represent approximately 80% of self-identified AI bot traffic. - Most single-purpose AI bot traffic is associated with model training, with search-related crawling a distant second. ## The Three Defining Characteristics of AI Crawlers - **High unique URL ratio:** More than 90% of pages observed in large-scale Common Crawl datasets are unique by content. - **Content diversity:** Different crawlers target different materials, including source code, technical documentation, media, and blog posts. - **Crawling inefficiency:** Many requests lead to 404 errors or redirects because of poor URL handling. - AI crawlers generally lack browser-side caching and shared session behavior, so independent crawler instances may repeatedly appear as new visitors. - They can also repeatedly revisit content while iteratively refining search results, but each iteration still tends to fetch mostly new pages. ## How AI Crawling Disrupts Traditional Caches - Conventional CDN caching keeps frequently requested content available near users and evicts less recently used objects when storage fills. - Cloudflare uses an LRU (least recently used) policy, but broad AI scans introduce large numbers of low-reuse objects. - These objects can evict content that human visitors are more likely to request. - AI-driven long-tail access increases cache misses and sends more requests back to origin servers. - Cache speculation and prefetching become less effective because crawler access patterns are difficult to predict. - Higher miss rates can cause: - Slower responses. - Increased origin-server load. - Greater egress costs. - Reduced cache hit rates for human traffic. ## Implications for Website Operators - Operators face a tradeoff between optimizing infrastructure for human visitors and accommodating AI crawlers. - Some organizations may want to encourage AI access: - Developers may want documentation represented in AI models. - E-commerce companies may want product information included in LLM search results. - Publishers may seek compensation through systems such as pay-per-crawl. - The challenge is supporting useful AI traffic without allowing it to degrade the cache performance experienced by human users. Cloudflare’s analysis, conducted with ETH Zurich researchers, suggests that CDN caching strategies need to evolve beyond traditional assumptions about popularity and reuse. Cache systems designed specifically for AI-era traffic may need to isolate crawler workloads or otherwise prevent broad, low-reuse scans from displacing content valuable to human users.
