Shared Dictionaries: compression that keeps up with the agentic web
Shared dictionaries address a growing web-performance problem: pages are getting heavier, being rebuilt more frequently, and fetched repeatedly by agents. Instead of retransmitting entire assets after every deployment, servers can compress new versions against files already cached by the browser and send only the differences. The approach could dramatically reduce bandwidth and CPU use, though adoption depends on browser support, security safeguards, and complex server-side implementation. ## The Problem: More Shipping Means Less Caching - Web pages have become 6–9% heavier annually due to frameworks, interactivity, and media. - Agentic crawlers and other automated tools increasingly request full pages; they accounted for nearly 10% of Cloudflare requests in March 2026, up about 60% year over year. - AI-assisted development leads to more frequent deployments and experiments. - Small code changes can cause bundlers to re-chunk assets and generate new filenames, forcing clients to download entire bundles again. - Conventional compression reduces the size of each response but cannot exploit the fact that the client already has most of the previous version. - Frequent deployments therefore create substantial redundant bandwidth and CPU usage. ## How Shared Dictionaries Work - A compression dictionary is shared knowledge between the client and server. - The server compresses a new response using content the client already possesses as a reference. - The client uses that same reference to reconstruct the complete file. - Brotli includes a built-in dictionary of common web patterns, while Zstandard can generate custom dictionaries from representative content. - Gzip lacks a prebuilt or custom dictionary and discovers patterns only during compression. ## Delta Compression for Versioned Assets - Shared dictionaries use the previously cached resource as the compression dictionary. - The initial response includes a `Use-As-Dictionary` header, telling the browser to retain the resource for future compression. - On a later request, the browser sends an `Available-Dictionary` header identifying what it has cached. - The server sends only the differences between the old and new versions. - A 500 KB JavaScript bundle with a one-line change could become only a few kilobytes on the wire. - The technique is especially useful for versioned JavaScript bundles, CSS, framework updates, and other incrementally changing assets. - Each release can use the immediately preceding version as its dictionary, allowing savings to continue across many deployments. - Custom and dynamic dictionaries for non-static content remain an area for future development. ## Lessons from SDCH - Google introduced Shared Dictionary Compression for HTTP (SDCH) in Chrome in 2008. - Although early adopters reported significant performance improvements, SDCH had serious security and architectural issues. - Compression side-channel attacks such as CRIME and BREACH demonstrated that attackers could infer secrets by injecting content and observing compressed response sizes. - SDCH also conflicted with the Same-Origin Policy and CORS because of its cross-origin dictionary model. - Its specification did not adequately define interactions with APIs such as the Cache API. - Chrome removed SDCH in 2017 after adoption failed to materialize. ## The Modern Standard and Remaining Challenges - RFC 9842, Compression Dictionary Transport, addresses major SDCH shortcomings. - Dictionaries are restricted to responses from the same origin, reducing conditions that enabled earlier side-channel attacks. - Chrome and Edge support the standard, while Firefox is working toward support. - Implementing the system requires servers to: - Generate or select dictionaries. - Advertise them with correct headers. - Detect `Available-Dictionary` requests. - Delta-compress responses dynamically. - Fall back cleanly for clients without dictionary support. - Cache behavior becomes more complicated because responses vary by both content encoding and dictionary availability. Cloudflare plans to offer a beta of its shared compression dictionary support on April 30, 2026. The technology is promising for frequently deployed applications and agent-heavy traffic, but broad benefits will depend on cross-browser adoption and careful handling of security and caching complexity.
Read original(opens in new tab)