Releasing czlib and zstd Go bindings

Datadog released Go bindings for two C compression libraries, czlib and zstd, to improve compression performance in production data pipelines. czlib addresses the standard library’s slower pure-Go zlib implementation, while zstd offers faster decompression, competitive compression, and modern features such as dictionaries and compression levels. The post concludes that performance depends heavily on payload size and interface, so applications should benchmark representative data.

czlib: Faster zlib-Compatible Compression

• czlib began as a fork of Vitess’s cgzip package.
• Datadog modified it to use zlib wrapping instead of gzip headers, matching the format used by its primary data pipeline.
• It provides:
  • Non-streaming compression and decompression
  • Streaming interfaces
  • Batch-oriented interfaces
• The authors recommend benchmarking with realistic messages using:
  • PAYLOAD=path_to_message go test -run=NONE -bench .

Benchmark Results

• For a 2 KB plaintext message:
  • czlib compression: 44.42 MB/s
  • Streaming czlib compression: 34.11 MB/s
  • Standard compress/zlib: 9.27 MB/s
  • czlib decompression: 255.62 MB/s
  • Standard-library decompression: 66.72 MB/s
• For a 1.7 MB message:
  • czlib and standard zlib had similar compression speeds, around 24 MB/s.
  • czlib decompression reached roughly 257 MB/s.
  • Standard-library decompression reached about 121 MB/s.
• The results show that czlib’s advantage is especially significant for smaller messages and decompression workloads.

zstd: A Modern Alternative

• zstd, or Zstandard, was developed by Yann Collet, who also created lz4.
• At the time of publication, its format had recently been finalized and version 1.0 was pending.
• Compared with zlib at compression level 6, zstd:
  • Compresses slightly faster
  • Produces a slightly better compression ratio
  • Decompresses substantially faster
• Its features include:
  • Streaming compression
  • Configurable compression levels
  • Precomputed dictionaries
  • Fixed-length batch compression similar to the lz4 interface

Go Binding Design

• The zstd binding intentionally mirrors the zlib API.
• It is designed to be a functional drop-in replacement, aside from a few zstd functions that do not return errors.
• Dictionary-building support is available in the upstream repositories.
• The binding exposes both advanced streaming functionality and efficient batch compression.

Use czlib when compatibility with zlib is required but the standard implementation is too slow; consider zstd for new systems that benefit from faster decompression and dictionary support. Always benchmark with data and interfaces representative of the target workload.