Using Datadog APM to improve the performance of Homebrew
Andrew Robert McBurney describes using Datadog APM to diagnose and optimize Homebrew’s slow `brew linkage` command. Instrumentation identified `LinkageChecker#check_dylibs` as the main bottleneck, and replacing repeated dynamic-library processing with persistent caching reduced execution time from 11.5 seconds to 182 milliseconds for 106 packages. A later implementation used Ruby’s built-in PStore instead of SQLite3 to avoid an additional gem dependency. ## Finding the Bottleneck with APM - Homebrew is widely used at Datadog, so improving its performance provides broad benefits. - The `brew linkage` command checks the library links of installed formulas and can identify when a reinstall is needed. - The target was to scan roughly 50 packages, including large packages such as Boost, in under five seconds. - The author instrumented Homebrew with Datadog’s Ruby `ddtrace` gem. - Flame graphs showed that most execution time was spent in `LinkageChecker#check_dylibs`. ## Why Multithreading Was Not Effective - The author tested Ruby threads as a way to process libraries concurrently. - Ruby’s Global Interpreter Lock limited the achievable parallelism. - Threading failed to meet the required performance target, so a different approach was needed. ## SQLite3-Based Caching - The expensive library-processing results were stored in an on-disk SQLite database. - A `linkage` table recorded: - Formula names and library paths - Linkage categories such as `system_dylibs`, `broken_dylibs`, `undeclared_deps`, and `brewed_dylibs` - Optional labels for selected linkage types - A uniqueness constraint on `(name, path, type, label)` prevented duplicate cache entries. - Homebrew could insert and retrieve linkage data using SQL queries. ## Performance Improvements - Without caching, processing 106 packages took 11.5 seconds. - Boost alone required about 1.01 seconds for dynamic-library checks. - With caching enabled: - The full command completed in 182 milliseconds. - Boost’s check took approximately 1.38 milliseconds. - The cached implementation significantly exceeded the original five-second performance requirement. ## Moving to PStore - After submitting the SQLite3 implementation for review, Homebrew maintainers recommended Ruby’s PStore. - PStore provides file-based persistence built around Ruby’s `Hash` data structure. - Its main advantage is avoiding a third-party SQLite3 gem dependency while preserving the benefits of caching. The central lesson is that profiling should guide optimization: rather than adding ineffective threading, the author located the true bottleneck and achieved a dramatic speedup through persistent caching. For similar command-line performance problems, instrument the complete execution path first, then choose the simplest cache or storage mechanism that satisfies both speed and dependency constraints.
Read original(opens in new tab)