How we reduced the size of our Agent Go binaries by up to 77%
The Datadog Agent’s Linux artifact grew from 428 MiB in version 7.16.0 to 1.22 GiB in 7.60.0, creating problems for serverless, IoT, and containerized environments. Rather than remove features, Datadog reduced Go binary sizes by up to 77% between versions 7.60.0 and 7.68.0. The effort combined dependency analysis, targeted code refactoring, and renewed use of Go linker optimizations. ## Why the Agent Became So Large - The Agent supports many operating systems, architectures, distributions, and deployment environments. - Its codebase contains hundreds of dependencies, including cloud SDKs, container runtimes, and security tools. - Build tags and dependency injection determine which features are included in each binary. - The compressed Linux amd64 Debian package grew from 126 MiB to 265 MiB. - Its uncompressed size increased from 428 MiB to 1,248 MiB—a 192% increase over five years. - Go binaries represented a substantial portion of that growth and became the primary optimization target. ## How Go Selects Dependencies - Go compiles required packages individually before the linker combines them into a binary. - Files are included only when they: - Are not test files ending in `_test.go` - Match the current operating system, architecture, and build tags - Satisfy other constraints such as CGO settings, compiler version, or architecture features - Starting from the main package, Go transitively includes imported packages and the runtime required by every Go binary. - Unnecessary dependencies can be excluded by: - Adding a build tag to the file that imports them - Moving dependency-using symbols into a separate package imported only by relevant binaries ## Analyzing Imports and Dependencies - `go list` reveals all packages used for a specific OS, architecture, and set of build tags. - `goda` generates dependency graphs, including indirect imports. - `goda` can also show only the paths leading to a particular target package using its `reach` function. - These tools account for `GOOS`, `GOARCH`, and build constraints, making them useful for examining platform-specific builds. ## Why Package Lists Are Not Enough - A package’s presence does not directly indicate its binary size impact. - The linker removes symbols that are not reachable from the program’s entry points. - The same package can therefore contribute different amounts of code depending on how it is used. - Importing a package can still have significant side effects: - `init` functions execute. - Global variables are initialized. - These behaviors may force otherwise unnecessary symbols to remain in the binary. - Certain uses of reflection can also limit linker optimizations. - Datadog used `go-size-analyzer` to measure the contribution of individual dependencies more accurately than import graphs alone. ## Overall Optimization Strategy - Datadog systematically audited dependencies rather than removing product capabilities. - The work focused on restructuring imports, isolating optional functionality, and restoring linker optimizations that had been disabled or undermined over time. - The resulting improvements brought artifact sizes close to levels from roughly five years earlier. - Some compiler and linker behaviors uncovered during the effort led to improvements benefiting other large Go projects, including Kubernetes. The practical lesson is to treat binary size as an ongoing dependency and architecture concern: analyze actual symbol reachability, isolate optional features behind build constraints or packages, and verify each build variant independently.
Read original(opens in new tab)