How we tracked down a Go 1.24 memory regression across hundreds of pods
Go 1.24 initially caused an unexpected ~20% increase in memory usage across several services, despite its Swiss Tables implementation being expected to reduce memory consumption. The increase appeared in system-level RSS metrics but not in Go’s runtime metrics or heap profiles. Investigation showed that a runtime allocator refactor likely caused more of the Go heap’s virtual memory to be committed to physical RAM. ## The Unexpected Go 1.24 Memory Increase - The issue emerged during an internal rollout of Go 1.24. - Multiple environments showed approximately 20% higher memory usage. - A staging bisect directly linked the increase to the Go 1.24 upgrade. - The behavior was surprising because Go 1.24’s headline Swiss Tables feature promised lower CPU and memory overhead. ## Ruling Out Swiss Tables and Mutex Changes - Swiss Tables were disabled with: ```bash GOEXPERIMENT=noswissmap ``` - Memory usage did not improve, ruling out the new map implementation as the cause. - The new spin-bit mutex implementation was disabled with: ```bash GOEXPERIMENT=nospinbitmutex ``` - The memory increase remained, eliminating this runtime change as the likely culprit. ## System Metrics vs. Go Runtime Metrics - Go runtime metrics showed almost no change after the upgrade. - System metrics reported a significant increase in resident set size (RSS). - RSS measures physical memory currently used in RAM, while Go’s runtime accounting primarily reflects allocated virtual memory. - This discrepancy matters operationally because systems such as Kubernetes and the Linux OOM Killer rely on physical-memory metrics. ## Examining the Go Heap with `/proc/[pid]/smaps` - Linux’s `/proc/[pid]/smaps` exposed memory usage for individual mappings. - In Go 1.24, the main Go heap mapping had roughly: - 1.28 GiB of virtual memory allocated - 1.26 GiB resident in physical RAM - In Go 1.23, a similarly sized heap mapping had about 300 MiB less RSS than its virtual size. - Other memory regions were not significantly affected, indicating that the increased RSS was isolated to the Go heap. - Upstream changes to label Go-allocated memory regions should make future `maps` and `smaps` investigations easier. ## The Suspected Allocator Regression - The evidence suggested Go 1.24 was not requesting substantially more virtual memory. - Instead, previously uncommitted virtual memory was being committed to physical RAM, increasing RSS without changing Go’s internal memory totals. - A major refactoring of the runtime’s `mallocgc` function stood out in the Go 1.24 changelog. - The investigation therefore focused on this allocator change as the likely source of the regression. Go 1.24’s memory increase was caused not by Swiss Tables or mutex changes, but likely by altered heap allocation behavior in the runtime. Comparing RSS with Go’s runtime metrics—and inspecting `/proc/[pid]/smaps`—was essential for identifying the allocator-related discrepancy.
Read original(opens in new tab)