datadog·Feb 28, 2022·3 min readCurated summary

Profiling improvements in Go 1.18

Go 1.18 introduced major profiling improvements alongside features such as generics and fuzzing. Its Linux CPU profiler became substantially more accurate on multicore systems by addressing dropped `SIGPROF` signals, and profiler labels received an important correctness fix. These changes strengthened Go’s ability to connect continuous profiling data with tracing systems such as Datadog. ## More Accurate Linux CPU Profiling - Earlier Go versions used `setitimer(2)` to request a `SIGPROF` signal every 10 ms of CPU time. - On busy multicore systems, Linux could generate multiple signals within a single kernel “jiffy” window, but standard POSIX signals do not queue. - As a result, many signals were dropped: - A service using 20 CPU cores might generate roughly 2,000 profiling signals per second. - Its Go profile could contain only about 240 samples per second. - Linux’s software clock, commonly operating at 250 Hz, could only measure CPU time in roughly 4 ms intervals. This caused signal bursts and undercounted CPU usage. - `setitimer(2)` also distributed process-directed signals unevenly across threads, creating additional profiling bias. ## Combining `timer_create` and `setitimer` - `timer_create(2)` provided more reliable per-thread signal accounting and avoided most of the signal-dropping and thread-bias problems. - Its drawback was that the profiler needed awareness of every thread, including threads created independently by cgo code. - The Go 1.18 fix combined both timer mechanisms: - The signal handler identifies the signal source. - Signals from inferior sources are discarded. - The implementation accounts for short-lived threads and cgo edge cases. - The work originated from investigations into Go issues GH 35057 and GH 14434 and was developed through collaboration between contributors and Go maintainers. ## Profiler Label Correctness - Profiler labels, also called pprof labels or tags, associate key/value metadata with goroutines. - Labels are inherited by child goroutines and appear in CPU and goroutine profiles, enabling profiles to be filtered by request, service, or trace metadata. - Testing at Datadog revealed that some stack samples were missing labels they should have carried. - The cause was a CPU profiler lookup using the wrong goroutine reference. - The fix changed the profiler to use `gp.m.curg`, the thread’s actual current goroutine, rather than relying on `gp`, which can differ in certain runtime situations. Go 1.18’s profiling changes made CPU measurements more trustworthy on Linux and improved the accuracy of metadata attached to profile samples. Together, they provided a stronger foundation for correlating Go profiling with distributed tracing.

Read original(opens in new tab)

CgoGoLinuxCpu Profiling+4

datadog·Apr 16, 2018·2 min readCurated summary

Cgo and Python

Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API. ## Why Embed Python in Go? - Supports incremental migration from an existing Python codebase. - Reuses mature Python libraries without reimplementing them in Go. - Enables runtime loading and execution of custom or updated Python scripts. - This dynamic extensibility is especially important for Datadog checks. ## Introducing cgo - CPython exposes a C API, while Go requires a Foreign Function Interface to call C code. - cgo provides that integration while preserving the normal `go build` workflow. - A C preamble placed immediately before `import "C"` can include headers and C code. - The pseudo-package `C` exposes C constants, functions, and types to Go. - `go build -x` reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary. ## Initializing the CPython Interpreter - A Go program must initialize Python with `Py_Initialize()` before executing Python code. - It should shut down the interpreter with `Py_Finalize()` when finished. - `Py_GetVersion()` demonstrates retrieving Python information through the C API. - `#cgo` directives can use `pkg-config` to locate Python development headers and libraries, such as `python-2.7`. - The examples use Python 2, but the same approach largely applies to Python 3. ## Using a Go Wrapper - Direct cgo interaction is mostly boilerplate, so Datadog uses the `go-python` library. - The wrapper exposes operations such as: - `python.Initialize()` - `python.PyRun_SimpleString(...)` - `python.Finalize()` - This hides cgo details and makes embedded Python code look more idiomatic from Go. ## Importing and Calling Python Code - A Python module can be imported with `PyImport_ImportModule`. - Go retrieves a function using `GetAttrString`. - The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments. - The Go code must check for failures when importing modules or locating functions. - A simple `foo.py` module containing a `hello()` function can therefore be loaded and executed from disk. Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as `go-python` makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.

Read original(opens in new tab)

CgoDatabase DesignPythonGo+4