Cgo and Python

Embedding Python in Go lets applications gradually migrate from Python, reuse existing libraries, and load scripts dynamically without recompiling. Datadog uses this approach in its Go-based Agent so checks can remain in Python while the core application moves to Go. The key is combining cgo with a Go-friendly wrapper around CPython’s C API.

Why Embed Python in Go?

• Supports incremental migration from an existing Python codebase.
• Reuses mature Python libraries without reimplementing them in Go.
• Enables runtime loading and execution of custom or updated Python scripts.
• This dynamic extensibility is especially important for Datadog checks.

Introducing cgo

• CPython exposes a C API, while Go requires a Foreign Function Interface to call C code.
• cgo provides that integration while preserving the normal go build workflow.
• A C preamble placed immediately before import "C" can include headers and C code.
• The pseudo-package C exposes C constants, functions, and types to Go.
• go build -x reveals how cgo generates intermediate C and Go files, compiles them, and links the final binary.

Initializing the CPython Interpreter

• A Go program must initialize Python with Py_Initialize() before executing Python code.
• It should shut down the interpreter with Py_Finalize() when finished.
• Py_GetVersion() demonstrates retrieving Python information through the C API.
• #cgo directives can use pkg-config to locate Python development headers and libraries, such as python-2.7.
• The examples use Python 2, but the same approach largely applies to Python 3.

Using a Go Wrapper

• Direct cgo interaction is mostly boilerplate, so Datadog uses the go-python library.
• The wrapper exposes operations such as:
  • python.Initialize()
  • python.PyRun_SimpleString(...)
  • python.Finalize()
• This hides cgo details and makes embedded Python code look more idiomatic from Go.

Importing and Calling Python Code

• A Python module can be imported with PyImport_ImportModule.
• Go retrieves a function using GetAttrString.
• The function is invoked through the Python API, passing empty tuple and dictionary objects even when it accepts no arguments.
• The Go code must check for failures when importing modules or locating functions.
• A simple foo.py module containing a hello() function can therefore be loaded and executed from disk.

Embedding CPython through cgo provides a practical bridge between Go and Python. A wrapper such as go-python makes the integration easier to maintain, while allowing applications like the Datadog Agent to combine a Go core with dynamically executed Python components.