datadog·Jun 15, 2017·3 min readCurated summary

Hackathon project: Viewing Datadog metrics in Minecraft

Datadog engineers used a two-day hackathon to display real-time Datadog metrics inside Minecraft. They connected Minecraft’s Python API with Datadog’s metrics API, then built configurable, live-updating graphs and monitor indicators in the game world. The project demonstrated that even an unconventional visualization environment can be practical to prototype with familiar tools. ## Controlling Minecraft with Python - The team used Raspberry Juice and a Minecraft Pi Edition server to expose Minecraft controls. - They ran the setup on laptops for better performance and faster development. - The `py3minepi` library enabled Python code to create, remove, and query blocks. - Creating a block required only a connection to the server and a call such as `mc.setBlock(...)`. ## Retrieving Datadog Metrics - The Datadog Python library provided access to the Metrics API. - The prototype authenticated with an API key and application key. - It queried recent data, such as average system CPU idle time over the previous five minutes. - The Minecraft and Datadog components were then combined so metric values could be rendered as blocks and structures. - Monitor status indicators changed between green and red depending on whether an alert was active. ## YAML-Based Dashboard Configuration - The team moved dashboard definitions out of Python code into YAML files. - Configuration specified: - Graph position, size, and orientation - Visual properties such as colors, transparency, and borders - Datadog queries and time ranges - Monitor IDs and display locations - This allowed complete dashboards containing multiple graphs and monitor indicators to be updated in real time. ## Handling Minecraft’s Persistence - Minecraft blocks remain in the world after being created, while metric graphs change constantly. - Early experiments left behind random cubes that made the world difficult to navigate. - The team implemented “vacuum” functions to remove everything generated by the visualization code before redrawing it. ## Rendering and Performance Challenges - Without browser technologies such as JavaScript and CSS, graphs had to be reduced to rows of data and represented with Minecraft blocks. - Large graphs could overwhelm the data pipeline. - Caching was added to reduce bandwidth usage and avoid repeatedly requesting the same data. - The performance concerns mirrored Datadog’s everyday engineering work, where caching and efficient data handling are essential. ## Hackathon Experience - The first four hours focused on configuring the environment and connecting the systems. - The remaining time was spent experimenting with building, viewing, destroying, and rebuilding metric displays. - The project’s main value was creative exploration rather than production monitoring. The prototype shows how quickly APIs can be combined to create unusual monitoring interfaces. While Minecraft is not intended to replace conventional dashboards, the project is a playful demonstration of real-time data visualization and rapid experimentation.

Read original(opens in new tab)

PythonDatabase DesignDatadogAPI+3

datadog·Jun 8, 2017·1 min readCurated summary

Protobuf parsing in Python | Datadog

The provided content does not include the blog post itself. It contains Datadog’s site navigation and a link whose URL suggests the article concerns Protobuf parsing in Python, but no technical claims, explanations, or conclusions are available to summarize. ## Available Information - The linked article appears to be: - **“Protobuf parsing in Python”** - Located on Datadog’s engineering blog. - The rest of the content is primarily Datadog product navigation, covering: - Infrastructure and application monitoring - Logs, security, digital experience, CI, and AI products - A separate banner promotes Datadog’s recognition as a Leader in the Gartner Magic Quadrant for Observability Platforms. Please provide the article’s body text or a complete page extract for a substantive summary.

Read original(opens in new tab)

PythonSerializationProtobufParsing

datadog·Jun 8, 2017·2 min readCurated summary

Protobuf parsing in Python

Protocol Buffers provides a compact, efficient binary format for structured data, making it suitable for APIs and inter-machine communication. The post introduces Protobuf through a Python metrics example and explains how to serialize and deserialize messages. It also shows how to stream multiple messages by prefixing each with its length, since Protobuf messages are not inherently self-delimiting. ## Protocol Buffers Basics - A `.proto` file defines the structure of a message. - The example `Metric` message contains: - A name - A type - A floating-point value - Repeated string tags - The `protoc` compiler generates language-specific code, such as Python’s `metric_pb2.py`. - Python can serialize a message with `SerializeToString()` and restore it with `ParseFromString()`. ## Streaming Multiple Messages - A single Protobuf message can be parsed directly, but consecutive messages need delimiters. - Protobuf does not automatically indicate where one message ends and the next begins. - The recommended approach is to prepend each serialized message with its byte length. - The length is encoded as a Varint, which uses fewer bytes for smaller integers. - This mirrors Java’s `writeDelimitedTo` and `parseDelimitedFrom` behavior and is also how the kube-state-metrics API chains messages. ## Varints and Python Implementation - Python’s Protobuf library does not provide public convenience methods for delimited messages. - The implementation uses internal helpers: - `_VarintBytes` to encode message lengths - `_DecodeVarint32` to read them - Serialization writes the length followed by the message bytes. - Deserialization reads the length, extracts the corresponding byte range, and parses it as a `Metric`. - The example loads the entire stream into memory, though a production implementation could process data incrementally. For APIs that exchange sequences of structured records, length-prefixed Protobuf messages offer an efficient and interoperable alternative to plain-text formats. Teams should account for message framing explicitly and use generated code plus appropriate streaming logic when handling multiple messages.

Read original(opens in new tab)

PythonKubernetesDatadogMetrics+4

datadog·Jan 23, 2017·2 min readCurated summary

The trouble with mounting

Datadog found that some agents stopped reporting all metrics because they became stuck in an unkillable state during disk checks. The root cause was `os.statvfs`, whose glibc implementation can hang while inspecting NFS mounts configured with hard-mount behavior. Since agents run in unpredictable customer environments, Datadog isolated the call in a separate thread and allowed the main process to continue after a timeout. ## Detecting the Hang - Customers reported gaps across every metric, indicating that the agent—not an individual check—had stopped functioning. - Logs showed the agent sometimes hung without producing an error. - A watchdog failed to terminate it because the process was stuck in an unkillable system call. - Developer-mode timing data identified `os.statvfs` as the consistently slow operation. ## How NFS Causes Unkillable Processes - `os.statvfs` calls the Linux `statvfs` function through CPython and glibc. - `statvfs` can hang when examining a remote directory mounted through NFS. - NFS hard mounts retry indefinitely and do not time out system calls. - Soft mounts eventually return an error, while the `intr` option allows interruption of the calling process. - Hard mounts may be appropriate when reads and writes must eventually succeed, but they are risky with unreliable NFS connections because they are the default in many configurations. ## The `/proc/mounts` Complication - Glibc’s `statvfs` implementation checks each directory listed in `/proc/mounts` until it finds the requested mount. - Consequently, a disconnected NFS mount can block `statvfs` even when the agent is checking a different filesystem. - This made changing NFS mount options impractical as a universal fix because Datadog cannot control customers’ system configurations. ## Datadog’s Workaround - The agent now runs `statvfs` on a separate thread. - If the call exceeds a timeout, the main agent thread continues operating. - This approach avoids total metric loss across heterogeneous environments. - The trade-off is a modest increase in memory usage on systems with hard-mounted NFS volumes. The practical lesson is to treat filesystem statistics as potentially blocking operations, especially in environments with NFS. Isolating such calls behind timeouts provides more reliable monitoring than assuming system calls will always return promptly.

Read original(opens in new tab)

PythonDatabase DesignDatadogMetrics+3

datadog·Oct 27, 2016·2 min readCurated summary

Cheering on coworkers: Building culture with Datadog dashboards

Christian’s colleagues built a Datadog dashboard to remotely track his progress in a six-day, 850 km ultramarathon. They scraped live race data from the event website, converted it into Datadog metrics, and visualized his distance, ranking, and elapsed time alongside video and other dashboard elements. At publication, Christian was leading by more than 47 km with 44 hours remaining. ## Extracting Race Data - The event website regularly published runners’ statistics and race progress in plain HTML. - A Python crawler using `Requests` retrieved the webpage. - `BeautifulSoup` parsed the HTML to extract: - Current ranking - Total distance run - Elapsed time - Other race information ## Sending Metrics to Datadog - The team used the Datadog Python client and StatsD to emit metrics through the Datadog Agent. - For each runner, the script sent gauge metrics for: - `runner.distance` - `runner.ranking` - `runner.elapsed_time` - Metrics were tagged with each runner’s name, enabling individual tracking and comparisons. ## Building the Dashboard - The collected metrics were combined into a Datadog dashboard. - The dashboard included: - Live race statistics - A live video feed - Animated GIFs for entertainment - Visualizations of meaningful progress metrics - Screens displaying the dashboard were placed in the company’s New York and Paris offices so colleagues could follow and encourage Christian throughout the race. The project demonstrates how a lightweight web scraper, StatsD metrics, and a monitoring dashboard can turn publicly available data into a live, engaging team experience.

Read original(opens in new tab)

PythonDatadogDashboard DesignBeautifulsoup+2

datadog·Oct 27, 2016Original article

Cheering on coworkers: Building culture with Datadog dashboards | Datadog (opens in new tab)

Datadog engineers developed a real-time tracking dashboard to monitor a colleague’s progress during an 850km, six-day ultra-marathon challenge. By scraping public race statistics and piping the data into their monitoring platform, the team created a centralized visualization tool to provide remote support and office-wide engagement. ### Data Extraction and Parsing The team needed to harvest race data that was only available as plain HTML on the event’s official website. * A crawler was built using the Python `Requests` library to automate the retrieval of the webpage's source code. * The team utilized `BeautifulSoup` to parse the HTML and isolate specific data points, such as the runner's current ranking and total distance covered. ### Ingesting Metrics with StatsD Once the data was structured, it was converted into telemetry using the Datadog agent and the `statsd` Python library. * The script utilized `dog.gauge` to emit three primary metrics: `runner.distance`, `runner.ranking`, and `runner.elapsed_time`. * Each metric was assigned a "name" tag corresponding to the runner, allowing the team to filter data and compare participants within the Datadog interface. * The data was updated periodically to ensure the dashboard reflected the most current race standings. ### Dashboard Visualization and Results The final phase involved synthesizing the metrics into a high-visibility dashboard displayed in the company’s New York and Paris offices. * The dashboard combined technical performance graphs with multimedia elements, including live video feeds and GIFs, to create an interactive cheering station. * The system successfully tracked the athlete's 47km lead in real-time, providing the team with immediate updates on his physical progress and elapsed time over the 144-hour event. This project demonstrates how standard observability tools can be repurposed for creative "life-graphing" applications. By combining simple web scraping with metric ingestion, engineers can quickly build custom monitoring solutions for any public data source.

PythonDatadogMetricsBeautifulsoup+3

datadog·Aug 23, 2016·3 min readCurated summary

Restroom hacks

Datadog built an office bathroom-availability monitor to reduce contention without compromising privacy or existing door functionality. Raspberry Pi 2 devices, GPIO-connected sensors, and simple Unix tools provided a low-maintenance way to report whether bathrooms were occupied. The project showed that the hardest parts were adapting to varied real-world hardware, mounting sensors cleanly, and dealing with unreliable Wi-Fi—not writing software. ## Project Goals - Avoid intrusive monitoring: - No cameras or sensors that could feel invasive. - Provide reliable occupancy information with minimal false positives and negatives. - Use door-lock status where possible as the occupancy signal. - Avoid interfering with existing locks and doors. - Keep devices secure, professional-looking, easy to maintain, and remotely updateable. - Treat the project as a fun hardware experiment. ## Adapting to Different Bathrooms - Bathrooms differed significantly in: - Lock styles, including push-button handles and rotary stall locks. - Number of rooms or stalls. - Availability and location of power outlets. - Wi-Fi quality, especially near concrete walls and older electrical equipment. - These variations required different sensor designs rather than one universal installation. ## Raspberry Pi and Sensor Hardware - Raspberry Pi 2 Model Bs served as the project’s controllers because they: - Ran Linux. - Supported Wi-Fi and SSH administration. - Were compact enough to conceal. - The team used several sensor types: - Magnetic reed switches for detecting door position. - Pin switches for detecting sliding stall-lock positions. - Photoresistors were purchased as a possible way to detect darkness but were not needed in the MVP. - For push-button locks, reed switches detected whether the door was open or closed. Although this could theoretically misreport a closed but unoccupied bathroom, it worked reliably in practice. - Stall-lock sensors were hidden inside hollow metal panels. Automotive-style pin switches were mounted using simple carved wooden blocks that contacted the sliding lock without obstructing it. - Wiring was concealed in wiremolding, with Raspberry Pis placed inside outlet boxes where possible. ## GPIO and Unix-Based Monitoring - Raspberry Pi GPIO pins were accessed through files in `/sys/class/gpio/`. - A Python script read sensor values and translated them into bathroom availability. - Configuration handled differences between normally open and normally closed sensors. - The service was exposed through `tcpserver` and managed with `daemontools`. - A basic command-line client could query status with Netcat, for example: ```sh nc 11.bathrooms.datadog-internal.com 50 ``` ## Making Availability Easy to Use - Employees could check status from the command line. - Some added bathroom availability to TextBar. - Datadog dashboards displayed bathroom status throughout the New York office. - The implementation required very little code; most effort went into sensor selection, physical installation, and network troubleshooting. The project demonstrates that inexpensive, hackable hardware combined with simple Linux tools can solve a practical office problem. For similar systems, prioritize non-intrusive sensors, flexible installation designs, and secure remote management; the resulting software can remain remarkably small.

Read original(opens in new tab)

PythonOpen SourceNetwork SecurityMonitoring+4