gitlab·May 7, 2026·3 min readCurated summary

Consolidate your GitLab stack with Gitaly on Kubernetes

Gitaly on Kubernetes is now generally available with GitLab 18.11, allowing teams to run their entire GitLab stack in Kubernetes instead of maintaining Gitaly on separate virtual machines. GitLab addressed Kubernetes-specific challenges involving cgroup isolation, pod restarts, and request reliability. The result is a more unified deployment model, though full high availability still depends on Gitaly Cluster support for Kubernetes. ### Challenges of Running Gitaly on Kubernetes - Git operations can consume unpredictable amounts of memory. - Gitaly isolates individual Git processes in dedicated cgroups so an out-of-memory failure does not bring down the main Gitaly process. - Kubernetes deployments required special handling because containerd traditionally restricted cgroupfs writes to privileged containers. - GitLab solved this by using an init container to mount `/sys/fs/cgroup` and make it writable. ### Handling Pod Restarts - Virtual-machine deployments can upgrade Gitaly in place and reload gracefully while preserving the socket. - Kubernetes StatefulSet replacements cause pods to stop and restart abruptly during upgrades, node drains, or configuration changes. - This could cause downtime, particularly for Gitaly Sharded deployments without built-in high availability. - GitLab made Gitaly client retries configurable, allowing clients such as Rails to retry requests until Gitaly becomes available again. - Users may experience slightly higher latency during restarts, but requests generally succeed without visible downtime. ### Benchmark Results and High Availability - GitLab tested common Git operations against VM-based and Kubernetes-based Gitaly installations during upgrades. - Success rates were nearly identical in both environments despite Kubernetes abruptly terminating pods and closing sockets. - Achieving complete success across every operation still requires Gitaly Cluster with Praefect. - Praefect does not yet support Kubernetes, but Kubernetes support is being developed. ### Benefits for GitLab Deployments - Teams with hybrid infrastructure can move Gitaly from virtual machines into their existing Kubernetes cluster. - This removes the need to maintain and monitor a separate VM fleet. - Organizations adopting GitLab on Kubernetes can use a fully Kubernetes-native deployment through the official Helm chart. - Gitaly can run as part of a complete GitLab installation or as an external component. ### Installation - The recommended deployment method is the GitLab Helm chart. - Users should review the Gitaly on Kubernetes documentation before installation. - The documentation covers configuration guidance, common pitfalls, full installations, and external Gitaly deployments. Gitaly on Kubernetes is a practical option for consolidating GitLab infrastructure and simplifying operations. Teams should use the Helm chart and configure client retries carefully, while recognizing that Kubernetes-based high availability through Praefect is still forthcoming.

Read original(opens in new tab)

ContainerdKubernetesGitlabGit+4

netflix·Feb 28, 2026·3 min readCurated summary

Mount Mayhem at Netflix: Scaling Containers on Modern CPUs

Netflix’s effort to modernize its container runtime exposed a hardware-level bottleneck rather than an application problem. Under heavy startup concurrency, containers with many image layers triggered massive mount and unmount activity, causing kernel lock contention, systemd stalls, and container startup failures. The issue was especially severe on older dual-socket NUMA instances, while newer single-socket systems scaled much more reliably. ## Container Startup at Netflix - New AWS capacity is rapidly filled with pods as applications scale. - Some nodes became unresponsive, with: - Health checks timing out for more than 30 seconds - Kubelet requests to containerd timing out - systemd processing huge numbers of mount events - The mount table taking tens of seconds to read - The problem primarily affected `r5.metal` instances running images with more than 50 layers. ## Mount Lock Contention - With user namespaces, containerd performs several mount operations for every image layer: - `open_tree()` references the layer. - `mount_setattr()` applies the container’s ID mapping. - `move_mount()` creates an ID-mapped bind mount. - These bind mounts become OverlayFS lower directories and are later unmounted. - The Linux VFS uses global mount-related locks, so concurrent container creation causes CPUs to contend on the same kernel locks. - For 100 containers with 50 layers each, containerd performs the process twice: - `100 × 2 × (1 + 50 + 50) = 20,200` mount operations - This makes startup cost depend heavily on both container concurrency and image layer count. ## Why the New Runtime Exposed the Problem - The old Docker-based runtime shifted file ownership while unpacking images. - All containers shared one host user range, avoiding repeated per-container mount work. - The new containerd-based runtime assigns each container a unique host user range for stronger isolation. - Instead of rewriting file ownership during extraction, it uses Linux ID-mapped mounts to apply ownership mappings efficiently. - This improves security and avoids expensive image copying, but creates many additional mount operations during startup. ## Differences Between AWS Instance Types Netflix compared: - `r5.metal`: 5th-generation Intel, dual-socket, multiple NUMA domains - `m7i.metal-24xl`: 7th-generation Intel, single-socket, single NUMA domain - `m7a.24xlarge`: 7th-generation AMD, single-socket, single NUMA domain Results showed: - At low concurrency—around 20 containers or fewer—all systems performed similarly. - `r5.metal` began failing at roughly 100 concurrent container launches. - Newer Intel instances maintained lower startup times and better success rates. - AMD-based `m7a` instances scaled most consistently and had the fewest failures. ## Kernel and CPU-Level Diagnosis - Profiling showed that containerd spent most of its time in Linux VFS path lookup code. - Specifically, threads were spinning in `path_init()` while waiting on a sequence lock. - Intel Topdown Microarchitecture Analysis found: - 95.5% of pipeline slots stalled on contested accesses - 57% attributed to false sharing - Cache-line bouncing and global lock contention, rather than raw CPU capacity, dominated performance. ## NUMA as a Contributing Factor - NUMA systems divide memory among processor sockets. - Local memory access is faster, while remote access crosses an interconnect and introduces additional latency. - The dual-socket layout of `r5.metal` amplified contention around shared mount-related data. - The better behavior of newer single-socket instances indicated that CPU topology and memory locality were key contributors to the container startup bottleneck. ## Practical Conclusion High-concurrency container launches can overwhelm kernel mount infrastructure, especially when using per-container ID mapping and images with many layers. Netflix’s results suggest minimizing image layers, controlling startup concurrency, and favoring newer single-socket hardware can substantially improve reliability and scaling.

Read original(opens in new tab)

ContainerdDatabase DesignKubernetesAWS+4

datadog·May 24, 2023·3 min readCurated summary

2023-03-08 incident: A deep dive into the platform-level impact

Datadog’s March 8, 2023 outage was caused by an unexpected interaction between Ubuntu 22.04, systemd-networkd, and an automated security patch. A systemd change introduced behavior that flushed unfamiliar IP routing rules whenever systemd-networkd restarted; a CVE patch triggered that restart across many hosts. Because the patch was installed automatically and outside Datadog’s carefully staged deployment process, infrastructure across regions and cloud providers was affected simultaneously. ## A Systemd Behavior Change - systemd v248 introduced a systemd-networkd startup behavior that removed IP rules it did not recognize. - systemd v249 added the `ManageForeignRoutingPolicyRules` setting, which could disable this behavior, but the default configuration continued managing foreign rules. - These changes were backported to older systemd releases. - Ubuntu 20.04 used systemd v245, which did not flush IP rules during a systemd-networkd restart. - Ubuntu 22.04, adopted progressively by Datadog beginning in November 2022, used systemd v249 with the behavior enabled. Initially, the change caused no visible problems because systemd-networkd generally started only when new hosts were created, before Datadog’s custom routing rules existed. ## The Security Patch That Triggered the Problem - On March 7, 2023, Ubuntu released systemd patch `249.11-0ubuntu3.7` for a CVE. - Installing the patch restarted all systemd components, including systemd-networkd. - That restart caused systemd-networkd to flush routing policy rules on affected Ubuntu 22.04 hosts. - Ubuntu 20.04 hosts received a similar patch but were not affected because systemd v245 did not exhibit the problematic restart behavior. ## Unattended Upgrades Created Broad Exposure Datadog used Ubuntu’s default unattended-upgrade configuration: - Package metadata was downloaded twice daily using `apt-daily.timer`, with randomized delays of up to 12 hours. - Upgrades ran daily using `apt-daily-upgrade.timer`, between 06:00 and 07:00 UTC. - Only security updates and required dependencies were automatically installed. - Regular updates from the `-updates` repository were excluded. This meant many hosts automatically installed the systemd security patch during the same daily upgrade window. Not every host was affected: more than 90% of the fleet used Ubuntu 22.04, and some nodes had not yet downloaded the patch when their upgrade ran. ## Conflict with Datadog’s Deployment Process - Datadog normally updates nodes by replacing them automatically rather than relying on unattended upgrades. - Its standard process validates changes on experimental clusters, then progressively deploys them through staging and production. - Deployments are normally limited to selected clusters, availability zones, and regions before expanding. - The unattended systemd patch bypassed this process because it was installed directly on existing hosts. - As a result, a low-level networking change propagated across otherwise isolated regions and cloud providers at nearly the same time. Datadog’s experience demonstrates that even security-only automated updates can introduce coordinated infrastructure risk. Critical system packages should be tested and rolled out through the same staged process as other production changes, or their automated upgrades should be carefully constrained and monitored.

Read original(opens in new tab)

ContainerdKubernetesCloud InfrastructureSystemd+4