gitlabMay 7, 2026

Consolidate your GitLab stack with Gitaly on Kubernetes (opens in new tab)

Gitaly on Kubernetes is now generally available with GitLab 18.11, allowing teams to run their entire GitLab stack in Kubernetes instead of maintaining Gitaly on separate virtual machines. GitLab addressed Kubernetes-specific challenges involving cgroup isolation, pod restarts, and request reliability. The result is a more unified deployment model, though full high availability still depends on Gitaly Cluster support for Kubernetes. ### Challenges of Running Gitaly on Kubernetes - Git operations can consume unpredictable amounts of memory. - Gitaly isolates individual Git processes in dedicated cgroups so an out-of-memory failure does not bring down the main Gitaly process. - Kubernetes deployments required special handling because containerd traditionally restricted cgroupfs writes to privileged containers. - GitLab solved this by using an init container to mount `/sys/fs/cgroup` and make it writable. ### Handling Pod Restarts - Virtual-machine deployments can upgrade Gitaly in place and reload gracefully while preserving the socket. - Kubernetes StatefulSet replacements cause pods to stop and restart abruptly during upgrades, node drains, or configuration changes. - This could cause downtime, particularly for Gitaly Sharded deployments without built-in high availability. - GitLab made Gitaly client retries configurable, allowing clients such as Rails to retry requests until Gitaly becomes available again. - Users may experience slightly higher latency during restarts, but requests generally succeed without visible downtime. ### Benchmark Results and High Availability - GitLab tested common Git operations against VM-based and Kubernetes-based Gitaly installations during upgrades. - Success rates were nearly identical in both environments despite Kubernetes abruptly terminating pods and closing sockets. - Achieving complete success across every operation still requires Gitaly Cluster with Praefect. - Praefect does not yet support Kubernetes, but Kubernetes support is being developed. ### Benefits for GitLab Deployments - Teams with hybrid infrastructure can move Gitaly from virtual machines into their existing Kubernetes cluster. - This removes the need to maintain and monitor a separate VM fleet. - Organizations adopting GitLab on Kubernetes can use a fully Kubernetes-native deployment through the official Helm chart. - Gitaly can run as part of a complete GitLab installation or as an external component. ### Installation - The recommended deployment method is the GitLab Helm chart. - Users should review the Gitaly on Kubernetes documentation before installation. - The documentation covers configuration guidance, common pitfalls, full installations, and external Gitaly deployments. Gitaly on Kubernetes is a practical option for consolidating GitLab infrastructure and simplifying operations. Teams should use the Helm chart and configure client retries carefully, while recognizing that Kubernetes-based high availability through Praefect is still forthcoming.

cgroupsk8sgitlabgit+4

githubApr 16, 2026

How GitHub uses eBPF to improve deployment safety (opens in new tab)

GitHub uses eBPF to prevent deployment scripts from depending on GitHub or other services that may be unavailable during an outage. The approach applies network restrictions only to deployment processes, preserving normal traffic for stateful production hosts. By combining Linux cgroups with eBPF’s `BPF_PROG_TYPE_CGROUP_SKB` hooks, GitHub can detect or block unsafe outbound calls before they create circular deployment dependencies. ## The Deployment Circular Dependency - GitHub hosts its own source code on `github.com`, creating a basic dependency: GitHub may need GitHub to deploy a fix. - GitHub mitigates this with: - A code mirror used for “fix-forward” deployments. - Prebuilt assets used to roll back changes. - Additional circular dependencies can still be introduced by deployment scripts, internal services, or tools that download binaries dynamically. ## Three Types of Circular Dependencies The post uses a hypothetical MySQL outage to illustrate how deployment recovery can fail: - **Direct dependencies** - A deployment script downloads the latest release of an open-source tool from GitHub. - If GitHub cannot serve release data, the deployment cannot complete. - **Hidden dependencies** - A required tool is already installed locally but checks GitHub for updates when it runs. - The tool may fail or hang if that update check cannot connect. - **Transient dependencies** - The deployment calls another internal service, such as a migrations service. - That service then attempts to download a binary from GitHub, causing the failure to propagate back to the deployment. ## Why Manual Review Is Insufficient - Teams responsible for stateful hosts traditionally review deployment scripts for circular dependencies. - Many indirect or unexpected dependencies are discovered only during incidents, when they can delay recovery. - Blocking `github.com` at the host level would be too disruptive because stateful hosts continue serving customer traffic during deploys, drains, and restarts. ## Per-Process Network Filtering with eBPF - eBPF allows custom programs to run inside the Linux kernel and attach to low-level operations such as networking. - GitHub focused on `BPF_PROG_TYPE_CGROUP_SKB`, which can inspect network egress for a specific cgroup. - Linux cgroups provide process grouping, isolation, and resource controls without requiring Docker. - The proposed design: - Create a dedicated cgroup. - Place only the deployment script and its processes inside it. - Restrict or monitor outbound network access for that group. - Leave the host’s ordinary production traffic unaffected. ## Proof of Concept with Go and eBPF - The proof of concept uses Go and the `cilium/ebpf` library. - The library simplifies: - Compiling and loading eBPF programs. - Attaching programs to kernel hooks. - Reading and updating eBPF maps. - The example attaches an egress program to `/sys/fs/cgroup/system.slice`. - An eBPF array map tracks the number of egress packets, while the Go program periodically reads and reports the counter. - The same mechanism can be extended from packet counting to selectively allowing or blocking network traffic. GitHub’s approach moves dependency validation from manual inspection into enforcement at runtime. Restricting only deployment processes with cgroups and eBPF provides a practical way to make emergency deployments independent of services that may be down, without blocking the production workloads sharing the same host.

cgroupsgoebpflinux-kernel+2

figmaOct 23, 2023

Server-side sandboxing: Containers and seccomp | Figma Blog (opens in new tab)

Containers and seccomp provide lightweight alternatives to virtual machines for isolating untrusted server-side workloads. Containers rely on OS-level features such as namespaces, cgroups, privilege dropping, and mandatory access controls, while seccomp restricts which system calls a process can invoke. Figma’s conclusion is that containers are not secure by default: effective sandboxing depends on the runtime, kernel, configuration, and broader infrastructure design. ## Container Isolation and Its Attack Surface - Container escapes depend on three main components: - The container runtime implementation - Operating-system primitives and interfaces exposed to the runtime - Runtime configuration - On Linux, Docker commonly uses the `runC` runtime alongside: - Namespaces and cgroups - Privilege dropping - Seccomp - SELinux or AppArmor - Vulnerabilities in the kernel or runtime, as well as configuration mistakes, can allow malicious workloads to modify host files or execute host-level code. - Unlike many VM solutions, containers place more responsibility on operators to configure isolation correctly. ## Risks from Compromised Containers - Simply running untrusted code inside a container does not guarantee safety. - Container settings should be strengthened to prevent host takeover. - The surrounding architecture should limit what a compromised container can access. - A safer design may use containers with: - No mounted network devices - No credentials - No access to unrelated data - Containers can be placed in an isolated network, with orchestration systems passing inputs and collecting outputs through controlled channels. ## Seccomp as an Additional Boundary - Seccomp, or “secure computing mode,” restricts the system calls available to a process. - This can reduce the kernel attack surface available to malicious code running inside a container. - Seccomp works alongside container mechanisms rather than replacing them; isolation depends on combining syscall restrictions with carefully configured namespaces, privileges, access controls, and infrastructure. ## Figma’s Evaluation Criteria Figma assesses sandboxing technologies using two questions: - Can a malicious workload escape its container and affect the host? - If it cannot escape, can it misuse the container’s permissions to reach other systems or cause harm? The practical recommendation is to treat containers as configurable security primitives, not automatically secure sandboxes. Use restrictive seccomp and container configurations, minimize credentials and connectivity, and design the surrounding system so that a compromised workload has limited impact.

cgroupsdockerseccompcontainer-sandboxing+4