cloudflare·Mar 26, 2026·2 min readCurated summary

A one-line Kubernetes fix that saved 600 hours a year

Atlantis restarts were taking about 30 minutes, blocking infrastructure changes and consuming more than 50 engineering hours monthly. The delay was caused by Kubernetes recursively changing ownership on a large Ceph-backed PersistentVolume containing millions of files. Setting `fsGroupChangePolicy: OnRootMismatch` avoided unnecessary recursive ownership changes and reduced restart time dramatically. ### The Restart Bottleneck - Atlantis runs as a singleton Kubernetes `StatefulSet`. - Its PersistentVolume stores repository and Terraform state. - Credential rotations, onboarding, and offboarding required restarting Atlantis. - With roughly 100 restarts per month, each 30-minute delay created more than 600 hours of annual lost engineering time. - The volume had grown large enough to exhaust inodes, making storage expansion and pod restarts necessary. ### Kubernetes Made the Delay Look Like a Scheduling Problem - `kubectl rollout restart statefulset atlantis` terminated the old pod and created a replacement. - The new pod was scheduled quickly but remained stuck in `Init:0/1`. - Kubernetes events showed the image pulling successfully, but revealed no obvious cause for the long gap. - Kubelet logs showed the PersistentVolume mounting successfully, followed by repeated `context deadline exceeded` errors while syncing the pod. ### The Hidden Cost of `fsGroup` - Searching logs using the PersistentVolume name exposed the relevant message: - Kubernetes was “setting volume ownership” because an `fsGroup` was configured. - Kubernetes warned that ownership changes could be slow when a volume contained many files. - The default behavior recursively changed ownership across the entire mounted volume. - As Atlantis’s volume accumulated millions of files, this initialization step became the 30-minute bottleneck. ### The One-Line Fix - The volume configuration was changed to: ```yaml fsGroupChangePolicy: OnRootMismatch ``` - With this policy, Kubernetes checks the root directory’s ownership and only performs recursive changes when necessary. - Existing volumes with the correct ownership no longer require a full filesystem traversal during every restart. The practical lesson is to inspect kubelet and volume logs when a pod appears scheduled but remains stuck before initialization. For large persistent volumes, explicitly setting `fsGroupChangePolicy: OnRootMismatch` can eliminate costly recursive ownership changes and prevent substantial operational downtime.

Read original(opens in new tab)

Persistent StorageDatabase DesignKubernetesTerraform+4

datadog·Oct 15, 2025·3 min readCurated summary

Failure is inevitable: Learning from a large outage, and building for reliability in depth at Datadog

Datadog’s March 2023 outage exposed a fundamental weakness in its reliability strategy: although 40–50% of production Kubernetes nodes remained operational, customers experienced the platform as entirely unavailable. The incident showed that preventing every failure is impossible and that systems must instead continue delivering useful, accurate service when components fail. Datadog consequently began redesigning products around graceful degradation, prioritizing data preservation, fresh information, and partial results. ## Lessons from the March 2023 Incident - An unsupervised global update triggered a restart interaction that disconnected roughly 50–60% of production Kubernetes nodes. - The web interface recovered quickly, but logs, metrics, alerts, traces, and other core features became unavailable. - Pages loaded without displaying customer data, creating a nearly complete outage from the user’s perspective. ## Limits of Traditional Root-Cause Analysis - Datadog identified the legacy global security-update mechanism as the immediate trigger and disabled it. - Fixing that mechanism alone could not address the broader class of failures caused by certificates, configuration changes, overloads, date-handling bugs, or other unexpected events. - The company concluded that resilience requires reducing the impact of failures, not merely preventing one specific failure mode. ## Why Partial Infrastructure Became a Total User-Facing Failure - Datadog’s systems historically favored complete correctness over partial visibility. - For example, metric queries could wait until all relevant tags were processed to avoid showing misleading values or triggering false alerts. - During a large outage, this behavior created a “square-wave” failure: missing some data caused the system to show no data. - Ordered queues could stall fresh results behind stuck work, retries could overload already-strained services, and node-specific processing could make surviving capacity ineffective. - The underlying design assumption was that systems should either function fully or stop, rather than degrade while continuing to provide value. ## Prioritizing Graceful Degradation Datadog shifted from relying primarily on redundancy and “never-fail” architectures to explicitly designing for inevitable failures. - Customer data should never be lost, even if delivery is delayed. - Fresh, real-time data should take priority over stale backlog processing. - Systems should provide partial but accurate results whenever possible instead of returning nothing. ## Persistent Storage at the Start of Processing Pipelines - The outage caused a limited but non-zero amount of irreversible customer data loss. - Some pipelines acknowledged data before writing it to replicated storage, leaving unreplicated data only in memory or on a local disk. - When a node failed, that data disappeared and could not be recovered through agent retries. - After the node loss, surviving intake nodes also struggled to write to downstream replicated stores. - Their memory and local-disk buffers eventually filled, causing additional data loss as the outage continued. - Datadog therefore identified persistent intake storage as a key requirement for preserving data during large-scale failures. The broader recommendation is to design systems not only to prevent outages, but also to remain useful during them: preserve every accepted event, prioritize current information, and expose accurate partial results instead of failing completely.

Read original(opens in new tab)

Persistent StorageDatabase DesignKubernetesData Loss Prevention+4