cloudflare·Dec 22, 2025·3 min readCurated summary

How Workers powers our internal maintenance scheduling pipeline

Cloudflare built an automated maintenance scheduler on Cloudflare Workers to prevent overlapping infrastructure changes from disrupting connectivity or customer-specific routing. The system evaluates the full network state, identifies conflicts across maintenance events, and alerts operators before unsafe schedules are approved. Its key design shift was from loading all operational data into one Worker to using graph-based, on-demand data retrieval that respects Workers’ memory limits. ## Why Manual Maintenance Planning Was No Longer Enough - Cloudflare operates data centers in more than 330 cities, making manual coordination increasingly unreliable. - Maintenance can create conflicts when: - Redundant edge routers in the same metro area are taken offline simultaneously. - All data centers selected by a customer’s Dedicated CDN Egress IPs (“Aegis”) pool become unavailable. - These failures could cause higher latency, connectivity loss, or 5xx errors. - The scheduler centralizes network state and warns operators when maintenance windows overlap in unsafe ways. ## Modeling Operational Safety as Constraints - Each safety rule begins with proposed maintenance items, such as routers or server groups. - The system finds calendar events whose time windows overlap with the proposed change. - It then combines those events with product data, including Aegis pools and their associated data center IDs. - For example, if an Aegis customer’s pool uses data centers 21 and 45, scheduling both for simultaneous downtime violates the constraint that at least one must remain online. - Operators receive conflict notifications and can reschedule maintenance before it affects customers. ## Reducing Data Usage on Workers - The initial design loaded server relationships, product configurations, and health metrics into a single Worker. - This quickly caused out-of-memory errors. - The scheduler instead loads only data relevant to the maintenance location and affected relationships. - A router maintenance request in Frankfurt, for example, does not need unrelated infrastructure data from Australia. ## Graph Processing with Typed Associations - Cloudflare modeled infrastructure and product relationships as a graph: - **Objects** represent entities such as routers, data centers, and Aegis pools. - **Associations** represent relationships between those entities. - Inspired by Facebook’s TAO system, the team created an interface supporting operations such as: - `object_get()` to retrieve an object. - `assoc_get()` to stream typed relationships. - `assoc_count()` to count related objects. - Constraints can retrieve only the associations they need, such as which Aegis pools include a particular data center and how many data centers each pool contains. - Parallel lookups and deduplication reduce both execution time and memory consumption. Cloudflare’s scheduler demonstrates how Workers can serve as a centralized safety layer for complex infrastructure operations. The practical recommendation is to represent operational dependencies as typed graphs and fetch relationship data incrementally, rather than loading the entire network and product state into each execution.

Read original(opens in new tab)

Infrastructure AutomationCloudflare WorkersServerlessGraph Processing+2

line·Aug 7, 2025Original article

Flexible Multi-site Architecture Designed with N (opens in new tab)

LINE NEXT optimized its web server infrastructure by transitioning from fragmented, manual Nginx setups to a centralized native Nginx multi-site architecture. By integrating global configurations and automating the deployment pipeline with Ansible, the team successfully reduced service launch lead times by over 80% while regaining the ability to use advanced features like GeoIP and real client IP tracking. This evolution ensures that the infrastructure can scale to support over 100 subdomains across diverse global services with high reliability and minimal manual overhead. ## Evolution of Nginx Infrastructure * **PMC-based Structure**: The initial phase relied on a Project Management Console using `rsync` via SSH; this created security risks and led to fragmented, siloed configurations that were difficult to maintain. * **Ingress Nginx Structure**: To improve speed, the team moved to Kubernetes-based Ingress using Helm charts, which automated domain and certificate settings but limited the use of native Nginx modules and complicated the retrieval of real client IP addresses. * **Native Nginx Multi-site Structure**: The current hybrid approach utilizes native Nginx managed by Ansible, combining the speed of configuration-driven setups with the flexibility to use advanced modules like GeoIP and Loki for log collection. ## Configuration Integration and Multi-site Management * **Master Configuration Extraction**: Common directives such as `timeouts`, `keep-alive` settings, and `log formats` were extracted into a master Nginx configuration file to eliminate redundancy across services. * **Hierarchical Directory Structure**: Inspired by Apache, the team adopted a `sites-available` structure where individual `server` blocks for different services (alpha, beta, production) are managed in separate files. * **Operational Efficiency**: This integrated structure allows a single Nginx instance to serve multiple sites simultaneously, significantly reducing the time required to add and deploy new service domains. ## Automated Deployment with Ansible * **Standardized Workflow**: The team replaced manual processes with Ansible playbooks that handle everything from cloning the latest configuration from Git to extracting environment-specific files. * **Safety and Validation**: The automated pipeline includes mandatory Nginx syntax verification (`nginx -t`) and process status checks to ensure stability before a deployment is finalized. * **Rolling Deployments**: To minimize service impact, updates are pushed sequentially across servers; the process automatically halts if an error is detected at any stage of the rollout. To effectively manage a rapidly expanding portfolio of global services, infrastructure teams should move toward a "configuration-as-code" model that separates common master settings from service-specific logic. Leveraging automation tools like Ansible alongside a native Nginx multi-site structure provides the necessary balance between rapid deployment and the granular control required for complex logging and security requirements.

Infrastructure AutomationKubernetesNginxAnsible+4