How we reduced core unit boot time from hours to minutes
Cloudflare traced a dramatic increase in core-server boot times—from minutes to nearly four hours—to a firmware update that caused UEFI to probe network boot interfaces sequentially. Each failed IPv4 HTTPS or iPXE attempt waited roughly five minutes before the firmware reached the working IPv6 interface. By explicitly configuring the correct boot interface and adapting their automation to firmware limitations, Cloudflare reduced later boot times to under a minute. ## Network Boot in Cloudflare’s Core - Core servers use automated network booting because they are bare-metal machines distributed across many environments. - The fleet relies primarily on: - PXE, often through the open-source iPXE firmware. - Native UEFI HTTPS boot on hardware that supports it. - iPXE enables scripted provisioning and can retrieve operating-system images over HTTP or HTTPS. ## The Linear Search That Caused Four-Hour Boots - After a firmware update, affected machines completed POST and hardware initialization normally but stalled before reaching the operating system. - UEFI tried network interfaces in sequence: - IPv4 HTTPS boot - IPv4 iPXE - Repeated attempts - IPv6 HTTPS boot, which eventually succeeded - Each failed attempt consumed about five minutes waiting for a timeout. - Multiple attempts added roughly 20 minutes to every boot. - Firmware upgrades required several reboots, causing the delays to compound to nearly four hours per server. ## Declaring the Correct Boot Interface - Cloudflare eliminated the linear search by configuring the preferred network boot interface order in advance. - The configuration was moved earlier into the PXE pre-boot stage and tailored to each hardware type and use case. - This removed repeated interface probing during firmware-upgrade reboots and reduced total upgrade time by about an hour. - Subsequent boots fell from approximately 20 minutes to less than one minute. ## Handling Firmware Limitations - Older UEFI versions did not support boot-order configuration. - Firmware upgrades could reset the boot-order settings. - Cloudflare added state validation to its automation: - Check the configuration after changes or upgrades. - Reapply the desired settings if they were lost. - Reboot when necessary. - The first boot could take slightly longer, but future boots became substantially faster. ## Working Around Lazy-Loaded Firmware Settings - The network boot configuration was represented by an `EFI_IFR_REF3` structure that firmware loaded only when accessed through a graphical BIOS callback. - Because the structure was not instantiated during programmatic scans, automation could not see or modify the network boot priorities. - Cloudflare worked with hardware vendors to enable tokens in the fixed “Boot Order Module.” - This forced the firmware to expose the network boot interface during startup without requiring manual BIOS interaction. ## Practical Conclusion The incident demonstrates that boot delays can come from firmware behavior rather than hardware or operating-system failures. Explicitly declaring boot interfaces, validating configuration persistence, and coordinating with vendors to expose hidden firmware settings turned repeated timeout waits into predictable, automated boot cycles.
Read original(opens in new tab)