Advancing Our Chef Infrastructure: Safety Without Disruption
Slack chose to improve its existing Chef and EC2 infrastructure rather than migrate to Policyfiles, avoiding disruptive cookbook and role changes. The central strategy is to divide production into six Availability Zone–based Chef environments, limiting deployment blast radius while preserving existing workflows. A canary environment and staggered release train provide earlier detection of configuration problems and safer fleet-wide rollouts. ## Why Slack Avoided Policyfiles - Policyfiles could have improved long-term safety by replacing roles and environments. - Migrating dozens of teams and their cookbooks would have required substantial effort. - Slack concluded that the short-term disruption and migration risk outweighed the benefits. - Instead, the team extended its existing EC2 framework without requiring cookbook or role changes. ## Splitting Production Chef Environments - Previously, all production nodes used one shared Chef environment. - Cron-triggered Chef runs were staggered across Availability Zones to prevent simultaneous fleet-wide changes. - This reduced the impact of bad changes on existing nodes, but newly provisioned instances immediately consumed the latest version from the shared environment. - During large scale-out events, a broken configuration could therefore spread rapidly to many new nodes. - Slack split production into six environments: `prod-1` through `prod-6`. - Service teams still launch instances as `prod`; internally, nodes are assigned to a numbered environment based on their Availability Zone. - Updates to one environment now affect only the nodes mapped to that environment. ## Extending Poptart Bootstrap - Slack’s base AMIs include `Poptart Bootstrap`, which runs through `cloud-init` during instance startup. - It creates the node’s Chef object, configures DNS, and posts success or failure notifications to Slack. - Slack extended it to inspect the node’s AZ ID and select the appropriate numbered production environment. - This automatically distributes new nodes across isolated Chef environments without requiring service teams to change their provisioning process. ## Canary Deployments and the Release Train - Cookbook changes are promoted: - To sandbox at the top of the hour - To development environments through a Kubernetes cron job - To production beginning at 30 minutes past the hour - `prod-1` acts as the canary production environment. - It receives the latest changes hourly when new cookbook artifacts exist. - This tests changes in real production conditions soon after they are created. - `prod-2` through `prod-6` follow a release train. - A version advances gradually through the production environments. - The next rollout begins only after the previous version has reached `prod-6`. - This sequencing limits the number of affected nodes and makes regressions easier to identify. ## Why `prod-1` Updates Frequently - If the canary waited until a version had passed through every production environment, it would test artifacts containing larger batches of accumulated changes. - Updating `prod-1` frequently keeps the feedback loop close to the originating change. - The remaining production environments provide progressively broader validation after the canary stage. - For example, a new artifact can move from sandbox and dev to `prod-1`, then advance through `prod-2` to `prod-6` while newer artifacts continue entering the canary path. Slack’s approach preserves its existing Chef ecosystem while adding isolation, automated environment assignment, and staged promotion. The result is a safer deployment process that reduces blast radius and catches production issues earlier without forcing widespread application changes.
Read original(opens in new tab)