Chef | Techlist.io

slackJul 14, 20263 min readCurated summary

Shipyard: How We Built Slack’s Next-Generation EC2 Platform

Slack’s Shipyard is a next-generation EC2 platform that replaces continuously modified, long-lived instances with immutable, deployable infrastructure artifacts. It combines layered machine images, service-level deployments, progressive rollouts, automated rollback, and short-lived instances to make EC2 operations more predictable and secure. The platform preserves EC2’s flexibility for workloads that cannot easily move to containers while adopting modern application-delivery practices. ## Why Slack Built Shipyard - Slack previously improved its Chef infrastructure with: - Multiple resilient Chef stacks - Versioned cookbook deployments - Safer promotion workflows - Split production environments and signal-based Chef runs - Despite these improvements, the traditional model of continuously updating instances still caused: - Infrastructure drift - Difficult service-level deployments - Coordination problems across infrastructure layers - Increasing operational complexity - Shipyard shifts infrastructure management from mutable instances and constant configuration enforcement to build pipelines, deployable artifacts, and automated safety mechanisms. ## Shipyard’s Core Capabilities - Supports AMD64 and ARM-based Graviton processors. - Supports Ubuntu, RHEL, and Amazon Linux. - Targets workloads that cannot easily migrate to containers, including: - Infrastructure services - Kubernetes worker nodes - Egress network stacks - Integrates with Slack’s Gondola deployment orchestrator for progressive rollouts. - Uses service health metrics to automatically pause deployments or roll back to a known-good version. - Uses layered images so instances perform less work during startup and provision quickly and consistently across regions. ## Configuration Management Without Continuous Mutation - Under the previous model, scheduled Chef jobs repeatedly checked and reapplied configuration. - Shipyard applies configuration during defined lifecycle stages, primarily: - Image baking - Initial instance provisioning - Service deployment - Configuration management tools no longer continuously modify the entire running system. - This reduces background workload, prevents unexpected overwrites, and makes instance behavior easier to understand. ## Fleet Inventory with Peekaboo - Shipyard introduces Peekaboo, a near-real-time inventory system for EC2. - Peekaboo uses AWS EventBridge, OpenSearch, and Lambda to collect cloud events and instance metadata. - It provides: - A fleet-exploration UI - An API for integrations - A command-line interface - Unlike Chef Server, Peekaboo is not limited to Shipyard-managed instances and can provide visibility across the entire EC2 fleet. ## Short-Lived Immutable Instances - Shipyard regularly rotates instances instead of relying on in-place updates. - Limited instance lifespans: - Reduce the window in which vulnerabilities can persist - Prevent long-term configuration drift - Encourage teams to replace instances rather than repair them manually - This makes infrastructure more closely resemble immutable application artifacts. ## The `slack-zero` Golden Image - `slack-zero` is Slack’s shared foundational AMI, maintained by the Compute Platform Team with security and monitoring teams. - It includes: - Operating-system hardening - Networking and service discovery - Monitoring and security agents - Common tools and foundational configuration - Service-specific images are built on top of `slack-zero`, similar to layering application images on a Docker base image. - When foundational components change, Slack produces a new immutable base image and rebuilds downstream service images to inherit updates. ## AWS Image Builder - Slack uses AWS Image Builder instead of Packer to create `slack-zero`. - Image Builder provides lifecycle policies that automatically remove old AMIs and reduce storage costs. - Each new image publishes its latest AMI identifier through an AWS Systems Manager parameter, helping downstream systems discover the current base image. Shipyard’s central recommendation is to treat EC2 infrastructure as versioned, replaceable artifacts rather than mutable machines. Combining trusted base images, automated provisioning, metric-driven deployment controls, fleet-wide inventory, and scheduled instance replacement gives Slack safer and more predictable EC2 operations at scale.

Read original

Chef AWS Amazon Ec2 Opensearch+4

slackOct 23, 20253 min readCurated summary

Advancing Our Chef Infrastructure: Safety Without Disruption

Slack chose to improve its existing Chef and EC2 infrastructure rather than migrate to Policyfiles, avoiding disruptive cookbook and role changes. The central strategy is to divide production into six Availability Zone–based Chef environments, limiting deployment blast radius while preserving existing workflows. A canary environment and staggered release train provide earlier detection of configuration problems and safer fleet-wide rollouts. ## Why Slack Avoided Policyfiles - Policyfiles could have improved long-term safety by replacing roles and environments. - Migrating dozens of teams and their cookbooks would have required substantial effort. - Slack concluded that the short-term disruption and migration risk outweighed the benefits. - Instead, the team extended its existing EC2 framework without requiring cookbook or role changes. ## Splitting Production Chef Environments - Previously, all production nodes used one shared Chef environment. - Cron-triggered Chef runs were staggered across Availability Zones to prevent simultaneous fleet-wide changes. - This reduced the impact of bad changes on existing nodes, but newly provisioned instances immediately consumed the latest version from the shared environment. - During large scale-out events, a broken configuration could therefore spread rapidly to many new nodes. - Slack split production into six environments: `prod-1` through `prod-6`. - Service teams still launch instances as `prod`; internally, nodes are assigned to a numbered environment based on their Availability Zone. - Updates to one environment now affect only the nodes mapped to that environment. ## Extending Poptart Bootstrap - Slack’s base AMIs include `Poptart Bootstrap`, which runs through `cloud-init` during instance startup. - It creates the node’s Chef object, configures DNS, and posts success or failure notifications to Slack. - Slack extended it to inspect the node’s AZ ID and select the appropriate numbered production environment. - This automatically distributes new nodes across isolated Chef environments without requiring service teams to change their provisioning process. ## Canary Deployments and the Release Train - Cookbook changes are promoted: - To sandbox at the top of the hour - To development environments through a Kubernetes cron job - To production beginning at 30 minutes past the hour - `prod-1` acts as the canary production environment. - It receives the latest changes hourly when new cookbook artifacts exist. - This tests changes in real production conditions soon after they are created. - `prod-2` through `prod-6` follow a release train. - A version advances gradually through the production environments. - The next rollout begins only after the previous version has reached `prod-6`. - This sequencing limits the number of affected nodes and makes regressions easier to identify. ## Why `prod-1` Updates Frequently - If the canary waited until a version had passed through every production environment, it would test artifacts containing larger batches of accumulated changes. - Updating `prod-1` frequently keeps the feedback loop close to the originating change. - The remaining production environments provide progressively broader validation after the canary stage. - For example, a new artifact can move from sandbox and dev to `prod-1`, then advance through `prod-2` to `prod-6` while newer artifacts continue entering the canary path. Slack’s approach preserves its existing Chef ecosystem while adding isolation, automated environment assignment, and staged promotion. The result is a safer deployment process that reduces blast radius and catches production issues earlier without forcing widespread application changes.

Read original

Chef Kubernetes Aws Ec2 Canary Deployment+3

datadogAug 23, 20173 min readCurated summary

Scaling support with Vagrant and Terraform

Datadog’s Solutions Team uses reproducible virtual environments to investigate customer issues across diverse operating systems, kernels, and integrations. Vagrant simplifies local VM creation, while provisioning scripts eliminate repeated installation and configuration work. Terraform extends the same approach to shared AWS environments, enabling teams to provision, preserve, and collaborate on sandboxes quickly. ## Reproducing Customer Environments with Vagrant - Containers are useful, but virtual machines are better when reproducing specific operating systems, kernels, orchestrators, or complex infrastructure. - Vagrant provides a simple workflow: - `vagrant init` - `vagrant up` - `vagrant ssh` - The main challenge is not creating a VM, but installing and configuring the technologies needed to match a customer’s environment. - With more than 200 integrations, engineers cannot be experts in every technology they may need to troubleshoot. ## Standardizing Setup with Provisioning Scripts - Vagrant provisioning supports tools such as Chef, Puppet, Ansible, and ordinary shell scripts. - Datadog stores reusable reproduction environments in a shared GitHub repository. - Each sandbox includes: - A `Vagrantfile` - A `setup.sh` provisioning script - A `data` directory for configuration files and supporting scripts - A `README.md` with usage information - Engineer-specific values, such as hostnames and tags, are kept in a local `.sandbox.conf.sh` file. - Once a sandbox exists, an engineer can run `vagrant up` and begin reproducing the customer issue within minutes. - The directory hierarchy organizes sandboxes by operating system, version or provider, and technology—for example, Ubuntu Xenial with Kafka. ## Sharing Remote Environments with Terraform - Terraform provides similar infrastructure management for remote cloud instances, including AWS EC2. - The team reuses the same `setup.sh` and `data` files for both Vagrant and Terraform, avoiding duplicate configuration work. - Each sandbox adds a `.tf` file that: - Creates an EC2 instance - Copies required data files - Executes the provisioning script remotely - A shared Terraform module handles common infrastructure tasks, while a `tf.example` file helps engineers create new configurations. - This preserves the same repository structure and workflow while extending sandboxes from local VMs to remote environments. ## Benefits for Team Collaboration - Remote sandboxes can remain available without consuming engineers’ local RAM. - Proper network security allows teammates to access and share environments. - Engineers can reproduce previously configured integrations during live customer interactions. - Investigations can continue across time zones, allowing teams to hand off urgent issues without rebuilding the environment. The overall recommendation is to treat reproduction environments as reusable infrastructure: encode installation and configuration steps once, store them in version control, and use Vagrant for local testing and Terraform for persistent, shared cloud sandboxes.

Read original

Chef PostgreSQL Docker & Containers Terraform+4