slack·Jul 14, 2026·3 min readCurated summary

Shipyard: How We Built Slack’s Next-Generation EC2 Platform

Slack’s Shipyard is a next-generation EC2 platform that replaces continuously modified, long-lived instances with immutable, deployable infrastructure artifacts. It combines layered machine images, service-level deployments, progressive rollouts, automated rollback, and short-lived instances to make EC2 operations more predictable and secure. The platform preserves EC2’s flexibility for workloads that cannot easily move to containers while adopting modern application-delivery practices. ## Why Slack Built Shipyard - Slack previously improved its Chef infrastructure with: - Multiple resilient Chef stacks - Versioned cookbook deployments - Safer promotion workflows - Split production environments and signal-based Chef runs - Despite these improvements, the traditional model of continuously updating instances still caused: - Infrastructure drift - Difficult service-level deployments - Coordination problems across infrastructure layers - Increasing operational complexity - Shipyard shifts infrastructure management from mutable instances and constant configuration enforcement to build pipelines, deployable artifacts, and automated safety mechanisms. ## Shipyard’s Core Capabilities - Supports AMD64 and ARM-based Graviton processors. - Supports Ubuntu, RHEL, and Amazon Linux. - Targets workloads that cannot easily migrate to containers, including: - Infrastructure services - Kubernetes worker nodes - Egress network stacks - Integrates with Slack’s Gondola deployment orchestrator for progressive rollouts. - Uses service health metrics to automatically pause deployments or roll back to a known-good version. - Uses layered images so instances perform less work during startup and provision quickly and consistently across regions. ## Configuration Management Without Continuous Mutation - Under the previous model, scheduled Chef jobs repeatedly checked and reapplied configuration. - Shipyard applies configuration during defined lifecycle stages, primarily: - Image baking - Initial instance provisioning - Service deployment - Configuration management tools no longer continuously modify the entire running system. - This reduces background workload, prevents unexpected overwrites, and makes instance behavior easier to understand. ## Fleet Inventory with Peekaboo - Shipyard introduces Peekaboo, a near-real-time inventory system for EC2. - Peekaboo uses AWS EventBridge, OpenSearch, and Lambda to collect cloud events and instance metadata. - It provides: - A fleet-exploration UI - An API for integrations - A command-line interface - Unlike Chef Server, Peekaboo is not limited to Shipyard-managed instances and can provide visibility across the entire EC2 fleet. ## Short-Lived Immutable Instances - Shipyard regularly rotates instances instead of relying on in-place updates. - Limited instance lifespans: - Reduce the window in which vulnerabilities can persist - Prevent long-term configuration drift - Encourage teams to replace instances rather than repair them manually - This makes infrastructure more closely resemble immutable application artifacts. ## The `slack-zero` Golden Image - `slack-zero` is Slack’s shared foundational AMI, maintained by the Compute Platform Team with security and monitoring teams. - It includes: - Operating-system hardening - Networking and service discovery - Monitoring and security agents - Common tools and foundational configuration - Service-specific images are built on top of `slack-zero`, similar to layering application images on a Docker base image. - When foundational components change, Slack produces a new immutable base image and rebuilds downstream service images to inherit updates. ## AWS Image Builder - Slack uses AWS Image Builder instead of Packer to create `slack-zero`. - Image Builder provides lifecycle policies that automatically remove old AMIs and reduce storage costs. - Each new image publishes its latest AMI identifier through an AWS Systems Manager parameter, helping downstream systems discover the current base image. Shipyard’s central recommendation is to treat EC2 infrastructure as versioned, replaceable artifacts rather than mutable machines. Combining trusted base images, automated provisioning, metric-driven deployment controls, fleet-wide inventory, and scheduled instance replacement gives Slack safer and more predictable EC2 operations at scale.

Read original(opens in new tab)

AWSAmazon Ec2OpensearchChef+4

slack·Jun 11, 2026·3 min readCurated summary

Agentic Testing: Where Agents Fit in the E2E Testing Stack

Agentic E2E testing validates whether users can achieve goals rather than enforcing one fixed sequence of UI actions. Slack’s experiment with more than 200 runs found that agents can reliably explore workflows, especially through Playwright MCP, but they are slower and more expensive than deterministic tests. The conclusion is that agents should complement—not replace—traditional E2E tests. ## Goal-Based Testing vs. Fixed Journeys - Traditional tests follow predefined steps: click, type, navigate, and assert. - Agent-driven tests receive a goal and adapt their actions to reach it. - Agents may use different paths to achieve the same result, such as: - Selecting a search suggestion or pressing Enter - Reusing existing navigation state or reopening a view - Adding or skipping intermediate actions - This flexibility improves exploration but introduces tradeoffs in reliability, runtime, and cost. ## Experiment Design Slack evaluated three execution models across more than 200 runs: - **Agent + Playwright MCP** - Uses predefined browser actions and persistent DOM snapshots and logs. - **Agent + Playwright CLI** - Runs Playwright commands through the shell and reassesses the UI after each step. - **Generated Playwright tests** - Produces deterministic test code from natural language, then iteratively refines it. The experiments used Claude Sonnet 4.5 for MCP and CLI workflows and Claude Opus 4.6 for generated tests. All tests ran in non-production Slack workspaces using test data. Two workflows were tested 20 times per configuration: - **Thread Reply:** A simple 15–20-step flow involving channel creation, messaging, thread replies, and verification. - **Search Discovery:** A 25–30-step flow involving search, result navigation, channels, threads, and state verification. Inputs were provided either as detailed natural-language instructions or structured YAML describing actions and expected outcomes. ## Results: Reliability, Cost, and Runtime | Approach | Thread Reply failures | Search Discovery failures | Average runtime | |---|---:|---:|---:| | Agent with Playwright MCP | 0% | Approximately 12% | 5–8 minutes | | Agent with Playwright CLI | Approximately 12% | Approximately 20% | 9–11 minutes | | Generated Playwright tests | Approximately 8% | Approximately 48% | About 3 minutes | - Playwright MCP was the most reliable agentic approach, particularly for simple workflows. - Playwright CLI failed more often due to authentication, navigation timing, and session instability. - Generated tests were fast and reasonably successful on simple flows but degraded sharply as workflows became more complex. - Generated tests often completed 70–80% of a complex workflow before failing on a final interaction or assertion. ## Why Complexity Exposes Differences - MCP maintains a live, stable view of the application through persistent context. - CLI-based agents reconstruct state from updated snapshots, allowing small timing or interpretation inconsistencies to accumulate. - Generated tests can suffer from: - Variable UI state - Imprecise element targeting - Mismatches between generated code and existing page-object abstractions - The results suggest agent-native execution models handle increasingly complex exploratory flows better than generated deterministic tests, despite taking longer. Agentic testing is best used as an exploratory layer for validating user goals and discovering unexpected paths. Deterministic Playwright tests remain preferable for fast, repeatable regression checks, while Playwright MCP appears to be the strongest option when flexible, goal-oriented E2E coverage is needed.

Read original(opens in new tab)

Large Language ModelsAI AgentsModel Context ProtocolClaude+3

slack·May 28, 2026·3 min readCurated summary

Slack AI: The Path to Multi-Cloud

Slack’s AI infrastructure evolved from self-managed SageMaker deployments to managed Amazon Bedrock as Slack sought enterprise-grade security, reliability, lower operational overhead, and faster access to new models. SageMaker provided strong isolation and compliance but required extensive capacity and regional management, while Bedrock simplified operations through managed throughput and on-demand options. Slack’s carefully staged migration achieved zero customer-facing incidents and established a practice of measuring parity, shifting traffic gradually, and monitoring continuously. ## Phase 1: The SageMaker Era - SageMaker offered: - Security and FedRAMP compliance - Control over model availability - An escrow VPC strategy that kept Slack data private while preventing access to providers’ model weights - Slack deployed model containers across multiple AWS regions to support global availability. - Engineering teams had to manage: - Cross-region IAM roles - Routing across model endpoints - Capacity planning - Auto-scaling - The main operational challenges were: - **Scaling latency:** New instances could not start instantly. - **GPU scarcity:** A100 and H100 capacity was often unavailable. - **Over-provisioning:** Idle resources had to be maintained for peak demand. - On-Demand Capacity Reservations and scheduled scaling reduced some problems, but required substantial manual coordination. - SageMaker also created model feature lag because AWS prioritized releasing newer Anthropic models and optimizations through Bedrock. ## Phase 2: Migrating to Amazon Bedrock By mid-2024, Bedrock had matured enough to meet Slack’s security and FedRAMP requirements. - The migration provided: - Managed infrastructure instead of individual GPU instances - Faster access to newly released LLMs - Flexible capacity options for different workload patterns - Slack used: - **Provisioned Throughput (PT)** for predictable, latency-sensitive features such as channel summaries - **On Demand (OD)** for bursty scheduled workloads such as Recap - Bedrock measured capacity in **Model Units (MUs)**, letting Slack plan around token throughput rather than hardware details. ## Executing a Zero-Incident Migration Slack used a gradual, validation-heavy process: - Obtained Legal, Security, and FedRAMP approval before moving production traffic. - Performed load testing to determine the number of MUs needed to match SageMaker capacity. - Compared model quality and latency through A/B tests and evaluation frameworks. - Used feature flags for incremental traffic shifting and immediate rollback. - Ran shadow requests and extensive tests to verify behavioral and performance parity. This approach allowed Slack to move live production traffic without customer-facing incidents. ## Operational Improvements and Remaining Gaps - Bedrock reduced infrastructure maintenance, allowing engineers to focus more on model quality and product features. - Slack could adopt new models weeks or months earlier, improving features such as AI Search with higher-reasoning models. - Capacity planning shifted from reactive scaling to forecasting demand several weeks ahead. - Provisioned Throughput still required maintaining a high baseline of MUs to handle large regional workday surges, creating an over-provisioning challenge for traffic that varies significantly throughout the day. Slack’s migration demonstrates that managed AI infrastructure can improve agility and reliability, but success depends on careful capacity modeling, comprehensive parity testing, gradual rollout, and continuous monitoring.

Read original(opens in new tab)

Large Language ModelsModel ServingGpuAws Bedrock+3

slack·May 5, 2026·3 min readCurated summary

From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

Slack had more than 700 SSH-based operators running critical EMR workloads, creating security risks, operational failures, and barriers to infrastructure modernization. The company replaced these connections with REST-based job submission across eight data regions without downtime. YARN Distributed Shell was the key enabler for migrating arbitrary command-line jobs that lacked dedicated REST APIs. ## How Slack’s SSH Architecture Developed - Airflow originally connected directly to EMR master nodes using `SSHOperator`. - Over time, teams created more than 700 SSH-based jobs for: - Spark and MapReduce workloads - AWS CLI commands - Custom Python scripts - Data-transfer operations such as `hadoop distcp` - The approach was simple but tightly coupled orchestration workers to production clusters. ## Security and Operational Costs of SSH - Direct SSH access expanded the attack surface. - SSH keys had to be distributed and rotated across orchestration workers. - Auditing required correlating activity across multiple systems. - Permissions became complicated, often involving custom security groups and configurations. - Jobs ran on EMR master nodes, causing resource contention. - Restarted Kubernetes pods could break SSH connections. - Long-running processes could become orphaned “zombie” jobs. - Connection failures made job success or failure difficult to determine. - SSH dependencies blocked Spark-on-Kubernetes, EMR on EKS, AWS child-account migration, and better observability. - Slack’s search-indexing pipeline was especially sensitive because it processed terabytes of data daily and supported search for millions of users. ## REST-Based Job Submission - SSH creates a stateful connection whose failure can leave job status ambiguous. - REST APIs provide a durable, server-managed lifecycle: - `POST` submits a job and returns an ID. - `GET` retrieves its status. - `DELETE` cancels it cleanly. - Clients can crash or restart without terminating the underlying job. - Existing systems such as YARN, Trino, and Snowflake use this model. - YARN provides REST submission for Hadoop, Spark, Hive, and MapReduce workloads, but not arbitrary shell commands. ## YARN Distributed Shell - Spark and Hive already had REST-compatible options through Livy and HiveServer2. - The difficult cases were MapReduce and more than 300 CLI-based jobs. - Slack considered custom wrapper services, Ansible or Salt, and creating a new YARN job type. - These alternatives added complexity, security work, or long-term maintenance. - YARN Distributed Shell—implemented through `ApplicationMaster`—could execute arbitrary scripts inside YARN containers. - It used existing YARN APIs and authentication mechanisms, avoiding a custom security layer. ## The Distributed Shell Workflow - Upload a command script to S3, such as an `aws s3 sync` operation. - Submit a YARN application specifying: - The Distributed Shell application master - The S3 script location - Script metadata such as length and timestamp - YARN then: - Allocates a resource-managed container - Downloads and executes the script - Enforces memory and vCore limits - Provides isolation, retries, cancellation, and centralized logging By using REST submission and YARN Distributed Shell, Slack could remove SSH from its EMR data pipelines while preserving support for both standard data-processing jobs and arbitrary command-line workloads.

Read original(opens in new tab)

KubernetesRest ApiApache SparkApache Airflow+3

slack·Apr 13, 2026·4 min readCurated summary

Managing context in long-run agentic applications

Long-running multi-agent applications cannot rely on unlimited conversation history: model APIs are stateless, and growing context windows eventually reduce quality or hit hard limits. Slack’s security-investigation system addresses this by giving agents complementary, purpose-specific context rather than exposing every agent to the full investigation history. Its three main channels—the Director’s Journal, Critic’s Review, and Critic’s Timeline—preserve coherence while leaving room for independent reasoning. ## The Challenge of Long-Run Coherence - Agent frameworks usually maintain continuity by resending the complete message history with every inference request. - Long investigations can involve hundreds of requests and megabytes of generated output. - Context windows impose both: - A hard limit on how much history can be supplied. - A quality limit, because performance may degrade before the window is completely full. - Multi-agent systems need carefully scoped views: - Too little shared context makes agents disconnected from the investigation. - Too much shared context can suppress creativity and encourage confirmation bias. ## Three Complementary Context Channels Slack uses separate information sources for different purposes: - **Director’s Journal** - Structured working memory for the orchestrating Director. - Records decisions, observations, findings, questions, actions, and hypotheses. - **Critic’s Review** - An annotated report evaluating Expert findings. - Includes credibility scores to distinguish reliable evidence from weaker claims. - **Critic’s Timeline** - A consolidated chronological view of findings. - Also attaches credibility scores, helping agents understand the sequence and evidential strength of events. Together, these channels provide continuity without forcing every agent to process the entire raw conversation. ## The Director’s Journal The Director coordinates the investigation by choosing questions, assigning specialist Experts, assessing progress, and deciding when to stop. The Journal gives it persistent working memory across phases and rounds. - The Director is encouraged to update the Journal frequently with short notes. - Entries can represent: - **Decisions** about investigative strategy - **Observations** about emerging patterns - **Findings** representing confirmed facts - **Questions** that remain unresolved - **Actions** taken or planned - **Hypotheses** about what may be happening - Entries can also include: - Priority levels - Follow-up actions - References to supporting evidence - Investigation phase, round number, and timestamp - The journaling tool itself simply accumulates entries; the agents’ prompts explain how to interpret them. ## Maintaining Alignment Across Agents - The Journal creates a shared narrative around the Director’s evolving plan. - It helps the Director: - Track progress - Identify dead ends - Revise investigative direction - Preserve decisions between rounds - Guide other agents toward a conclusion - Every agent receives the current Journal chronologically, along with instructions describing: - The Director’s role - Each agent’s relationship to the Director - The Journal’s purpose - How its entries should influence their work - This approach keeps specialists anchored to the overall investigation without requiring them to read every prior interaction. ## Example Investigation Context The sample Journal comes from an investigation into an apparent kernel-module-loading alert that turned out to be a false positive. - The Director recorded that: - The event originated from a package-installation hook rather than a direct `modprobe` command. - The host appeared to be a personal development workstation. - Root access was expected in that environment. - The detection rule matched “kmod” in a script path rather than confirming module loading. - The Director identified relevant Expert domains, including: - Endpoint telemetry - Identity and access - Configuration management - User behavior - The Journal captured both the preliminary conclusion and remaining verification tasks, such as checking the parent process chain. The design therefore preserves the reasoning trail while keeping it structured and compact. A practical design for long-running agentic systems is to replace indiscriminate transcript accumulation with multiple, curated context channels. Persistent journals can maintain leadership and continuity, while independent reviews and timelines provide evidence-focused context without overwhelming agents or biasing their reasoning.

Read original(opens in new tab)

Large Language ModelsAI AgentsMulti Agent SystemsContext Management+2

slack·Mar 31, 2026·2 min readCurated summary

From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus

Slack needed better client-side observability while migrating edge services to HTTP/3, which uses QUIC over UDP rather than TCP. Existing SaaS tools and Prometheus Blackbox Exporter could not probe HTTP/3 endpoints, so an intern added QUIC support using Go’s `quic-go` library and open-sourced it. The result unified HTTP/1.1, HTTP/2, and HTTP/3 monitoring while making the capability available to the broader Prometheus community. ## Limitations of Legacy Monitoring - Slack used a mix of commercial monitoring services and internal tools for network measurements. - HTTP/3 introduced a major observability gap because it runs over QUIC/UDP. - Existing SaaS solutions lacked built-in HTTP/3 probing. - Prometheus Blackbox Exporter had no native QUIC support. - Without probing at scale, Slack could not reliably measure round-trip times, detect regressions to HTTP/2, or monitor hundreds of thousands of HTTP/3 endpoints. ## Adding QUIC Support to Blackbox Exporter - Intern Sebastian Feliciano selected `quic-go` because of its adoption and first-class Go HTTP client support. - The implementation used an `http3.Transport` with TLS and QUIC configuration: ```go http3Transport := &http3.Transport{ TLSClientConfig: tlsConfig, QUICConfig: &quic.Config{}, } ``` - The new transport was attached to a standard Go `http.Client`. - The implementation preserved Blackbox Exporter’s existing configuration and composability patterns. - Sebastian open-sourced the feature and eventually got it accepted upstream. ## In-House Integration and Operational Benefits - Because upstream review could take longer than the internship timeline, Slack built an internal system around the new functionality. - Grafana now provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics. - Operators can compare protocol performance and correlate it with other telemetry. - Improved visibility supports more accurate alerts and faster debugging of HTTP/3 issues. ## Future Enhancements - **SNI routing tests:** Verify that shared edge infrastructure routes hostnames to the correct backend and presents the correct TLS certificate. - **End-to-end path visualization:** Map network hops between monitoring agents and endpoints to identify latency spikes or packet loss more precisely. ## Broader Lessons - Observability should be established before a major protocol or infrastructure migration. - Filling gaps through open source can benefit both the organization and the wider engineering community. - Supporting emerging protocols such as QUIC early helps future-proof monitoring systems. Slack recommends trying the new QUIC functionality in Prometheus Blackbox Exporter and contributing to its continued development.

Read original(opens in new tab)

ObservabilityGoPrometheusQuic+4

slack·Mar 19, 2026·2 min readCurated summary

How Slack Rebuilt Notifications 📣

Slack rebuilt its notification system to reduce noise by replacing years of inconsistent, tightly coupled behavior with a unified model. The redesign separates what activity users see from how they receive interruptions, while aligning desktop and mobile settings. By combining backend migration strategies, auto-saving controls, and shared UI patterns, Slack aims to make notifications predictable and easier to manage. ## Diagnosing Notification Overload - Notification frustration is common, especially for users in many channels. - Notification issues are among Slack’s top three sources of Customer Experience tickets. - The underlying problem was architectural as well as behavioral: - Desktop and mobile used conflicting preference systems. - Equivalent settings, such as “Nothing” and “Off,” behaved differently across clients. - Activity preferences were coupled to push delivery. - Settings could fall out of sync between desktop and mobile. - Advanced options were scattered or difficult to discover. ## A Unified Notification Model Slack introduced a simpler set of controls: - Channel notifications now offer: - **All new posts** - **Mentions** - **Mute** - Push notifications have separate on/off controls across desktop and mobile. - Advanced features, including mobile “badge all unreads,” are easier to find. - Global preferences use consistent structure and language. - Simplified preference logic improves synchronization between clients. ## Refactoring Preferences Safely - Slack migrated users from four conflicting preference systems to a unified model. - The new model separates: - Desktop activity: **Everything** or **Mentions** - Desktop push: `desktop_push_enabled` set to `true` or `false` - Mobile activity and push behavior: **Everything**, **Mentions**, or **Nothing** - Rather than changing millions of database records directly, Slack used read-time interpretation to preserve backward compatibility and allow rollback. - Existing “Off” settings now behave as “Mentions” with push disabled. - A backfill populated the new desktop push preference based on users’ previous settings. - This preserves in-app awareness while allowing push interruptions to be controlled independently. ## Auto-Saving and Clearer Controls - The previous modal required users to press **Save**, which caused accidental abandoned changes. - The redesigned interface applies changes immediately through auto-save. - Users can independently choose what activity to see and how they want to receive it. - Shared React components replaced legacy mobile-specific UI code, improving consistency across platforms. - Users can now, for example, view all activity while receiving push notifications only for mentions. Slack’s approach demonstrates that reducing notification noise requires more than a visual redesign. Separating activity from delivery, simplifying preference states, and keeping clients synchronized gives users clearer and more reliable control over interruptions.

Read original(opens in new tab)

ReactState ManagementDatabase MigrationCross Platform Development+1

slack·Dec 1, 2025·3 min readCurated summary

Streamlining Security Investigations with Agents

Slack’s Security Engineering team uses a multi-agent AI system to investigate security alerts across billions of daily events. After finding that a single prompt produced inconsistent results, the team replaced it with a controlled workflow of specialized model invocations, structured outputs, and application-level orchestration. The resulting system improves consistency, enables evidence review, and allocates more capable models only where they add the most value. ## From Prompt Prototype to Controlled Workflow - The initial prototype consisted of a roughly 300-word prompt defining: - The analyst’s role - Available data sources - Investigation methodology - Report formatting - Response classifications - A stdio-based MCP server safely exposed selected security data sources to the model. - A coding-agent CLI served as the prototype execution environment. - Results varied significantly: - Sometimes the model cross-referenced evidence effectively. - Other times it reached convenient or unsupported conclusions too quickly. - Prompt refinements helped somewhat, but prompts were ultimately too limited for fine-grained process control. ## Structured Investigation Tasks - Slack decomposed the investigation into a sequence of model invocations. - Each invocation performs one well-defined task and returns a structured output. - The application chains these tasks together and passes only the necessary context between stages. - Structured outputs use JSON schemas to constrain model responses. - They improve predictability, but can still fail when schemas are too complex and remain vulnerable to hallucination or attempts to circumvent constraints. - Guidance such as “question your evidence” became an explicit workflow step rather than merely an instruction in a prompt. ## Persona-Based Agent Architecture - Slack drew inspiration from research on meta-prompting, multi-persona collaboration, and security tabletop exercises. - Instead of representing multiple personas within one model call, Slack implemented each persona as an independent model invocation. - Every agent/task pair has: - A defined responsibility - A carefully designed output structure - Specific prompts and instructions - Potentially different models and tools - The application orchestrates the agents and controls how knowledge moves through the investigation. ## The Investigation Loop - **Director agent** - Guides the investigation from beginning to end. - Forms questions for domain experts. - Uses a journaling tool to plan and organize progress. - Decides how to continue based on reviewed findings and timelines. - **Expert agents** - Investigate questions using specialized knowledge and data sources. - Slack currently uses four domains: - **Access:** Authentication, authorization, and perimeter services - **Cloud:** Infrastructure, compute, orchestration, and networking - **Code:** Source code and configuration management - **Threat:** Threat intelligence and analysis - **Critic agent** - Acts as a meta-expert reviewing domain findings. - Applies a defined rubric to assess quality. - Adds analysis and credibility scores to individual findings. - Helps identify reliable evidence and reduce hallucinations. - Returns its conclusions to the Director, closing the investigation loop. - The Critic’s mildly adversarial role provides an independent challenge to expert conclusions. ## The Knowledge Pyramid - Different stages use different model capabilities and costs. - Domain experts operate at the base: - They query complex data sources. - They may make many tool calls. - Processing their results can consume substantial tokens. - The Critic reviews the larger set of expert findings and identifies the most valuable or credible evidence. - Higher-level reasoning can therefore use a smaller, more focused context. - Because each stage is independent, Slack can select different model versions, prompts, tools, and output formats for different tasks. Slack’s main recommendation is to treat complex agent behavior as an explicitly orchestrated workflow rather than relying on one increasingly elaborate prompt. Specialized agents, structured outputs, independent critique, and tiered model usage provide more predictable investigations while preserving the flexibility of AI-assisted analysis.

Read original(opens in new tab)

Large Language ModelsAI AgentsModel Context ProtocolPrompt Engineering+3

slack·Nov 19, 2025·3 min readCurated summary

Android VPAT journey

Slack’s Android VPAT review uncovered recurring accessibility issues after the company’s 2024 UI redesign. Slack addressed problems involving error announcements, headings, form labels, list counts, and workspace reordering through changes to UI components and TalkBack support. The review also showed that some vendor recommendations required interpretation against Android conventions, while keyboard navigation remains future work. ## Background and Triage - A VPAT documents how a product aligns with accessibility standards and helps customers evaluate software. - Slack commissioned a third-party VPAT in 2024 after its IA4 redesign. - Straightforward issues, such as poor color contrast and missing image labels, were assigned immediately. - The remaining Android findings were grouped into recurring themes: - Inaccessible error messages - Missing semantic headings - Unclear edit-field labels - Incorrect list item counts - Inaccessible workspace drag-and-drop - Strikethrough information not conveyed to screen readers - Errors communicated through color alone - Keyboard navigation and focus, which remain future work because Android’s large-form-factor support is limited ## Making Error Messages Accessible - Invalid form submissions displayed errors visually, but TalkBack did not announce them. - Slack addressed both primary error patterns: - `OutlinedTextField` now announces errors positioned below the field. - Error-type `SKBanner` components now announce their messages to screen-reader users. - Users can therefore understand both that a field is invalid and why it failed without swiping through the screen. ## Adding Semantic Headings - Missing headings made page structure difficult for screen-reader users to understand and navigate. - Slack added headings in lists, including the Preferences page. - The team did not classify top app-bar titles as headings because testing with other Android applications showed that this is not a consistent Android convention. ## Providing Persistent Edit-Field Context - Some fields relied only on placeholder text, which disappears after text is entered. - This can make the field’s purpose unclear, particularly for users with cognitive impairments. - The team explored difficult cases such as the main search field and message input area. - Because of space limitations, the message input received no ideal redesign. - Slack added a search icon to the search field, giving it a persistent visual cue even after the placeholder disappears. ## Correcting List Item Counts - TalkBack incorrectly counted decorative dividers as list items. - For example, a bottom sheet with five actual rows and two dividers was announced as containing seven items. - Slack introduced `SKListAccessibilityDelegate` for `SKListAdapter`. - The delegate overwrites accessibility `CollectionInfo` with the correct number of meaningful list items. ## Making Workspace Reordering Accessible - Dragging workspaces requires dexterity that some users may not have. - Slack added an explicit Edit mode with visible six-dot drag handles for each workspace. - TalkBack users can now use custom “Move before” and “Move after” actions from the accessibility context menu. - These actions are available through a three-finger tap or TalkBack gestures such as `L` and `r`. - A Done button exits Edit mode and removes the drag handles. Slack’s experience demonstrates that Android accessibility improvements often require both component-level fixes and alternative interaction models. Teams should validate screen-reader behavior directly, distinguish decorative elements from meaningful content, provide persistent context for inputs, and offer accessible alternatives to gesture- or dexterity-dependent interactions.

Read original(opens in new tab)

AccessibilityAndroidKeyboard NavigationJetpack Compose+2

slack·Nov 6, 2025·3 min readCurated summary

Build better software to build software better

Slack’s backend build pipeline for Quip and Slack Canvas once took 60 minutes, delaying feedback and slowing delivery. The team improved build performance by applying familiar software-engineering techniques—caching, parallelization, precise interfaces, and careful decomposition—using Bazel. The central argument is that build systems should be designed like high-performance programs: do less work, distribute unavoidable work, and define work units rigorously. ## Modeling Builds as Dependency Graphs - Applications can be represented as directed acyclic graphs of source files, intermediate artifacts, and deployable outputs. - A backend artifact depends on Python files, while a frontend artifact depends on TypeScript files. - Changing a Python file should rebuild the backend but not unrelated frontend components. - Clearly defined graph nodes allow build systems to optimize work rather than rebuilding everything. ## Caching and Hermetic Work - Caching avoids repeating expensive operations by storing outputs for known inputs. - The article uses a cached recursive `factorial()` function as an analogy: - The input is the cache key. - The return value is the cached artifact. - Effective caching requires work to be: - **Hermetic:** dependent only on explicitly provided inputs. - **Idempotent:** producing the same output for the same inputs. - Cache hit rate matters: poorly defined work units produce more cache misses. ## Granular Cache Units - Caching an entire `process_images(images, transforms)` operation is inefficient because changing one image invalidates the result for every image. - A more granular design caches `process_image(image, transform)` independently. - The higher-level operation can then reuse cached results and process only new image-transform combinations. - Smaller, well-defined units generally improve cache reuse and reduce rebuild time. ## Parallelizing Independent Work - Image processing can also be distributed across CPU threads using `ThreadPoolExecutor`. - Parallel work requires: - Completely specified inputs and outputs. - The ability to transfer data across thread, process, or network boundaries. - Handling completion and failure in any order. - APIs must document ordering guarantees; the threaded example returns images in completion order rather than input order. - Work-unit granularity affects scalability: - Too few large tasks limit available parallelism. - Too many tiny tasks may introduce coordination overhead. - The appropriate balance depends on the workload. ## Applying These Principles to Bazel - Bazel represents builds as directed acyclic graphs made of targets. - Each target defines: - Its input or dependency files. - Its output files. - The commands that transform inputs into outputs. - This structure provides the foundation for caching and parallel execution, just as explicit function inputs and outputs enable those optimizations in application code. The practical recommendation is to design build steps as small, hermetic, idempotent, and independently executable units. Combined with Bazel’s dependency graph, this lets teams avoid unnecessary work, maximize cache hits, and run independent tasks concurrently—turning slow build pipelines into faster sources of developer feedback.

Read original(opens in new tab)

TypeScriptPythonCachingBazel+4

slack·Oct 23, 2025·3 min readCurated summary

Advancing Our Chef Infrastructure: Safety Without Disruption

Slack chose to improve its existing Chef and EC2 infrastructure rather than migrate to Policyfiles, avoiding disruptive cookbook and role changes. The central strategy is to divide production into six Availability Zone–based Chef environments, limiting deployment blast radius while preserving existing workflows. A canary environment and staggered release train provide earlier detection of configuration problems and safer fleet-wide rollouts. ## Why Slack Avoided Policyfiles - Policyfiles could have improved long-term safety by replacing roles and environments. - Migrating dozens of teams and their cookbooks would have required substantial effort. - Slack concluded that the short-term disruption and migration risk outweighed the benefits. - Instead, the team extended its existing EC2 framework without requiring cookbook or role changes. ## Splitting Production Chef Environments - Previously, all production nodes used one shared Chef environment. - Cron-triggered Chef runs were staggered across Availability Zones to prevent simultaneous fleet-wide changes. - This reduced the impact of bad changes on existing nodes, but newly provisioned instances immediately consumed the latest version from the shared environment. - During large scale-out events, a broken configuration could therefore spread rapidly to many new nodes. - Slack split production into six environments: `prod-1` through `prod-6`. - Service teams still launch instances as `prod`; internally, nodes are assigned to a numbered environment based on their Availability Zone. - Updates to one environment now affect only the nodes mapped to that environment. ## Extending Poptart Bootstrap - Slack’s base AMIs include `Poptart Bootstrap`, which runs through `cloud-init` during instance startup. - It creates the node’s Chef object, configures DNS, and posts success or failure notifications to Slack. - Slack extended it to inspect the node’s AZ ID and select the appropriate numbered production environment. - This automatically distributes new nodes across isolated Chef environments without requiring service teams to change their provisioning process. ## Canary Deployments and the Release Train - Cookbook changes are promoted: - To sandbox at the top of the hour - To development environments through a Kubernetes cron job - To production beginning at 30 minutes past the hour - `prod-1` acts as the canary production environment. - It receives the latest changes hourly when new cookbook artifacts exist. - This tests changes in real production conditions soon after they are created. - `prod-2` through `prod-6` follow a release train. - A version advances gradually through the production environments. - The next rollout begins only after the previous version has reached `prod-6`. - This sequencing limits the number of affected nodes and makes regressions easier to identify. ## Why `prod-1` Updates Frequently - If the canary waited until a version had passed through every production environment, it would test artifacts containing larger batches of accumulated changes. - Updating `prod-1` frequently keeps the feedback loop close to the originating change. - The remaining production environments provide progressively broader validation after the canary stage. - For example, a new artifact can move from sandbox and dev to `prod-1`, then advance through `prod-2` to `prod-6` while newer artifacts continue entering the canary path. Slack’s approach preserves its existing Chef ecosystem while adding isolation, automated environment assignment, and staged promotion. The result is a safer deployment process that reduces blast radius and catches production issues earlier without forcing widespread application changes.

Read original(opens in new tab)

KubernetesAws Ec2ChefCanary Deployment+3

slack·Oct 7, 2025·4 min readCurated summary

Deploy Safety: Reducing customer impact from change

Slack’s Deploy Safety Program reduced customer-impact hours by 90% from its peak by focusing on safer change across all deployment systems, rather than optimizing individual services in isolation. The program combines measurable reliability goals, automated detection and rollback, blast-radius reduction, and cultural change. Its core lesson is to invest broadly, measure results, and expand approaches that demonstrably reduce customer impact without slowing development. ## Defining the Reliability Problem - Slack became increasingly mission-critical, raising customer expectations for reliability. - In analysis of customer-facing incidents, 73% were triggered by Slack-induced change, especially code deployments. - Incidents occurred across hundreds of services and multiple deployment systems, producing inconsistent levels of customer impact. - Customers reported that interruptions became significantly more disruptive after roughly 10 minutes. - Earlier reliability efforts often focused on individual deployment systems or services, leading to manual processes that slowed innovation and reduced engineering morale. ## North Star Goals and the Deploy Safety Manifesto The initial program goals applied to Slack’s highest-importance services: - Detect and automatically remediate deployment problems within 10 minutes. - Detect and manually remediate problems within 20 minutes. - Identify problematic deployments before they reach 10% of the fleet. - Preserve Slack’s engineering and development velocity. These goals later evolved into a Deploy Safety Manifesto covering all deployment systems and processes, including: - Automated safety improvements. - Deployment guardrails. - Changes to engineering practices and safety culture. ## Measuring Customer Impact Slack defined its primary program metric as: - **Hours of customer impact from high-severity and selected medium-severity change-triggered incidents.** The metric is an imperfect proxy for customer sentiment because: - Incident severity reflects current or anticipated impact, not always the final customer experience. - Medium-severity incidents require additional filtering to determine whether their actual impact is relevant. - It can be difficult to connect an individual engineering project directly to changes in customer sentiment. Slack evaluates the metric using four principles: - Measure outcomes rather than activity. - Distinguish real measurements from proxy metrics. - Apply subjective criteria consistently. - Regularly validate the metric against feedback from leaders who speak directly with customers. ## Choosing Where to Invest At the beginning of the program, Slack did not know which projects would produce the greatest benefit or when results would appear. Incident data is inherently delayed, while customers are experiencing reliability problems immediately. The investment strategy therefore emphasized: - Broad initial investment and a bias toward action. - Addressing known customer pain first. - Expanding successful projects and repeatable patterns. - Reducing investment in areas with limited impact. - Maintaining a flexible roadmap that could change as results emerged. Projects were prioritized according to whether they could: - Detect deployment problems earlier. - Improve automatic remediation time. - Improve manual rollback and remediation time. - Reduce severity by limiting deployment blast radius. ## Improving Webapp Backend Deployments Slack identified Webapp backend deployments as the largest source of change-triggered incidents and iteratively improved their safety: - Built automated metric monitoring. - Added automatic alerts and manual rollback procedures to validate alignment with customer impact. - Introduced automatic deployments and rollback. - Demonstrated that repeated automatic rollbacks could keep customer impact below 10 minutes. - Expanded monitoring to additional metrics. - Optimized manual rollback processes. - Added manual rollback capability for the frontend. - Began consolidating deployment practices through a centralized orchestration system inspired by ReleaseBot and AWS Pipelines. - Extended metrics-based deployment and automatic remediation beyond Bedrock and Kubernetes. These improvements made Webapp backend, frontend, and some infrastructure deployments significantly safer, with continued quarter-over-quarter improvement. ## Iterative Expansion Slack applied the same pattern across other areas: - Try an intervention. - Measure whether customer impact improves. - Invest further when the approach succeeds. - Reuse successful patterns in other systems. - Reduce or redirect investment when results are limited. The article notes that some efforts, such as faster mobile-app issue detection, were successful, while others produced less noticeable improvements. Slack’s experience suggests that deployment safety works best as an ongoing program: establish measurable customer-focused goals, automate detection and recovery, control blast radius, and continuously replicate proven practices without sacrificing delivery speed.

Read original(opens in new tab)

KubernetesIncident ManagementAws PipelinesDeployment Orchestration+2

slack·Sep 4, 2025·3 min readCurated summary

Building Slack’s Anomaly Event Response

Slack’s Anomaly Event Response (AER) is designed to close the gap between detecting suspicious activity and stopping it. By combining real-time monitoring, adaptive analytics, and automated session termination, AER can disrupt high-confidence attacks within minutes rather than hours or days. Slack presents it as a built-in security capability for Enterprise Grid customers that works without additional tools or security staff. ## Shared Responsibility for Securing Slack - Slack processes billions of daily interactions from tens of millions of weekly users. - Enterprise customers receive audit logs covering hundreds of platform actions. - Specialized anomaly logs flag activity such as: - Irregular logins - Malware uploads - Unexpected data transfers - Audit logs provide early warning but traditionally require security personnel or third-party systems to interpret and act on them. - AER provides automated response for customers that lack the resources or infrastructure to build those integrations. - Advanced customers can still combine AER with customized security controls. ## Configurable Threat Detection AER focuses on common indicators of account compromise, data exfiltration, and automated abuse: - Access from Tor exit nodes - Excessive downloading - Data scraping through non-native automation tools - Session fingerprint mismatches - Unexpected API-call volumes or patterns - Unusual user agents, including virtual or non-standard clients Organizations can choose which anomaly types should terminate sessions and which should only be logged. Notification settings are also configurable, with alerts available for organization owners and security administrators through email or Slack. ## Detection Engine - The detection engine analyzes billions of Slack events each day. - It combines rule-based heuristics with dynamic thresholds. - Thresholds are calibrated to each enterprise’s historical usage patterns. - This prevents normal high-volume activity in one organization from being treated as anomalous in another. - Adaptive thresholds help reduce false positives while allowing Slack to refine detection sensitivity over time. ## AER Architecture AER consists of three main components: - **Detection engine:** Identifies suspicious activity and creates anomaly audit payloads. - **Decision framework:** Validates detected behavior and determines whether it qualifies for automated response. - **Response orchestrator:** Carries out the configured response, including terminating user sessions. The overall flow is: 1. Suspicious user activity is analyzed. 2. An anomaly is detected. 3. The AER controller determines whether it is a supported anomaly and whether the organization’s settings require action. 4. Associated user sessions may be terminated. 5. The event is always recorded in audit logs. 6. Customer notifications are sent according to configured preferences. AER’s practical value is that it turns anomaly detection into immediate containment, helping organizations interrupt attack chains before attackers can complete data theft or compromise.

Read original(opens in new tab)

CybersecurityAnomaly DetectionThreat DetectionSession Management+3

slack·Apr 14, 2025·3 min readCurated summary

Optimizing Our E2E Pipeline

Slack optimized its monorepo E2E pipeline by avoiding frontend rebuilds when a pull request contains no frontend changes. Using `git diff` to detect relevant changes and serving recent frontend artifacts from S3 through an internal CDN, the team reduced build frequency by 60% and cut end-to-end pipeline time from roughly 10 minutes to 2 minutes. The changes also lowered storage and compute costs and improved test reliability. ## The Cost of Unnecessary Frontend Builds - Slack’s E2E pipeline validates frontend, backend, database, and service changes before merging into `main`. - Previously, every run rebuilt the frontend, even when a pull request changed only backend or unrelated files. - A typical pipeline included: - About 5 minutes for the frontend build - Deployment to QA - More than 200 E2E tests taking another 5 minutes - With hundreds of pull requests merged daily, redundant builds caused: - Thousands of unnecessary builds each week - Nearly a gigabyte of S3 data per build - Terabytes of duplicate stored artifacts - Significant developer and cloud-compute costs ## Conditional Frontend Builds - Slack used `git diff` with three-dot notation to compare the checked-out branch against `main`. - If frontend files had changed, the pipeline ran a new frontend build. - If no frontend changes were detected, the build step was skipped. - Git analyzed the repository’s more than 100,000 tracked files in only a few seconds. ## Reusing Prebuilt Assets - When a new build was unnecessary, the pipeline located a recent frontend build already stored in AWS S3. - The selected artifact was still in production, ensuring the E2E tests used sufficiently current frontend assets. - An internal CDN served those assets to the QA environment. - S3 naming and asset-management conventions made it possible to find an appropriate artifact in under three seconds on average. ## Results and Additional Benefits - Frontend build frequency fell by 60%. - Average E2E pipeline time dropped from about 10 minutes to 2 minutes. - Monthly savings included hundreds of hours of compute and developer waiting time. - S3 usage decreased by several terabytes per month. - Test flakiness reached its lowest measured level, partly because asset delivery became more consistent. - The work also exposed legacy systems and generated a backlog of future maintenance improvements. Slack’s experience demonstrates that pipelines should not automatically repeat expensive steps when their inputs have not changed. Detecting affected files and reusing trustworthy build artifacts can substantially improve speed, reliability, and cost without requiring a wholesale rewrite of the CI/CD system.

Read original(opens in new tab)

CI/CDGitMonorepoCdn+4