Large Language Models - Slack

slackJun 11, 20263 min readCurated summary

Agentic Testing: Where Agents Fit in the E2E Testing Stack

Agentic E2E testing validates whether users can achieve goals rather than enforcing one fixed sequence of UI actions. Slack’s experiment with more than 200 runs found that agents can reliably explore workflows, especially through Playwright MCP, but they are slower and more expensive than deterministic tests. The conclusion is that agents should complement—not replace—traditional E2E tests. ## Goal-Based Testing vs. Fixed Journeys - Traditional tests follow predefined steps: click, type, navigate, and assert. - Agent-driven tests receive a goal and adapt their actions to reach it. - Agents may use different paths to achieve the same result, such as: - Selecting a search suggestion or pressing Enter - Reusing existing navigation state or reopening a view - Adding or skipping intermediate actions - This flexibility improves exploration but introduces tradeoffs in reliability, runtime, and cost. ## Experiment Design Slack evaluated three execution models across more than 200 runs: - **Agent + Playwright MCP** - Uses predefined browser actions and persistent DOM snapshots and logs. - **Agent + Playwright CLI** - Runs Playwright commands through the shell and reassesses the UI after each step. - **Generated Playwright tests** - Produces deterministic test code from natural language, then iteratively refines it. The experiments used Claude Sonnet 4.5 for MCP and CLI workflows and Claude Opus 4.6 for generated tests. All tests ran in non-production Slack workspaces using test data. Two workflows were tested 20 times per configuration: - **Thread Reply:** A simple 15–20-step flow involving channel creation, messaging, thread replies, and verification. - **Search Discovery:** A 25–30-step flow involving search, result navigation, channels, threads, and state verification. Inputs were provided either as detailed natural-language instructions or structured YAML describing actions and expected outcomes. ## Results: Reliability, Cost, and Runtime | Approach | Thread Reply failures | Search Discovery failures | Average runtime | |---|---:|---:|---:| | Agent with Playwright MCP | 0% | Approximately 12% | 5–8 minutes | | Agent with Playwright CLI | Approximately 12% | Approximately 20% | 9–11 minutes | | Generated Playwright tests | Approximately 8% | Approximately 48% | About 3 minutes | - Playwright MCP was the most reliable agentic approach, particularly for simple workflows. - Playwright CLI failed more often due to authentication, navigation timing, and session instability. - Generated tests were fast and reasonably successful on simple flows but degraded sharply as workflows became more complex. - Generated tests often completed 70–80% of a complex workflow before failing on a final interaction or assertion. ## Why Complexity Exposes Differences - MCP maintains a live, stable view of the application through persistent context. - CLI-based agents reconstruct state from updated snapshots, allowing small timing or interpretation inconsistencies to accumulate. - Generated tests can suffer from: - Variable UI state - Imprecise element targeting - Mismatches between generated code and existing page-object abstractions - The results suggest agent-native execution models handle increasingly complex exploratory flows better than generated deterministic tests, despite taking longer. Agentic testing is best used as an exploratory layer for validating user goals and discovering unexpected paths. Deterministic Playwright tests remain preferable for fast, repeatable regression checks, while Playwright MCP appears to be the strongest option when flexible, goal-oriented E2E coverage is needed.

Read original

Large Language Models AI Agents Model Context Protocol Claude+3

slackMay 28, 20263 min readCurated summary

Slack AI: The Path to Multi-Cloud

Slack’s AI infrastructure evolved from self-managed SageMaker deployments to managed Amazon Bedrock as Slack sought enterprise-grade security, reliability, lower operational overhead, and faster access to new models. SageMaker provided strong isolation and compliance but required extensive capacity and regional management, while Bedrock simplified operations through managed throughput and on-demand options. Slack’s carefully staged migration achieved zero customer-facing incidents and established a practice of measuring parity, shifting traffic gradually, and monitoring continuously. ## Phase 1: The SageMaker Era - SageMaker offered: - Security and FedRAMP compliance - Control over model availability - An escrow VPC strategy that kept Slack data private while preventing access to providers’ model weights - Slack deployed model containers across multiple AWS regions to support global availability. - Engineering teams had to manage: - Cross-region IAM roles - Routing across model endpoints - Capacity planning - Auto-scaling - The main operational challenges were: - **Scaling latency:** New instances could not start instantly. - **GPU scarcity:** A100 and H100 capacity was often unavailable. - **Over-provisioning:** Idle resources had to be maintained for peak demand. - On-Demand Capacity Reservations and scheduled scaling reduced some problems, but required substantial manual coordination. - SageMaker also created model feature lag because AWS prioritized releasing newer Anthropic models and optimizations through Bedrock. ## Phase 2: Migrating to Amazon Bedrock By mid-2024, Bedrock had matured enough to meet Slack’s security and FedRAMP requirements. - The migration provided: - Managed infrastructure instead of individual GPU instances - Faster access to newly released LLMs - Flexible capacity options for different workload patterns - Slack used: - **Provisioned Throughput (PT)** for predictable, latency-sensitive features such as channel summaries - **On Demand (OD)** for bursty scheduled workloads such as Recap - Bedrock measured capacity in **Model Units (MUs)**, letting Slack plan around token throughput rather than hardware details. ## Executing a Zero-Incident Migration Slack used a gradual, validation-heavy process: - Obtained Legal, Security, and FedRAMP approval before moving production traffic. - Performed load testing to determine the number of MUs needed to match SageMaker capacity. - Compared model quality and latency through A/B tests and evaluation frameworks. - Used feature flags for incremental traffic shifting and immediate rollback. - Ran shadow requests and extensive tests to verify behavioral and performance parity. This approach allowed Slack to move live production traffic without customer-facing incidents. ## Operational Improvements and Remaining Gaps - Bedrock reduced infrastructure maintenance, allowing engineers to focus more on model quality and product features. - Slack could adopt new models weeks or months earlier, improving features such as AI Search with higher-reasoning models. - Capacity planning shifted from reactive scaling to forecasting demand several weeks ahead. - Provisioned Throughput still required maintaining a high baseline of MUs to handle large regional workday surges, creating an over-provisioning challenge for traffic that varies significantly throughout the day. Slack’s migration demonstrates that managed AI infrastructure can improve agility and reliability, but success depends on careful capacity modeling, comprehensive parity testing, gradual rollout, and continuous monitoring.

Read original

Large Language Models Model Serving Gpu Aws Bedrock+3

slackApr 13, 20264 min readCurated summary

Managing context in long-run agentic applications

Long-running multi-agent applications cannot rely on unlimited conversation history: model APIs are stateless, and growing context windows eventually reduce quality or hit hard limits. Slack’s security-investigation system addresses this by giving agents complementary, purpose-specific context rather than exposing every agent to the full investigation history. Its three main channels—the Director’s Journal, Critic’s Review, and Critic’s Timeline—preserve coherence while leaving room for independent reasoning. ## The Challenge of Long-Run Coherence - Agent frameworks usually maintain continuity by resending the complete message history with every inference request. - Long investigations can involve hundreds of requests and megabytes of generated output. - Context windows impose both: - A hard limit on how much history can be supplied. - A quality limit, because performance may degrade before the window is completely full. - Multi-agent systems need carefully scoped views: - Too little shared context makes agents disconnected from the investigation. - Too much shared context can suppress creativity and encourage confirmation bias. ## Three Complementary Context Channels Slack uses separate information sources for different purposes: - **Director’s Journal** - Structured working memory for the orchestrating Director. - Records decisions, observations, findings, questions, actions, and hypotheses. - **Critic’s Review** - An annotated report evaluating Expert findings. - Includes credibility scores to distinguish reliable evidence from weaker claims. - **Critic’s Timeline** - A consolidated chronological view of findings. - Also attaches credibility scores, helping agents understand the sequence and evidential strength of events. Together, these channels provide continuity without forcing every agent to process the entire raw conversation. ## The Director’s Journal The Director coordinates the investigation by choosing questions, assigning specialist Experts, assessing progress, and deciding when to stop. The Journal gives it persistent working memory across phases and rounds. - The Director is encouraged to update the Journal frequently with short notes. - Entries can represent: - **Decisions** about investigative strategy - **Observations** about emerging patterns - **Findings** representing confirmed facts - **Questions** that remain unresolved - **Actions** taken or planned - **Hypotheses** about what may be happening - Entries can also include: - Priority levels - Follow-up actions - References to supporting evidence - Investigation phase, round number, and timestamp - The journaling tool itself simply accumulates entries; the agents’ prompts explain how to interpret them. ## Maintaining Alignment Across Agents - The Journal creates a shared narrative around the Director’s evolving plan. - It helps the Director: - Track progress - Identify dead ends - Revise investigative direction - Preserve decisions between rounds - Guide other agents toward a conclusion - Every agent receives the current Journal chronologically, along with instructions describing: - The Director’s role - Each agent’s relationship to the Director - The Journal’s purpose - How its entries should influence their work - This approach keeps specialists anchored to the overall investigation without requiring them to read every prior interaction. ## Example Investigation Context The sample Journal comes from an investigation into an apparent kernel-module-loading alert that turned out to be a false positive. - The Director recorded that: - The event originated from a package-installation hook rather than a direct `modprobe` command. - The host appeared to be a personal development workstation. - Root access was expected in that environment. - The detection rule matched “kmod” in a script path rather than confirming module loading. - The Director identified relevant Expert domains, including: - Endpoint telemetry - Identity and access - Configuration management - User behavior - The Journal captured both the preliminary conclusion and remaining verification tasks, such as checking the parent process chain. The design therefore preserves the reasoning trail while keeping it structured and compact. A practical design for long-running agentic systems is to replace indiscriminate transcript accumulation with multiple, curated context channels. Persistent journals can maintain leadership and continuity, while independent reviews and timelines provide evidence-focused context without overwhelming agents or biasing their reasoning.

Read original

Large Language Models AI Agents Multi Agent Systems Context Management+2

slackDec 1, 20253 min readCurated summary

Streamlining Security Investigations with Agents

Slack’s Security Engineering team uses a multi-agent AI system to investigate security alerts across billions of daily events. After finding that a single prompt produced inconsistent results, the team replaced it with a controlled workflow of specialized model invocations, structured outputs, and application-level orchestration. The resulting system improves consistency, enables evidence review, and allocates more capable models only where they add the most value. ## From Prompt Prototype to Controlled Workflow - The initial prototype consisted of a roughly 300-word prompt defining: - The analyst’s role - Available data sources - Investigation methodology - Report formatting - Response classifications - A stdio-based MCP server safely exposed selected security data sources to the model. - A coding-agent CLI served as the prototype execution environment. - Results varied significantly: - Sometimes the model cross-referenced evidence effectively. - Other times it reached convenient or unsupported conclusions too quickly. - Prompt refinements helped somewhat, but prompts were ultimately too limited for fine-grained process control. ## Structured Investigation Tasks - Slack decomposed the investigation into a sequence of model invocations. - Each invocation performs one well-defined task and returns a structured output. - The application chains these tasks together and passes only the necessary context between stages. - Structured outputs use JSON schemas to constrain model responses. - They improve predictability, but can still fail when schemas are too complex and remain vulnerable to hallucination or attempts to circumvent constraints. - Guidance such as “question your evidence” became an explicit workflow step rather than merely an instruction in a prompt. ## Persona-Based Agent Architecture - Slack drew inspiration from research on meta-prompting, multi-persona collaboration, and security tabletop exercises. - Instead of representing multiple personas within one model call, Slack implemented each persona as an independent model invocation. - Every agent/task pair has: - A defined responsibility - A carefully designed output structure - Specific prompts and instructions - Potentially different models and tools - The application orchestrates the agents and controls how knowledge moves through the investigation. ## The Investigation Loop - **Director agent** - Guides the investigation from beginning to end. - Forms questions for domain experts. - Uses a journaling tool to plan and organize progress. - Decides how to continue based on reviewed findings and timelines. - **Expert agents** - Investigate questions using specialized knowledge and data sources. - Slack currently uses four domains: - **Access:** Authentication, authorization, and perimeter services - **Cloud:** Infrastructure, compute, orchestration, and networking - **Code:** Source code and configuration management - **Threat:** Threat intelligence and analysis - **Critic agent** - Acts as a meta-expert reviewing domain findings. - Applies a defined rubric to assess quality. - Adds analysis and credibility scores to individual findings. - Helps identify reliable evidence and reduce hallucinations. - Returns its conclusions to the Director, closing the investigation loop. - The Critic’s mildly adversarial role provides an independent challenge to expert conclusions. ## The Knowledge Pyramid - Different stages use different model capabilities and costs. - Domain experts operate at the base: - They query complex data sources. - They may make many tool calls. - Processing their results can consume substantial tokens. - The Critic reviews the larger set of expert findings and identifies the most valuable or credible evidence. - Higher-level reasoning can therefore use a smaller, more focused context. - Because each stage is independent, Slack can select different model versions, prompts, tools, and output formats for different tasks. Slack’s main recommendation is to treat complex agent behavior as an explicitly orchestrated workflow rather than relying on one increasingly elaborate prompt. Specialized agents, structured outputs, independent critique, and tiered model usage provide more predictable investigations while preserving the flexibility of AI-assisted analysis.

Read original

Large Language Models AI Agents Model Context Protocol Prompt Engineering+3