Model Context Protocol - Slack

slackJun 11, 20263 min readCurated summary

Agentic Testing: Where Agents Fit in the E2E Testing Stack

Agentic E2E testing validates whether users can achieve goals rather than enforcing one fixed sequence of UI actions. Slack’s experiment with more than 200 runs found that agents can reliably explore workflows, especially through Playwright MCP, but they are slower and more expensive than deterministic tests. The conclusion is that agents should complement—not replace—traditional E2E tests. ## Goal-Based Testing vs. Fixed Journeys - Traditional tests follow predefined steps: click, type, navigate, and assert. - Agent-driven tests receive a goal and adapt their actions to reach it. - Agents may use different paths to achieve the same result, such as: - Selecting a search suggestion or pressing Enter - Reusing existing navigation state or reopening a view - Adding or skipping intermediate actions - This flexibility improves exploration but introduces tradeoffs in reliability, runtime, and cost. ## Experiment Design Slack evaluated three execution models across more than 200 runs: - **Agent + Playwright MCP** - Uses predefined browser actions and persistent DOM snapshots and logs. - **Agent + Playwright CLI** - Runs Playwright commands through the shell and reassesses the UI after each step. - **Generated Playwright tests** - Produces deterministic test code from natural language, then iteratively refines it. The experiments used Claude Sonnet 4.5 for MCP and CLI workflows and Claude Opus 4.6 for generated tests. All tests ran in non-production Slack workspaces using test data. Two workflows were tested 20 times per configuration: - **Thread Reply:** A simple 15–20-step flow involving channel creation, messaging, thread replies, and verification. - **Search Discovery:** A 25–30-step flow involving search, result navigation, channels, threads, and state verification. Inputs were provided either as detailed natural-language instructions or structured YAML describing actions and expected outcomes. ## Results: Reliability, Cost, and Runtime | Approach | Thread Reply failures | Search Discovery failures | Average runtime | |---|---:|---:|---:| | Agent with Playwright MCP | 0% | Approximately 12% | 5–8 minutes | | Agent with Playwright CLI | Approximately 12% | Approximately 20% | 9–11 minutes | | Generated Playwright tests | Approximately 8% | Approximately 48% | About 3 minutes | - Playwright MCP was the most reliable agentic approach, particularly for simple workflows. - Playwright CLI failed more often due to authentication, navigation timing, and session instability. - Generated tests were fast and reasonably successful on simple flows but degraded sharply as workflows became more complex. - Generated tests often completed 70–80% of a complex workflow before failing on a final interaction or assertion. ## Why Complexity Exposes Differences - MCP maintains a live, stable view of the application through persistent context. - CLI-based agents reconstruct state from updated snapshots, allowing small timing or interpretation inconsistencies to accumulate. - Generated tests can suffer from: - Variable UI state - Imprecise element targeting - Mismatches between generated code and existing page-object abstractions - The results suggest agent-native execution models handle increasingly complex exploratory flows better than generated deterministic tests, despite taking longer. Agentic testing is best used as an exploratory layer for validating user goals and discovering unexpected paths. Deterministic Playwright tests remain preferable for fast, repeatable regression checks, while Playwright MCP appears to be the strongest option when flexible, goal-oriented E2E coverage is needed.

Read original

Model Context Protocol Large Language Models AI Agents Claude+3

slackDec 1, 20253 min readCurated summary

Streamlining Security Investigations with Agents

Slack’s Security Engineering team uses a multi-agent AI system to investigate security alerts across billions of daily events. After finding that a single prompt produced inconsistent results, the team replaced it with a controlled workflow of specialized model invocations, structured outputs, and application-level orchestration. The resulting system improves consistency, enables evidence review, and allocates more capable models only where they add the most value. ## From Prompt Prototype to Controlled Workflow - The initial prototype consisted of a roughly 300-word prompt defining: - The analyst’s role - Available data sources - Investigation methodology - Report formatting - Response classifications - A stdio-based MCP server safely exposed selected security data sources to the model. - A coding-agent CLI served as the prototype execution environment. - Results varied significantly: - Sometimes the model cross-referenced evidence effectively. - Other times it reached convenient or unsupported conclusions too quickly. - Prompt refinements helped somewhat, but prompts were ultimately too limited for fine-grained process control. ## Structured Investigation Tasks - Slack decomposed the investigation into a sequence of model invocations. - Each invocation performs one well-defined task and returns a structured output. - The application chains these tasks together and passes only the necessary context between stages. - Structured outputs use JSON schemas to constrain model responses. - They improve predictability, but can still fail when schemas are too complex and remain vulnerable to hallucination or attempts to circumvent constraints. - Guidance such as “question your evidence” became an explicit workflow step rather than merely an instruction in a prompt. ## Persona-Based Agent Architecture - Slack drew inspiration from research on meta-prompting, multi-persona collaboration, and security tabletop exercises. - Instead of representing multiple personas within one model call, Slack implemented each persona as an independent model invocation. - Every agent/task pair has: - A defined responsibility - A carefully designed output structure - Specific prompts and instructions - Potentially different models and tools - The application orchestrates the agents and controls how knowledge moves through the investigation. ## The Investigation Loop - **Director agent** - Guides the investigation from beginning to end. - Forms questions for domain experts. - Uses a journaling tool to plan and organize progress. - Decides how to continue based on reviewed findings and timelines. - **Expert agents** - Investigate questions using specialized knowledge and data sources. - Slack currently uses four domains: - **Access:** Authentication, authorization, and perimeter services - **Cloud:** Infrastructure, compute, orchestration, and networking - **Code:** Source code and configuration management - **Threat:** Threat intelligence and analysis - **Critic agent** - Acts as a meta-expert reviewing domain findings. - Applies a defined rubric to assess quality. - Adds analysis and credibility scores to individual findings. - Helps identify reliable evidence and reduce hallucinations. - Returns its conclusions to the Director, closing the investigation loop. - The Critic’s mildly adversarial role provides an independent challenge to expert conclusions. ## The Knowledge Pyramid - Different stages use different model capabilities and costs. - Domain experts operate at the base: - They query complex data sources. - They may make many tool calls. - Processing their results can consume substantial tokens. - The Critic reviews the larger set of expert findings and identifies the most valuable or credible evidence. - Higher-level reasoning can therefore use a smaller, more focused context. - Because each stage is independent, Slack can select different model versions, prompts, tools, and output formats for different tasks. Slack’s main recommendation is to treat complex agent behavior as an explicitly orchestrated workflow rather than relying on one increasingly elaborate prompt. Specialized agents, structured outputs, independent critique, and tiered model usage provide more predictable investigations while preserving the flexibility of AI-assisted analysis.

Read original

Model Context Protocol Large Language Models AI Agents Prompt Engineering+3