slack·Jun 11, 2026·3 min readCurated summary

Agentic Testing: Where Agents Fit in the E2E Testing Stack

Agentic E2E testing validates whether users can achieve goals rather than enforcing one fixed sequence of UI actions. Slack’s experiment with more than 200 runs found that agents can reliably explore workflows, especially through Playwright MCP, but they are slower and more expensive than deterministic tests. The conclusion is that agents should complement—not replace—traditional E2E tests. ## Goal-Based Testing vs. Fixed Journeys - Traditional tests follow predefined steps: click, type, navigate, and assert. - Agent-driven tests receive a goal and adapt their actions to reach it. - Agents may use different paths to achieve the same result, such as: - Selecting a search suggestion or pressing Enter - Reusing existing navigation state or reopening a view - Adding or skipping intermediate actions - This flexibility improves exploration but introduces tradeoffs in reliability, runtime, and cost. ## Experiment Design Slack evaluated three execution models across more than 200 runs: - **Agent + Playwright MCP** - Uses predefined browser actions and persistent DOM snapshots and logs. - **Agent + Playwright CLI** - Runs Playwright commands through the shell and reassesses the UI after each step. - **Generated Playwright tests** - Produces deterministic test code from natural language, then iteratively refines it. The experiments used Claude Sonnet 4.5 for MCP and CLI workflows and Claude Opus 4.6 for generated tests. All tests ran in non-production Slack workspaces using test data. Two workflows were tested 20 times per configuration: - **Thread Reply:** A simple 15–20-step flow involving channel creation, messaging, thread replies, and verification. - **Search Discovery:** A 25–30-step flow involving search, result navigation, channels, threads, and state verification. Inputs were provided either as detailed natural-language instructions or structured YAML describing actions and expected outcomes. ## Results: Reliability, Cost, and Runtime | Approach | Thread Reply failures | Search Discovery failures | Average runtime | |---|---:|---:|---:| | Agent with Playwright MCP | 0% | Approximately 12% | 5–8 minutes | | Agent with Playwright CLI | Approximately 12% | Approximately 20% | 9–11 minutes | | Generated Playwright tests | Approximately 8% | Approximately 48% | About 3 minutes | - Playwright MCP was the most reliable agentic approach, particularly for simple workflows. - Playwright CLI failed more often due to authentication, navigation timing, and session instability. - Generated tests were fast and reasonably successful on simple flows but degraded sharply as workflows became more complex. - Generated tests often completed 70–80% of a complex workflow before failing on a final interaction or assertion. ## Why Complexity Exposes Differences - MCP maintains a live, stable view of the application through persistent context. - CLI-based agents reconstruct state from updated snapshots, allowing small timing or interpretation inconsistencies to accumulate. - Generated tests can suffer from: - Variable UI state - Imprecise element targeting - Mismatches between generated code and existing page-object abstractions - The results suggest agent-native execution models handle increasingly complex exploratory flows better than generated deterministic tests, despite taking longer. Agentic testing is best used as an exploratory layer for validating user goals and discovering unexpected paths. Deterministic Playwright tests remain preferable for fast, repeatable regression checks, while Playwright MCP appears to be the strongest option when flexible, goal-oriented E2E coverage is needed.

Read original(opens in new tab)

PlaywrightLarge Language ModelsAI AgentsModel Context Protocol+3

cloudflare·Apr 15, 2026·3 min readCurated summary

Browser Run: give your agents a browser

Cloudflare is renaming Browser Rendering to Browser Run and positioning it as a full browser platform for AI agents. It provides remotely hosted Chrome sessions that agents can control, observe, debug, record, and scale globally, while allowing humans to intervene when necessary. The update expands access through CDP and MCP, making existing automation tools and AI coding assistants compatible with Cloudflare’s browser infrastructure. ## Browser Run for AI Agents - Agents can navigate websites, read content, fill out forms, extract data, take screenshots, and verify results. - Browser sessions run on Cloudflare’s global network, reducing infrastructure and browser-maintenance requirements. - Sessions can scale dynamically and open near users for lower latency. - The platform now supports up to 120 concurrent browsers, up from 30. ## Observability and Human Intervention - **Live View** shows an agent’s browser session in real time, making it easier to confirm success or diagnose failures. - **Human in the Loop** allows agents to transfer control when they encounter login screens or unusual edge cases. - A human can resolve the issue and return control to the agent. - **Session Recordings** capture DOM changes, interactions, and navigation for debugging and postmortem analysis. ## Browser Control Options Browser Run supports several levels of automation: - Low-level control through the Chrome DevTools Protocol (CDP). - Higher-level automation with Puppeteer and Playwright. - Quick Actions for simpler tasks. - WebMCP for websites that expose agent-discoverable actions. ### Chrome DevTools Protocol - Browser Run now exposes CDP directly through a WebSocket endpoint. - Existing CDP-based frameworks, scripts, and agent tools can connect with minimal changes. - CDP provides capabilities beyond Puppeteer and Playwright, including JavaScript debugging. - Raw protocol messages can be sent directly to models, potentially reducing token usage. - Developers can connect from any language or environment without creating a Cloudflare Worker. - Self-hosted Chrome scripts can be migrated by changing the browser WebSocket URL and adding Cloudflare authentication headers. ### MCP Client Support - MCP clients such as Claude Desktop, Cursor, Codex, and OpenCode can use Browser Run as a remote browser. - Cloudflare supports the `chrome-devtools-mcp` package, which provides browser automation, debugging, and performance-analysis capabilities. - Configuration requires pointing the MCP server to Browser Run’s CDP endpoint and supplying an API token. ### WebMCP - WebMCP is intended to make websites more reliable for AI agents. - Websites can declare actions that agents can discover and call directly. - This addresses the limitations of a web originally designed primarily for human navigation. ## Overall Direction Cloudflare’s update combines hosted browser infrastructure, multiple automation interfaces, real-time visibility, replayable sessions, and human fallback. The goal is to make browser-based agents more dependable in production while avoiding the operational burden of managing Chrome infrastructure themselves. For teams building web-using agents, Browser Run offers a practical path from self-hosted or local browser automation to scalable, observable remote sessions, especially when existing CDP, Puppeteer, Playwright, or MCP tooling is already in use.

Read original(opens in new tab)

PlaywrightAI AgentsCloudflareHuman In The Loop+4

toss·Dec 23, 2025Original article

Tax Refund Automation: AI (opens in new tab)

At Toss Income, QA Manager Suho Jung successfully automated complex E2E testing for diverse tax refund services by leveraging AI as specialized virtual team members. By shifting from manual coding to a "human-as-orchestrator" model, a single person achieved the productivity of a four-to-five-person automation team within just five months. This approach overcame the inherent brittleness of testing long, React-based flows that are subject to frequent policy changes and external system dependencies. ### Challenges in Tax Service Automation The complexity of tax refund services presented unique hurdles that made traditional manual automation unsustainable: * **Multi-Step Dependencies:** Each refund flow averages 15–20 steps involving internal systems, authentication providers, and HomeTax scraping servers, where a single timing glitch can fail the entire test. * **Frequent UI and Policy Shifts:** Minor UI updates or new tax laws required total scenario reconfigurations, making hard-coded tests obsolete almost immediately. * **Environmental Instability:** Issues such as "Target closed" errors during scraping, differing domain environments, and React-specific hydration delays caused constant test flakiness. ### Building an AI-Driven QA Team Rather than using AI as a simple autocomplete tool, the project assigned specific "personas" to different AI models to handle distinct parts of the lifecycle: * **SDET Agent (Claude Sonnet 4.5):** Acted as the lead developer, responsible for designing the Page Object Model (POM) architecture, writing test logic, and creating utility functions. * **Documentation Specialist:** Automatically generated daily retrospectives and updated technical guides by analyzing daily git commits. * **Git Master:** Managed commit history and PR descriptions to ensure high-quality documentation of the project’s evolution. * **Pair Programmers (Cursor & Codex):** Handled real-time troubleshooting, type errors, and comparative analysis of different test scripts. ### Technical Solutions for React and Policy Logic The team implemented several sophisticated technical strategies to ensure test stability: * **React Interaction Readiness:** To solve "Element is not clickable" errors, they developed a strategy that waits not just for visibility, but for event handlers to bind to the DOM (Hydration). * **Safe Interaction Fallbacks:** A standard `click` utility was created that attempts a Playwright click, then a native keyboard 'Enter' press, and finally a JS dispatch to ensure interactions succeed even during UI transitions. * **Dynamic Consent Flow Utility:** A specialized system was built to automatically detect and handle varying "Terms of Service" agreements across different sub-services (Tax Secretary, Hidden Refund, etc.) through a single unified function. * **Test Isolation:** Automated scripts were used to prevent `userNo` (test ID) collisions, ensuring 35+ complex scenarios could run in parallel without data interference. ### Integrated Feedback and Reporting The automation was integrated directly into internal communication channels to create a tight feedback loop: * **Messenger Notifications:** Every test run sends a report including execution time, test IDs, and environment data to the team's messenger. * **Automated Failure Analysis:** When a test fails, the AI automatically posts the error log, the specific failed step, a tracking EventID, and a screenshot as a thread reply for immediate debugging. * **Human-AI Collaboration:** This structure shifted the QA's role from writing code to discussing failures and policy changes within the messenger threads. The success of this 5-month experiment suggests that for high-complexity environments, the future of QA lies in "AI Orchestration." Instead of focusing on writing selectors, QA engineers should focus on defining problems and managing the AI agents that build the architecture.

PlaywrightAIAI AgentsReact+5

toss·Dec 1, 2025Original article

Toss Income Tax Refund Service: (opens in new tab)

Toss Income’s QA team transitioned from traditional manual testing and rigid class-based Page Object Models (POM) to a stateless Functional POM to keep pace with rapid deployment cycles. This shift allowed them to manage complex tax refund logic and frequent UI changes with high reliability and minimal maintenance overhead. By treating automation as a modular assembly of functions, they successfully reduced verification times from four hours to twenty minutes while significantly increasing test coverage. ### Transitioning to Functional POM * Replaced stateful classes and complex inheritance with stateless functions that receive a `page` object as input and return the updated `page` as output. * Adopted a clear naming convention (e.g., `gotoLoginPage`, `enterPhonePin`, `verifyRefundAmount`) to ensure that test cases read like human-readable scenarios. * Centralized UI selectors and interaction logic within these functions, allowing developers to update a single point of truth when UI text or button labels change. ### Modularizing the User Journey * Segmented the complex tax refund process into four distinct modules: Login/Terms, Deduction Checks, Refund/Payment Info, and Reporting. * Developed independent, reusable functions for specific data inputs—such as medical or credit card deductions—which can be assembled like "Lego blocks" to create new test scenarios rapidly. * Decoupled business logic from UI interactions, enabling the team to create diverse test cases by simply varying parameters like amounts or dates. ### Robust Interaction and Page Management * Implemented a 4-step "Robust Click Strategy" to eliminate flakiness caused by React rendering timings, sequentially trying an Enter key press, a standard click, a forced click, and finally a direct JavaScript execution. * Created a `waitForNetworkIdleSafely` utility that prevents test failures during polling or background network activity by prioritizing UI anchors over strict network idleness. * Standardized page transition handling with a `getLatestNonScrapePage` utility, ensuring the `currentPage` object always points to the most recent active tab or redirect window. ### Integration and Performance Outcomes * Achieved a 600% increase in test coverage, expanding from 5 core scenarios to 35 comprehensive automated flows. * Reduced the time required to respond to UI changes by 98%, as modifications are now localized to a single POM function rather than dozens of test files. * Established a 24/7 automated validation system that provides immediate feedback on functional correctness, data integrity (tax amount accuracy), and performance metrics via dedicated communication channels. For engineering teams operating in high-velocity environments, adopting a stateless, functional approach to test automation is a highly effective way to reduce technical debt. By focusing on modularity and implementing fallback strategies for UI interactions, teams can transform QA from a final bottleneck into a continuous, data-driven validation layer that supports rapid experimentation.

PlaywrightTypeScriptTest AutomationFunctional Programming+3