Regression Testing | Techlist.io

tossAug 11, 20265 min readCurated summary

Toss’s Speed and Quality: Are Commercial Tools Enough? — Tossion

Toss’s QA Platform team built Tossion to replace a rigid commercial test case management system with a platform they could continuously adapt. It unifies test cases, manual and automated results, evidence, history, and release snapshots while preserving a clear record of what was tested at each point in time. The platform later expanded into AI-assisted PR analysis, test case generation, and real-device regression testing, enabling QA processes to evolve at Toss’s speed. ## Why Tossion Was Needed - Previous QA information was scattered across automation results, manual test results, test cases, and decision evidence. - Answering questions such as “How did this test perform last time?” required checking multiple systems. - The commercial TCM could not be modified quickly, and requests for new features often stalled. - Toss needed a platform that could be changed directly as new QA requirements emerged. ## Preserving Test History - Tossion organizes work as: - Project - Suite - Section - Test Case - Test cases continue to change as products evolve, but test runs must preserve historical reality. - When a test run is created, it copies the test case’s assignee, steps, and description instead of merely referencing the current test case. - Status changes create history entries showing who made the decision, when, and against which version. - Closed test runs store snapshots of test cases, comments, and automation results, so reports remain available even if the original test cases are later deleted. - Manually selected test cases override Type or Platform filters. ## Collaborative Test Execution - Test runs progress through `Active → Completed → Closed`. - Assignee-based charts show each person’s remaining work at a glance. - Fields such as Status, Type, Assignee, Version, Platform, RNR, and History are added or removed based on actual usage. - Multiple users can work in the same test run simultaneously: - Active viewers appear as avatars. - Editing locks prevent conflicts. - Locks are automatically released when users leave or disconnect. - Status changes are synchronized without requiring a refresh. - The main benefit is short feedback loops: requested improvements can be built and deployed immediately. ## AI-Assisted Release and PR Analysis - Toss planned to use AI for test case creation, PR analysis, regression automation, and execution. - Tossion analyzes every PR included in a release, separating those with QA labels from those marked as not requiring QA. - The goal is not merely to summarize changes, but to audit whether “no QA needed” classifications are correct. - An agent running on a QA server: - Registers with Tossion. - Polls for work. - Executes the AI already authenticated on the server. - Hundreds of PRs are divided into smaller batches and analyzed in parallel for deeper review. - Results are checked for vague or unusable content, such as: - Missing screens or conditions. - Repeated PR titles. - Raw function names. - Missing reproduction steps, expected results, failure symptoms, or reasoning. - Invalid analyses are retried, and the number of merged PRs is compared with the number of analyzed PRs to detect omissions. - Historical incident data increases risk when a new PR touches files associated with past outages. - Results are uploaded after each batch, allowing interrupted jobs to resume without repeating completed analysis. - The resulting “must-test” list defines the scope of the Sprint test run. ## AI-Generated Test Cases - AI generates test cases for new features, but Tossion controls their placement in the test case tree. - AI returns paths such as `Assets > Account Connection > Select Bank`; Tossion converts them into actual sections, reusing existing sections or creating missing ones. - Reliability is checked in three layers: - AI reviews its own output for missing branches, error cases, and boundary values. - Scripts validate naming, numbering, screen coverage, and requirements coverage. - A separate AI creates an independent test plan based on ISTQB and ISO/IEC 29119 practices. - The plan and generated cases are compared: - Planned but missing cases indicate omissions. - Cases outside the plan indicate unplanned scope. - This catches state-based scenarios that screen-oriented test generation might miss. - Generated cases are uploaded through a portable CLI rather than direct UI integration, reducing dependency on local packages, runtimes, and paths. - Stable cases can later become regression tests. ## Running Regression Tests on Real Devices - A Tossion execution modal specifies: - Device - Build - Test scope - Target test run - Runners connected to Android and iOS physical devices register themselves with Tossion but require administrator approval before receiving work. - Runners report device status every 30 seconds. - Tossion supplies the correct build for installation, ensuring results are tied to a known version. - Users can run the full regression suite or a selected section. - Progress is streamed as scenarios finish, including duration and failure messages. - Results are stored at the step level: - Status - Duration - Error message - Screenshot - Scenario-level video - Automation results can be attached directly to the relevant Sprint test run, making them part of the same record used for manual testing. ## Linking Automation Results to Test Cases - Aggregate reports such as “200 regression tests, 3 failures” do not identify which manual test case rows were covered. - Tossion aims to connect test cases and automation in both directions: - Generate automation code from test cases. - Write automation results back into individual test cases. - This removes the need for QA engineers to manually reconcile separate reports with test case lists. Tossion’s central value is not just test management, but ownership and adaptability. By combining immutable test history, collaborative execution, AI validation, and real-device automation in one extensible platform, Toss’s QA team can adjust its tools and processes as quickly as product requirements change.

Read original

Regression Testing AI Android iOS+3

tossJun 26, 20264 min readCurated summary

While Others Test Toss, We Build the Way to Test.

Every week, Toss releases a new version containing roughly 300–400 code changes, making quality assurance essential to protecting users from broken critical features. The QA Platform team combines smoke tests, regression testing, release monitoring, automation, and AI-assisted tools to make releases reliable and efficient. Its central lesson is that teams often want QA work handled responsibly—not merely more tools to operate themselves—so the team is shifting toward directly owning more of the testing process. ## Weekly Release Quality - QA begins when a Release Candidate is available. - **Toss Doctor** runs smoke tests covering core flows such as login through account deletion. - A pull-request analyzer identifies changed areas, potential impact, bug risk, and testing priorities. - **Toss Checker** performs regression testing to ensure new changes have not damaged stable functionality. - After release, the team monitors crash metrics and decides whether an immediate hotfix is necessary or whether a safer fix can wait for the next release. - Custom dashboards track crashes and hotfixes, including causes and prevention measures. - The team also supports product groups starting QA, improves internal tools, and helps establish organization-wide QA processes. ## Defining Toss’s Quality Standard The team’s goal is to go beyond basic testing and establish consistent quality standards across Toss. - **Reliable releases every time:** Quality must remain dependable week after week, not just during individual successful launches. - **High-quality testing:** The focus is on finding defects that could become real incidents, rather than simply increasing test volume. - **Efficient quality assurance:** Manual repetition alone cannot keep pace with the company’s release speed, so automation and sustainable workflows are necessary. - AI is intended to handle suitable decisions and repetitive work, allowing people to focus on areas requiring human judgment. ## Building the Tossion Platform Commercial tools did not provide enough flexibility for Toss’s release pace and evolving AI experiments, so the team built its own platform, **Tossion**. - Tossion replaced TestRail and brought test-case creation, execution, and result tracking into one system. - Multiple bots were consolidated into **Toss Butler**, optimized for the team’s workflow. - **PRCheck** analyzes pull requests and highlights where testers should focus. - **tcgen** uses PRDs, design documents, and surrounding context to generate initial test cases for review. - An automation testing platform displays manual and automated test results together. - **Crash Trend** tracks crash patterns using metrics tailored to Toss. - A hotfix dashboard categorizes causes and records measures intended to prevent recurrence. - These tools are connected by one objective: handling the growing volume of weekly changes more effectively. ## Learning What Teams Really Need The team initially assumed that making test-case creation easier would encourage more people to test. However, tcgen received less adoption than expected. - Users did not necessarily want better tools for doing QA themselves. - What they really wanted was for someone to perform testing quickly and accurately while taking responsibility for its quality. - Providing a tool could feel less like removing work and more like assigning a new task. - As a result, the team shifted toward directly handling more testing and pursuing tenfold efficiency without transferring responsibility to product teams. ## Staying Flexible as AI Evolves AI has solved many problems but has also made long-term planning difficult. - Initial hypotheses may be only partly correct, as the tcgen experience demonstrated. - Tools and methods can become outdated rapidly as AI capabilities change. - The team discarded an API testing tool, **API Labs**, after only eight hours when it proved misaligned with its goals. - Tossion, Toss Doctor, Toss Checker, and internal skills are designed with replacement in mind rather than as permanent, finished systems. - AI can accelerate tool creation, but people must still define what quality means, establish priorities, and decide what standards must be preserved. The QA Platform team’s ongoing approach is to build adaptable systems, learn from actual usage, and remain willing to replace anything that no longer serves its purpose. Its upcoming work will explore Tossion, the release-gate tools, regression automation, and intelligent AI bots in greater detail.

Read original

Regression Testing AI Test Automation Release Management+4

kakaoJun 15, 20264 min readCurated summary

Automating KakaoTalk Recommendation Metric Analysis with an AI Agent

The post describes Kakao’s use of an AI agent to automate repetitive analysis of KakaoTalk recommendation metrics on an existing Hadoop environment. Rather than building a new platform or granting the model broad permissions, the team documented existing procedures, data definitions, and decision rules in Markdown-based agent skills and context files. The resulting system helps analysts produce draft reports and explore follow-up questions, while humans remain responsible for validating results and making final decisions. ## Repetitive Analysis Is an Ideal Automation Target - Recommendation analysis often begins with simple questions about CTR changes, experiments, or user-group anomalies. - Answering them typically requires: - Connecting to the analysis environment - Finding the right tables - Writing and executing queries - Interpreting results - Repeating the process across dimensions such as age, category, and time - Much of the effort lies in data preparation and extraction rather than interpretation. - The initial goal was for the AI to follow these steps and produce a first-pass analysis without requiring users to handle queries directly. ## Teaching the Agent to Use Hadoop - The team did not build a new analytics platform or add an MCP integration layer. - Existing Hadoop access scripts were sufficient; the missing component was documentation explaining how to use them. - These procedures were packaged as Agent Skills—Markdown files such as `SKILL.md` describing: - How to connect to Hadoop - How to submit queries - How to retrieve and organize results - The `hadoop-butler` plugin bundled these skills for internal use. - The main lesson was that existing infrastructure can often be extended by converting undocumented operational knowledge into instructions an agent can follow. ## Context Documents Improve Analytical Accuracy - Access to data does not guarantee correct analysis. - Context files such as `CLAUDE.md` or `AGENTS.md` documented: - Relevant tables and clusters - Feature definitions, such as `watch_length` and `valid_view` - User and session aggregation rules - Standard metric definitions - This prevented the agent from repeatedly guessing which tables, columns, or aggregation rules to use. - The documentation also captured institutional knowledge that could help new team members, not only AI systems. - Output quality was determined by the quality and precision of the available context. ## AI Produces Drafts; Analysts Continue the Investigation - Natural-language analysis was most useful for recurring tasks such as: - Detecting anomalies - Comparing experiments - Reviewing weekly performance - The agent’s first report helped identify areas for deeper investigation. - Analysts could then ask follow-up questions and refine the analysis conversationally. - AI-generated reports were treated as reviewable drafts, not final conclusions. - Query logic, selected columns, metric definitions, and interpretations still required human verification. ## Plausible but Incorrect Results The agent’s most dangerous errors were not syntax failures; they were queries that executed successfully but produced misleading results. - **Semantic errors** - To count users, the correct field was `user_id`. - The agent once selected the similarly named `session_user_id`, which represented a session-oriented identifier. - The query ran normally, but the resulting user count was wrong. - **Performance errors** - The agent combined several `COUNT(DISTINCT ...)` expressions in one Hive query. - Although valid SQL, this could force processing through a single reducer and make the query extremely slow. - The better approach was to split the calculations by column and run them in parallel. ## Documentation and Regression Testing - Explicit rules were added to context files and skills, including: - Which identifier to use for user-level aggregation - Wrapping column names in backticks - Splitting multiple `COUNT(DISTINCT)` operations into separate queries - Because natural-language instructions can break other behaviors when modified, the team tested them like software. - An MLflow-based end-to-end evaluation pipeline: - Defines expected behavior for each skill - Runs the agent headlessly with `claude -p` - Uses an LLM judge to evaluate tool-call order, execution traces, and final output - Runs regression scenarios before deployment - This made it possible to catch unintended behavior changes before release. The recommended architecture combines four elements: an AI model, precise domain context, an existing execution environment such as Hadoop, and a verification loop. Organizations should first document their established procedures and analytical definitions, then connect the agent to existing tools and test its behavior systematically.

Read original

Regression Testing AI Agents Sql Context Engineering+4

metaFeb 11, 20262 min readCurated summary

The Death of Traditional Testing: Agentic Development Broke a 50-Year-Old Field, JiTTesting Can Revive It

Just-in-Time Tests (JiTTests) are an LLM-driven testing approach designed for fast, agentic software development. Instead of maintaining static test suites, the system generates tests for each code change, simulates likely faults, and reports only meaningful regressions. The goal is to reduce test maintenance and false positives while catching serious bugs before production. ## Limitations of Traditional Testing - Tests are manually written as code changes enter the system. - They must account for both current behavior and unknown future changes. - This often leads to: - Tests that fail to detect relevant bugs. - False positives when intended changes break outdated assumptions. - Ongoing maintenance and review costs. - Agentic development increases the volume and speed of changes, making these problems harder and more expensive to manage. ## How Catching JiTTests Work - A new code change or pull request is submitted. - An LLM infers the likely intent of the change. - The system creates mutants—versions of the code containing deliberately introduced faults. - It generates and runs tests designed to expose those faults. - Rule-based and LLM-based assessors evaluate failures and filter out likely false positives. - Engineers receive a focused report when the system identifies an unexpected behavior change. Because the tests are tailored to a specific change, they can reason about intended behavior and distinguish legitimate updates from regressions. ## Benefits for Agentic Development - Tests are generated on demand and do not remain in the codebase. - There is no ongoing test maintenance or test-code review. - Each test is specific to the change being evaluated. - Tests automatically adapt as the code evolves. - Human attention is required mainly when an actual bug is detected. - Testing shifts from measuring generic code quality to determining whether a specific change introduces a real fault. Catching JiTTests are presented as a way to make testing scale with AI-assisted development by moving routine test creation and maintenance from engineers to automated systems.

Read original

Regression Testing Large Language Models Automated Testing Just In Time Testing+3