While Others Test Toss, We Build the Way to Test.
Every week, Toss releases a new version containing roughly 300–400 code changes, making quality assurance essential to protecting users from broken critical features. The QA Platform team combines smoke tests, regression testing, release monitoring, automation, and AI-assisted tools to make releases reliable and efficient. Its central lesson is that teams often want QA work handled responsibly—not merely more tools to operate themselves—so the team is shifting toward directly owning more of the testing process. ## Weekly Release Quality - QA begins when a Release Candidate is available. - **Toss Doctor** runs smoke tests covering core flows such as login through account deletion. - A pull-request analyzer identifies changed areas, potential impact, bug risk, and testing priorities. - **Toss Checker** performs regression testing to ensure new changes have not damaged stable functionality. - After release, the team monitors crash metrics and decides whether an immediate hotfix is necessary or whether a safer fix can wait for the next release. - Custom dashboards track crashes and hotfixes, including causes and prevention measures. - The team also supports product groups starting QA, improves internal tools, and helps establish organization-wide QA processes. ## Defining Toss’s Quality Standard The team’s goal is to go beyond basic testing and establish consistent quality standards across Toss. - **Reliable releases every time:** Quality must remain dependable week after week, not just during individual successful launches. - **High-quality testing:** The focus is on finding defects that could become real incidents, rather than simply increasing test volume. - **Efficient quality assurance:** Manual repetition alone cannot keep pace with the company’s release speed, so automation and sustainable workflows are necessary. - AI is intended to handle suitable decisions and repetitive work, allowing people to focus on areas requiring human judgment. ## Building the Tossion Platform Commercial tools did not provide enough flexibility for Toss’s release pace and evolving AI experiments, so the team built its own platform, **Tossion**. - Tossion replaced TestRail and brought test-case creation, execution, and result tracking into one system. - Multiple bots were consolidated into **Toss Butler**, optimized for the team’s workflow. - **PRCheck** analyzes pull requests and highlights where testers should focus. - **tcgen** uses PRDs, design documents, and surrounding context to generate initial test cases for review. - An automation testing platform displays manual and automated test results together. - **Crash Trend** tracks crash patterns using metrics tailored to Toss. - A hotfix dashboard categorizes causes and records measures intended to prevent recurrence. - These tools are connected by one objective: handling the growing volume of weekly changes more effectively. ## Learning What Teams Really Need The team initially assumed that making test-case creation easier would encourage more people to test. However, tcgen received less adoption than expected. - Users did not necessarily want better tools for doing QA themselves. - What they really wanted was for someone to perform testing quickly and accurately while taking responsibility for its quality. - Providing a tool could feel less like removing work and more like assigning a new task. - As a result, the team shifted toward directly handling more testing and pursuing tenfold efficiency without transferring responsibility to product teams. ## Staying Flexible as AI Evolves AI has solved many problems but has also made long-term planning difficult. - Initial hypotheses may be only partly correct, as the tcgen experience demonstrated. - Tools and methods can become outdated rapidly as AI capabilities change. - The team discarded an API testing tool, **API Labs**, after only eight hours when it proved misaligned with its goals. - Tossion, Toss Doctor, Toss Checker, and internal skills are designed with replacement in mind rather than as permanent, finished systems. - AI can accelerate tool creation, but people must still define what quality means, establish priorities, and decide what standards must be preserved. The QA Platform team’s ongoing approach is to build adaptable systems, learn from actual usage, and remain willing to replace anything that no longer serves its purpose. Its upcoming work will explore Tossion, the release-gate tools, regression automation, and intelligent AI bots in greater detail.
Read original(opens in new tab)