AI Didn’t Replace QA; It Expanded It (opens in new tab)
Generative AI has not replaced QA at LINE Album; it has expanded QA’s scope and influence. The team found that QA productivity depends less on executing tests quickly than on organizing and interpreting large amounts of scattered information. By embedding AI into event-driven quality workflows, QA engineers now focus more on risk assessment, test strategy, and final decisions. ## QA as a Quality Architect - QA operates across the entire product lifecycle: planning, development, testing, release, and post-release feedback. - Its responsibilities include: - Identifying design risks during planning - Assessing the impact of code changes - Designing test strategies - Validating releases - Connecting user feedback and operational data to product improvements - QA information comes from many sources: - Planning and technical documents - Slack discussions and decisions - Jira tickets and pull requests - Automated test scripts and logs - App Store and Google Play reviews - The central challenge is therefore managing information volume and complexity, not merely increasing testing speed. ## From AI Assistant to AI-Driven Workflow - Initially, AI was used interactively to: - Summarize documents - Draft test cases - Organize bug reports - Document reproduction steps - This improved individual productivity but required QA engineers to manually collect and prepare information. - LINE Album QA instead built an automated quality platform with more than 30 workflows. - AI now reacts automatically to events such as: - Jira issue creation - Code changes and pull requests - Test execution - User feedback collection - AI gathers, analyzes, and structures quality information, while QA engineers interpret risks and make decisions. ## Scheduling and Webhook Automation ### Scheduled Analysis - Scheduled workflows periodically collect and summarize quality data. - Examples include: - Daily App Store review classification - API test result summaries sent to Slack - UI automation reports - Weekly QA activity and issue reports - QA engineers spend less time gathering data and more time evaluating risks and verifying important findings. ### Webhook-Based Analysis - Webhook workflows run immediately when quality-related events occur. - Examples include: - Summarizing the potential impact of merged code changes - Creating meeting notes when Slack discussions end - Analyzing and visualizing automated test results - This allows the team to recognize important quality signals much earlier. ## The AI-Supported QA Workday - UI tests run through MagicPod for Android and iOS, with results updated in Jira and shared in Slack. - Failed tests trigger analysis to determine whether they are flaky tests and identify possible causes. - Pytest-based API tests are similarly reported to Jira and Slack. - Daily Scrum workflows automatically provide: - Current test progress - Scrum board and issue dashboard links - Unresolved issues - Jira mentions requiring QA attention - App reviews are analyzed daily, classified as positive or negative, translated into Japanese and Korean, and summarized for the team. - During focused work periods, QA engineers use AI-generated information to plan quality activities, execute tests, monitor workflows, and summarize relevant discussions or documents. - End-of-day workflows summarize completed work and remaining issues. ## AI as a Test Design Partner - By 2026, AI generated approximately 90% of LINE Album QA’s test-case drafts. - Simple prompting produced many generic scenarios but failed to capture: - The reason a feature was introduced - Historical defect patterns - Effects on existing user flows - The team improved results by supplying broader context, including: - Specifications and development tickets - Change rationale - Previous Jira issues - Test history - Recurring bug patterns ### Multi-Agent Test Generation - An orchestrator coordinates five specialized sub-agents: - **Plan-Analyzer:** Examines requirements, feature descriptions, and images. - **Dev-Analyzer:** Adds implementation and development-ticket context. - **TestCase-Generator:** Produces normal, exceptional, boundary-value, platform-specific, and prioritized scenarios. - **TestCase-Validator:** Checks coverage, traceability, completeness, Given/When/Then structure, priorities, and platform coverage. - **Quality-Inspector:** Uses prior feedback and quality evaluations to improve future generations. - The workflow expands testing beyond stated requirements by incorporating defects that have historically occurred. - Validation feedback creates an iterative loop, making the output more executable and useful over time. AI is most valuable when it is connected to the organization’s accumulated context and quality signals—not when it is used merely as a chat-based drafting tool. The recommended approach is to automate information collection and analysis while keeping QA responsible for interpretation, prioritization, and final quality decisions.