Toss/AI

17 posts

toss5 min readCurated summary

Toss’s Speed and Quality: Are Commercial Tools Enough? — Tossion

Toss’s QA Platform team built Tossion to replace a rigid commercial test case management system with a platform they could continuously adapt. It unifies test cases, manual and automated results, evidence, history, and release snapshots while preserving a clear record of what was tested at each point in time. The platform later expanded into AI-assisted PR analysis, test case generation, and real-device regression testing, enabling QA processes to evolve at Toss’s speed. ## Why Tossion Was Needed - Previous QA information was scattered across automation results, manual test results, test cases, and decision evidence. - Answering questions such as “How did this test perform last time?” required checking multiple systems. - The commercial TCM could not be modified quickly, and requests for new features often stalled. - Toss needed a platform that could be changed directly as new QA requirements emerged. ## Preserving Test History - Tossion organizes work as: - Project - Suite - Section - Test Case - Test cases continue to change as products evolve, but test runs must preserve historical reality. - When a test run is created, it copies the test case’s assignee, steps, and description instead of merely referencing the current test case. - Status changes create history entries showing who made the decision, when, and against which version. - Closed test runs store snapshots of test cases, comments, and automation results, so reports remain available even if the original test cases are later deleted. - Manually selected test cases override Type or Platform filters. ## Collaborative Test Execution - Test runs progress through `Active → Completed → Closed`. - Assignee-based charts show each person’s remaining work at a glance. - Fields such as Status, Type, Assignee, Version, Platform, RNR, and History are added or removed based on actual usage. - Multiple users can work in the same test run simultaneously: - Active viewers appear as avatars. - Editing locks prevent conflicts. - Locks are automatically released when users leave or disconnect. - Status changes are synchronized without requiring a refresh. - The main benefit is short feedback loops: requested improvements can be built and deployed immediately. ## AI-Assisted Release and PR Analysis - Toss planned to use AI for test case creation, PR analysis, regression automation, and execution. - Tossion analyzes every PR included in a release, separating those with QA labels from those marked as not requiring QA. - The goal is not merely to summarize changes, but to audit whether “no QA needed” classifications are correct. - An agent running on a QA server: - Registers with Tossion. - Polls for work. - Executes the AI already authenticated on the server. - Hundreds of PRs are divided into smaller batches and analyzed in parallel for deeper review. - Results are checked for vague or unusable content, such as: - Missing screens or conditions. - Repeated PR titles. - Raw function names. - Missing reproduction steps, expected results, failure symptoms, or reasoning. - Invalid analyses are retried, and the number of merged PRs is compared with the number of analyzed PRs to detect omissions. - Historical incident data increases risk when a new PR touches files associated with past outages. - Results are uploaded after each batch, allowing interrupted jobs to resume without repeating completed analysis. - The resulting “must-test” list defines the scope of the Sprint test run. ## AI-Generated Test Cases - AI generates test cases for new features, but Tossion controls their placement in the test case tree. - AI returns paths such as `Assets > Account Connection > Select Bank`; Tossion converts them into actual sections, reusing existing sections or creating missing ones. - Reliability is checked in three layers: - AI reviews its own output for missing branches, error cases, and boundary values. - Scripts validate naming, numbering, screen coverage, and requirements coverage. - A separate AI creates an independent test plan based on ISTQB and ISO/IEC 29119 practices. - The plan and generated cases are compared: - Planned but missing cases indicate omissions. - Cases outside the plan indicate unplanned scope. - This catches state-based scenarios that screen-oriented test generation might miss. - Generated cases are uploaded through a portable CLI rather than direct UI integration, reducing dependency on local packages, runtimes, and paths. - Stable cases can later become regression tests. ## Running Regression Tests on Real Devices - A Tossion execution modal specifies: - Device - Build - Test scope - Target test run - Runners connected to Android and iOS physical devices register themselves with Tossion but require administrator approval before receiving work. - Runners report device status every 30 seconds. - Tossion supplies the correct build for installation, ensuring results are tied to a known version. - Users can run the full regression suite or a selected section. - Progress is streamed as scenarios finish, including duration and failure messages. - Results are stored at the step level: - Status - Duration - Error message - Screenshot - Scenario-level video - Automation results can be attached directly to the relevant Sprint test run, making them part of the same record used for manual testing. ## Linking Automation Results to Test Cases - Aggregate reports such as “200 regression tests, 3 failures” do not identify which manual test case rows were covered. - Tossion aims to connect test cases and automation in both directions: - Generate automation code from test cases. - Write automation results back into individual test cases. - This removes the need for QA engineers to manually reconcile separate reports with test case lists. Tossion’s central value is not just test management, but ownership and adaptability. By combining immutable test history, collaborative execution, AI validation, and real-device automation in one extensible platform, Toss’s QA team can adjust its tools and processes as quickly as product requirements change.

Read original(opens in new tab)
toss4 min readCurated summary

While Others Test Toss, We Build the Way to Test.

Every week, Toss releases a new version containing roughly 300–400 code changes, making quality assurance essential to protecting users from broken critical features. The QA Platform team combines smoke tests, regression testing, release monitoring, automation, and AI-assisted tools to make releases reliable and efficient. Its central lesson is that teams often want QA work handled responsibly—not merely more tools to operate themselves—so the team is shifting toward directly owning more of the testing process. ## Weekly Release Quality - QA begins when a Release Candidate is available. - **Toss Doctor** runs smoke tests covering core flows such as login through account deletion. - A pull-request analyzer identifies changed areas, potential impact, bug risk, and testing priorities. - **Toss Checker** performs regression testing to ensure new changes have not damaged stable functionality. - After release, the team monitors crash metrics and decides whether an immediate hotfix is necessary or whether a safer fix can wait for the next release. - Custom dashboards track crashes and hotfixes, including causes and prevention measures. - The team also supports product groups starting QA, improves internal tools, and helps establish organization-wide QA processes. ## Defining Toss’s Quality Standard The team’s goal is to go beyond basic testing and establish consistent quality standards across Toss. - **Reliable releases every time:** Quality must remain dependable week after week, not just during individual successful launches. - **High-quality testing:** The focus is on finding defects that could become real incidents, rather than simply increasing test volume. - **Efficient quality assurance:** Manual repetition alone cannot keep pace with the company’s release speed, so automation and sustainable workflows are necessary. - AI is intended to handle suitable decisions and repetitive work, allowing people to focus on areas requiring human judgment. ## Building the Tossion Platform Commercial tools did not provide enough flexibility for Toss’s release pace and evolving AI experiments, so the team built its own platform, **Tossion**. - Tossion replaced TestRail and brought test-case creation, execution, and result tracking into one system. - Multiple bots were consolidated into **Toss Butler**, optimized for the team’s workflow. - **PRCheck** analyzes pull requests and highlights where testers should focus. - **tcgen** uses PRDs, design documents, and surrounding context to generate initial test cases for review. - An automation testing platform displays manual and automated test results together. - **Crash Trend** tracks crash patterns using metrics tailored to Toss. - A hotfix dashboard categorizes causes and records measures intended to prevent recurrence. - These tools are connected by one objective: handling the growing volume of weekly changes more effectively. ## Learning What Teams Really Need The team initially assumed that making test-case creation easier would encourage more people to test. However, tcgen received less adoption than expected. - Users did not necessarily want better tools for doing QA themselves. - What they really wanted was for someone to perform testing quickly and accurately while taking responsibility for its quality. - Providing a tool could feel less like removing work and more like assigning a new task. - As a result, the team shifted toward directly handling more testing and pursuing tenfold efficiency without transferring responsibility to product teams. ## Staying Flexible as AI Evolves AI has solved many problems but has also made long-term planning difficult. - Initial hypotheses may be only partly correct, as the tcgen experience demonstrated. - Tools and methods can become outdated rapidly as AI capabilities change. - The team discarded an API testing tool, **API Labs**, after only eight hours when it proved misaligned with its goals. - Tossion, Toss Doctor, Toss Checker, and internal skills are designed with replacement in mind rather than as permanent, finished systems. - AI can accelerate tool creation, but people must still define what quality means, establish priorities, and decide what standards must be preserved. The QA Platform team’s ongoing approach is to build adaptable systems, learn from actual usage, and remain willing to replace anything that no longer serves its purpose. Its upcoming work will explore Tossion, the release-gate tools, regression automation, and intelligent AI bots in greater detail.

Read original(opens in new tab)
toss4 min readCurated summary

5. Technical Writer, A Decision to Disappear

Toss’s technical writing team argues that documentation is essential context for AI, but manually maintaining thousands of documents is impossible with only three technical writers serving roughly 4,000 people. Their solution is to automate the technical writer’s work by teaching AI the team’s implicit standards and embedding those standards into reusable Skills. The initial system supported document creation and review, but adoption remained low because users still had to install, invoke, and supply information to the AI manually. ## Why Toss Wanted to Automate Technical Writing - Documentation gives AI the organizational context it needs to work effectively. - Toss has approximately 4,000 employees but only three technical writers. - Reviewing documents individually does not scale, especially in a fast-moving organization where features change or disappear before documentation is complete. - The team’s goal to “eliminate technical writers” means transferring routine writing and editing work to AI, not abandoning documentation quality. ## Teaching AI Technical Writing Principles - The team analyzed existing technical writing review comments to identify how writers evaluate documents. - Existing writing guidelines were converted into explicit principles, such as: - Focus each page on one subject. - Present value before implementation details. - Each principle was supplemented with incorrect and correct examples so AI would understand the intent rather than apply rules mechanically. - Common document types were converted into templates. - Templates include: - Instructions explaining what each section should contain. - `(required)` markers for information that must not be omitted. - For example, an ADR template requires an overview, context, considered alternatives, decision, and rationale, while also allowing optional sections such as expected outcomes and related references. ## Skill for Writing New Documents The document-writing Skill reproduces the four stages a technical writer typically follows: - **Clarify the purpose:** Ask about the project, document goal, audience, level of detail, source materials, and expected structure. - **Design the structure:** Use a standard structure or select a relevant template, such as onboarding guides, meeting notes, or PRDs. - **Write the content:** Apply technical writing and MDX rules while using templates as structural guidance. - **Review the draft:** Check for awkward wording, missing information, and other quality issues. The Skill also distinguishes between required and optional template sections: - Required sections remain in the draft even when source information is incomplete. - Missing information is represented with questions or comments rather than guesses. - Optional sections are omitted when there is not enough source material to complete them. ## Skill for Reviewing and Improving Documents - The team initially converted past review comments into a checklist. - This produced poor results: AI overlooked important issues while generating unnecessary comments. - The problem was that good writing follows relatively stable principles, whereas bad writing can fail in many different ways. - The revised workflow lets AI independently: - Read the technical writing principles. - Analyze the document. - Identify violations. - Explain the issue and suggest revised wording. - Perform a final checklist-based review. - Previous review comments are now used as examples of how principles apply, rather than as a rigid list of required findings. - One example principle requires descriptions of parameters or properties to include their meaning, accepted format, and usage example—not merely a type such as `date: string`. ## Low Adoption Revealed a Usability Problem - Despite creating both Skills, the team found that few employees used them. - Users still had to: - Download and install the Skill manually. - Understand CLI-based setup, which was unfamiliar to non-developers. - Remember to invoke the Skill whenever they began writing documentation. - Find and provide all relevant source materials themselves. - The team concluded that improving the AI’s capabilities was not enough; the workflow also had to reduce the effort required from users. The main lesson is that AI-based documentation succeeds only when organizational knowledge, writing principles, and templates are encoded clearly—and when the system is integrated into everyday work so employees do not have to remember to use it or prepare everything manually.

Read original(opens in new tab)
toss4 min readCurated summary

6. Beyond Tools: Standards and Responsibility

Toss’s commerce domain found that reliable organizational knowledge cannot be created by writing more documents or adding automation alone. Sustainable knowledge management requires clear standards for what should be documented, who owns it, how it is maintained, and which sources can be trusted. The proposed solution combines AI-assisted documentation with domain-level responsibility and company-wide governance. ## The Limits of Writing Alone - A commerce wiki consolidated terminology, onboarding material, code references, and policy documents. - This reduced confusion over terms such as “seller” and “store” and gave teams a shared starting point. - However, product and policy changes happened faster than one Technical Writer could document them. - Important knowledge also appeared in policy changes, temporary experiments, and chat discussions that were difficult to track manually. ## Why Culture and Participation Were Not Enough - The team promoted documentation through: - A weekly “Commerce Wiki News” newsletter - A policy-question channel and bot - AI documentation workshops - A documentation guild - These efforts increased requests, wiki usage, and adoption of official terminology. - Participation rarely continued beyond an individual’s first document because documentation was not part of normal work priorities. - Writers lacked guidance on: - What information to preserve - How much detail to include - Which audience to target - How to verify whether a document was correct - Documentation became sustainable only when it was treated as a team responsibility embedded in existing workflows. ## AI Automation Reveals the Governance Problem - AI now creates draft documents nightly from two signals: - Product deployment and policy-change announcements - Questions that the commerce Q&A bot cannot answer - AI gathers supporting context and produces drafts, while humans verify the evidence and approve them. - This removes the burden of starting documents from a blank page. - Automation also exposed new problems: - Duplicate or overlapping documents - Unclear authoritative sources - Outdated policies being used in bot answers - Difficulty distinguishing current policies from completed experiments - Automation can collect and draft information, but it cannot decide who owns a policy or whether a document should still be trusted. ## Knowledge Standards and Governance - The focus shifted from “How do we create more documents?” to “How do we create knowledge people can trust?” - Toss’s knowledge-management standards state that teams should: - Preserve recurring questions, important decisions, and information needed by newcomers. - Organize knowledge so both people and AI can find it. - Connect documents to work tools such as Q&A bots and GitHub. - Assign owners and review cycles to keep information accurate and current. - Possible classification systems include: - **Technical layers** for teams with clear data or system flows - **Service domains** for teams responsible for multiple service areas - **Functional units** for systems with distinct feature boundaries - Information becomes organizational knowledge only when it helps people understand situations and make better decisions, with sufficient context and verification. ## The Role of the Knowledge Committee - The Knowledge Committee defines and maintains company-wide documentation standards and resolves conflicts between organizational rules. - Unlike a voluntary guild, it has designated members with decision-making authority. - Governance operates at two levels: - The Technical Writing Chapter manages shared standards for sources, ownership, document status, and lifecycle. - Individual domains decide how those standards apply locally, including ownership, update schedules, and retirement rules. - This balance prevents both inconsistent practices across teams and overly centralized rules that ignore local realities. - For example, commerce teams may need separate handling for permanent deployments and temporary experiments so expired policies do not remain authoritative. The practical recommendation is to treat knowledge management as an operating system for the organization, not a documentation project. AI can reduce the effort of capturing knowledge, but clear ownership, review processes, lifecycle rules, and governance are necessary to keep that knowledge reliable and useful.

Read original(opens in new tab)
toss3 min readCurated summary

Why Does Our Team’s Documentation Fail? (2)

Documentation succeeds not through individual resolve, but through centralized knowledge, clear purpose, and systems that reduce the fear of being wrong. Interviews across a commerce domain and a server-development chapter show that documentation strategies must match an organization’s existing maturity, audience, and work patterns. AI lowers the cost of writing and sharing knowledge, while also requiring more deliberate document organization. ## Lessons from Early Attempts - The commerce wiki was initially designed as a self-service platform supported by workshops and guilds. - Workshops could encourage a first contribution, but sustaining second, third, and later contributions was difficult. - In the Ads domain, documentation was already strong, so the better approach was to respect existing conventions and help people locate knowledge rather than create another system. - Organizations with little documentation need to build foundational knowledge; organizations with mature documentation need better discovery and maintenance. ## Reducing the Fear of Asking and Writing - Developers often avoid asking questions because doing so publicly reveals what they do not know. - They may also hesitate to publish documentation because they fear their knowledge could be inaccurate. - The team addresses this through: - **“Lee’s Development Consultation Week,”** which normalizes questions and encourages teammates to answer questions Lee cannot address. - **A daily knowledge bot, “Ha,”** which shares short server-development tips automatically. - Correcting or adding to an existing shared post feels easier than writing a complete document from scratch. ## How AI Has Changed Documentation - AI makes it faster to create initial drafts and distribute knowledge through chatbots. - It also enables measurement of knowledge flow, including: - The number of questions asked. - Whether teammates provide answers. - The quality and content of those answers. - The number of new documents and week-over-week growth. - These metrics reveal knowledge gaps and recurring questions without manually reviewing every channel. - AI also creates a need for more detailed internal context than human readers typically require. - Commerce therefore separates: - Central, human-friendly documentation managed by technical writers. - Team-repository documentation containing detailed, team-specific context useful to AI but unnecessary for everyone else. ## Shared Principles and Organizational Differences - Both domains and chapters should centralize knowledge and avoid spreading it across too many channels. - Domains typically: - Connect documentation to products and code. - Change rapidly. - Serve a broad and varied audience. - Chapters typically: - Document conventions, working methods, and professional knowledge. - Change more slowly. - Focus on productivity and capability development. - Have a clearer audience, such as a specific role or discipline. - Domain documentation should be understandable even to non-developers, since developers may also lack context outside their specialties. - Separating guides, capability-based policies, glossaries, and metrics helps different readers find documents suited to their needs. ## Where to Begin - First diagnose the organization’s current documentation maturity by asking what people do when they get stuck: - **Ask coworkers or search chat:** foundational documentation is largely missing. - **Search documents:** assess whether information is easy to find; fill gaps if searches fail. - **Ask an AI or bot:** evaluate whether answers are accurate and whether the underlying documents are complete, current, centralized, and sufficiently contextualized. - Define the specific problem driving documentation, rather than starting with a vague goal. - Begin with a focused need, such as creating a glossary for inconsistent terminology or building references for sharing knowledge with other teams. The practical recommendation is to centralize knowledge, identify the organization’s biggest documentation gap, and build a low-friction system where people can contribute, correct, and consume information without relying solely on personal effort.

Read original(opens in new tab)
toss4 min readCurated summary

Going Beyond Expertise

Technical Writers (TWs) can contribute far beyond writing documentation: they can lead product teams and build systems that turn knowledge into an organizational asset. Toss’s Knowledge System Team created “todoc,” an internal platform that makes documentation easier to write, centralizes scattered knowledge, and enables AI access. Its broader goal is to make documentation emerge naturally from daily work and remain accurate without constant manual maintenance. ## TWs as Product Owners and Makers - The author leads a product team of developers, designers, and TWs. - Their responsibilities include: - Setting product direction, roadmap, and priorities - Interviewing users and bringing insights to the team - Planning features - Building features directly with AI tools - TW expertise is especially valuable because TWs have deeply considered: - Why documents are difficult to read - What makes documentation effective - How information should be structured for AI consumption ## Why Toss Built Todoc Todoc was launched to address weaknesses in Toss’s existing documentation environment. - Static-site-generated documentation required users to: - Clone a repository - Write Markdown - Submit pull requests - Wait for review - This workflow was familiar to developers but created major barriers for designers, PMs, and other non-developers. - Existing documentation tools accumulated outdated policies, unfinished notes, and unexplained content, creating “documentation debt.” - Knowledge was fragmented across: - Static sites - Documentation tools - Code - Collaboration messengers - Individual employees’ knowledge After its beta launch, Todoc grew to more than 500 documents and 40,000 valid pages, with over 1,000 monthly users. ## Todoc’s Four Core Values ### Easy Documentation for Everyone - Anyone can create or edit documents immediately. - Content can be connected from GitHub, documentation tools, internal messengers, and other sources. - The platform removes the technical and procedural barriers to documentation. ### AI-Ready Knowledge - Well-organized documentation can be used by team bots and other AI tools. - Todoc supports API, CLI, and MCP access. - Teams use it for request bots, product specifications, and other workflows. ### A Single Source of Truth - Todoc consolidates scattered sources into complete, centralized documents. - Users can determine which information is current without searching across multiple systems. - The platform serves as the organization’s SSoT (Single Source of Truth). ### Scalable Infrastructure - Teams no longer need to select, build, or maintain their own documentation infrastructure. - Each team can have its own space on a shared platform. - The model is being expanded to Toss affiliates. ## Automating Documentation Quality and Maintenance Lowering the barrier to writing creates a new challenge: maintaining quality. - TW judgment is being converted into: - AI proofreading - Automated document reviews - Bots that generate initial drafts - Todoc is also designed to create documentation automatically from: - Decisions and discussions in internal messengers - Code changes - Ongoing project conversations - The system aims to update documents without relying on someone remembering to maintain them. - It evaluates whether knowledge is still valid by checking: - Whether policies match implemented code - Whether information is actively used - How recently it was updated ## The Evolution of TW Expertise The role is shifting from writing excellent documents manually to designing systems that consistently produce and maintain excellent documentation. - Experience understanding why documents are hard to read becomes standards for human- and AI-readable content. - Judgments about what makes a good document become criteria for AI review and automated editing. - Expertise in identifying outdated information becomes a system for validating knowledge. - TWs increasingly focus on: - Creating places where knowledge can gather - Defining quality standards - Encoding human judgment into systems - Generating documentation through normal work - Keeping knowledge continuously updated The practical vision is an organization where outdated documents trigger their own notifications, project work leaves behind organized records, and recurring explanations are preserved for future employees. Toss’s Technical Writing Chapter is therefore working to systematize TW expertise and establish documentation governance so teams can document effectively without constant manual intervention.

Read original(opens in new tab)
toss3 min readCurated summary

If You Asked a Designer to Make Anything with AI

Toss Design Chapter’s AI Contest invited designers to build anything with AI, resulting in 122 projects over one month. The examples show that designers primarily used AI to improve existing work—making it faster, more persuasive, and higher quality—rather than creating entirely new kinds of work. The article recommends starting with a frustrating, repetitive task or a frequently repeated communication problem. ## Automating Repetitive Work - A color-extraction tool automatically identifies and adjusts colors from images for use in UI. - Color extraction had been an unresolved challenge at Toss because results varied widely by image. - Designers used AI to draft the logic, test it against many sample images, and rapidly refine it. - The resulting system is now used for product-card colors in Toss Shopping. ## Reducing Collaboration Costs with a Personal Bot - A Slack bot was trained on a designer’s knowledge, past discussions, and reference materials. - It creates draft answers to the many design and requirements questions the designer receives each day. - Team members can send the draft as-is or revise it before responding. - The bot learns from those revisions, improving its answers to similar questions over time. - The designer described the result as feeling like becoming “1.5 people,” and other Toss designers began creating their own bots. ## Persuading Through Interactive Prototypes - A designer built a functioning prototype of a stock-trading desktop interface instead of presenting only static screens. - Users could drag panels, rearrange them, and resize windows, with the interface responding accordingly. - Showing the intended interactions directly reduced the risk that design ideas would be misunderstood during development. - The working prototype helped align designers and developers and persuade the product owner. ## Pushing Quality Within Tight Deadlines - AI-generated motion graphics were created for the key visual of Toss Bank’s recruitment website. - Each job category needed its own animation despite a very short schedule. - The designer created the foundational images manually and repeatedly refined Kling prompts to achieve the desired results. - Human-designed starting and ending frames combined with AI-generated motion allowed all category animations to be completed in a single day. ## Four Ways to Start Using AI - **Efficiency:** Hand off one especially annoying repetitive task to AI. - **Replication:** Build a bot to answer questions you repeatedly handle yourself. - **Persuasion:** Turn designs that require verbal explanation into working prototypes. - **Quality:** Use AI to reach a higher level of polish within a limited timeframe. The practical recommendation is to begin with an existing task rather than searching for an entirely new AI application. Choose one area where AI can save time, communicate intent more clearly, or help raise the final quality.

Read original(opens in new tab)
toss4 min readCurated summary

Designing the Work You Do Every Day

A product designer at Toss Bank transformed her personal task-management routine instead of accepting repetitive manual work as unavoidable. She built an AI-powered desktop widget that collects Slack messages, summarizes them into actionable tasks, preserves their context, and helps her focus on prioritization. What began as a personal solution revealed a broader problem shared across roles and spread throughout her team. ## From Manual Task Tracking to Workflow Design - For two and a half years, she manually copied tasks, feedback, discussion links, and requests from Slack into Notion or Slack lists. - As her responsibilities expanded to three teams, daily tasks grew from roughly 10 to more than 20. - She reframed the issue as a product-design problem: - **User:** herself - **Real goal:** completing the most important work without missing anything - **Main friction:** copying, organizing, and locating context - **Ideal state:** tasks collected automatically, leaving only prioritization to manage - This led to three core requirements: - AI should register tasks directly from Slack. - Each task should retain its source thread and document links. - Priorities should remain visible in an always-present widget. ## Teaching AI to Understand Work Context - Adding a specific emoji to a Slack message sends it to a designated channel. - Claude Code reads the message and converts it into a task with: - A concise summary - The relevant team tag - A link to the original Slack thread - The hardest part was turning long, contextual Slack conversations into one clear action. - For example, a request about an error during a loan-extension application becomes “Check loan-extension error case.” - She created writing guidelines and examples defining: - What qualifies as a good task - How teams should be categorized - Which expressions and sentence structures to use - The goal was for AI-generated tasks to sound like something she would have written herself. - Refining the AI’s output was less about coding than encoding her judgment about what constitutes a real task. ## Designing the Widget Experience - Making the widget feel natural required detailed interaction design and repeated implementation. - She rebuilt the code to refine the expand-and-collapse behavior. - The drag interaction took nearly a week to complete. - Explaining seemingly obvious behaviors to AI forced her to define her own requirements more precisely. - In this sense, working with AI became a process of clarifying thoughts and translating them into explicit language. ## Replacing Anxiety with Prioritization - She no longer needs to open Slack or Notion repeatedly to remember her tasks. - The always-visible widget removed a previously unnoticed source of friction. - AI now handles collecting and organizing work, reducing the mental energy spent on administration. - She can concentrate on deciding what matters most instead of worrying that something has been forgotten. ## A Personal Problem Shared by the Team - Although the widget was initially built for personal use, many colleagues adopted it. - Developers unexpectedly became active users, reporting bugs and suggesting features. - The usual designer–developer relationship reversed: developers raised issues while she fixed and redeployed the tool. - This showed that task collection, prioritization, and context management are common problems across job functions. - The tool spread not because its concept was revolutionary, but because it addressed an existing, widely felt inconvenience. ## Applying the Method - Identify the most frequent “not really work” task from the past week: - Copying information - Searching for context - Organizing lists - Define the problem as a product: - Who is the user? - What are they truly trying to accomplish? - Where is the greatest friction? - What does success look like? - Examine why existing tools do not solve the problem. - Start with the smallest version that can be useful immediately. The practical lesson is to treat repetitive coordination work as something that can be designed away. Instead of searching for a perfect general-purpose tool, build a small solution around the specific context, habits, and judgments that existing products cannot know.

Read original(opens in new tab)
toss4 min readCurated summary

How the Toss Team Faces the AI Wave: AI Surf Day

Toss created **AI Surf Day**, a dedicated weekly time for employees to experiment with AI, share lessons, and redesign their workflows. Running on Fridays from April through June, the initiative aims to reduce the AI gap across technical and nontechnical roles by making experimentation collaborative and accessible. Its broader conclusion is that successful AI transformation depends less on formal programs than on culture, time, and people who actively share what they learn. ## AI Surf Day’s Purpose - Employees focus on their core work Monday through Thursday and reserve Friday for AI experimentation and practical application. - The program addresses anxiety and knowledge gaps, especially among nondevelopers who may struggle to identify useful AI information or find time to learn it. - Its concept comes from Jon Kabat-Zinn’s phrase: “You can’t stop the waves, but you can learn to surf.” - The goal is to help Toss become a company that works with AI as a foundation, not merely a workplace where individuals use AI tools. ## AI Surf Club - Employees can create or join informal groups focused on AI topics; roughly 200 clubs were formed at launch. - An **AI Antipattern Study** focused on failures and mistakes, turning participants’ experiences into a practical guide for avoiding common problems. - An **LLM Wiki** group explored how to organize scattered organizational knowledge across data engineering, machine learning, and business teams. - A beginner-focused “Step 0” group helped employees overcome basic technical barriers, such as installing agent tools and asking questions they felt were too fundamental. - A customer-protection team built an external-complaint monitoring portal in one month, along with automation for complaint-response drafts and classification. - A marketing team divided AI work into roles such as: - **Builder:** creates AI-powered tools and workflows - **Curator:** collects useful examples and resources - **Operator:** applies AI to repetitive work - **Scouter:** identifies new opportunities - The clubs emphasized reusable outputs and shared confidence, rather than isolated individual experimentation. ## AI Surf Weekly - Weekly sessions share successful internal AI applications, lessons learned, and current industry insights. - Toss connected employees with similar needs across different departments, enabling them to solve problems quickly by learning from existing internal examples. - Rather than prescribing specific tools, the program presents ideas and use cases that encourage employees to adapt solutions to their own work. - Examples included connecting a sales employee with an HR colleague who had built a similar tool, and pairing a marketer with a designer experienced in AI-powered automation. ## AI Surf Evangelists - Toss selected 142 employees across its affiliated companies and teams to promote AI adoption in their own organizations. - Evangelists were chosen through peer nominations, recognizing people who already shared useful discoveries and helped colleagues overcome AI-related obstacles. - Their responsibilities over three months include: - Reporting effective AI use cases - Sharing useful insights with colleagues - Hosting at least one meetup or workshop - Toss’s Culture team provides workshop templates and facilitation support. - Many teams have conducted workshops around redesigning their existing workflows with AI. - The program treats AI adoption as a team-level workflow redesign challenge, rather than simply measuring individual proficiency with AI tools. ## OpenAI Collaboration and Mini-Hackathon - Toss held a special AI Surf Day with OpenAI on May 15. - Hands-on sessions covered: - Codex-based development workflows for developers - ChatGPT Agent-based automation for nondevelopers - A 2.5-hour hackathon produced two notable projects: - An iOS workflow where Codex implements features, operates the simulator, tests the result, iterates on problems, and produces verification footage. - An agent that classifies thousands of daily Toss Place product records, sends reviewers links, and supports approval or rejection through an admin interface. - These projects demonstrated how AI can become a reusable agentic workflow rather than a one-time assistant. ## Culture Over Programs - Toss does not claim to have a fixed answer for managing AI’s rapid evolution. - The lasting value of AI Surf Day is the protected time for learning and experimentation, along with a culture where employees openly share results and failures. - Successful examples spread naturally across teams, while evangelist-led workshops translate experimentation into concrete changes in how work is performed. Organizations pursuing AI transformation can take a similar approach: create dedicated experimentation time, encourage peer-led learning, recognize existing champions, and focus on reusable workflow improvements rather than tool adoption alone.

Read original(opens in new tab)
toss4 min readCurated summary

Why High-Performing Organizations Need Toss-Style TPMs in the AI Era

TPM roles are often associated with coordinating schedules, dependencies, risks, and stakeholders. Toss argues that this is no longer enough: as organizations grow and AI increases cross-team complexity, the most important problems often fall into gray areas with no clear owner. Its TPM is therefore redefined as a strategic execution problem-solver who structures ambiguous problems and drives them to measurable resolution. ## Why TPM Needs to Be Redefined - Traditional TPMs typically deliver already-defined technical programs by managing: - Schedules - Risks - Dependencies - Cross-functional communication - At Toss, many difficult problems do not begin as clearly named programs. - Common examples include: - Problems spanning multiple teams with no accountable owner - Strategies without an execution model - Issues recognized as important but lacking priority or authority - Frequent status updates without meaningful change - These problems may involve product, technology strategy, organization design, and operations simultaneously. - AI adoption is accelerating this trend by increasing dependencies across data, security, quality, productivity, and organizational practices. ## How Toss’s TPM Differs from Related Roles - **Product Owner:** Defines what to build, product priorities, and customer or business value. - **Engineering Manager or SDM:** Builds the conditions for a team to execute consistently, including people, quality, and team health. - **Traditional TPM or Technical Project Manager:** Manages delivery of an already-defined initiative. - **Toss TPM:** Addresses the structural problems left between or outside these roles. - Finds important but undefined problems - Establishes ownership and decision rights - Creates an executable structure - Drives the work through to completion - The role is not primarily a project scheduler or people manager; it is a problem solver for organizational gray areas. ## Why Cross-Team Problems Matter in Strong Organizations - In less mature organizations, bottlenecks such as unclear responsibility or poor prioritization are usually visible within teams. - In high-performing organizations, individual teams may operate effectively while problems remain between teams. - Organizational structures clarify accountability and speed decisions, but they can also leave boundary-spanning issues without an owner. - These issues include: - Company-wide problems that local optimization cannot solve - Important long-term work that is not urgent - Responsibilities shared by several teams but owned by none - AI makes these boundary problems more frequent because technical, operational, and organizational concerns increasingly overlap. ## What a Toss TPM Does - **Finds problems proactively** - Identifies recurring gaps, structural bottlenecks, and unnamed problems rather than waiting for assigned work. - **Turns strategy into execution** - Determines which teams must act, in what order, who should be the DRI, and what must be deprioritized. - **Creates value between teams** - Designs solutions where different goals, constraints, and working speeds collide. - **Removes blockers** - Goes beyond reporting risks by changing decision structures, assembling the right people, resetting priorities, or redesigning collaboration. - **Considers people and systems together** - Examines leadership, team composition, authority, and operating mechanisms—not just timelines. - **Measures success through real change** - Success means execution resumes, direction improves, recurring bottlenecks decrease, and future solutions become easier. - Coordination is a useful skill, but problem-solving is the role’s core identity. ## Capabilities Needed to Become This Kind of TPM - **Problem structuring:** Separating symptoms from root problems, identifying stakeholders, and locating decision bottlenecks. - **Execution design:** Translating strategic direction into concrete workflows, sequencing, and ownership. - **Influence and mobilization:** Moving teams without relying solely on formal authority, including handling difficult conversations. - **Systems thinking:** Addressing repeated problems by changing mechanisms rather than relying on individual heroics. - **Follow-through:** Carrying work from discovery and alignment through execution, measurable results, and prevention of recurrence. Toss’s recommendation is to look for important problems that everyone recognizes but no one owns. People who cannot ignore those gaps can begin acting as informal TPMs in their current organizations—turning ambiguous, cross-functional problems into executable solutions and driving them to completion.

Read original(opens in new tab)
toss5 min readCurated summary

Introducing Toss Place's Data Bot 'PANDA': How every team member works like a data expert

PANDA, short for Place Analytics & Data, is Toss Place’s AI data-analysis assistant, designed to let employees retrieve and interpret approved data without waiting for analysts. It was created after the team found that 70% of data requests involved simple metric lookups rather than complex analysis. The project’s main conclusion is that reliable AI analytics depends less on prompting alone and more on standardized data, business definitions, controlled table selection, and iterative validation. ## Why Toss Place Built PANDA - Employees previously relied on analysts to search dashboards, write SQL, or manually investigate data requests. - PANDA provides self-service access within each employee’s security permissions. - It reduces routine extraction work for analysts, allowing them to focus on deeper analysis. - The goal is to establish a stronger culture of “data democracy,” where employees can access and use data immediately. ## Challenges with a Simple AI Chatbot Early experiments showed that asking an AI model to search all company data produced unreliable and expensive results: - Referencing thousands of tables and internal documents consumed excessive tokens. - The model sometimes selected different tables for identical questions, producing inconsistent answers. - It often misunderstood business definitions. For example, “active stores” could mean stores with completed installations or stores that had processed payments. - Inefficient SQL caused unnecessary Snowflake data scans and higher warehouse costs. ## Standardized Data Marts as a Single Source of Truth Toss Place collaborated across its Data Analysis and Data Platform teams to establish reliable standard data marts. - Core concepts, such as store information, were consolidated into standardized tables. - Naming conventions made table and column purposes easier for both people and AI to understand: - Tables follow `{mart_type}_{domain}_{subject}`, such as `fact_device_error_log`. - Columns follow `{prefix}_{entity}_{attribute}_{suffix}`, such as `is_merchant_active`. - Table and column descriptions were documented comprehensively. - The standardization effort reduced ambiguity by ensuring the same business concepts were represented consistently. ## Connecting Business Language to Data Data structures alone could not answer questions about terms such as “installed store” or “store category.” - Domain-specific terms and metric definitions were documented. - These business definitions were linked to the relevant standard data marts. - Data analysts helped reconcile differing interpretations and establish shared organizational definitions. - This gave PANDA the context needed to apply the correct business logic. ## Scoring and Ranking for Reliable Table Selection PANDA limits its search to well-managed tables and uses dbt tags to import selected metadata into a Manifest file. - Tables are ranked using: - **Similarity score:** Based on relationships between the question and table, including table-name matches and description relevance. - **Hierarchy weight:** Reflecting the reliability of the data layer. - The final score is calculated as: `similarity score × hierarchy weight` - Weights are assigned as follows: - Company-wide SSOT metrics: ×4 - Validated standard marts: ×3 - Domain analysis marts: ×2 - Raw bronze data and logs: ×1 - This improves accuracy, consistency, and trustworthiness while reducing unnecessary warehouse exploration. ## Agentic Loop for Querying and Validation Rather than expecting a correct answer in one attempt, PANDA uses an agentic loop. - It selects appropriate tools based on the question. - It explores tables, generates and executes queries, and reviews the results. - If the result appears inaccurate, it can inspect the schema again, modify the query, and retry. - If necessary, it asks the user for clarification. - This approach allows PANDA to handle exceptions dynamically instead of relying only on predefined rules. ## Answers Designed for Practical Use PANDA structures responses so users can understand and apply the results: - **Result:** The requested data or metric. - **Query criteria:** The period, filters, and aggregation method used. - **Insight:** An interpretation that can support practical decisions. This makes PANDA more than a number-retrieval chatbot; it also exposes part of the reasoning process normally provided by a data analyst. ## Adoption and User Response PANDA quickly became part of everyday work at Toss Place. - One-third of employees used it on its first day. - Half of the organization had tried it within a week. - More than 4,000 messages were exchanged during that period. - Current adoption is approximately 70%. - Employees reported feeling more comfortable asking small questions and using data while away from their desks. - Users particularly valued receiving insights alongside raw figures. - Unexpectedly, developers and even data professionals used PANDA actively, suggesting that its answers achieved a meaningful level of trust. ## Future Development PANDA was developed and launched in just one month, but the team plans further improvements. - Increase data coverage to more than 90%. - Raise answer accuracy above 97%. - Use real user questions, follow-up behavior, and abandonment patterns to identify unmet needs. - Expand beyond basic data retrieval to reduce more of the data team’s workload. PANDA’s central lesson is that effective enterprise AI does not require the most complicated technology. It requires solving a real business pain point with trustworthy data foundations, clear definitions, and a workflow that users can rely on.

Read original(opens in new tab)
tossOriginal article

Will developers be replaced by AI? (opens in new tab)

The current AI hype cycle is a significant economic bubble where massive infrastructure investments of $560 billion far outweigh the modest $35 billion in generated revenue. However, drawing parallels to the 1995 dot-com era, the author argues that while short-term expectations are overblown, the long-term transformation of the developer role is inevitable. The conclusion is that developers won't be replaced but will instead evolve into "Code Creative Directors" who manage AI through the lens of technical abstraction and delegation. ### The Economic Bubble and Amara’s Law * The industry is experiencing a 16:1 imbalance between AI investment and revenue, with 95% of generative AI implementations reportedly failing to deliver clear efficiency improvements. * Amara’s Law suggests that we are overestimating AI's short-term impact while potentially underestimating its long-term necessity. * Much of the current "AI-driven" job market contraction is actually a result of companies cutting personnel costs to fund expensive GPU infrastructure and AI research. ### Jevons Paradox and the Evolution of Roles * Jevons Paradox indicates that as the "cost" of producing code drops due to AI efficiency, the total demand for software and the complexity of systems will paradoxically increase. * The developer’s identity is shifting from "code producer" to "system architect," focusing on agent orchestration, result verification, and high-level design. * AI functions as a "power tool" similar to game engines, allowing small teams to achieve professional-grade output while amplifying the capabilities of senior engineers. ### Delegation as a Form of Abstraction * Delegating a task to AI is an act of "work abstraction," which involves choosing which low-level details a developer can afford to ignore. * The technical boundary of what is "hard to delegate" is constantly shifting; for example, a complex RAG (Retrieval-Augmented Generation) pipeline built for GPT-4 might become obsolete with the release of a more capable model like GPT-5. * The focus for developers must shift from "what is easy to delegate" to "what *should* be delegated," distinguishing between routine boilerplate and critical human judgment. ### The Risks of Premature Abstraction * Abstraction does not eliminate complexity; it simply moves it into the future. If the underlying assumptions of an AI-generated system change, the abstraction "leaks" or breaks. * Sudden shifts in scaling (traffic surges), regulation (GDPR updates), or security (zero-day vulnerabilities) expose the limitations of AI-delegated work, requiring senior intervention. * Poorly managed AI delegation can lead to "abstraction debt," where the cost of fixing a broken AI-generated system exceeds the cost of having written it manually from the start. To thrive in this environment, developers should embrace AI not as a replacement, but as a layer of abstraction. Success requires mastering the ability to define clear boundaries for AI—delegating routine CRUD operations and boilerplate while retaining human control over architecture, security, and complex business logic.

tossOriginal article

Tax Refund Automation: AI (opens in new tab)

At Toss Income, QA Manager Suho Jung successfully automated complex E2E testing for diverse tax refund services by leveraging AI as specialized virtual team members. By shifting from manual coding to a "human-as-orchestrator" model, a single person achieved the productivity of a four-to-five-person automation team within just five months. This approach overcame the inherent brittleness of testing long, React-based flows that are subject to frequent policy changes and external system dependencies. ### Challenges in Tax Service Automation The complexity of tax refund services presented unique hurdles that made traditional manual automation unsustainable: * **Multi-Step Dependencies:** Each refund flow averages 15–20 steps involving internal systems, authentication providers, and HomeTax scraping servers, where a single timing glitch can fail the entire test. * **Frequent UI and Policy Shifts:** Minor UI updates or new tax laws required total scenario reconfigurations, making hard-coded tests obsolete almost immediately. * **Environmental Instability:** Issues such as "Target closed" errors during scraping, differing domain environments, and React-specific hydration delays caused constant test flakiness. ### Building an AI-Driven QA Team Rather than using AI as a simple autocomplete tool, the project assigned specific "personas" to different AI models to handle distinct parts of the lifecycle: * **SDET Agent (Claude Sonnet 4.5):** Acted as the lead developer, responsible for designing the Page Object Model (POM) architecture, writing test logic, and creating utility functions. * **Documentation Specialist:** Automatically generated daily retrospectives and updated technical guides by analyzing daily git commits. * **Git Master:** Managed commit history and PR descriptions to ensure high-quality documentation of the project’s evolution. * **Pair Programmers (Cursor & Codex):** Handled real-time troubleshooting, type errors, and comparative analysis of different test scripts. ### Technical Solutions for React and Policy Logic The team implemented several sophisticated technical strategies to ensure test stability: * **React Interaction Readiness:** To solve "Element is not clickable" errors, they developed a strategy that waits not just for visibility, but for event handlers to bind to the DOM (Hydration). * **Safe Interaction Fallbacks:** A standard `click` utility was created that attempts a Playwright click, then a native keyboard 'Enter' press, and finally a JS dispatch to ensure interactions succeed even during UI transitions. * **Dynamic Consent Flow Utility:** A specialized system was built to automatically detect and handle varying "Terms of Service" agreements across different sub-services (Tax Secretary, Hidden Refund, etc.) through a single unified function. * **Test Isolation:** Automated scripts were used to prevent `userNo` (test ID) collisions, ensuring 35+ complex scenarios could run in parallel without data interference. ### Integrated Feedback and Reporting The automation was integrated directly into internal communication channels to create a tight feedback loop: * **Messenger Notifications:** Every test run sends a report including execution time, test IDs, and environment data to the team's messenger. * **Automated Failure Analysis:** When a test fails, the AI automatically posts the error log, the specific failed step, a tracking EventID, and a screenshot as a thread reply for immediate debugging. * **Human-AI Collaboration:** This structure shifted the QA's role from writing code to discussing failures and policy changes within the messenger threads. The success of this 5-month experiment suggests that for high-complexity environments, the future of QA lies in "AI Orchestration." Instead of focusing on writing selectors, QA engineers should focus on defining problems and managing the AI agents that build the architecture.

tossOriginal article

Automating Service Vulnerability Analysis (opens in new tab)

Toss has developed a high-precision automated vulnerability analysis system by integrating Large Language Models (LLMs) with traditional security testing tools. By evolving their architecture from a simple prompt-based approach to a multi-agent system utilizing open-source models and static analysis, the team achieved over 95% accuracy in threat detection. This project demonstrates that moving beyond a technical proof-of-concept requires solving real-world constraints such as context window limits, output consistency, and long-term financial sustainability. ### Navigating Large Codebases with MCP * Initial attempts to use RAG (Retrieval Augmented Generation) and repository compression tools failed because the LLM could not maintain complex code relationships within token limits. * The team implemented a "SourceCode Browse MCP" (Model Context Protocol) which allows the LLM agent to dynamically query the codebase. * By indexing the code, the agent can perform specific tool calls to find function definitions or variable usages only when necessary, effectively bypassing context window restrictions. ### Ensuring Consistency via SAST Integration * Testing revealed that standalone LLMs produced inconsistent results, often missing known vulnerabilities or generating hallucinations across different runs. * To solve this, the team integrated Semgrep, a Static Application Security Testing (SAST) tool, to identify all potential "Source-to-Sink" paths. * Semgrep was chosen over CodeQL due to its lighter resource footprint and faster execution, acting as a structured roadmap that ensures the LLM analyzes every suspicious input path without omission. ### Optimizing Costs with Multi-Agent Architectures * Analyzing every possible code path identified by SAST tools was prohibitively expensive due to high token consumption. * The workflow was divided among three specialized agents: a Discovery Agent to filter out irrelevant paths, an Analysis Agent to perform deep logic checks, and a Verification Agent to confirm findings. * This "sieve" strategy ensured that the most resource-intensive analysis was only performed on high-probability vulnerabilities, significantly reducing operational costs. ### Transitioning to Open Models for Sustainability * Scaling the system to hundreds of services and daily PRs made proprietary cloud models financially unviable. * After benchmarking models like Llama 3.1 and GPT-OSS, the team selected **Qwen3:30B** for its 100% coverage rate and high true-positive accuracy in vulnerability detection. * To bridge the performance gap between open-source and proprietary models, the team utilized advanced prompt engineering, one-shot learning, and enforced structured JSON outputs to improve reliability. To build a production-ready AI security tool, teams should focus on the synergy between specialized open-source models and traditional static analysis tools. This hybrid approach provides a cost-effective and sustainable way to achieve enterprise-grade accuracy while maintaining full control over the analysis infrastructure.

tossOriginal article

Toss's AI Technology Recognized (opens in new tab)

Toss ML Engineer Jin-woo Lee presents FedLPA, a novel Federated Learning algorithm accepted at NeurIPS 2025 that addresses the critical challenges of data sovereignty and non-uniform data distributions. By allowing AI models to learn from localized data without transferring sensitive information across borders, this research provides a technical foundation for expanding services like Toss Face Pay into international markets with strict privacy regulations. ### The Challenge of Data Sovereignty in Global AI * Traditional AI development requires centralizing data on a single server, which is often impossible due to international privacy laws and data sovereignty regulations. * Federated Learning offers a solution by sending the model to the user’s device (client) rather than moving the data, ensuring raw biometric information never leaves the local environment. * Standard Federated Learning fails in real-world scenarios where data is non-IID (Independent and Identically Distributed), meaning user patterns in different countries or regions vary significantly. ### Overcoming Limitations in Category Discovery * Existing models assume all users share similar data distributions and that all data classes are known beforehand, which leads to performance degradation when encountering new demographics. * FedLPA incorporates Generalized Category Discovery (GCD) to identify both known classes and entirely "novel classes" (e.g., new fraud patterns or ethnic features) that were not present in the initial training set. * This approach prevents the model from becoming obsolete as it encounters new environments, allowing it to adapt to local characteristics autonomously. ### The FedLPA Three-Step Learning Pipeline * **Confidence-guided Local Structure Discovery (CLSD):** The system builds a similarity graph by comparing feature vectors of local data. It refines these connections using "high-confidence" samples—data points the model is certain about—to strengthen the quality of the relational map. * **InfoMap Clustering:** Instead of requiring a human to pre-define the number of categories, the algorithm uses the InfoMap community detection method. This allows the client to automatically estimate the number of unique categories within its own local data through random walks on the similarity graph. * **Local Prior Alignment (LPA):** The model uses self-distillation to ensure consistent predictions across different views of the same data. Most importantly, an LPA regularizer forces the model’s prediction distribution to align with the "Empirical Prior" discovered in the clustering phase, preventing the model from becoming biased toward over-represented classes. ### Business Implications and Strategic Value * **Regulatory Compliance:** FedLPA removes technical barriers to entry for markets like the EU or Southeast Asia by maintaining high model performance while strictly adhering to local data residency requirements. * **Hyper-personalization:** Financial services such as Fraud Detection Systems (FDS) and Credit Scoring Systems (CSS) can be trained on local patterns, allowing for more accurate detection of region-specific scams or credit behaviors. * **Operational Efficiency:** By enabling models to self-detect and learn from new patterns without manual labeling or central intervention, the system significantly reduces the cost and time required for global maintenance. Implementing localized Federated Learning architectures like FedLPA is a recommended strategy for tech organizations seeking to scale AI services internationally while navigating the complex landscape of global privacy regulations and diverse data distributions.