toss·Jun 19, 2026·4 min readCurated summary

Going Beyond Expertise

Technical Writers (TWs) can contribute far beyond writing documentation: they can lead product teams and build systems that turn knowledge into an organizational asset. Toss’s Knowledge System Team created “todoc,” an internal platform that makes documentation easier to write, centralizes scattered knowledge, and enables AI access. Its broader goal is to make documentation emerge naturally from daily work and remain accurate without constant manual maintenance. ## TWs as Product Owners and Makers - The author leads a product team of developers, designers, and TWs. - Their responsibilities include: - Setting product direction, roadmap, and priorities - Interviewing users and bringing insights to the team - Planning features - Building features directly with AI tools - TW expertise is especially valuable because TWs have deeply considered: - Why documents are difficult to read - What makes documentation effective - How information should be structured for AI consumption ## Why Toss Built Todoc Todoc was launched to address weaknesses in Toss’s existing documentation environment. - Static-site-generated documentation required users to: - Clone a repository - Write Markdown - Submit pull requests - Wait for review - This workflow was familiar to developers but created major barriers for designers, PMs, and other non-developers. - Existing documentation tools accumulated outdated policies, unfinished notes, and unexplained content, creating “documentation debt.” - Knowledge was fragmented across: - Static sites - Documentation tools - Code - Collaboration messengers - Individual employees’ knowledge After its beta launch, Todoc grew to more than 500 documents and 40,000 valid pages, with over 1,000 monthly users. ## Todoc’s Four Core Values ### Easy Documentation for Everyone - Anyone can create or edit documents immediately. - Content can be connected from GitHub, documentation tools, internal messengers, and other sources. - The platform removes the technical and procedural barriers to documentation. ### AI-Ready Knowledge - Well-organized documentation can be used by team bots and other AI tools. - Todoc supports API, CLI, and MCP access. - Teams use it for request bots, product specifications, and other workflows. ### A Single Source of Truth - Todoc consolidates scattered sources into complete, centralized documents. - Users can determine which information is current without searching across multiple systems. - The platform serves as the organization’s SSoT (Single Source of Truth). ### Scalable Infrastructure - Teams no longer need to select, build, or maintain their own documentation infrastructure. - Each team can have its own space on a shared platform. - The model is being expanded to Toss affiliates. ## Automating Documentation Quality and Maintenance Lowering the barrier to writing creates a new challenge: maintaining quality. - TW judgment is being converted into: - AI proofreading - Automated document reviews - Bots that generate initial drafts - Todoc is also designed to create documentation automatically from: - Decisions and discussions in internal messengers - Code changes - Ongoing project conversations - The system aims to update documents without relying on someone remembering to maintain them. - It evaluates whether knowledge is still valid by checking: - Whether policies match implemented code - Whether information is actively used - How recently it was updated ## The Evolution of TW Expertise The role is shifting from writing excellent documents manually to designing systems that consistently produce and maintain excellent documentation. - Experience understanding why documents are hard to read becomes standards for human- and AI-readable content. - Judgments about what makes a good document become criteria for AI review and automated editing. - Expertise in identifying outdated information becomes a system for validating knowledge. - TWs increasingly focus on: - Creating places where knowledge can gather - Defining quality standards - Encoding human judgment into systems - Generating documentation through normal work - Keeping knowledge continuously updated The practical vision is an organization where outdated documents trigger their own notifications, project work leaves behind organized records, and recurring explanations are preserved for future employees. Toss’s Technical Writing Chapter is therefore working to systematize TW expertise and establish documentation governance so teams can document effectively without constant manual intervention.

Read original(opens in new tab)

AIModel Context ProtocolGithubAPI+4

toss·Jun 19, 2026·3 min readCurated summary

If You Asked a Designer to Make Anything with AI

Toss Design Chapter’s AI Contest invited designers to build anything with AI, resulting in 122 projects over one month. The examples show that designers primarily used AI to improve existing work—making it faster, more persuasive, and higher quality—rather than creating entirely new kinds of work. The article recommends starting with a frustrating, repetitive task or a frequently repeated communication problem. ## Automating Repetitive Work - A color-extraction tool automatically identifies and adjusts colors from images for use in UI. - Color extraction had been an unresolved challenge at Toss because results varied widely by image. - Designers used AI to draft the logic, test it against many sample images, and rapidly refine it. - The resulting system is now used for product-card colors in Toss Shopping. ## Reducing Collaboration Costs with a Personal Bot - A Slack bot was trained on a designer’s knowledge, past discussions, and reference materials. - It creates draft answers to the many design and requirements questions the designer receives each day. - Team members can send the draft as-is or revise it before responding. - The bot learns from those revisions, improving its answers to similar questions over time. - The designer described the result as feeling like becoming “1.5 people,” and other Toss designers began creating their own bots. ## Persuading Through Interactive Prototypes - A designer built a functioning prototype of a stock-trading desktop interface instead of presenting only static screens. - Users could drag panels, rearrange them, and resize windows, with the interface responding accordingly. - Showing the intended interactions directly reduced the risk that design ideas would be misunderstood during development. - The working prototype helped align designers and developers and persuade the product owner. ## Pushing Quality Within Tight Deadlines - AI-generated motion graphics were created for the key visual of Toss Bank’s recruitment website. - Each job category needed its own animation despite a very short schedule. - The designer created the foundational images manually and repeatedly refined Kling prompts to achieve the desired results. - Human-designed starting and ending frames combined with AI-generated motion allowed all category animations to be completed in a single day. ## Four Ways to Start Using AI - **Efficiency:** Hand off one especially annoying repetitive task to AI. - **Replication:** Build a bot to answer questions you repeatedly handle yourself. - **Persuasion:** Turn designs that require verbal explanation into working prototypes. - **Quality:** Use AI to reach a higher level of polish within a limited timeframe. The practical recommendation is to begin with an existing task rather than searching for an entirely new AI application. Choose one area where AI can save time, communicate intent more clearly, or help raise the final quality.

Read original(opens in new tab)

AIPrototypingSlackPrompt Engineering+3

toss·Jun 17, 2026·3 min readCurated summary

How AI Has Changed the Product Design Process

Toss redesigned its customer-service chatbot after finding that 60% of users abandoned the existing menu-based experience. Instead of forcing customers to identify the correct category, the team used AI to interpret natural-language requests and connect users directly to solutions. The project’s key insight was to prototype and validate experiences early, then improve the system through reusable rules rather than endlessly editing individual scenarios. ## Replacing Menu Navigation with Natural Language - The customer center serves roughly 600,000 visitors monthly, including about 170,000 chat users. - Users often know their problem—such as a failed payment or mistaken transfer—but not how Toss categorizes it. - The team therefore aimed to build a chatbot that understood intent and guided users directly to the appropriate resolution. ## Generating Scenario Drafts from Real 상담 Data - AI created initial scenarios for the 20 most common customer-service issues. - Personal information was removed and data was anonymized or pseudonymized before use. - Instead of relying primarily on policy documents, the team trained the AI on real consultation data. - This revealed: - The language customers actually use - The questions agents ask to narrow down causes - The order in which problems are diagnosed and resolved - Ways to explain complex policies in understandable language - As a result, the chatbot behaved more like a human 상담사 than a system that simply listed policies. ## Validating Many Situations with a Scenario Hub - A single request can hide dozens of possible conditions. For example, membership cancellation may depend on whether: - The monthly fee was already paid - Benefits were used - Cancellation was already scheduled - The team built a scenario hub where these conditions could be saved and selected. - Each selection immediately opened a prototype conversation with the chosen conditions applied. - This enabled rapid testing after scenario changes, new branches, or rule updates. - The prototype became an experimentation environment rather than a static mockup, allowing the team to judge realistic conversations instead of relying on assumptions. ## Replacing Individual Fixes with General Rules - Testing revealed recurring problems, including: - Asking for information the chatbot already knew - Repeating the same explanation - Guessing when it lacked reliable information - Rather than fixing each scenario separately, the team created general rules such as: - Resolve the customer’s problem before giving explanations - Admit uncertainty instead of guessing - Escalate to a human agent only under defined conditions - Clearly limit the AI’s authority - Improving one scenario helped only that scenario, while improving a rule improved many scenarios at once. ## Designing the Experience Before the System - The team first envisioned the desired customer experience, then worked backward to determine the required: - Data structures - APIs - Operational tools - System architecture - This reversed the conventional process of designing infrastructure first and fitting the user experience onto it. - Within approximately three weeks, the team completed analysis, experience design, scenario generation, prototyping, testing, and refinement. - AI changed not only productivity but also the order of product design: designers could validate more possibilities before committing to implementation. ## Practical Lessons - Build an imperfect prototype early instead of waiting for a complete design. - Study real user data before relying solely on guides or documentation. - When similar problems recur, identify a shared rule instead of repeatedly patching individual cases. The broader recommendation is to use AI as a fast experimentation tool. Its greatest value is not simply producing artifacts, but making it cheaper and faster to test which experiences and design principles actually work.

Read original(opens in new tab)

Natural Language ProcessingConversational AiChatbotRapid Prototyping+3

toss·Jun 17, 2026·4 min readCurated summary

Designing the Work You Do Every Day

A product designer at Toss Bank transformed her personal task-management routine instead of accepting repetitive manual work as unavoidable. She built an AI-powered desktop widget that collects Slack messages, summarizes them into actionable tasks, preserves their context, and helps her focus on prioritization. What began as a personal solution revealed a broader problem shared across roles and spread throughout her team. ## From Manual Task Tracking to Workflow Design - For two and a half years, she manually copied tasks, feedback, discussion links, and requests from Slack into Notion or Slack lists. - As her responsibilities expanded to three teams, daily tasks grew from roughly 10 to more than 20. - She reframed the issue as a product-design problem: - **User:** herself - **Real goal:** completing the most important work without missing anything - **Main friction:** copying, organizing, and locating context - **Ideal state:** tasks collected automatically, leaving only prioritization to manage - This led to three core requirements: - AI should register tasks directly from Slack. - Each task should retain its source thread and document links. - Priorities should remain visible in an always-present widget. ## Teaching AI to Understand Work Context - Adding a specific emoji to a Slack message sends it to a designated channel. - Claude Code reads the message and converts it into a task with: - A concise summary - The relevant team tag - A link to the original Slack thread - The hardest part was turning long, contextual Slack conversations into one clear action. - For example, a request about an error during a loan-extension application becomes “Check loan-extension error case.” - She created writing guidelines and examples defining: - What qualifies as a good task - How teams should be categorized - Which expressions and sentence structures to use - The goal was for AI-generated tasks to sound like something she would have written herself. - Refining the AI’s output was less about coding than encoding her judgment about what constitutes a real task. ## Designing the Widget Experience - Making the widget feel natural required detailed interaction design and repeated implementation. - She rebuilt the code to refine the expand-and-collapse behavior. - The drag interaction took nearly a week to complete. - Explaining seemingly obvious behaviors to AI forced her to define her own requirements more precisely. - In this sense, working with AI became a process of clarifying thoughts and translating them into explicit language. ## Replacing Anxiety with Prioritization - She no longer needs to open Slack or Notion repeatedly to remember her tasks. - The always-visible widget removed a previously unnoticed source of friction. - AI now handles collecting and organizing work, reducing the mental energy spent on administration. - She can concentrate on deciding what matters most instead of worrying that something has been forgotten. ## A Personal Problem Shared by the Team - Although the widget was initially built for personal use, many colleagues adopted it. - Developers unexpectedly became active users, reporting bugs and suggesting features. - The usual designer–developer relationship reversed: developers raised issues while she fixed and redeployed the tool. - This showed that task collection, prioritization, and context management are common problems across job functions. - The tool spread not because its concept was revolutionary, but because it addressed an existing, widely felt inconvenience. ## Applying the Method - Identify the most frequent “not really work” task from the past week: - Copying information - Searching for context - Organizing lists - Define the problem as a product: - Who is the user? - What are they truly trying to accomplish? - Where is the greatest friction? - What does success look like? - Examine why existing tools do not solve the problem. - Start with the smallest version that can be useful immediately. The practical lesson is to treat repetitive coordination work as something that can be designed away. Instead of searching for a perfect general-purpose tool, build a small solution around the specific context, habits, and judgments that existing products cannot know.

Read original(opens in new tab)

AISlackAutomationClaude Code+2

toss·Jun 16, 2026·4 min readCurated summary

Why the Designer Built an App Instead of a Mockup

AI is changing design by letting designers create working prototypes rather than only static mockups. While building Toss’s “underlay” component, the author used SwiftUI, AI, and Metal shaders to design interactions directly on an iPhone. The experience reduced the translation gap between design and development and showed that a functional code prototype can become a more accurate specification than a traditional design document. ## Turning End Screens into New Starting Points - Toss identified screens that appear after users finish tasks, such as completing a transfer or payment. - These “dead ends” usually signal the end of an experience and encourage users to leave the app. - The project aimed to turn them into starting points for new experiences through a reusable component that could work throughout the app. ## The Idea Behind Underlay - Existing notification patterns—bottom sheets, toasts, push notifications, and overlays—attract attention but can interrupt what users are doing. - The author found inspiration in removing a shipping label from a moving box and revealing text that had already existed underneath. - This led to the concept of an **underlay**: instead of placing new information on top of the current screen, the component reveals something beneath it. - The component needed to be recognizable as a feature, preserve the existing experience, and remain reusable across the app. ## Designing Interaction with AI and Code - Figma prototypes showed that underlay’s movement mattered more than its static appearance. - Without experience in Protopie or Framer, the author built an iOS playground in SwiftUI using Xcode and AI. - The designer’s role was to: - Describe the intended experience - Select the best options proposed by AI - Evaluate the result on a real device - The process became a cycle of designing, choosing, testing, and refining. ## Building and Refining the Prototype - The author first created a personal playground for quickly experimenting with layouts, buttons, and text. - Each change was tested through actual device interaction rather than static previews. - The prototype was revised hundreds of times because imagined motion often felt different once implemented. - A scanning-light interaction was created to suggest that AI was reading the screen and finding relevant information. - The visual effect used a Metal shader to control pixel-level properties such as: - Light diffusion - Tint and width - Movement speed - Background darkening - The author gradually began adjusting shader values directly, despite having no prior shader or interaction-tool experience. ## Replacing Interaction Guidelines with a Repository - Instead of preparing a detailed document specifying timing, easing curves, and delays, the author gave developers the working prototype repository. - The developers reproduced the core structure almost immediately because they could inspect and run the implementation directly. - During refinement, the designer adjusted values and interactions directly on the developer’s laptop with AI. - A working reference made communication more precise than phrases such as “the motion feels wrong.” - Both the AI and developers could respond to the same concrete behavior rather than interpreting abstract descriptions. ## Designing the Implementation Structure - The production iOS code ended up closely resembling the prototype’s structure. - This happened because the author had built the prototype for rapid user testing and iteration, not specifically for production. - A well-structured prototype allowed design changes to be communicated through code and reused during implementation. - The article argues that a good design is not only visually convincing; its underlying construction should also be viable. - When a prototype is built in a development-ready way, it can function as the specification rather than disposable code. ## Applying the Approach - Start with the best possible experience instead of limiting ideas to what a familiar design tool can express. - Use AI and code to create a working artifact, since real devices reveal details and textures static tools may hide. - When handing off complex interactions, provide executable code or a repository instead of relying solely on written guidelines. - As AI handles more of the “how,” designers increasingly differentiate themselves through what they choose to imagine and build.

Read original(opens in new tab)

FigmaiOSDesign To CodeInteraction Design+3

toss·Jun 16, 2026·4 min readCurated summary

User Segmentation for Understanding 28 Million MAU, TUES

Toss developed TUES (Toss User Engagement Segment) to analyze its 28 million monthly active users from a platform-wide perspective. It groups users by their service-use patterns, enabling Toss to understand user motivations, design segment-specific strategies, and explain changes in company-wide metrics. TUES V2 improves on the original by capturing usage depth, multi-service behavior, and engagement with individual service categories. ## Platform-Wide User Segmentation - Service-specific segments such as “users of Service A” are not mutually exclusive or collectively exhaustive because users may use multiple services. - TUES groups users with similar patterns across Toss’s entire service ecosystem. - It helps identify: - Which services users primarily use - How engaged they are with the app - Which user groups may be suitable for particular growth or marketing strategies ## How TUES V1 Worked - Toss calculated each user’s service-use rate per app open. - For example, a user who opened the app 60 times and used Toss Pay during 20 of those sessions had a 33% usage rate. - Users with similar service-usage distributions were grouped using K-Means clustering. - The raw clusters were interpreted and renamed to make them more useful for product and strategy teams. - V1 included: - **Highly engaged users:** Users who regularly use several services - **Service-oriented users:** Users primarily focused on Toss Bank, Toss Securities, inquiry services, benefits, transfers, or other services - **Simple visitors:** Users who open the app but rarely use its services ## How Toss Uses TUES - **Transition strategy:** Teams can plan how to move users from simple visits to service-oriented engagement and eventually to highly engaged usage. - **Product growth:** Product teams can quickly identify which user segments use their service most and combine that insight with transition strategies. - **Behavior analysis:** TUES reveals when users change segments, begin churning, or return after inactivity. - **Top-line metric analysis:** When MAU changes, Toss can identify which user segments moved and which services likely caused the change. - **Targeted marketing:** Marketers use TUES segments for campaigns such as push notifications. The segments are also available in Toss’s internal marketing tool, TUBA. ## Limitations of TUES V1 After roughly two years of use, Toss identified several weaknesses: - V1 measured only the probability of using a service during an app open, not the number of times it was used. - Users who engaged with a service once and users who used it ten times could appear equivalent. - It could not show engagement with secondary service categories. - K-Means is a hard-clustering method, so each user belonged to only one segment despite often using multiple services. - New major services, including Toss Shopping, App in Toss, and Toss Pay, were grouped into a generic “ETC” category. ## TUES V2 Improvements - **Usage-depth measurement:** V2 uses the number of service interactions per app open as a feature, capturing the intensity of engagement. - **Soft clustering:** Instead of assigning each user to one segment, V2 calculates each user’s degree of association with multiple segments and selectively uses those results. - **Three-layer structure:** Users are described through: 1. Overall app engagement 2. Primary service orientation 3. Engagement with each individual service category - The layers are built sequentially, making it clearer why a user belongs to a segment and what action may be appropriate next. ## New Strategic Capabilities in V2 - Teams can identify which service-category engagement should increase first to move users from a semi-engaged segment to a highly engaged one. - Individual service teams, or silos, can quantitatively connect actions that increase service engagement with company-wide segment and performance changes. - Products can more clearly compare the engagement profiles of users who do and do not use their services. - Cross-activation strategies now have a more precise starting point based on service-level engagement. ## Future Development Toss plans to combine TUES with additional analytical frameworks to: - Create faster and more detailed transition strategies using concepts such as service similarity. - Build strategic user maps based on user profiles and service-use patterns. - Quantify segment-specific value by combining TUES with frameworks such as MTVi. TUES demonstrates how platform-level segmentation can make a growing MAU base easier to understand and act upon. By combining overall engagement, primary service use, and service-level depth, TUES helps Toss develop more targeted growth strategies and connect individual product actions to broader company outcomes.

Read original(opens in new tab)

Machine LearningData AnalyticsSoft ClusteringUser Segmentation+4

toss·Jun 9, 2026·5 min readCurated summary

The History of Facial Recognition and the Future of Face Pay

Face recognition has evolved from manually recorded facial measurements to deep-learning systems capable of identifying people with near-perfect accuracy. The technology’s progression depended on automated feature extraction, robust local descriptors, neural networks, and increasingly large datasets. Face payment represents its next major application, combining convenience with layered security designed to address biometric privacy risks. ## From Manual Measurements to Automated Recognition - In the 1960s, Woodrow Wilson Bledsoe pioneered face recognition through a classified government project. - Researchers manually recorded coordinates for features such as eye spacing, nose-to-mouth distance, and ear positions. - Computers performed calculations, but humans still identified the facial landmarks. - In 1973, Takeo Kanade developed a system that automatically detected the eyes, nose, and mouth. - His work demonstrated that computers could extract meaningful facial information without manually supplied coordinates. ## Eigenfaces and Statistical Representation - In 1991, Matthew Turk and Alex Pentland introduced Eigenfaces using Principal Components Analysis (PCA). - The system created an average face and represented individual faces by how they differed from that average. - These differences formed “Eigenfaces,” mathematical components that worked like an alphabet for describing faces. - Recognition became a matter of comparing the combination weights assigned to two facial images. ## Coping with Lighting and Real-World Conditions - Early systems struggled when the same face appeared under different lighting, angles, or shadows. - Researchers shifted from analyzing the entire face to examining local regions and textures. - Local Binary Patterns (LBP) compared each pixel with its neighbors, making facial descriptions less dependent on absolute brightness. - Hand-crafted features were combined with classifiers such as: - **SVM**, which finds separating boundaries between identities. - **AdaBoost**, which combines weak classifiers into a stronger recognition system. - These methods improved robustness even though they could not learn features automatically like modern deep-learning models. ## Deep Learning Surpasses Human-Level Benchmarks - Facebook’s 2014 DeepFace system used deep neural networks trained on approximately four million face images. - It achieved 97.35% accuracy on the LFW benchmark, approaching the reported human score of 97.53%. - Google FaceNet later introduced Triplet Loss and reached 99.63% on LFW. - SphereFace, CosFace, and ArcFace continued improving discriminative facial representations. - The field shifted from proving that recognition was possible to optimizing accuracy and reliability. ## The Role of Large-Scale Datasets - **FERET:** Established an early standardized benchmark with 1,199 people and 14,126 grayscale images. - **LFW:** Contained 13,233 images of 5,749 people collected from natural, uncontrolled environments. - **VGGFace:** Provided 2.7 million images of 2,600 people, reinforcing the relationship between data scale and deep-learning performance. - **MS-Celeb-1M:** Pursued celebrity recognition at massive scale but was later withdrawn over privacy concerns. - **WebFace260M:** Started with 260 million images and refined them into roughly 42 million images covering two million people. - These datasets improved recognition while also highlighting the tension between technical progress, data quality, and privacy. ## Face Recognition as a Payment Method - Face recognition had already reached smartphones, airports, and office access systems, but payments required stricter standards. - Unlike device unlocking, payment errors can directly cause financial loss. - Toss FacePay launched in South Korea in September 2025, aiming to combine strong security with a frictionless experience. - Face payment offers: - Availability without carrying a wallet or charged phone. - Hands-free checkout. - Faster payment without searching for cards or opening an app. ## How FacePay Works - A terminal camera identifies the customer among registered users. - The system must also reject unregistered users and apply additional authentication in ambiguous cases, such as similar-looking individuals or twins. - Once identified, the system connects the customer to a registered credit card, debit card, or another selected payment method. - Face payment adds customer identification and payment-method lookup to the conventional POS, VAN, and card-issuer approval flow. ## Edge and Cloud Processing - **Edge processing** offers low latency and better privacy because images need not leave the terminal. - Its limitations include restricted hardware, smaller models, lower potential accuracy, and difficult device-by-device updates. - **Cloud processing** enables stronger models, centralized logging, and simpler model updates. - It introduces transmission delays and increases the importance of network and server security. - FacePay combines both approaches: the terminal performs initial processing, while the server handles facial-feature extraction, recognition, and payment operations. ## Layered Biometric Security - Communications between terminals and servers are protected with TLS, while images receive additional AES-256 encryption. - Matrix Projection creates cancellable biometric vectors: changing the key produces a different representation of the same face. - If a biometric vector were compromised, it could theoretically be reissued with a new key, similar to changing a password. - The stored biometric representation is designed not to correspond one-to-one with the original face image or allow easy reconstruction. - Access to biometric data is restricted, logged, and limited to operational needs. - FacePay also underwent review by South Korea’s Personal Information Protection Commission and offers full compensation for fraudulent payments through its protection program. The development of face payment reflects six decades of progress in computer vision, machine learning, data infrastructure, and security engineering. Its practical success depends not only on recognition accuracy, but also on cancellable biometric protection, strict access controls, privacy oversight, and effective defenses against spoofing.

Read original(opens in new tab)

Cloud ComputingDeep LearningComputer VisionData Privacy+4

toss·Jun 9, 2026·3 min readCurated summary

How Do TAMs Solve Problems in Fast-Moving Organizations?

TAM CONNECT 2025 brought together Technical Account Managers from Toss and Kakao Pay to compare how they connect technology, business, customers, and internal teams. Although their organizations differ, they face similar challenges: operational complexity, cross-team coordination, recurring incidents, and the need to improve customer experience. The event framed TAMs not as basic support staff, but as technology-driven problem solvers whose role is expanding through automation and AI. ## The Scope of a TAM’s Role - TAMs resolve partner integration issues and advise on API adoption. - They coordinate multiple teams during incidents and improve operational processes. - Their work includes automating recurring problems and influencing product and platform design. - Depending on the situation, a TAM may act like a developer, product manager, or incident leader. - At Toss, their responsibilities span authentication, Face Connect, financial platforms, online and offline payments, and partner APIs. ## Reducing Alert Noise and Preventing Recurring Problems - Toss’s Dayoung Park presented a problem-solving framework that began by redefining which alerts truly require attention. - Excessive notifications can obscure serious incidents, so the team focused on identifying meaningful operational signals. - They structured incident patterns, detected recurring issues automatically, and analyzed the root causes of settlement discrepancies. - The goal was not merely to resolve incidents faster, but to build systems that prevent them from recurring. ## Making Operations Independent of Individual Owners - PayToss’s Gimun Lee discussed reducing dependency on specific people’s knowledge. - Response histories and operational information were shared transparently so anyone could handle an issue. - Their Discord developer community used n8n workflows, LLM-based log analysis, and automatically generated incident-cause and resolution suggestions. - These tools helped the team maintain an average response time of under ten minutes. ## Using Customer Experience to Improve TAM Work - Toss’s Seongmin Chun drew on previous experience working for a customer organization. - Understanding customer frustrations and the information needed during incidents influenced his communication and support practices. - The team used the PDCA cycle to continually improve integration guides, standardize repeated communications, and structure operational processes. - Effective TAM work means not only solving current problems but also ensuring the same problems do not happen again. ## Shared Challenges Across Companies - TAMs must balance customer expectations with internal development teams. - They operate in rapidly changing services with increasingly complex systems. - The role requires simultaneous understanding of technology, business priorities, and customer needs. - TAMs often appear to sit ambiguously between support, operations, development, and business, but their actual work involves structuring complex problems and mobilizing multiple teams. - The event characterized TAMs as technology-based problem solvers rather than simple support personnel. ## AI and the Future of TAM Work - AI is already being applied to: - Log analysis - Incident-cause recommendations - Operations guide generation - Automated responses to recurring inquiries - Anomaly detection - Document search and summarization - As AI handles repetitive responses, TAMs are likely to focus more on complex problem-solving, structural improvements, cross-team coordination, customer-experience design, and operational strategy. TAM CONNECT demonstrated that TAM organizations can learn significantly from one another. As technology and services become more complex, TAMs will likely become increasingly strategic, combining automation and AI with the judgment needed to improve systems, processes, and customer experiences.

Read original(opens in new tab)

Large Language ModelsAPIIncident ManagementFintech+4

toss·Jun 8, 2026·4 min readCurated summary

Rubric Design and System Implementation for Skill Quality Management

Toss’s AI DX Team created a 30-item rubric to improve the quality of internal Skills used by coding agents. The central conclusion is that deterministic defects should be checked with rules, while semantic questions—especially whether a Skill will be triggered—should be evaluated by an LLM. This separation improves accuracy, cost efficiency, and developer feedback. ## Why Skill Evaluation Is Difficult Skills are artifacts that are both invoked and read by LLMs, so they lack the compiler and test-based validation available for code. - Defects can accumulate silently: - A Skill may never be invoked. - It may be invoked but have little practical effect. - Two especially common problems are: - **Trigger failure:** Trigger conditions are placed in the Skill body instead of its description. Agents inspect the description when deciding whether to invoke a Skill; the body is read only afterward. - **Format failure:** Invalid naming conventions, mismatched folder names, or malformed metadata can prevent the agent from recognizing the Skill at all. ## Rules for Deterministic and Semantic Checks The rubric explicitly separates the 30 checks into: - **17 rule-based checks** - Use regular expressions, counts, and AST parsing. - Handle objective issues consistently and cheaply. - **13 model-based checks** - Use an LLM for meaning-dependent judgments. - Evaluate questions such as whether a description adequately communicates when the Skill should be used. Mixing the two approaches causes problems: - LLMs may overlook clear format violations. - Regular expressions produce false positives when trying to understand varied natural-language intent. - Rule checks can run on every pull request at nearly no cost. - Model checks run only after structural blockers have passed, reducing LLM expenses. ## Rubric Structure and Severity The rubric contains six sections and 30 evaluation items. - Each item is classified as: - **BLOCKER** - **MAJOR** - **MINOR** - Results are summarized using grades from **S to F**. - Any single BLOCKER automatically produces an **F**. - The grade is primarily a compact signal for authors; merge eligibility is simplified to whether the result is F or not-F. ## Validity: Does the Skill Need to Exist? The validity section contains three MAJOR checks. - It asks whether the Skill: - Has a legitimate reason to exist. - Provides recurring or reusable value. - Offers something more useful than simply asking the coding agent to perform the task directly. - This section is intended to identify Skills that should not have been created in the first place. ## Structure: Catching Format Errors The structure section has eight checks, including five BLOCKER-level checks. The rule-based implementation verifies items such as: - Presence and parseability of YAML frontmatter. - `name` following lowercase kebab-case. - Consistency between the Skill’s `name` and its folder name. - Description length between 1 and 1,024 characters. - Absence of XML tags in the body. The checks collect all failures and return them together so authors can fix multiple problems from a single pull-request comment. Only an unrecoverable frontmatter parsing failure causes an immediate return. ## Trigger Design: Making Skills Discoverable The trigger section contains six checks, including one BLOCKER. - A description must communicate both: - **WHAT** the Skill does. - **WHEN** it should be used. - A detailed “when to use” section in the body is insufficient because the agent cannot see the body during invocation selection. - The team initially tried regular expressions to detect trigger signals such as: - “when” - “use when” - Korean expressions meaning “when using” or “at the time of.” - This produced failures because trigger intent can be expressed through emojis, indirect wording, and many other forms that keyword lists cannot cover. - The final approach assigns the semantic question—whether the description covers the body’s trigger conditions—to an LLM. The resulting design favors simple, repeatable rule checks for formal correctness and model-based evaluation only where natural-language meaning is unavoidable.

Read original(opens in new tab)

Large Language ModelsPythonYamlCoding Agents+3

toss·Jun 5, 2026·4 min readCurated summary

How the Toss Team Faces the AI Wave: AI Surf Day

Toss created **AI Surf Day**, a dedicated weekly time for employees to experiment with AI, share lessons, and redesign their workflows. Running on Fridays from April through June, the initiative aims to reduce the AI gap across technical and nontechnical roles by making experimentation collaborative and accessible. Its broader conclusion is that successful AI transformation depends less on formal programs than on culture, time, and people who actively share what they learn. ## AI Surf Day’s Purpose - Employees focus on their core work Monday through Thursday and reserve Friday for AI experimentation and practical application. - The program addresses anxiety and knowledge gaps, especially among nondevelopers who may struggle to identify useful AI information or find time to learn it. - Its concept comes from Jon Kabat-Zinn’s phrase: “You can’t stop the waves, but you can learn to surf.” - The goal is to help Toss become a company that works with AI as a foundation, not merely a workplace where individuals use AI tools. ## AI Surf Club - Employees can create or join informal groups focused on AI topics; roughly 200 clubs were formed at launch. - An **AI Antipattern Study** focused on failures and mistakes, turning participants’ experiences into a practical guide for avoiding common problems. - An **LLM Wiki** group explored how to organize scattered organizational knowledge across data engineering, machine learning, and business teams. - A beginner-focused “Step 0” group helped employees overcome basic technical barriers, such as installing agent tools and asking questions they felt were too fundamental. - A customer-protection team built an external-complaint monitoring portal in one month, along with automation for complaint-response drafts and classification. - A marketing team divided AI work into roles such as: - **Builder:** creates AI-powered tools and workflows - **Curator:** collects useful examples and resources - **Operator:** applies AI to repetitive work - **Scouter:** identifies new opportunities - The clubs emphasized reusable outputs and shared confidence, rather than isolated individual experimentation. ## AI Surf Weekly - Weekly sessions share successful internal AI applications, lessons learned, and current industry insights. - Toss connected employees with similar needs across different departments, enabling them to solve problems quickly by learning from existing internal examples. - Rather than prescribing specific tools, the program presents ideas and use cases that encourage employees to adapt solutions to their own work. - Examples included connecting a sales employee with an HR colleague who had built a similar tool, and pairing a marketer with a designer experienced in AI-powered automation. ## AI Surf Evangelists - Toss selected 142 employees across its affiliated companies and teams to promote AI adoption in their own organizations. - Evangelists were chosen through peer nominations, recognizing people who already shared useful discoveries and helped colleagues overcome AI-related obstacles. - Their responsibilities over three months include: - Reporting effective AI use cases - Sharing useful insights with colleagues - Hosting at least one meetup or workshop - Toss’s Culture team provides workshop templates and facilitation support. - Many teams have conducted workshops around redesigning their existing workflows with AI. - The program treats AI adoption as a team-level workflow redesign challenge, rather than simply measuring individual proficiency with AI tools. ## OpenAI Collaboration and Mini-Hackathon - Toss held a special AI Surf Day with OpenAI on May 15. - Hands-on sessions covered: - Codex-based development workflows for developers - ChatGPT Agent-based automation for nondevelopers - A 2.5-hour hackathon produced two notable projects: - An iOS workflow where Codex implements features, operates the simulator, tests the result, iterates on problems, and produces verification footage. - An agent that classifies thousands of daily Toss Place product records, sends reviewers links, and supports approval or rejection through an admin interface. - These projects demonstrated how AI can become a reusable agentic workflow rather than a one-time assistant. ## Culture Over Programs - Toss does not claim to have a fixed answer for managing AI’s rapid evolution. - The lasting value of AI Surf Day is the protected time for learning and experimentation, along with a culture where employees openly share results and failures. - Successful examples spread naturally across teams, while evangelist-led workshops translate experimentation into concrete changes in how work is performed. Organizations pursuing AI transformation can take a similar approach: create dedicated experimentation time, encourage peer-led learning, recognize existing champions, and focus on reusable workflow improvements rather than tool adoption alone.

Read original(opens in new tab)

AILarge Language ModelsAI AgentsAutomation+4

toss·May 22, 2026·4 min readCurated summary

Tips for Growing the Skills to Solve Cross-Functional Technical Problems

As organizations grow, their hardest technical problems increasingly arise between teams rather than within them. These cross-functional, cross-domain problems cannot be solved through more meetings, status updates, or risk tracking alone; they require redefining the problem, structuring it, and creating an execution model that moves people to action. The post presents Toss’s Technical Program Manager (TPM) role as an example of this approach. ## Why Cross-Functional Problems Persist - Individual teams may perform well while the organization still fails to optimize as a whole. - Technical issues often span product, infrastructure, data, security, operations, strategy, and organizational design. - Common symptoms include: - Unclear ownership - Missing decision-makers - Conflicting priorities - Dependencies across multiple teams - Important “gray areas” with no formal owner - As organizations mature, these boundary problems become more common because team responsibilities become clearer while cross-team gaps remain. ## The Core Principle: Redefine the Problem - Cross-functional technical problems are not solved by increasing management activity. - More frequent meetings, status reports, risk registers, and stakeholder alignment may be useful but often address symptoms. - The real bottleneck may be: - An absent decision structure - Ambiguous ownership - Conflicting priorities - A system that does not connect individual team efforts - Effective problem-solving starts by identifying the underlying issue rather than merely describing delays or communication problems. ## Capabilities Required to Solve These Problems ### Reframing the Problem - Identify why schedules slip or decisions stall. - Determine which responsibilities or decisions are missing. - Find the structural conditions that repeatedly create the same gap. - Without accurate problem definition, organizations continue managing symptoms. ### Turning Ambiguity into Structure - Make decisions, options, responsibilities, dependencies, and sequencing explicit. - Break complex issues into manageable units. - Replace vague discussion with concrete decision points and ownership. ### Exercising Strategic Judgment - Distinguish temporary incidents from recurring structural problems. - Decide whether the issue can be solved within one team or requires broader intervention. - Assess whether immediate action is necessary. - Prioritize problems that improve the organization’s overall execution capability. ### Converting Plans into Execution - Identify who must act and which decisions must happen first. - Remove blockers and turn unclear discussions into explicit decisions. - Secure agreement on action plans and ensure those actions actually occur. - The goal is not merely to monitor execution, but to make execution possible. ### Influencing Without Formal Authority - Cross-functional work rarely succeeds through hierarchy alone. - TPMs need trust, sound judgment, and the ability to translate between teams with different goals and constraints. - Their influence should come from credibility and problem-solving results rather than title. ### Seeing People and Structure Together - Many technical problems are also caused by unclear roles, unsuitable team structures, or outdated operating mechanisms. - Effective intervention may require changing processes, redistributing responsibilities, or involving leadership—not just modifying technology. ## A Practical Starting Point for Less Autonomous Organizations - **Solve a small, concrete bottleneck first:** Demonstrate that involvement makes work clearer and faster. - **Add structure within existing coordination duties:** Use meetings and schedule management to expose decisions, dependencies, and blockers. - **Clarify ownership in a limited scope:** Define the real owner, decision rights, and completion criteria for a small initiative. - **Build evidence through successful cases:** Organizations often recognize new roles through demonstrated results rather than role descriptions. ## Important Cautions - Coordination remains valuable, but it should serve problem-solving rather than become the goal. - Lack of formal authority does not mean lack of influence; trust, structure, and results can be more powerful. - Introducing an idealized role too quickly may trigger resistance. It is better to make the approach work within the organization’s current environment and expand from proven examples. The central recommendation is to stop treating cross-functional technical problems as coordination exercises. First ask what the real bottleneck is, who is missing, and what execution structure would enable progress; then use that understanding to drive concrete organizational change.

Read original(opens in new tab)

Cross Functional CollaborationProblem SolvingOrganizational DesignTechnical Program Management+2

toss·May 14, 2026·4 min readCurated summary

Why High-Performing Organizations Need Toss-Style TPMs in the AI Era

TPM roles are often associated with coordinating schedules, dependencies, risks, and stakeholders. Toss argues that this is no longer enough: as organizations grow and AI increases cross-team complexity, the most important problems often fall into gray areas with no clear owner. Its TPM is therefore redefined as a strategic execution problem-solver who structures ambiguous problems and drives them to measurable resolution. ## Why TPM Needs to Be Redefined - Traditional TPMs typically deliver already-defined technical programs by managing: - Schedules - Risks - Dependencies - Cross-functional communication - At Toss, many difficult problems do not begin as clearly named programs. - Common examples include: - Problems spanning multiple teams with no accountable owner - Strategies without an execution model - Issues recognized as important but lacking priority or authority - Frequent status updates without meaningful change - These problems may involve product, technology strategy, organization design, and operations simultaneously. - AI adoption is accelerating this trend by increasing dependencies across data, security, quality, productivity, and organizational practices. ## How Toss’s TPM Differs from Related Roles - **Product Owner:** Defines what to build, product priorities, and customer or business value. - **Engineering Manager or SDM:** Builds the conditions for a team to execute consistently, including people, quality, and team health. - **Traditional TPM or Technical Project Manager:** Manages delivery of an already-defined initiative. - **Toss TPM:** Addresses the structural problems left between or outside these roles. - Finds important but undefined problems - Establishes ownership and decision rights - Creates an executable structure - Drives the work through to completion - The role is not primarily a project scheduler or people manager; it is a problem solver for organizational gray areas. ## Why Cross-Team Problems Matter in Strong Organizations - In less mature organizations, bottlenecks such as unclear responsibility or poor prioritization are usually visible within teams. - In high-performing organizations, individual teams may operate effectively while problems remain between teams. - Organizational structures clarify accountability and speed decisions, but they can also leave boundary-spanning issues without an owner. - These issues include: - Company-wide problems that local optimization cannot solve - Important long-term work that is not urgent - Responsibilities shared by several teams but owned by none - AI makes these boundary problems more frequent because technical, operational, and organizational concerns increasingly overlap. ## What a Toss TPM Does - **Finds problems proactively** - Identifies recurring gaps, structural bottlenecks, and unnamed problems rather than waiting for assigned work. - **Turns strategy into execution** - Determines which teams must act, in what order, who should be the DRI, and what must be deprioritized. - **Creates value between teams** - Designs solutions where different goals, constraints, and working speeds collide. - **Removes blockers** - Goes beyond reporting risks by changing decision structures, assembling the right people, resetting priorities, or redesigning collaboration. - **Considers people and systems together** - Examines leadership, team composition, authority, and operating mechanisms—not just timelines. - **Measures success through real change** - Success means execution resumes, direction improves, recurring bottlenecks decrease, and future solutions become easier. - Coordination is a useful skill, but problem-solving is the role’s core identity. ## Capabilities Needed to Become This Kind of TPM - **Problem structuring:** Separating symptoms from root problems, identifying stakeholders, and locating decision bottlenecks. - **Execution design:** Translating strategic direction into concrete workflows, sequencing, and ownership. - **Influence and mobilization:** Moving teams without relying solely on formal authority, including handling difficult conversations. - **Systems thinking:** Addressing repeated problems by changing mechanisms rather than relying on individual heroics. - **Follow-through:** Carrying work from discovery and alignment through execution, measurable results, and prevention of recurrence. Toss’s recommendation is to look for important problems that everyone recognizes but no one owns. People who cannot ignore those gaps can begin acting as informal TPMs in their current organizations—turning ambiguous, cross-functional problems into executable solutions and driving them to completion.

Read original(opens in new tab)

AIData GovernanceCross Functional CollaborationOrganizational Design+3

toss·Apr 27, 2026·3 min readCurated summary

Why We Adopted Post-Quantum Cryptography a Decade Before Quantum Computers Arrive

Toss Payments’ biggest legacy-overhaul challenge was not the technology itself, but improving security without disrupting tens of thousands of merchants using decades-old integrations. Because payment systems depend on outdated client environments and small businesses with limited technical resources, security upgrades had to be gradual and carefully communicated. The effort ultimately led from modernizing transport security to adopting post-quantum cryptography in 2026. ## The Challenge of Changing a Legacy Payment Network - Toss Payments supports merchants integrated with its PG system for many years, sometimes decades. - Server-side clients are harder to update than browsers, which update automatically to support new standards. - Security changes such as upgrading TLS, removing weak ciphers, or changing encryption can affect every API call, payment window, and server connection. - Many merchants are small businesses without dedicated developers, making complex security requirements difficult to understand and implement. - As a result, security is a shared responsibility: Toss Payments can strengthen its systems, but legacy merchant environments may still leave connections partially exposed. ## Why Existing Encryption Is Becoming Unsafe - Modern HTTPS and payment systems commonly rely on public-key algorithms such as RSA and ECDSA. - These algorithms are considered secure because conventional computers cannot practically factor enormous numbers or solve elliptic-curve problems. - Quantum algorithms have been mathematically shown to solve these problems efficiently once sufficiently powerful quantum computers exist. - This would make current encryption systems vulnerable, undermining decades of digital-security assumptions. ## Q-Day and “Harvest Now, Decrypt Later” - “Q-Day” refers to the point when quantum computers can break today’s widely used encryption. - Attackers can already intercept and store encrypted payment communications that they cannot currently decrypt. - Once quantum computers become practical, the stored data could be decrypted in bulk. - Payment information is especially valuable because it can remain sensitive for years; data transmitted today could be exposed in the 2030s. - The threat therefore requires action before quantum computers are fully operational. ## A Four-Year Security Upgrade Toss Payments chose a phased approach rather than replacing its security stack all at once: - **2022:** Became the first payment gateway in Korea’s PG industry to implement HTTP/3. - **2022–2025:** Removed weak TLS cipher suites. - **2022–2025:** Completed the rollout of TLS 1.3. - **April 2026:** Implemented post-quantum cryptography (PQC). Each stage balanced stronger protection against the risk of disrupting merchant payments. The gradual rollout gave merchants time to update their systems while ensuring that security improvements continued instead of being postponed indefinitely. ## Starting with HTTP/3 - HTTP/3 is a newer web-transport protocol designed to improve speed and stability, especially on unreliable networks. - It requires TLS 1.3, meaning that adopting HTTP/3 also enforces the use of a modern security protocol. - Toss Payments began with HTTP/3 because it offered both performance improvements and a relatively direct path toward stronger encryption. The broader lesson is that legacy security cannot be improved through a single disruptive upgrade. A phased migration, combined with clear communication and preparation for post-quantum cryptography, allows payment providers to raise security standards while keeping existing merchants operational.

Read original(opens in new tab)

Database DesignApi IntegrationPost Quantum CryptographyQuantum Computing+4

toss·Apr 23, 2026·5 min readCurated summary

Introducing Toss Place's Data Bot 'PANDA': How every team member works like a data expert

PANDA, short for Place Analytics & Data, is Toss Place’s AI data-analysis assistant, designed to let employees retrieve and interpret approved data without waiting for analysts. It was created after the team found that 70% of data requests involved simple metric lookups rather than complex analysis. The project’s main conclusion is that reliable AI analytics depends less on prompting alone and more on standardized data, business definitions, controlled table selection, and iterative validation. ## Why Toss Place Built PANDA - Employees previously relied on analysts to search dashboards, write SQL, or manually investigate data requests. - PANDA provides self-service access within each employee’s security permissions. - It reduces routine extraction work for analysts, allowing them to focus on deeper analysis. - The goal is to establish a stronger culture of “data democracy,” where employees can access and use data immediately. ## Challenges with a Simple AI Chatbot Early experiments showed that asking an AI model to search all company data produced unreliable and expensive results: - Referencing thousands of tables and internal documents consumed excessive tokens. - The model sometimes selected different tables for identical questions, producing inconsistent answers. - It often misunderstood business definitions. For example, “active stores” could mean stores with completed installations or stores that had processed payments. - Inefficient SQL caused unnecessary Snowflake data scans and higher warehouse costs. ## Standardized Data Marts as a Single Source of Truth Toss Place collaborated across its Data Analysis and Data Platform teams to establish reliable standard data marts. - Core concepts, such as store information, were consolidated into standardized tables. - Naming conventions made table and column purposes easier for both people and AI to understand: - Tables follow `{mart_type}_{domain}_{subject}`, such as `fact_device_error_log`. - Columns follow `{prefix}_{entity}_{attribute}_{suffix}`, such as `is_merchant_active`. - Table and column descriptions were documented comprehensively. - The standardization effort reduced ambiguity by ensuring the same business concepts were represented consistently. ## Connecting Business Language to Data Data structures alone could not answer questions about terms such as “installed store” or “store category.” - Domain-specific terms and metric definitions were documented. - These business definitions were linked to the relevant standard data marts. - Data analysts helped reconcile differing interpretations and establish shared organizational definitions. - This gave PANDA the context needed to apply the correct business logic. ## Scoring and Ranking for Reliable Table Selection PANDA limits its search to well-managed tables and uses dbt tags to import selected metadata into a Manifest file. - Tables are ranked using: - **Similarity score:** Based on relationships between the question and table, including table-name matches and description relevance. - **Hierarchy weight:** Reflecting the reliability of the data layer. - The final score is calculated as: `similarity score × hierarchy weight` - Weights are assigned as follows: - Company-wide SSOT metrics: ×4 - Validated standard marts: ×3 - Domain analysis marts: ×2 - Raw bronze data and logs: ×1 - This improves accuracy, consistency, and trustworthiness while reducing unnecessary warehouse exploration. ## Agentic Loop for Querying and Validation Rather than expecting a correct answer in one attempt, PANDA uses an agentic loop. - It selects appropriate tools based on the question. - It explores tables, generates and executes queries, and reviews the results. - If the result appears inaccurate, it can inspect the schema again, modify the query, and retry. - If necessary, it asks the user for clarification. - This approach allows PANDA to handle exceptions dynamically instead of relying only on predefined rules. ## Answers Designed for Practical Use PANDA structures responses so users can understand and apply the results: - **Result:** The requested data or metric. - **Query criteria:** The period, filters, and aggregation method used. - **Insight:** An interpretation that can support practical decisions. This makes PANDA more than a number-retrieval chatbot; it also exposes part of the reasoning process normally provided by a data analyst. ## Adoption and User Response PANDA quickly became part of everyday work at Toss Place. - One-third of employees used it on its first day. - Half of the organization had tried it within a week. - More than 4,000 messages were exchanged during that period. - Current adoption is approximately 70%. - Employees reported feeling more comfortable asking small questions and using data while away from their desks. - Users particularly valued receiving insights alongside raw figures. - Unexpectedly, developers and even data professionals used PANDA actively, suggesting that its answers achieved a meaningful level of trust. ## Future Development PANDA was developed and launched in just one month, but the team plans further improvements. - Increase data coverage to more than 90%. - Raise answer accuracy above 97%. - Use real user questions, follow-up behavior, and abandonment patterns to identify unmet needs. - Expand beyond basic data retrieval to reduce more of the data team’s workload. PANDA’s central lesson is that effective enterprise AI does not require the most complicated technology. It requires solving a real business pain point with trustworthy data foundations, clear definitions, and a workflow that users can rely on.

Read original(opens in new tab)

AIDatabase DesignMachine LearningSql+4

toss·Apr 23, 2026·4 min readCurated summary

StarRocks Operations: Isolating Multi-tenant Workloads with Resource Groups

Toss adopted StarRocks as a real-time OLAP engine to consolidate service queries, analytics, validation, and dashboard workloads on one platform. As different workloads began competing within the same clusters, the key operational challenge became deciding which queries to protect during CPU contention. The article describes a gradual strategy: classify workloads, use `cpu_weight` by default, and introduce `exclusive_cpu_cores` only when stronger isolation is required. ## Why StarRocks - Toss previously relied on separate MySQL and Hadoop-based paths for serving, validation, monitoring, and analytics. - StarRocks reduced this duplication by providing: - A MySQL-compatible SQL interface - Large-scale analytical processing - Real-time service-oriented reads - Workloads eventually included: - Advertising and loan-underwriting services - Dashboards and monitoring tools - Kafka Connect ingestion - Batch jobs and backfills - Average traffic varied by cluster: - Service cluster: approximately 69 QPS over 24 hours and 87 QPS over a week - Monitoring and batch cluster: approximately 20 QPS, plus heavier batch workloads - Peak contention between different workloads mattered more than average QPS. ## Workload Classification Toss prioritized workloads in the following order: 1. Service queries 2. Server-side batch jobs 3. Large-scale ingestion and backfills 4. Monitoring and user query tools such as Grafana, Tableau, and Redash - Service queries required strict SLA protection. - Batch jobs needed to finish reliably but did not require real-time responses. - Ingestion and backfills could overwhelm the cluster and therefore needed explicit limits. - Monitoring queries received the lowest priority. ## Using `cpu_weight` for Shared Capacity - `cpu_weight` distributes CPU proportionally when workloads compete. - Higher-weight groups receive more CPU during contention. - When the cluster is idle, all groups can use available CPU regardless of weight. - Toss used this as the default mechanism for multi-tenant workload control. - Example priorities: - `service_wg`: weight 50 - `batch_wg`: weight 10 - `dashboard_wg`: weight 5 - Resource groups could also specify `mem_limit` and `concurrency_limit`. - StarRocks uses a scheduler inspired by Linux CFS, with pipeline drivers yielding in roughly 100 ms time slices. ## Using `exclusive_cpu_cores` for Strong Isolation - `exclusive_cpu_cores` reserves physical CPU cores for a resource group. - StarRocks binds worker threads to those cores using `pthread_setaffinity_np`. - The group receives separate pools for: - `DriverExecutor` - `ScanExecutor` - `ConnectorScanExecutor` - This prevents the protected workload from competing with shared thread pools. - `exclusive_cpu_cores` and `cpu_weight` cannot be used together within the same resource group, although both types can coexist in one cluster. - The setting is limited to `(0, min_be_cpu_cores - 1]`. - Because it is more rigid and consumes dedicated capacity, Toss recommends using it only when relative priority is insufficient. ## Toss Shopping Case - A cluster handled both real-time queries from `shopping_service` and heavy workloads from `commerce_batch`. - Initially, both workloads had similar priority, allowing large batch queries to degrade service latency. - First adjustment: - Increase `shopping_service`’s `cpu_weight` - Lower `commerce_batch`’s weight - This improved prioritization but did not eliminate latency spikes when heavy batch work overlapped with roughly 1,500 service requests per minute. - Second adjustment: - Place `shopping_service` in its own resource group - Assign dedicated CPU cores with `exclusive_cpu_cores` - Afterward, service latency remained stable even during heavy batch execution. - The operational approach was therefore incremental: begin with weights and escalate to dedicated cores only when necessary. ## Classifier Design and Resource Controls - Resource Groups control how resources are allocated; Classifiers determine which queries enter each group. - Classifiers can match attributes such as: - User - Role - Query type - Source IP - Database - The article recommends using stable identifiers such as `user` or `db` for reliable production behavior. - Examples include mapping service `SELECT` queries by service account and assigning server-side batch queries according to their dedicated user. - CPU isolation alone is insufficient for memory-heavy full scans or sudden spikes involving hundreds of concurrent queries, so memory and concurrency limits are also important. Toss’s practical recommendation is to start with clear workload classification and `cpu_weight`, then add memory and concurrency limits. Use `exclusive_cpu_cores` selectively for latency-sensitive workloads whose SLAs cannot be protected through proportional CPU scheduling alone.

Read original(opens in new tab)

Database DesignMysqlHadoopStarrocks+4