toss·Apr 27, 2026·3 min readCurated summary

Why We Adopted Post-Quantum Cryptography a Decade Before Quantum Computers Arrive

Toss Payments’ biggest legacy-overhaul challenge was not the technology itself, but improving security without disrupting tens of thousands of merchants using decades-old integrations. Because payment systems depend on outdated client environments and small businesses with limited technical resources, security upgrades had to be gradual and carefully communicated. The effort ultimately led from modernizing transport security to adopting post-quantum cryptography in 2026. ## The Challenge of Changing a Legacy Payment Network - Toss Payments supports merchants integrated with its PG system for many years, sometimes decades. - Server-side clients are harder to update than browsers, which update automatically to support new standards. - Security changes such as upgrading TLS, removing weak ciphers, or changing encryption can affect every API call, payment window, and server connection. - Many merchants are small businesses without dedicated developers, making complex security requirements difficult to understand and implement. - As a result, security is a shared responsibility: Toss Payments can strengthen its systems, but legacy merchant environments may still leave connections partially exposed. ## Why Existing Encryption Is Becoming Unsafe - Modern HTTPS and payment systems commonly rely on public-key algorithms such as RSA and ECDSA. - These algorithms are considered secure because conventional computers cannot practically factor enormous numbers or solve elliptic-curve problems. - Quantum algorithms have been mathematically shown to solve these problems efficiently once sufficiently powerful quantum computers exist. - This would make current encryption systems vulnerable, undermining decades of digital-security assumptions. ## Q-Day and “Harvest Now, Decrypt Later” - “Q-Day” refers to the point when quantum computers can break today’s widely used encryption. - Attackers can already intercept and store encrypted payment communications that they cannot currently decrypt. - Once quantum computers become practical, the stored data could be decrypted in bulk. - Payment information is especially valuable because it can remain sensitive for years; data transmitted today could be exposed in the 2030s. - The threat therefore requires action before quantum computers are fully operational. ## A Four-Year Security Upgrade Toss Payments chose a phased approach rather than replacing its security stack all at once: - **2022:** Became the first payment gateway in Korea’s PG industry to implement HTTP/3. - **2022–2025:** Removed weak TLS cipher suites. - **2022–2025:** Completed the rollout of TLS 1.3. - **April 2026:** Implemented post-quantum cryptography (PQC). Each stage balanced stronger protection against the risk of disrupting merchant payments. The gradual rollout gave merchants time to update their systems while ensuring that security improvements continued instead of being postponed indefinitely. ## Starting with HTTP/3 - HTTP/3 is a newer web-transport protocol designed to improve speed and stability, especially on unreliable networks. - It requires TLS 1.3, meaning that adopting HTTP/3 also enforces the use of a modern security protocol. - Toss Payments began with HTTP/3 because it offered both performance improvements and a relatively direct path toward stronger encryption. The broader lesson is that legacy security cannot be improved through a single disruptive upgrade. A phased migration, combined with clear communication and preparation for post-quantum cryptography, allows payment providers to raise security standards while keeping existing merchants operational.

Read original(opens in new tab)

Database DesignApi IntegrationPost Quantum CryptographyQuantum Computing+4

toss·Apr 23, 2026·5 min readCurated summary

Introducing Toss Place's Data Bot 'PANDA': How every team member works like a data expert

PANDA, short for Place Analytics & Data, is Toss Place’s AI data-analysis assistant, designed to let employees retrieve and interpret approved data without waiting for analysts. It was created after the team found that 70% of data requests involved simple metric lookups rather than complex analysis. The project’s main conclusion is that reliable AI analytics depends less on prompting alone and more on standardized data, business definitions, controlled table selection, and iterative validation. ## Why Toss Place Built PANDA - Employees previously relied on analysts to search dashboards, write SQL, or manually investigate data requests. - PANDA provides self-service access within each employee’s security permissions. - It reduces routine extraction work for analysts, allowing them to focus on deeper analysis. - The goal is to establish a stronger culture of “data democracy,” where employees can access and use data immediately. ## Challenges with a Simple AI Chatbot Early experiments showed that asking an AI model to search all company data produced unreliable and expensive results: - Referencing thousands of tables and internal documents consumed excessive tokens. - The model sometimes selected different tables for identical questions, producing inconsistent answers. - It often misunderstood business definitions. For example, “active stores” could mean stores with completed installations or stores that had processed payments. - Inefficient SQL caused unnecessary Snowflake data scans and higher warehouse costs. ## Standardized Data Marts as a Single Source of Truth Toss Place collaborated across its Data Analysis and Data Platform teams to establish reliable standard data marts. - Core concepts, such as store information, were consolidated into standardized tables. - Naming conventions made table and column purposes easier for both people and AI to understand: - Tables follow `{mart_type}_{domain}_{subject}`, such as `fact_device_error_log`. - Columns follow `{prefix}_{entity}_{attribute}_{suffix}`, such as `is_merchant_active`. - Table and column descriptions were documented comprehensively. - The standardization effort reduced ambiguity by ensuring the same business concepts were represented consistently. ## Connecting Business Language to Data Data structures alone could not answer questions about terms such as “installed store” or “store category.” - Domain-specific terms and metric definitions were documented. - These business definitions were linked to the relevant standard data marts. - Data analysts helped reconcile differing interpretations and establish shared organizational definitions. - This gave PANDA the context needed to apply the correct business logic. ## Scoring and Ranking for Reliable Table Selection PANDA limits its search to well-managed tables and uses dbt tags to import selected metadata into a Manifest file. - Tables are ranked using: - **Similarity score:** Based on relationships between the question and table, including table-name matches and description relevance. - **Hierarchy weight:** Reflecting the reliability of the data layer. - The final score is calculated as: `similarity score × hierarchy weight` - Weights are assigned as follows: - Company-wide SSOT metrics: ×4 - Validated standard marts: ×3 - Domain analysis marts: ×2 - Raw bronze data and logs: ×1 - This improves accuracy, consistency, and trustworthiness while reducing unnecessary warehouse exploration. ## Agentic Loop for Querying and Validation Rather than expecting a correct answer in one attempt, PANDA uses an agentic loop. - It selects appropriate tools based on the question. - It explores tables, generates and executes queries, and reviews the results. - If the result appears inaccurate, it can inspect the schema again, modify the query, and retry. - If necessary, it asks the user for clarification. - This approach allows PANDA to handle exceptions dynamically instead of relying only on predefined rules. ## Answers Designed for Practical Use PANDA structures responses so users can understand and apply the results: - **Result:** The requested data or metric. - **Query criteria:** The period, filters, and aggregation method used. - **Insight:** An interpretation that can support practical decisions. This makes PANDA more than a number-retrieval chatbot; it also exposes part of the reasoning process normally provided by a data analyst. ## Adoption and User Response PANDA quickly became part of everyday work at Toss Place. - One-third of employees used it on its first day. - Half of the organization had tried it within a week. - More than 4,000 messages were exchanged during that period. - Current adoption is approximately 70%. - Employees reported feeling more comfortable asking small questions and using data while away from their desks. - Users particularly valued receiving insights alongside raw figures. - Unexpectedly, developers and even data professionals used PANDA actively, suggesting that its answers achieved a meaningful level of trust. ## Future Development PANDA was developed and launched in just one month, but the team plans further improvements. - Increase data coverage to more than 90%. - Raise answer accuracy above 97%. - Use real user questions, follow-up behavior, and abandonment patterns to identify unmet needs. - Expand beyond basic data retrieval to reduce more of the data team’s workload. PANDA’s central lesson is that effective enterprise AI does not require the most complicated technology. It requires solving a real business pain point with trustworthy data foundations, clear definitions, and a workflow that users can rely on.

Read original(opens in new tab)

Database DesignAIMachine LearningSql+4

toss·Apr 23, 2026·4 min readCurated summary

StarRocks Operations: Isolating Multi-tenant Workloads with Resource Groups

Toss adopted StarRocks as a real-time OLAP engine to consolidate service queries, analytics, validation, and dashboard workloads on one platform. As different workloads began competing within the same clusters, the key operational challenge became deciding which queries to protect during CPU contention. The article describes a gradual strategy: classify workloads, use `cpu_weight` by default, and introduce `exclusive_cpu_cores` only when stronger isolation is required. ## Why StarRocks - Toss previously relied on separate MySQL and Hadoop-based paths for serving, validation, monitoring, and analytics. - StarRocks reduced this duplication by providing: - A MySQL-compatible SQL interface - Large-scale analytical processing - Real-time service-oriented reads - Workloads eventually included: - Advertising and loan-underwriting services - Dashboards and monitoring tools - Kafka Connect ingestion - Batch jobs and backfills - Average traffic varied by cluster: - Service cluster: approximately 69 QPS over 24 hours and 87 QPS over a week - Monitoring and batch cluster: approximately 20 QPS, plus heavier batch workloads - Peak contention between different workloads mattered more than average QPS. ## Workload Classification Toss prioritized workloads in the following order: 1. Service queries 2. Server-side batch jobs 3. Large-scale ingestion and backfills 4. Monitoring and user query tools such as Grafana, Tableau, and Redash - Service queries required strict SLA protection. - Batch jobs needed to finish reliably but did not require real-time responses. - Ingestion and backfills could overwhelm the cluster and therefore needed explicit limits. - Monitoring queries received the lowest priority. ## Using `cpu_weight` for Shared Capacity - `cpu_weight` distributes CPU proportionally when workloads compete. - Higher-weight groups receive more CPU during contention. - When the cluster is idle, all groups can use available CPU regardless of weight. - Toss used this as the default mechanism for multi-tenant workload control. - Example priorities: - `service_wg`: weight 50 - `batch_wg`: weight 10 - `dashboard_wg`: weight 5 - Resource groups could also specify `mem_limit` and `concurrency_limit`. - StarRocks uses a scheduler inspired by Linux CFS, with pipeline drivers yielding in roughly 100 ms time slices. ## Using `exclusive_cpu_cores` for Strong Isolation - `exclusive_cpu_cores` reserves physical CPU cores for a resource group. - StarRocks binds worker threads to those cores using `pthread_setaffinity_np`. - The group receives separate pools for: - `DriverExecutor` - `ScanExecutor` - `ConnectorScanExecutor` - This prevents the protected workload from competing with shared thread pools. - `exclusive_cpu_cores` and `cpu_weight` cannot be used together within the same resource group, although both types can coexist in one cluster. - The setting is limited to `(0, min_be_cpu_cores - 1]`. - Because it is more rigid and consumes dedicated capacity, Toss recommends using it only when relative priority is insufficient. ## Toss Shopping Case - A cluster handled both real-time queries from `shopping_service` and heavy workloads from `commerce_batch`. - Initially, both workloads had similar priority, allowing large batch queries to degrade service latency. - First adjustment: - Increase `shopping_service`’s `cpu_weight` - Lower `commerce_batch`’s weight - This improved prioritization but did not eliminate latency spikes when heavy batch work overlapped with roughly 1,500 service requests per minute. - Second adjustment: - Place `shopping_service` in its own resource group - Assign dedicated CPU cores with `exclusive_cpu_cores` - Afterward, service latency remained stable even during heavy batch execution. - The operational approach was therefore incremental: begin with weights and escalate to dedicated cores only when necessary. ## Classifier Design and Resource Controls - Resource Groups control how resources are allocated; Classifiers determine which queries enter each group. - Classifiers can match attributes such as: - User - Role - Query type - Source IP - Database - The article recommends using stable identifiers such as `user` or `db` for reliable production behavior. - Examples include mapping service `SELECT` queries by service account and assigning server-side batch queries according to their dedicated user. - CPU isolation alone is insufficient for memory-heavy full scans or sudden spikes involving hundreds of concurrent queries, so memory and concurrency limits are also important. Toss’s practical recommendation is to start with clear workload classification and `cpu_weight`, then add memory and concurrency limits. Use `exclusive_cpu_cores` selectively for latency-sensitive workloads whose SLAs cannot be protected through proportional CPU scheduling alone.

Read original(opens in new tab)

Database DesignMysqlHadoopStarrocks+4

toss·Apr 23, 2026·4 min readCurated summary

Applying Post-Quantum Cryptography for the Quantum Computing Era: Why Implement It 10 Years Early?

Toss Payments’ hardest legacy-modernization challenge was not replacing old systems, but improving security across tens of thousands of merchants with diverse, outdated environments. Over four years, it gradually introduced HTTP/3, removed vulnerable cipher suites, deployed TLS 1.3, and ultimately adopted post-quantum cryptography (PQC) in April 2026. The central lesson is that security upgrades must begin early and be introduced gradually, with merchant support and backward compatibility built in. ## Breaking the Inertia of Legacy Systems - Mission-critical payment services tend to follow the principle: “If it works, don’t touch it.” - Security protocol changes are particularly difficult because they can affect every merchant integration and may be hard to troubleshoot or roll back. - Many merchants still operate decades-old server-side systems that cannot support modern security policies. - Documentation alone is often insufficient, especially for merchants without dedicated development teams. - Because every API, SDK, payment window, and server connection is part of the security boundary, Toss Payments could not improve security independently of its merchants. ## Why Existing Encryption Must Evolve - Modern HTTPS, banking, and payment systems rely heavily on RSA and ECDSA. - These algorithms are considered secure because factoring large numbers and solving elliptic-curve problems is impractical for classical computers. - Quantum computers could solve these problems efficiently, making current public-key cryptography vulnerable. - The anticipated point at which quantum computers can break these systems is often called “Q-Day.” - The “Harvest Now, Decrypt Later” threat means attackers can collect encrypted payment data today and decrypt it years later when quantum computers become practical. ## A Four-Year Security Upgrade Program Toss Payments chose a gradual migration strategy to improve security without abruptly disrupting merchant payments: - **2022:** Introduced HTTP/3, which requires TLS 1.3. - **2022–2025:** Removed vulnerable TLS cipher suites. - **2022–2025:** Enabled TLS 1.3 across all endpoints. - **April 2026:** Introduced post-quantum cryptography. ## HTTP/3 as a Low-Impact Starting Point - HTTP/3 improves speed and reliability on unstable networks. - Because it requires TLS 1.3, enabling HTTP/3 also raised security standards. - Modern browsers automatically select HTTP/3, so merchants required no configuration changes. - This made HTTP/3 an effective first step with minimal migration risk. ## Gradual Cipher Suite Removal - A cipher suite defines the algorithms used by a client and server to establish encrypted communication. - Some legacy merchant servers supported only vulnerable suites, such as `TLS_RSA_WITH_AES_128_CBC_SHA`. - Removing them immediately could stop payments for affected merchants, while delaying removal would leave the wider ecosystem exposed. - Toss Payments used: - Merchant-by-merchant compatibility analysis - Individual notifications six months to a year in advance - Environment-specific documentation and configuration guidance - Technical consulting where necessary - The Technical Account Manager team was essential in coordinating these changes and communicating with merchants in accessible language. ## TLS 1.3 Deployment - TLS 1.2 remained the minimum supported version, while TLS 1.3 was added alongside it. - Clients capable of TLS 1.3 automatically use the stronger protocol. - Older clients continue using TLS 1.2 without forced changes. - TLS 1.3 was enabled endpoint by endpoint from 2022 and supported across all endpoints by 2025. - The process demonstrated that ecosystem-wide security improvements require more time helping merchants migrate than technically changing the servers. ## Post-Quantum Cryptography - Toss Payments began preparing for PQC in 2025 and completed deployment in April 2026. - Modern browsers and clients that support PQC automatically use stronger quantum-resistant channels. - Unsupported environments continue using established encryption methods, preserving compatibility. - Merchants do not need to change configurations or update their integrations. - The approach provides stronger protection against future quantum attacks while minimizing present-day disruption. ## Cross-Team Collaboration - **Infra Team:** Applied PQC within Toss Payments’ private data-center infrastructure and physical hardware. - **Server Platform Team:** Integrated PQC into live traffic paths in AWS. - **TAM Team:** Used its experience from the cipher-suite migration to guide merchants and assess integration environments. - The result was a large-scale, proactive security deployment across the private payment ecosystem. Toss Payments’ experience suggests that organizations should start security migrations well before threats become immediate. Compatibility layers, staged enforcement, and sustained technical support allow legacy ecosystems to adopt stronger security without sacrificing availability.

Read original(opens in new tab)

Database DesignApi IntegrationPost Quantum CryptographyHttp 3+3

toss·Apr 16, 2026·5 min readCurated summary

Extending Real-time Ad Frequency Capping Aggregation to One Week with Apache Flink + RocksDB Tuning

The post describes Toss’s expansion of real-time advertising frequency-capping from short Flink windows to periods of up to seven days. The new system provides accurate sliding counts from one minute to seven days through a single Redis lookup, while treating Flink state as the authoritative source and Redis as its projection. The migration addressed architectural complexity, backfill consistency, and distinct RocksDB bottlenecks across three specialized Flink applications. ## Frequency Capping and Its Business Impact - Frequency capping controls how many times an individual user sees an advertisement. - Incorrect counts can: - Waste an advertiser’s budget through excessive exposure. - Prevent valid impressions when the system believes a limit has already been reached. - Different products require different windows, such as: - Three impressions per day. - One impression over the previous seven days. - The target system therefore needed accurate, real-time sliding counts from one minute through seven days. ## Limitations of the Previous Batch-Oriented System The original architecture combined three Airflow-managed layers: - **Head** - Stored current-day and previous-day events in Redis through a Spring Kafka consumer. - Updated counts immediately per event. - **Mid** - Used daily Spark jobs to pre-aggregate data from D-2 through D-7. - **Tail** - Added hourly correction data around the boundary between Head and Mid. - Airflow workflows ran approximately 75 times per day. At serving time, the API could perform up to four Redis lookups and combine the results. - This structure was difficult to maintain because of the dependencies and boundary conditions between Head, Mid, and Tail. - Time-based truncation made precise event-level sliding windows difficult. - The architecture remains useful for longer windows such as 30 days and fixed daily aggregates, especially when data exceeds Kafka retention and must be recovered from batch storage. - Extending the existing short-window Flink system was chosen to simplify serving and reduce DAG complexity. ## Three Flink Applications Rather than place all windows in one Flink job, the team split processing into three applications with shared code but independent RocksDB configurations: - **Minutes** - Handles one- to 30-minute windows. - Frequent event expiration creates heavy write traffic. - Its main concern is RocksDB Write Buffer Manager pressure and resulting Write Stalls. - **Hours** - Handles windows up to 12 hours. - Maintains many more advertisement IDs in state. - Filter Block Cache misses can saturate CPU. - Redis synchronization requires an O(N) scan over advertisement IDs in each window. - Filter Block tuning and additional managed memory are important. - **Days** - Handles the largest state volume. - A seven-day window can produce approximately 68 GB of live SST files and 220–230 GB savepoints. - Checkpoint I/O becomes the primary bottleneck, motivating a Flink Changelog design. Separating the applications allowed each workload’s RocksDB and runtime bottlenecks to be optimized independently without affecting the others. ## Backfill and Catch-up Architecture The most difficult migration problem was maintaining correctness at the transition point between historical data and live processing. - **Backfill** - Loads seven days of historical events. - Only increments counts. - Does not register expiration timers. - Synchronizes the initialized values to Redis once and then finishes. - **Catch-up** - Re-reads historical events from Kafka. - Rebuilds both counts and expiration timers. - Begins writing to Redis after reaching the historical scan end. - Enables each window only after sufficient lookback data has been reconstructed. The two phases cannot safely share one pipeline: - Backfill must only add historical counts. - Live or catch-up processing must both add new events and subtract events that leave the sliding window. - If expiration timers ran while backfill was incomplete, decrements could occur before all historical increments had been applied, producing incorrect results. - Flink batch mode was rejected because state is discarded when the job finishes. - A Spark and Hive-based approach was also rejected because it would introduce additional systems and complicate the single-source-of-truth model. Separate Kafka consumer groups were required so that backfill offsets would not cause catch-up events to be skipped. ## State as the Single Source of Truth - Flink state stores the authoritative aggregate. - Redis is treated only as a serving projection. - If Redis becomes inconsistent, it can be reconstructed from Flink state. - This design preserves correctness during failures, restarts, and Redis resynchronization. ## Maintaining Transition Consistency Three mechanisms were combined to make the backfill-to-catch-up boundary reliable: - **Redis write condition** - Writes are based on each event’s `eventTime` being after the backfill completion point. - Using the global watermark directly could block all writes because one slow or idle partition can hold back the watermark. - **`withIdleness` set to 60 seconds** - Excludes inactive Kafka partitions from watermark progression. - A longer timeout avoids falsely marking a partition idle just before a bounded source emits `MAX_WATERMARK`. - **Timer state TTL** - Must exceed the sliding-window expiration period. - If the timer fires after its associated state has expired, `timerState.get()` returns null and the decrement is skipped. - This would leave counts artificially high after delays or recovery. - The state is manually cleaned up after timer processing. ## RocksDB and Flink Runtime Tuning Once the system was serving real-time results, operational metrics exposed different bottlenecks in each application. - The minutes application initially experienced RocksDB Write Stalls caused by pressure on the shared Write Buffer Manager. - RocksDB first stores writes in MemTables and flushes them into SST files organized across levels L0–L6. - Flink maps managed state types such as `MapState` and `ValueState` to separate RocksDB Column Families. - Because multiple Column Families share the Write Buffer Manager’s memory budget, write-heavy workloads require careful tuning of RocksDB memory and write paths. - The hours and days applications require different optimizations focused on cache misses, CPU usage, checkpoint I/O, and level management. ## Practical Conclusion For real-time frequency capping, a unified Flink-based design can simplify serving and improve sliding-window accuracy, but long windows should not automatically be combined with short ones in a single job. Separate applications, state-as-SSOT, distinct backfill and catch-up pipelines, and workload-specific RocksDB tuning are essential for maintaining correctness and operability at scale.

Read original(opens in new tab)

Database DesignRedisApache FlinkAirflow+3

toss·Mar 25, 2026·3 min readCurated summary

Metric Review, Driving Execution

Metric Review is Toss Place’s weekly operating system for turning data insights into product and business action. By connecting OKRs to a hierarchy of driver metrics, analysts continuously detect risks, test hypotheses, and encourage execution rather than merely reporting results. The approach has improved data literacy and helped teams contribute directly to company-level Key Results. ### Building a Data-Literate Organization - Toss Place aims for everyone—not only analysts—to perform effective analysis. - The Data Platform Team strengthens data quality and infrastructure, while the Data Analysis Team provides domain knowledge and delivery capabilities. - Analysts are expected to develop three complementary skills: - Technical expertise with data and analysis tools - Logical communication - Deep product and business knowledge ### Why Metric Review Matters - Metrics serve as a shared language for aligning teams around organizational goals. - Metric Review helps teams identify: - Whether goals are on track - Emerging risks - New opportunities - Analysts act as **Metric Owners**, providing insights that support better decisions and following through until actions and outcomes are verified. ### Operating Model #### OKR-Linked Metric Hierarchy - Company-level Key Results flow down to team and silo-level Key Results. - The levers that influence each team’s KR become its driver metrics. - This hierarchy provides the structure for identifying opportunities and threats. #### A Continuous Analysis Cycle - The operating cycle is: - Goal setting → hypothesis formation → validation and execution → insight discovery - Metric Review translates this into: - Metric analysis → hypothesis testing → insight sharing → driving action - Exploratory data analysis (EDA) is also conducted when metric movements suggest deeper questions. #### Weekly Consistency - Reviewing metrics weekly helps teams detect small changes before they become significant. - Regular analysis also builds domain knowledge by requiring analysts to understand why metrics rise or fall. - Monthly or occasional reporting may explain past performance but often misses the window for timely action. ### Examples of Business Impact #### Growth Tribe: Establishing Shared Metrics - Weekly metric reviews initially focused on reporting performance and interpretation. - Over time, the practice changed how teams worked: - Designers defined product hypotheses around target metrics and incorporated logging requirements into designs. - Backend developers collaborated with analysts on analysis-friendly data structures. - Client developers prioritized measurable events when implementing logs. - Product Owners combined qualitative feedback with quantitative results to determine whether goals were on track. - This created a feedback loop that contributed to successful product launches and improved company metrics. #### POS Tribe: Segment-Specific Solutions - POS adoption varied significantly across partner dealerships. - Analysts used clustering to identify groups with different adoption patterns. - Product teams combined cluster analysis with interviews to design tailored interventions: - Low-adoption groups received stronger education and onboarding. - High-adoption groups received simplified store creation and installation flows. - Segment-specific actions accelerated POS expansion more effectively than a single broad solution. #### Supply Chain: Forecast-Based Optimization - Because Toss Place manufactures and distributes hardware, supply-chain metrics are strategically important. - Analysts and the SCM team monitored: - Device shipments - Market installation rates - Inventory and ordering forecasts - Potential improvement areas - Hypothesis-driven actions helped optimize distribution and reduce costs. ### How the Organization Changed - Analysts became Metric Owners rather than report writers. - Product teams began asking, “Which metric should we move?” before asking what to build. - Business teams increasingly aligned strategies using quantitative evidence. - Repeated Metric Reviews strengthened organization-wide data literacy and contributed to meaningful company Key Result achievement. The practical recommendation is to evaluate analysis by whether it leads to measurable action. Teams should structure problems, create testable hypotheses, define follow-up metrics, and maintain a consistent review rhythm until the execution loop is closed.

Read original(opens in new tab)

Database DesignData AnalysisOkrsData Platform+4

toss·Mar 20, 2026·3 min readCurated summary

Automating Service Vulnerability Analysis using LLM #2

The post explains how Toss Security Research improved AI-driven vulnerability analysis in a research network. Its main challenges were efficiently providing large codebases to an AI and making analysis results consistent and complete. The solution combined a custom code-browsing MCP server with SAST tools used not to identify vulnerabilities directly, but to enumerate all input-to-function paths that the AI must review. ## Efficiently Providing Large Codebases - Tools such as Cursor and Claude Code can search large projects, but primarily rely on pattern matching with tools like ripgrep. - Without prebuilt indexes, they may miss relevant code or waste tokens exploring unnecessary files. - The team built an MCP server that: - Uses **ctags** to index symbol definitions. - Uses **tree-sitter** to parse function boundaries. - Allows AI to access code remotely, similar to IDE features such as “Go to Definition” and “Find References.” ### SourceCode Browse MCP The MCP server provides four main tools: - **`find_references()`** - Searches for symbols or patterns using ripgrep. - Returns file paths, line numbers, snippets, total matches, and whether results were truncated. - **`read_definition()`** - Looks up definitions through the ctags index. - Returns metadata such as file, line, symbol type, language, signature, and scope. - Uses tree-sitter to include the complete function body when requested. - **`read_source()`** - Reads a configurable number of lines before and after a target line. - Lets the AI retrieve only the relevant local context instead of entire files. - **`get_project_structure()`** - Returns the indexed project’s directory structure. - Provides the AI with a project “blueprint,” which is especially important in remote environments where it cannot inspect the repository locally. The MCP workflow is to locate relevant symbols with `find_references()` and `read_definition()`, inspect nearby code with `read_source()`, and use `get_project_structure()` to understand the overall project. ## Improving Consistency and Accuracy - AI analysis produced inconsistent results: for example, it might find all 10 XSS vulnerabilities in one run but only 8 in another. - This variability made the results difficult to trust. - The team combined AI analysis with SAST tooling to ensure complete coverage. ## Using SAST to Enumerate Review Candidates - Rather than passing SAST-detected vulnerabilities directly to the AI, the team used SAST as a candidate-generation tool. - This avoids limiting the AI to vulnerabilities that the SAST engine itself knows how to detect. - SAST extracts every location where untrusted input enters the application and tracks its possible flow to function calls. - Custom Semgrep taint rules identify sources such as: - Spring `@RequestParam` - `@PathVariable` - `@RequestHeader` - Fields read from `@RequestBody` DTOs - `@RequestPart` - `@ModelAttribute` - `@RequestAttribute` - Potential sinks include generic function calls and object method calls. - The AI then reviews every extracted source-to-sink path, combining the completeness of static analysis with the broader reasoning ability of an LLM. The overall approach is to use deterministic indexing and SAST for coverage, while relying on AI for deeper vulnerability interpretation.

Read original(opens in new tab)

Database DesignLarge Language ModelsKotlinAi Security+4

toss·Mar 6, 2026·3 min readCurated summary

Foreign User Research: Why

Toss investigated why many foreign users struggle to use Korea’s financial services, even after signing up. Research showed that confusing identity verification, name formatting, and address entry often prevented users from completing registration, forcing them to visit bank branches for routine tasks. By redesigning the name-entry and authentication process, Toss increased the foreign-user verification completion rate by about 15% and eliminated the gap with Korean users. ## Investigating Foreign Users’ Financial Experiences - Foreigners often perceive Korea’s banking system as complex and difficult to navigate without assistance. - Toss wanted to make its “finance for everyone” vision include foreign residents. - The team suspected that several verification steps caused users to abandon registration: - Preparing a foreigner registration card - Mismatches in telecom-provider information - One-won account verification - Difficulties entering names and personal details ## Field Research with Blue-Collar Workers - The team focused especially on blue-collar foreign workers, whose financial habits were less understood than those of students or white-collar workers. - Initial attempts to arrange factory interviews failed, so researchers visited the Siheung Industrial Complex during lunch hours. - Street interviews were difficult because formal clothing, identification badges, and consent documents made passersby cautious. - A more casual approach helped the team conduct several interviews. - Researchers later visited a multicultural center in Pocheon, where they met foreign residents from different countries and with varying lengths of stay. ## Why Foreign Users Rely on Bank Branches - Mobile banking often felt like a complicated system that users could access only after repeated trial and error. - Many users abandoned the process before reaching any financial-service features. ### Name Entry and Identity Verification - Users were unsure how to format their names: - Where to place spaces - Whether to enter family names first - Whether to match their foreigner registration card, bank account, or telecom records - A name such as “BRAD PITT” might need to be entered in an unexpected format, such as “BR AD.” - Some users repeatedly failed verification because their name format differed across institutions. - One participant had never successfully completed online identity verification under their own name in eight years. - Error messages rarely explained the actual cause of failure. - After five or more failed attempts, users could no longer continue. ### Address Entry - Entering Korean addresses was another major barrier, especially for users unfamiliar with typing Korean. - Users tried postal codes, English addresses, and lot numbers, then searched through address lists. - Search results often displayed too many options, making the correct address difficult to locate. - Repeated unsuccessful searches led some users to abandon registration and visit an offline branch instead. ## Improving the Authentication Funnel - Research identified name entry and authentication as the primary causes of foreign-user drop-off. - Toss’s product team redesigned the name-input structure and authentication flow. - The changes increased the foreign-user authentication completion rate by approximately 15%. - The completion-rate gap between Korean and foreign users was ultimately eliminated. Toss’s research demonstrates that inclusive financial services require understanding users who are often overlooked. Removing small but fundamental barriers in registration and authentication can make digital banking accessible to a much broader population.

Read original(opens in new tab)

Database DesignUser ExperienceUser ResearchMobile App+4

toss·Dec 10, 2025Original article

Legacy Settlement Modernization: From the (opens in new tab)

Toss Payments recently overhauled its 20-year-old legacy settlement system to overcome deep-seated technical debt and prepare for massive transaction growth. By shifting from monolithic SQL queries and aggregated data to a granular, object-oriented architecture, the team significantly improved system maintainability, traceability, and batch processing performance. The transition focused on breaking down complex dependencies and ensuring that every transaction is verifiable and reproducible. ### Replacing Monolithic SQL with Object-Oriented Logic * The legacy system relied on a "giant common query" filled with nested `DECODE`, `CASE WHEN`, and complex joins, making it nearly impossible to identify the impact of small changes. * The team applied a "Divide and Conquer" strategy, splitting the massive query into distinct domains and refined sub-functions. * Business logic was moved from the database layer into Kotlin-based objects (e.g., `SettlementFeeCalculator`), making business rules explicit and easier to test. * This modular approach allowed for "Incremental Migration," where specific features (like exchange rate conversions) could be upgraded to the new system independently. ### Improving Traceability through Granular Data Modeling * The old system stored data in an aggregated state (Sum), which prevented developers from tracing errors back to specific transactions or reusing data for different reporting needs. * The new architecture manages data at the minimum transaction unit (1:1), ensuring that every settlement result corresponds to a specific transaction. * "Setting Snapshots" were introduced to store the exact contract conditions (fee rates, VAT status) at the time of calculation, allowing the system to reconstruct the context of past settlements. * A state-based processing model was implemented to enable selective retries for failed transactions, significantly reducing recovery time compared to the previous "all-or-nothing" transaction approach. ### Optimizing High-Resolution Data and Query Performance * Managing data at the transaction level led to an explosion in data volume, necessitating specialized database strategies. * The team implemented date-based Range Partitioning and composite indexing on settlement dates to maintain high query speeds despite the increased scale. * To balance write performance and read needs, they created "Query-specific tables" that offload the processing burden from the main batch system. * Complex administrative queries were delegated to a separate high-performance data serving platform, maintaining a clean separation between core settlement logic and flexible data analysis. ### Resolving Batch Performance and I/O Bottlenecks * The legacy batch system struggled with long processing times that scaled poorly with transaction growth due to heavy I/O and single-threaded processing. * I/O was minimized by caching merchant contract information in memory at the start of a batch step, eliminating millions of redundant database lookups. * The team optimized the `ItemProcessor` in Spring Batch by implementing bulk lookups (using a Wrapper structure) to handle multiple records at once rather than querying the database for every individual item. This modernization demonstrates that scaling a financial system requires moving beyond "convenient" aggregations toward a granular, state-driven architecture. By decoupling business logic from the database and prioritizing data traceability, Toss Payments has built a foundation capable of handling the next generation of transaction volumes.

Database DesignKotlinRefactoringQuery Optimization+3

toss·Nov 30, 2025Original article

From Legacy Payment Ledger to Scalable System (opens in new tab)

Toss Payments successfully modernized a 20-year-old legacy payment ledger by transitioning to a decoupled, MySQL-based architecture designed for high scalability and consistency. By implementing strategies like INSERT-only immutability and event-driven domain isolation, they overcame structural limitations such as the inability to handle split payments. Ultimately, the project demonstrates that robust system design must be paired with resilient operational recovery mechanisms to manage the complexities of large-scale financial migrations. ### Legacy Ledger Challenges * **Inconsistent Schemas:** Different payment methods used entirely different table structures; for instance, a table named `REFUND` unexpectedly contained only account transfer data rather than all refund types. * **Domain Coupling:** Multiple domains (settlement, accounting, and payments) shared the same tables and columns, meaning a single schema change required impact analysis across several teams. * **Structural Limits:** A rigid 1:1 relationship between a payment and its method prevented the implementation of modern features like split payments or "Dutch pay" models. ### New Ledger Architecture * **Data Immutability:** The system shifted from updating existing rows to an **INSERT-only** principle, ensuring a reliable audit trail and preventing database deadlocks. * **Event-Driven Decoupling:** Instead of direct database access, the system uses Kafka to publish payment events, allowing independent domains to consume data without tight coupling. * **Payment-Approval Separation:** By separating the "Payment" (the transaction intent) from the "Approval" (the specific financial method), the system now supports multiple payment methods per transaction. ### Safe Migration and Data Integrity * **Asynchronous Mirroring:** To maintain zero downtime, data was initially written to the legacy system and then asynchronously loaded into the new MySQL ledger. * **Resource Tuning:** Developers used dedicated migration servers within the same AWS Availability Zone to minimize latency and implemented **Bulk Inserts** to handle hundreds of millions of rows efficiently. * **Verification Batches:** A separate batch process ran every five minutes against a Read-Only (RO) database to identify and correct any data gaps caused by asynchronous processing failures. ### Operational Resilience and Incident Response * **Query Optimization:** During a load spike, the MySQL optimizer chose "Full Scans" over indexes; the team resolved this by implementing SQL hints and utilizing a 5-version Docker image history for rapid rollbacks. * **Network Cancellation:** To handle timeouts between Toss and external card issuers, the system uses specific logic to automatically send cancellation requests and synchronize states. * **Timeout Standardization:** Discrepancies between microservices were resolved by calculating the maximum processing time of approval servers and aligning all upstream timeout settings to prevent merchant response mismatches. * **Reliable Event Delivery:** While using the **Outbox pattern** for events, the team added log-based recovery (Elasticsearch and local disk) and idempotency keys in event headers to handle both missing and duplicate messages. For organizations tackling significant technical debt, this transition highlights that initial design is only half the battle. True system reliability comes from building "self-healing" structures—such as automated correction batches and standardized timeout chains—that can survive the unpredictable nature of live production environments.

Database DesignApache KafkaMicroservicesMysql+4

toss·Nov 3, 2025Original article

Working as a QA in a (opens in new tab)

Toss Place implements a dual-role QA structure where managers are embedded directly within product Silos from the initial planning stages to final deployment. This shift moves QA from a final-stage bottleneck to a proactive partner that enhances delivery speed and stability through deep historical context and early risk mitigation. Consequently, the organization has transitioned to a culture where quality is viewed as a shared team responsibility rather than a siloed functional task. ### Integrating QA into Product Silos * QA managers belong to both a central functional team and specific product units (Silos) to ensure they are involved in the entire product lifecycle. * Participation begins at the OKR design phase, allowing QA to align testing strategies with specific product intentions and business goals. * Early involvement enables accurate risk assessment and scope estimation, preventing the "shallow testing" that often occurs when QA only sees the final product. ### Optimizing Spec Reviews and Sanity Testing * The team introduced a structured flow consisting of Spec Reviews followed by Q&A sessions to reduce repetitive discussions and information gaps. * All specification changes are centralized in shared design tools (such as Deus) or messenger threads to ensure transparency across all roles. * "Sanity Test" criteria were established where developers and QA agree on "Happy Case" validations and minimum spec requirements before development begins, ensuring everyone starts from the same baseline. ### Collaborative Live Monitoring * Post-release checklists were developed to involve the entire Silo in live monitoring, overcoming the limitations of having a single QA manager per unit. * This collaborative approach encourages non-technical roles to interact with the live product, reinforcing the culture that quality is a collective team responsibility. ### Streamlining Issue Tracking and Communication * The team implemented a "Send to Notion" workflow to instantly capture messenger-based feedback and ideas into a structured, prioritized backlog. * To reduce communication fragmentation, they transitioned from Jira to integrated Messenger Lists and Canvases, which allowed for centralized discussions and faster issue resolution. * Backlogs are prioritized based on user experience impact and release urgency, ensuring that critical bugs are addressed while minor improvements are tracked for future cycles. The success of these initiatives demonstrates that QA effectiveness is driven by integration and autonomy rather than rigid adherence to specific tools. To achieve both high velocity and high quality, organizations should empower QA professionals to act as product peers who can flexibly adapt their processes to the unique needs and data-driven goals of their specific product teams.

Database DesignJiraNotionQuality Assurance+4