Easily Operating Karrot (opens in new tab)

This blog post by the Daangn (Karrot) search platform team details their journey in optimizing Elasticsearch operations on Kubernetes (ECK). While their initial migration to ECK reduced deployment times, the team faced critical latency spikes during rolling restarts due to "cold caches" and high traffic volumes. To achieve a "deploy anytime" environment, they developed a data node warm-up system to ensure nodes are performance-ready before they begin handling live search requests. ## Scaling Challenges and Operational Constraints - Over two years, Daangn's search infrastructure expanded from a single cluster to four specialized clusters, with peak traffic jumping from 1,000 to over 10,000 QPS. - The initial strategy of "avoiding peak hours" for deployments became a bottleneck, as the window for safe updates narrowed while total deployment time across all clusters exceeded six hours. - Manual monitoring became a necessity rather than an option, as engineers had to verify traffic conditions and latency graphs before and during every ArgoCD sync. ## The Hazards of Rolling Restarts in Elasticsearch - Standard Kubernetes rolling restarts are problematic for stateful systems because a "Ready" Pod does not equate to a "Performant" Pod; Elasticsearch relies heavily on memory-resident caches (page cache, query cache, field data cache). - A version update in the Elastic Operator once triggered an unintended rolling restart that caused a 60% error rate and 3-second latency spikes because new nodes had to fetch all data from disk. - When a node restarts, the cluster enters a "Yellow" state where remaining replicas must handle 100% of the traffic, creating a single point of failure and increasing the load on the surviving nodes. ## Strategy for Reliable Node Warm-up - The primary goal was to reach a state where p99 latency remains stable during restarts, regardless of whether the deployment occurs during peak traffic hours. - The solution involves a "Warm-up System" designed to pre-load frequently accessed data into the filesystem and Elasticsearch internal caches before the node is allowed to join the load balancer. - By executing representative search queries against a newly started node, the system ensures that the necessary segments are already in the page cache, preventing the disk I/O thrashing that typically follows a cold start. ## Implementation Goals - Automate the validation of node readiness beyond simple health checks to include performance readiness. - Eliminate the need for human "eyes-on-glass" monitoring during the 90-minute deployment cycles. - Maintain high availability and consistent user experience even when shards are being reallocated and replicas are temporarily unassigned. To maintain a truly resilient search platform on Kubernetes, it is critical to recognize that for stateful applications, "available" is not the same as "ready." Implementing a customized warm-up controller or logic is a recommended practice for any high-traffic Elasticsearch environment to decouple deployment schedules from traffic patterns.

Gemini provides automated feedback for theoretical computer scientists at STOC 2026 (opens in new tab)

Google Research launched an experimental program for the STOC 2026 conference using a specialized Gemini model to provide automated, rigorous feedback on theoretical computer science submissions. By identifying critical logical errors and proof gaps within a 24-hour window, the tool demonstrated that advanced AI can serve as a powerful pre-vetting collaborator for high-level mathematical research. The overwhelmingly positive reception from authors indicates that AI can effectively augment the human peer-review process by improving paper quality before formal submission. ## Advanced Reasoning via Inference Scaling - The tool utilized an advanced version of Gemini 2.5 Deep Think specifically optimized for mathematical rigor. - It employed inference scaling methods, allowing the model to explore and combine multiple possible solutions and reasoning traces simultaneously. - This non-linear approach to problem-solving helps the model focus on the most salient technical issues while significantly reducing the likelihood of hallucinations. ## Structured Technical Feedback - Feedback was delivered in a structured format that included a high-level summary of the paper's core contributions. - The model provided a detailed analysis of potential mistakes, specifically targeting errors within lemmas, theorems, and logical proofs. - Authors also received a categorized list of minor corrections, such as inconsistent variable naming and typographical errors. ## Identified Technical Issues and Impact - The pilot saw high engagement, with over 80% of STOC 2026 submitters opting in for the AI-generated review. - The tool successfully identified "critical bugs" and calculation errors that had previously evaded human authors for months. - Survey results showed that 97% of participants found the feedback helpful, and 81% reported that the tool improved the overall clarity and readability of their work. ## Expert Verification and Hallucinations - Because the users were domain experts, they were able to act as a filter, distinguishing between deep technical insights and occasional model hallucinations. - While the model sometimes struggled to parse complex notation or interpret figures, authors valued the "neutral tone" and the speed of the two-day turnaround. - The feedback was used as a starting point for human verification, allowing researchers to refine their arguments rather than blindly following the model's output. ## Future Outlook and Educational Potential - Beyond professional research, 75% of surveyed authors see significant educational value in using the tool to train students in mathematical rigor. - The experiment's success has led to 88% of participants expressing interest in having continuous access to such a tool throughout their entire research and drafting process. The success of the STOC 2026 pilot suggests that researchers should consider integrating specialized LLMs early in the drafting phase to catch "embarrassing" or logic-breaking errors. While the human expert remains the final arbiter of truth, these tools provide a necessary layer of automated verification that can accelerate the pace of scientific discovery.

Painting the Wheels of a Moving (opens in new tab)

Toss Design System (TDS) underwent its first major color system overhaul in seven years to address deep-seated issues with perceptual inconsistency and fragmented cross-platform management. By transitioning to a perceptually uniform color space and an automated token pipeline, the team established a scalable infrastructure capable of supporting the brand's rapid expansion into global markets and diverse digital environments. ### Legacy Issues in Color Consistency * **Uneven luminosity across hues:** Colors sharing the same numerical value (e.g., Grey 100 and Blue 100) exhibited different perceptual brightness levels, leading to "patchy" layouts when used together. * **Discrepancies between Light and Dark modes:** Specific colors, such as Teal 50, appeared significantly more vibrant in dark mode than in light mode, forcing designers to manually customize colors for different themes. * **Accessibility hurdles:** Low-contrast colors often became invisible on low-resolution devices or virtual environments, failing to meet consistent accessibility standards. ### Technical Debt and Scaling Barriers * **Interconnected palettes:** Because the color scales were interdependent, modifying a single color required re-evaluating the entire palette across all hues and both light/dark modes. * **Fragmentation of truth:** Web, native apps, and design editors managed tokens independently, leading to "token drift" where certain colors existed on some platforms but not others. * **Business expansion pressure:** As Toss moved toward becoming a "super-app" and entering global markets, the manual process of maintaining design consistency became a bottleneck for development speed. ### Implementing Perceptually Uniform Color Spaces * **Adopting OKLCH:** Toss shifted from traditional HSL models to OKLCH to ensure that colors with the same lightness values are perceived as equally bright by the human eye. * **Automated color logic:** The team developed an automation logic that extracts accessible color combinations (backgrounds, text, and assets) for any input color, allowing third-party mini-apps to maintain brand identity without sacrificing accessibility. * **Chroma Clamping:** To ensure compatibility with standard RGB displays, the system utilizes chroma clamping to maintain intended hue and lightness even when hardware limitations arise. ### Refined Visual Correction and Contrast * **Solving the "Dark Yellow Problem":** Since mathematically consistent yellow often appears muddy or loses its "yellowness" at higher contrast levels, the team applied manual visual corrections to preserve the color's psychological impact. * **APCA-based Dark Mode optimization:** Utilizing the Advanced Perceptual Contrast Algorithm (APCA), the team increased contrast ratios in dark mode to compensate for human optical illusions and improve legibility at low screen brightness. ### Designer-Led Automation Pipeline * **Single Source of Truth:** By integrating Token Studio (Figma plugin) with GitHub, the team created a unified repository where design changes are synchronized across all platforms simultaneously. * **Automated deployment:** Designers can now commit changes and generate pull requests directly; pre-processing scripts then transform these tokens into platform-specific code for web, iOS, and Android without requiring manual developer intervention. The transition to a token-based, automated color system demonstrates that investing in foundational design infrastructure is essential for long-term scalability. For organizations managing complex, multi-platform products, adopting perceptually uniform color spaces like OKLCH can significantly reduce design debt and improve the efficiency of cross-functional teams.

Code Quality Improvement Techniques Part 26 (opens in new tab)

The quality of code documentation depends heavily on the hierarchy of information, specifically prioritizing high-level intent in the very first sentence. By focusing on abstract summaries rather than implementation details at the start, developers can ensure that readers understand a function's purpose instantly without parsing through sequential logic. This principle of "summary-first" communication enhances readability and developer productivity across documentation comments, inline explanations, and TODO tasks. ### Strategies for Effective Documentation * **Prioritize the first sentence:** Documentation comments should be written so that the overview is understandable from the first sentence alone. * **Increase abstraction levels:** Avoid simply repeating what the code does (e.g., "split by period and remove empty strings"). Instead, describe the result in domain terms, such as "returns a list of words grouped by sentences." * **Identify the most important element:** Since the primary goal of most functions is to produce a result, the summary should lead with what is being returned rather than how it is calculated. * **Layer the details:** Technical specifics—such as specific delimiters like `SENTENCE_SEPARATOR` ('.') or `WORD_SEPARATOR_REGEX` ([ ,]+)—and exclusion rules for empty strings should follow the initial summary. * **Use concrete examples:** For complex transformations or edge cases, include a sample input and output (e.g., showing how `" a bc. .d,,."` maps to `[["a", "bc"], ["d"]]`) to clarify boundary conditions. ### Prioritizing Intent in Non-Documentation Comments * **Focus on the "Why" for workarounds:** In inline comments, especially for "temporary fixes" or bug workarounds, the reason for the code's existence is more important than the action it performs. For instance, leading with "This is to avoid a bug in Device X" is more helpful than "Resetting the value to its previous state." * **Lead with the goal in TODOs:** When writing TODO comments, state the ideal state or the required change first. Explanations regarding current limitations or why the change cannot be made immediately should be relegated to the following sentences. * **Improve scannability:** Structuring comments this way allows developers to scan the codebase and understand the motivation behind complex logic without needing to read the entire comment block. To maintain a clean and maintainable codebase, always choose the most critical piece of information—whether it is the function's return value, a bug's context, or a future goal—and place it at the very beginning of your comments.

Spotlight on innovation: Google-sponsored Data Science for Health Ideathon across Africa (opens in new tab)

Google Research, in partnership with several pan-African machine learning communities, recently concluded the Africa-wide Data Science for Health Ideathon to address regional medical challenges. By providing access to specialized open-source health models and technical mentorship, the initiative empowered local researchers to develop tailored solutions for issues ranging from maternal health to oncology. The event demonstrated that localized innovation, supported by high-performance AI foundations, can effectively bridge healthcare gaps in resource-constrained environments. ## Collaborative Framework and Objectives * The Ideathon was launched at the 2025 Deep Learning Indaba in Kigali, Rwanda, in collaboration with SisonkeBiotik, Ro’ya, and DS-I Africa. * The primary goal was to foster capacity building within the African AI community, moving beyond theoretical research toward the execution of practical healthcare tools. * Participants received hands-on training on Google’s specialized health models and were supported with Google Cloud Vertex AI compute credits and mentorship from global experts. * Submissions were evaluated based on their innovation, technical feasibility, and contextual relevance to African health systems. ## Technical Foundations and Google Health Models * Developers focused on a suite of open health AI models, including MedGemma for clinical reasoning, TxGemma for therapeutics, and MedSigLIP for medical vision-language tasks. * The competition utilized a two-phase journey: an initial "Idea Development" stage where teams defined clinical problems and outlined AI approaches, followed by a "Prototype & Pitch" phase. * Technical implementations frequently involved advanced techniques such as Retrieval-Augmented Generation (RAG) to ensure alignment with local medical protocols and WHO guidelines. * Fine-tuning methods, specifically Low-Rank Adaptation (LoRA), were utilized by teams to specialize large-scale models like MedGemma-27B-IT for niche datasets. ## Innovative Solutions for Regional Health * **Dawa Health:** This first-place winner developed an AI-powered cervical cancer screening tool that uses MedSigLIP to identify abnormalities in colposcopy images uploaded via WhatsApp, combined with Gemini RAG for clinical guidance. * **Solver (CerviScreen AI):** This team built a web application for automated cervical-cytology screening by fine-tuning MedGemma-27B-IT on the CRIC dataset to assist cytopathologists with annotated images. * **Mkunga:** A maternal health call center that adapts MedGemma and Gemini to provide advice in Swahili using Speech-to-Text (STT) and Text-to-Speech (TTS) technologies. * **HexAI (DermaDetect):** Recognized for the best proof-of-concept, this offline-first mobile app allows community health workers to triage skin conditions using on-device versions of MedSigLIP, specifically designed for low-connectivity areas. The success of the Ideathon underscores the importance of "local solutions for local priorities." By making sophisticated models like MedGemma and MedSigLIP openly available, the technical barrier to entry is lowered, allowing African developers to build high-impact, culturally and linguistically relevant medical tools. For organizations looking to implement AI in global health, this model of providing foundational tools and cloud resources to local experts remains a highly effective strategy for sustainable innovation.

The Evolution of Kanana-o (opens in new tab)

Kakao has significantly advanced its integrated multimodal model, Kanana-o, by enhancing its ability to process complex instructions across text, image, and audio inputs while enriching its emotional vocal expression. By developing specialized datasets and sophisticated training techniques for prosody, the team has bridged the performance gap between text and audio modalities. The result is a more natural, human-like AI capable of nuanced interaction and high-performance instruction following, particularly within the Korean linguistic context. ## Advancing Multimodal Instruction Following * Addressed the "modality gap" where multimodal models often show decreased reasoning and reasoning performance when processing audio inputs compared to text. * Constructed a structured, high-quality dataset featuring complex, multi-step instructions such as summarizing a context and then translating it into a specific language or style. * Leveraged the Speech-KoMT-Bench to evaluate performance, showing that Kanana-o significantly outperforms global competitors of similar scale in Korean-specific tasks. * Focused on "Domain-generalization" to ensure the model's core intelligence remains stable regardless of whether the input is text, audio, or a combination of both. ## Image-Audio-Text Modality Alignment * Developed integrated datasets to ensure that reasoning capabilities learned in text-image or text-audio contexts generalize to complex image-audio scenarios. * Trained the model to handle tasks where users ask questions about visual information via voice, requiring the simultaneous alignment of three different data types. * Prioritized the maintenance of "World Knowledge" during multimodal training so that the addition of new modalities does not degrade the model’s factual accuracy. ## Enhancing Vocal Expressiveness and Prosody * Focused on "prosody"—the rhythm, pitch, and stress of speech—to move beyond robotic, flat text-to-speech (TTS) outputs. * Implemented a system of descriptive tokens and emotion tags (e.g., "warm voice," "excited tone") during training to give the model fine-grained control over its vocal persona. * Incorporated natural human speech elements, such as realistic breathing patterns and contextual variations in speech speed, to make interactions feel more intuitive and less synthetic. * Refined the model's ability to interpret the user's emotional state from their voice and respond with a matching emotional intensity. The evolution of Kanana-o highlights a shift from simply maximizing generic benchmarks to optimizing real-world user experiences through multimodal alignment and emotional intelligence. The success of this model underscores the necessity of high-quality, structured instruction data and fine-grained control over output styles to create truly conversational AI that feels natural to the user.

Korean and Images All at Once: Kak (opens in new tab)

Kakao has developed Kanana-v-embedding, a specialized multimodal embedding model designed to bridge the gap between Korean text and visual data within a unified semantic space. By leveraging a Vision-Language Model (VLM) framework, the model enables seamless search and recommendation across various combinations of text and images, offering a significant performance boost over existing English-centric models like CLIP. This development provides a robust technical foundation for enhancing Kakao’s services, including RAG-based systems and localized content discovery. ### Unified Multimodal Meaning Space * The model maps text and images into a single vector space where semantic similarity is measured via cosine similarity. * Unlike traditional CLIP models that use independent encoders, this architecture treats text and images as a single sequence, allowing for "text + image" combined queries. * It supports four primary interaction modes: Text-to-Text, Text-to-Image, Image-to-Image, and (Text+Image)-to-(Text+Image). ### VLM-Based Architecture and Instruction Tuning * The system utilizes a VLM consisting of an LLM and an image encoder, extracting embeddings from the final hidden state of the [EOS] token. * It employs instruction-based query embedding, where specific prompts (e.g., "Find an image matching this caption") guide the model to generate embeddings tailored to the specific task, such as retrieval or classification. * The model is optimized for the Korean language and cultural context, addressing the limitations of previous models that struggled with non-English data. ### Advanced Training for Scalability and Precision * **Gradient Caching:** To overcome GPU memory limitations, this technique allows the model to train with effectively large batch sizes, which is critical for the InfoNCE loss used in contrastive learning. * **Matryoshka Representation Learning (MRL):** The model supports flexible embedding sizes ranging from 64 to 2,048 dimensions. This allows services to choose between low-latency (smaller dimensions) or high-precision (larger dimensions) without retraining. * **Hard Negative Mining:** The training process incorporates "hard negatives"—items that are similar but incorrect—to sharpen the model’s ability to distinguish between subtle differences in data. ### Performance Benchmarks and Efficiency * Kanana-v-embedding significantly outperforms CLIP and VLM2Vec on the KoEmbed benchmark, particularly in Korean Text-to-Image and Image-to-Text retrieval tasks. * In the M-BEIR (Multimodal Benchmark for Retrieval), the model demonstrated superior performance in multimodal document retrieval and image-to-text tasks compared to established open-source models. * Evaluation of MRL showed that the model retains high accuracy even when dimensions are reduced to 256 or 512, providing a 4x to 8x improvement in storage and search efficiency with minimal loss in quality. For organizations looking to implement multimodal RAG or advanced recommendation systems in Korean-language environments, Kanana-v-embedding offers a highly adaptable solution. Its ability to balance computational cost and retrieval quality through Matryoshka learning makes it particularly suitable for large-scale production environments where latency is a primary concern.

Enhancing the “Frequently (opens in new tab)

Baedal Minjok (Baemin) has significantly improved its cart recommendation system by transitioning from a basic Item2Vec model to a sophisticated two-stage architecture that combines graph-based embeddings with Transformer sequence modeling. This evolution addresses the "substitutability bias" and lack of sequential context found in previous methods, allowing the system to understand the specific intent behind a user's shopping journey. By moving beyond simple item similarity, the new model effectively identifies cross-selling opportunities that align with the logical flow of a customer's purchase behavior. ### Limitations of the Item2Vec Approach * **Substitutability Bias:** The original Item2Vec model, based on the Skip-gram architecture, tended to map items from the same category into similar vector spaces. This resulted in recommending alternative brands of the same product (e.g., suggesting another brand of milk) rather than complementary goods (e.g., cereal or bread). * **Loss of Sequential Context:** Because Item2Vec treats a basket of goods as a "bag of words," it ignores the order in which items are added. This prevents the model from distinguishing between different user intents, such as a user starting with meat to grill versus a user starting with ingredients for a stew. * **Failure in Cross-Selling:** The primary goal of cart recommendations is to encourage cross-selling, but the reliance on embedding similarity alone limited the diversity of suggestions, often trapping users within a single product category. ### Stage 1: Graph-Based Product and Category Embeddings * **Node2Vec Implementation:** To combat data sparsity and the "long-tail" problem where many items have low purchase frequency, the team utilized Node2Vec. This method uses random walks to generate sequences that help the model learn structural relationships even when direct transaction data is thin. * **Heterogeneous Graph Construction:** The graph consists of both "Item Nodes" and "Category Nodes." Connecting items to their respective categories allows the system to generate initial vectors for new or low-volume products that lack sufficient historical purchase data. * **Association Rule Weighting:** Rather than using simple co-occurrence counts for edge weights, the team applied Association Rules. This ensures that weights reflect the actual strength of the complementary relationship, preventing popular "mega-hit" items from dominating all recommendation results. ### Stage 2: Transformer-Based Sequence Recommendation * **Capturing Purchase Context:** The second stage employs a Transformer model to analyze the sequence of items currently in the user's cart. This architecture is specifically designed to understand how the meaning of an item changes based on what preceded it. * **Next Item Prediction:** Using the pre-trained embeddings from Stage 1 as inputs, the Transformer predicts the most likely "next item" a user will add. This allows the system to provide dynamic recommendations that evolve as the user continues to shop. * **Integration of Category Data:** By feeding both item-level and category-level embeddings into the Transformer, the model maintains a high level of accuracy even when a user interacts with niche products, as the category context provides a fallback for the recommendation logic. ### Practical Conclusion For production-scale recommendation systems, relying solely on item similarity often leads to redundant suggestions that do not drive incremental sales. By decoupling the learning of structural relationships (via graphs) from the learning of temporal intent (via Transformers), engineers can build a system that is robust against data sparsity while remaining highly sensitive to the immediate context of a user's session. This two-stage approach is recommended for e-commerce environments where cross-category discovery is a key business metric.

[Internship] Introducing (opens in new tab)

The 2026 NAVER AI CHALLENGE offers undergraduate and graduate students a four-week intensive internship to tackle real-world AI challenges alongside Naver’s senior engineers. This program focuses on the full development lifecycle, allowing participants to experience Naver’s unique collaborative culture while moving from initial idea design to technical verification. By integrating interns directly into professional workflows, the challenge aims to foster the next generation of AI talent through hands-on industrial problem-solving. ### Application Timeline and Eligibility * The internship is open to all students currently enrolled in bachelor's or master's programs, regardless of their major or year of study. * Applications are accepted from December 10 until December 16 at 11:00 AM. * The selection process consists of document screening in late December, followed by technical interviews in early January, with final announcements scheduled for mid-January. * The official internship period runs for four weeks, from January 19 to February 13, 2026. ### AI Project Specializations * **Data Pipeline and Lineage:** One track focuses on AI-driven data pipeline log analysis to automate Data Asset mapping and construct comprehensive End-to-End Data Lineage systems. * **VLM-based Evaluation Systems:** The second track involves developing automated systems using Vision Language Models (VLM) to evaluate search and recommendation quality from a user-experience perspective. * **Technical Mentorship:** Participants work directly with Naver engineers to determine technical directions and validate their AI models within a professional production context. ### Support and Work Environment * Naver provides a dedicated workspace and the latest OA equipment to ensure a seamless development experience for all interns. * Participants receive project activity funds to support their research and development efforts during the four-week duration. * The program emphasizes networking and professional growth by providing direct access to the infrastructure and expertise used by one of Korea's leading tech companies. For students looking to transition academic AI knowledge into industrial-scale applications, this internship provides a high-impact environment to build professional-grade systems. Interested candidates should finalize their project selection between data pipeline automation and VLM evaluation before the December 16 deadline to secure a spot in the selection process.

Legacy Settlement Overhaul: From (opens in new tab)

Toss Payments recently overhauled its 20-year-old legacy settlement system to overcome deep-seated technical debt and prepare for massive transaction growth. By shifting from monolithic SQL queries and aggregated data to a granular, object-oriented architecture, the team significantly improved system maintainability, traceability, and batch processing performance. The transition focused on breaking down complex dependencies and ensuring that every transaction is verifiable and reproducible. ### Replacing Monolithic SQL with Object-Oriented Logic * The legacy system relied on a "giant common query" filled with nested `DECODE`, `CASE WHEN`, and complex joins, making it nearly impossible to identify the impact of small changes. * The team applied a "Divide and Conquer" strategy, splitting the massive query into distinct domains and refined sub-functions. * Business logic was moved from the database layer into Kotlin-based objects (e.g., `SettlementFeeCalculator`), making business rules explicit and easier to test. * This modular approach allowed for "Incremental Migration," where specific features (like exchange rate conversions) could be upgraded to the new system independently. ### Improving Traceability through Granular Data Modeling * The old system stored data in an aggregated state (Sum), which prevented developers from tracing errors back to specific transactions or reusing data for different reporting needs. * The new architecture manages data at the minimum transaction unit (1:1), ensuring that every settlement result corresponds to a specific transaction. * "Setting Snapshots" were introduced to store the exact contract conditions (fee rates, VAT status) at the time of calculation, allowing the system to reconstruct the context of past settlements. * A state-based processing model was implemented to enable selective retries for failed transactions, significantly reducing recovery time compared to the previous "all-or-nothing" transaction approach. ### Optimizing High-Resolution Data and Query Performance * Managing data at the transaction level led to an explosion in data volume, necessitating specialized database strategies. * The team implemented date-based Range Partitioning and composite indexing on settlement dates to maintain high query speeds despite the increased scale. * To balance write performance and read needs, they created "Query-specific tables" that offload the processing burden from the main batch system. * Complex administrative queries were delegated to a separate high-performance data serving platform, maintaining a clean separation between core settlement logic and flexible data analysis. ### Resolving Batch Performance and I/O Bottlenecks * The legacy batch system struggled with long processing times that scaled poorly with transaction growth due to heavy I/O and single-threaded processing. * I/O was minimized by caching merchant contract information in memory at the start of a batch step, eliminating millions of redundant database lookups. * The team optimized the `ItemProcessor` in Spring Batch by implementing bulk lookups (using a Wrapper structure) to handle multiple records at once rather than querying the database for every individual item. This modernization demonstrates that scaling a financial system requires moving beyond "convenient" aggregations toward a granular, state-driven architecture. By decoupling business logic from the database and prioritizing data traceability, Toss Payments has built a foundation capable of handling the next generation of transaction volumes.

We Refactor Culture Just Like (opens in new tab)

The Commerce Web Frontend Development team at Woowa Brothers recently underwent a significant organizational "refactoring" to manage the increasing complexity of their expanding commerce platform. By moving away from rigid, siloed roles and adopting a flexible "boundary-less" part system, the team successfully synchronized disparate services like B Mart and Baemin Store. This cultural shift demonstrates that treating organizational structure with the same iterative mindset as code can eliminate operational bottlenecks and foster a more resilient engineering environment. ### Transitioning to Boundary-less Parts * The team abandoned traditional division methods—such as project-based, funnel-based, or service-vs-backoffice splits—because they created resource imbalances and restricted developers' understanding of the overall service flow. * Traditional project-based splits often led to specific teams being overwhelmed during peak periods while others remained underutilized, creating significant delivery bottlenecks. * To solve these inefficiencies, the team introduced "boundary-less parts," where developers are not strictly tied to a single domain but are encouraged to work across the entire commerce ecosystem. * This structure allows the organization to remain agile, moving resources fluidly to address high-priority business needs without being hindered by departmental "walls." ### From R&R to Responsibility and Expandability (R&E) * The team replaced the traditional R&R (Role & Responsibility) model with "R&E" (Responsibility & Expandability), focusing on the core principle of "owning" a problem until it is fully resolved. * This shift encourages developers to expand their expertise beyond their immediate tasks, fostering a culture where helping colleagues and understanding neighboring domains is the standard. * Work is distributed through a strategic sync between team and part leaders, but team members maintain the flexibility to jump into different domains as project requirements evolve. * Regular "part shuffling" is utilized to ensure that domain knowledge is distributed across the entire 20-person frontend team, preventing the formation of information silos. ### Impact on Technical Integration and Team Resilience * The flexible structure was instrumental in the "ONE COMMERCE" initiative, which required integrating the technical stacks and user experiences of B Mart and Baemin Store. * Because developers had broad domain context, they were able to identify redundant logic across different services and abstract them into shared, common modules, ensuring architectural consistency. * The organization significantly improved its "Bus Factor"—the number of people who can leave before a project stalls—by ensuring multiple engineers understand the context of any given system. * Developers evolved into "domain-wide engineers" who understand the full lifecycle of a transaction, from the customer-facing UI to the backend administrative and logistics data flows. To prevent today's organizational solutions from becoming tomorrow's cultural legacy debt, engineering teams should proactively refactor their workflows. Moving from rigid role definitions to a model based on shared responsibility and cross-domain mobility is essential for maintaining velocity and technical excellence in large-scale platform environments.

won Park": Author. * (opens in new tab)

Daangn’s data governance team addressed the lack of transparency in their data pipelines by building a column-level lineage system using SQL parsing. By analyzing BigQuery query logs with specialized parsing tools, they successfully mapped intricate data dependencies that standard table-level tracking could not capture. This system now enables precise impact analysis and significantly improves data reliability and troubleshooting speed across the organization. **The Necessity of Column-Level Visibility** * Table-level lineage, while easily accessible via BigQuery’s `JOBS` view, fails to identify how specific fields—such as PII or calculated metrics—propagate through downstream systems. * Without granular lineage, the team faced "cascading failures" where a single pipeline error triggered a chain of broken tables that were difficult to trace manually. * Schema migrations, such as modifying a source MySQL column, were historically high-risk because the impact on derivative BigQuery tables and columns was unknown. **Evaluating Extraction Strategies** * BigQuery’s native `INFORMATION_SCHEMA` was found to be insufficient because it does not support column-level detail and often obscures original source tables when Views are involved. * Frameworks like OpenLineage were considered but rejected due to high operational costs; requiring every team to instrument their own Airflow jobs or notebooks was deemed impractical for a central governance team. * The team chose a centralized SQL parsing approach, leveraging the fact that nearly all data transformations within the company are executed as SQL queries within BigQuery. **Technical Implementation and Tech Stack** * **sqlglot:** This library serves as the core engine, parsing SQL strings into Abstract Syntax Trees (AST) to programmatically identify source and destination columns. * **Data Collection:** The system pulls raw query text from `INFORMATION_SCHEMA.JOBS` across all Google Cloud projects to ensure comprehensive coverage. * **Processing and Orchestration:** Spark is utilized to handle the parallel processing of massive query logs, while Airflow schedules regular updates to the lineage data. * **Storage:** The resulting mappings are stored in a centralized BigQuery table (`data_catalog.lineage`), making the dependency map easily accessible for impact analysis and data cataloging. By centralizing lineage extraction through SQL parsing rather than per-job instrumentation, organizations can achieve comprehensive visibility without placing an integration burden on individual developers. This approach is highly effective for BigQuery-centric environments where SQL is the primary language for data movement and transformation.

Why did Athenz engineers take on the (opens in new tab)

Security platform engineer Jung-woo Kim details his transition from a specialized Athenz developer to a "Kubestronaut," a prestigious CNCF designation awarded to those who master the entire Kubernetes ecosystem. By systematically obtaining five distinct certifications, he argues that deep, practical knowledge of container orchestration is essential for building secure, scalable access control systems in private cloud environments. His journey demonstrates that moving beyond application-level expertise to master cluster administration and security directly improves architectural design and operational troubleshooting. ## The Kubestronaut Framework * The title is awarded by the Cloud Native Computing Foundation (CNCF) to individuals who pass five specific certification exams: CKA, CKAD, CKS, KCNA, and KCSA. * The CKA (Administrator), CKAD (Application Developer), and CKS (Security Specialist) exams are performance-based, requiring candidates to solve real-world technical problems in a live terminal environment rather than answering multiple-choice questions. * Success in these exams demands a combination of deep technical knowledge, speed, and accuracy, as practitioners must configure clusters and resolve failures under strict time constraints. * The remaining Associate-level exams (KCNA and KCSA) provide a theoretical foundation in cloud-native security and ecosystem standards. ## A Progressive Path to Technical Mastery * **CKAD (Application Developer):** The initial focus was on mastering the deployment of Athenz—an open-source auth system—ensuring it runs efficiently from a developer's perspective. Preparation involved rigorous use of tools like killer.sh to simulate high-pressure environments. * **CKA (Administrator):** To manage multi-cluster environments and understand the underlying components that make Kubernetes function, the author moved to the administrator level, gaining insight into how various services interact within the cluster. * **CKS (Security Specialist):** Given his background in security, this was the most critical and difficult stage, focusing on cluster hardening, vulnerability analysis, and implementing strict network policies to ensure the entire infrastructure remains resilient. ## Organizational Impact and Open Source Governance * Obtaining these certifications provided a clearer understanding of open-source governance, specifically how Special Interest Groups (SIGs) and pull request (PR) workflows drive massive projects like Kubernetes. * This technical depth was applied to a high-stakes project providing Athenz services in a Bare Metal as a Service (BMaaS) environment, allowing for more stable and efficient architecture design. * The learning process was supported by corporate initiatives, including access to Udemy Business for technical training and a hybrid work culture that allowed for consistent, early-morning study habits. To achieve expert-level proficiency in complex systems like Kubernetes, engineers should adopt the "Ubo-cheonri" philosophy—making slow but steady progress. Starting with even one minute of study or a single GitHub commit per day can eventually lead to mastering the highest levels of cloud-native architecture. For those managing enterprise-grade infrastructure, pursuing the Kubestronaut path is highly recommended as it transforms theoretical knowledge into a broad, practical vision for system design.

A differentially private framework for gaining insights into AI chatbot use (opens in new tab)

Google Research has introduced Urania, a novel framework designed to extract high-level usage insights from AI chatbot conversations while maintaining rigorous differential privacy (DP) guarantees. Unlike previous heuristic methods that rely on simple redaction or LLM-based PII stripping, this pipeline ensures that no individual user's data can be reconstructed from the resulting summaries. By combining DP clustering and keyword extraction with LLM-based summarization, the system provides a formal, auditable approach to understanding platform trends without compromising sensitive information. ## Limitations of Heuristic Privacy * Existing frameworks often rely on large language models to manually strip personally identifiable information (PII) from text before analysis. * These heuristic protections are difficult to formalize or audit, and their effectiveness may diminish as models evolve or face sophisticated prompt injection attacks. * The Urania framework addresses these weaknesses by using mathematical privacy budgets (the epsilon parameter) to measure and limit the influence of any single user's data on the final output. ## The Differentially Private Pipeline * **DP Clustering**: The framework first converts conversation data into numerical embeddings. These are grouped using a DP clustering algorithm, ensuring that cluster centers reflect broad trends rather than specific individual inputs. * **DP Keyword Extraction**: The system identifies keywords for each cluster and generates a histogram of their frequency. By adding mathematical noise to these counts, the framework masks individual contributions and ensures that only keywords common to many users are retained. * **Keyword Generation Methods**: The researchers explored three methods for extraction: LLM-guided selection of relevant terms, a differentially private version of TF-IDF, and an LLM-guided approach that selects terms from a pre-defined list of public keywords. * **LLM Summarization**: In the final stage, an LLM generates a high-level summary of the cluster using only the noisy, anonymized keywords. Because the LLM never sees the raw conversation text, the "post-processing" property of DP guarantees that the final summary remains private. ## Privacy and Utility Trade-offs * The framework was tested against a non-private baseline (Simple-CLIO) to evaluate how privacy constraints affect the quality of the insights generated. * Stronger privacy settings (lower epsilon values) inherently result in a utility trade-off, as the added noise can obscure some niche usage patterns. * Despite these trade-offs, the framework provides a robust defense against data leakage, as the summarization model is structurally prevented from seeing sensitive original text, making it resilient to prompt injection. This framework offers a scalable way for platform providers to analyze chatbot usage patterns and enforce safety policies while providing mathematical certainty regarding user privacy. For organizations handling sensitive conversation data, moving from heuristic redaction to formal DP pipelines like Urania provides a more robust and auditable path for service improvement.

Improving Work Efficiency: Revisiting (opens in new tab)

The Toss Research Platform team argues that operational efficiency is best achieved by decomposing complex workflows into granular, atomic actions rather than attempting massive systemic overhauls. By systematically questioning the necessity of each step and prioritizing improvements based on stakeholder impact rather than personal workload, teams can eliminate significant waste through incremental automation. This approach demonstrates that even minor reductions in repetitive manual tasks can lead to substantial gains in team-wide productivity as an organization scales. ### Granular Action Mapping * Break down workflows into specific physical or digital actions—such as clicks, data entries, and channel switches—rather than high-level phases. * Document the "Who, Where, What, and Why" for every individual step to identify exactly where friction occurs. * Include exception cases and edge scenarios in the process map to uncover hidden gaps in the current operating model. ### Questioning Necessity and Identifying Automation Targets * Apply a critical filter to every mapped action by asking, "Why is this necessary?" to eliminate redundant tasks like manual cross-platform notifications. * Distinguish between essential human-centric tasks and mechanical actions, such as calendar entry creation, that are ripe for automation. * Address "micro-inefficiencies" that appear insignificant in isolation but aggregate into major resource drains when repeated multiple times daily across a large team. ### Stakeholder-Centric Prioritization * Shift the criteria for optimization from personal convenience to the impact on the broader organization. * Rank improvements based on three specific metrics: the number of people affected, the downstream influence on other workflows, and the total cumulative time consumed. * Recognize that automating a "small" task for an operator can unlock significant time and clarity for dozens of participants and observers. ### Incremental Implementation and Risk Mitigation * Avoid the "all-or-nothing" automation trap by deploying partial solutions that address solvable segments of a process immediately. * Utilize designated test periods for process changes to monitor for risks, such as team members missing interviews due to altered notification schedules. * Gather continuous feedback from stakeholders during small-scale experiments, allowing for iterative adjustments or quick reversals before a full rollout. To scale operations effectively, start by breaking your current workload into its smallest possible components and identifying the most frequent manual repetitions. True efficiency often comes from these small, validated adjustments and consistent feedback loops rather than waiting to build a perfect, fully automated end-to-end system.