Project Automation with AI: Faster and Sm (opens in new tab)

This session from NAVER Engineering Day 2025 explores how developers can transition AI from a simple assistant into a functional project collaborator through local automation. By leveraging local Large Language Models (LLMs) and the Model Context Protocol (MCP), development teams can automate high-friction tasks such as build failure diagnostics and crash log analysis. The presentation demonstrates that integrating these tools directly into the development pipeline significantly reduces the manual overhead required for routine troubleshooting and reporting. ### Integrating LLMs with Local Environments * Utilizing **Ollama** allows teams to run LLMs locally, ensuring data privacy and reducing latency compared to cloud-based alternatives. * The **mcp-agent** (Model Context Protocol) serves as the critical bridge, connecting the LLM to local file systems, tools, and project-specific data. * This infrastructure enables the AI to act as an "agent" that can autonomously navigate the codebase rather than just processing static text prompts. ### Build Failure and Crash Monitoring Automation * When a build fails, the AI agent automatically parses the logs to identify the root cause, providing a concise summary instead of requiring a developer to sift through thousands of lines of terminal output. * For crash monitoring, the system goes beyond simple summarization by analyzing stack traces and identifying the specific developer or team responsible for the affected code segment. * By automating the initial diagnostic phase, the time between an error occurring and a developer beginning the fix is dramatically shortened. ### Intelligent Reporting via Slack * The system integrates with **Slack** to deliver automated, context-aware reports that categorize issues by severity and impact. * These reports include actionable insights, such as suggested fixes or links to relevant documentation, directly within the communication channel used by the team. * This ensures that project stakeholders remain informed of the system's health without requiring manual status updates from engineers. ### Considerations for LLM and MCP Implementation * While powerful, the combination of LLMs and MCP agents is not a "silver bullet"; it requires careful prompt engineering and boundary setting to prevent hallucination in technical diagnostics. * Effective automation depends on the quality of the local context provided to the agent; the more structured the logs and metadata, the more accurate the AI's conclusions. * Organizations should evaluate the balance between the computational cost of running local models and the productivity gains achieved through automation. To successfully implement AI-driven automation, developers should start by targeting specific, repetitive bottlenecks—such as triaging build errors—before expanding the agent's scope to more complex architectural tasks. Focusing on the integration between Ollama and mcp-agent provides a secure, extensible foundation for building a truly "smart" development workflow.

@RequestCache: Developing a Custom (opens in new tab)

The development of `@RequestCache` addresses the performance degradation and network overhead caused by redundant external API calls or repetitive computations within a single HTTP request. By implementing a custom Spring-based annotation, developers can ensure that specific data is fetched only once per request and shared across different service layers. This approach provides a more elegant and maintainable solution than manual parameter passing or struggling with the limitations of global caching strategies. ### Addressing Redundant Operations in Web Services * Modern web architectures often involve multiple internal services (e.g., Order, Payment, and Notification) that independently request the same data, such as a user profile. * These redundant calls increase response times, put unnecessary load on external servers, and waste system resources. * `@RequestCache` provides a declarative way to cache method results within the scope of a single HTTP request, ensuring the actual logic or API call is executed only once. ### Limitations of Manual Data Passing * The common alternative of passing response objects as method parameters leads to "parameter drilling," where intermediate service layers must accept data they do not use just to pass it to a deeper layer. * In the "Strategy Pattern," adding a new data dependency to an interface forces every implementation to change, even those that have no use for the new parameter, which violates clean architecture principles. * Manual passing makes method signatures brittle and increases the complexity of refactoring as the call stack grows. ### The TTL Dilemma in Traditional Caching * Using Redis or a local cache with Time-To-Live (TTL) settings is often insufficient for request-level isolation. * If the TTL is set too short, the cache might expire before a long-running request finishes, leading to the very redundant calls the system was trying to avoid. * If the TTL is too long, the cache persists across different HTTP requests, which is logically incorrect for data that should be fresh for every new user interaction. ### Leveraging Spring’s Request Scope and Proxy Mechanism * The implementation utilizes Spring’s `@RequestScope` to manage the cache lifecycle, ensuring that data is automatically cleared when the request ends. * Under the hood, `@RequestScope` uses a Singleton Proxy that delegates calls to a specific instance stored in the `RequestContextHolder` for the current thread. * The cache relies on `RequestAttribute`, which uses `ThreadLocal` storage to guarantee isolation between different concurrent requests. * Lifecycle management is handled by Spring’s `FrameworkServlet`, which prevents memory leaks by automatically cleaning up request attributes after the response is sent. For applications dealing with deep call stacks or complex service interactions, a request-scoped caching annotation provides a robust way to optimize performance without sacrificing code readability. This mechanism is particularly recommended when the same data is needed across unrelated service boundaries within a single transaction, ensuring consistency and efficiency throughout the request lifecycle.

This is the First Click (opens in new tab)

This session from NAVER ENGINEERING DAY 2025 explores the implementation of visual data tools to interpret complex user behavior within Naver’s Integrated Search. By transforming raw quantitative click logs into intuitive heatmaps and histograms, the development team provides a clearer understanding of how users navigate and consume content. This approach serves as a critical bridge for stakeholders to find actionable evidence for service improvements that are often obscured by traditional data analysis. ### Visualizing User Intent through Heatmaps and Histograms * Click logs from Naver Integrated Search are converted into heatmaps to pinpoint exactly where users are focusing their attention and making their "first clicks." * Histograms are utilized alongside heatmaps to provide a temporal and frequency-based perspective on user interactions, making it easier to identify patterns in data consumption. * The visualization system aims to help developers and designers who struggle with raw quantitative data to gain an immediate, intuitive grasp of user experience (UX) performance. ### Handling Dynamic Data in Real-Time Search Services * The system is designed to respond to the "real-time evolution" of Naver Search, where content and UI layouts change frequently based on trends and algorithms. * The FE Infrastructure team shared technical know-how on collecting and processing client-side logs to ensure data accuracy even as the search interface evolves. * Significant trial and error were involved in developing a visualization framework that remains consistent and reliable across diverse search result types and user devices. ### Practical Application for Service Improvement * By using heatmaps as a primary diagnostic tool, teams can move beyond speculative design and base UI/UX updates on concrete visual evidence of user friction or engagement. * The technology allows for the identification of "dead zones" or overlooked features that may require repositioning or removal to streamline the search experience. * Integrating these visual tools into the development workflow enables faster feedback loops between data analysis and front-end implementation. For organizations managing high-traffic web platforms, moving from raw data tables to visual behavior mapping is essential for understanding the nuance of user interaction. Implementing a robust heatmap and histogram system allows teams to validate product hypotheses quickly and ensures that service updates are driven by actual user behavior rather than just aggregate metrics.

Building Data Lineage- (opens in new tab)

Naver Webtoon developed "Flow.er," an on-demand data lineage pipeline service designed to overcome the operational inefficiencies and high maintenance costs of legacy data workflows. By integrating dbt for modular modeling and Airflow for scalable orchestration, the platform automates complex backfill and recovery processes while maintaining high data integrity. This shift to a lineage-centric architecture allows the engineering team to manage data as a high-quality product rather than a series of disconnected tasks. ### Challenges in Traditional Data Pipelines * High operational burdens were caused by manual backfilling and recovery tasks, which became increasingly difficult as data volume and complexity grew. * Legacy systems lacked transparency in data dependencies, making it hard to predict the downstream impact of code changes or upstream data failures. * Fragmented development environments led to inconsistencies between local testing and production outputs, slowing down the deployment of new data products. ### Core Architecture and the Role of dbt and Airflow * dbt serves as the central modeling layer, defining transformations and establishing clear data lineage that maps how information flows between tables. * Airflow functions as the orchestration engine, utilizing the lineage defined in dbt to trigger tasks in the correct order and manage execution schedules. * Individual development instances provide engineers with isolated environments to test dbt models, ensuring that logic is validated before being merged into the main pipeline. * The system includes a dedicated model management page and a robust CI/CD pipeline to streamline the transition from development to production. ### Expanding the Platform with Tower and Playground * "Tower" and "Playground" were introduced as supplementary components to support a broader range of data organizations and facilitate easier experimentation. * A specialized Partition Checker was developed to enhance data integrity by automatically verifying that all required data partitions are present before downstream processing begins. * Improvements to the Manager DAG system allow the platform to handle large-scale pipeline deployments across different teams while maintaining a unified view of the data lineage. ### Future Evolution with AI and MCP * The team is exploring the integration of Model Context Protocol (MCP) servers to bridge the gap between data pipelines and AI applications. * Future developments focus on utilizing AI agents to further automate pipeline monitoring and troubleshooting, reducing the need for human intervention in routine maintenance. To build a sustainable and scalable data infrastructure, organizations should transition from simple task scheduling to a lineage-aware architecture. Adopting a framework like Flow.er, which combines the modeling strengths of dbt with the orchestration power of Airflow, enables teams to automate the most labor-intensive parts of data engineering—such as backfills and dependency management—while ensuring the reliability of the final data product.

[AI_TOP_1 (opens in new tab)

The AI TOP 100 contest was designed to shift the focus from evaluating AI model performance to measuring human proficiency in solving real-world problems through AI collaboration. By prioritizing the "problem-solving process" over mere final output, the organizers sought to identify individuals who can define clear goals and navigate the technical limitations of current AI tools. The conclusion of this initiative suggests that true AI literacy is defined by the ability to maintain a "human-in-the-loop" workflow where human intuition guides AI execution and verification. ### Core Philosophy of Human-AI Collaboration * **Human-in-the-Loop:** The contest emphasizes a cycle of human analysis, AI problem-solving, and human verification. This ensures that the human remains the "pilot" who directs the AI engine and takes responsibility for the quality of the result. * **Strategic Intervention:** Participants were encouraged to provide AI with structural context it might struggle to perceive (like complex table relationships) and to perform data pre-processing to improve AI accuracy. * **Task Delegation:** For complex iterative tasks, such as generating images for a montage, solvers were expected to build automated pipelines using AI agents to handle repetitive feedback loops while focusing human effort on higher-level strategy. ### Designing Against "One-Shot" Solutions * **Low Barrier, High Ceiling:** Problems were designed to be intuitive enough for anyone to understand but complex enough to prevent "one-shot" solutions (the "click-and-solve" trap). * **Targeting Technical Weaknesses:** Organizers intentionally embedded technical hurdles that current LLMs struggle with, forcing participants to demonstrate how they bridge the gap between AI limitations and a correct answer. * **The Difficulty Ladder:** To account for varying domain expertise (e.g., OCR experience), problems utilized a multi-part structure. This included "Easy" starting questions to build momentum and "Medium" hint questions that guided participants toward solving the more difficult "Killer" components. ### The 4-Pattern Problem Framework * **P1 - Insight (Analysis & Definition):** Identifying meaningful opportunities or problems within complex, unstructured data. * **P2 - Action (Implementation & Automation):** Developing functional code or workflows to execute a defined solution. * **P3 - Persuasion (Strategy & Creativity):** Generating logical and creative content to communicate technical solutions to non-technical stakeholders. * **P4 - Decision (Optimization):** Making optimal choices and simulations to maximize goals under specific constraints. ### Quality Assurance and Score Calibration * **4-Stage Pipeline:** Problems moved from Ideation to Drafting (testing for one-shot immunity), then to Candidate (analyzing abuse vulnerabilities), and finally to a Final selection based on difficulty balance. * **Cross-Model Validation:** Internal and alpha testers solved problems using various models including Claude, GPT, and Gemini to ensure that no single tool could bypass the intended human-led process. * **Effort-Based Scoring:** Instead of uniform points, scores were calibrated based on the "effort cost" and human competency required to solve them. This resulted in varying total points per problem to better reflect the true difficulty of the task. In the era of rapidly evolving AI, the ability to "use" a tool is becoming less valuable than the ability to "collaborate" with it. This shift requires a move toward building automated pipelines and utilizing a "difficulty ladder" approach to tackle complex, multi-stage problems that AI cannot yet solve in a single iteration.

Zoomer: Powering AI Performance at Meta's Scale Through Intelligent Debugging and Optimization - Engineering at Meta (opens in new tab)

Zoomer is Meta’s centralized, automated platform designed to solve performance bottlenecks and GPU underutilization across its massive AI training and inference infrastructure. By integrating deep analytics with scalable data collection, the tool has become the internal standard for optimizing workloads ranging from Llama 3 training to large-scale ads recommendation engines. Ultimately, Zoomer enables significant energy savings and hardware efficiency gains, allowing Meta to accelerate model iteration and increase throughput across its global fleet of GPUs. ### The Three-Layered Architecture * **Infrastructure and Platform Layer:** This foundation utilizes Meta’s Manifold blob storage for trace data and employs fault-tolerant processing pipelines to manage massive trace files across thousands of hosts. * **Analytics and Insights Engine:** This layer performs deep analysis using specialized tools such as Kineto for GPU traces, NVIDIA DCGM for hardware metrics, and StrobeLight for CPU profiling. It automatically detects performance anti-patterns and provides actionable optimization recommendations. * **Visualization and User Interface Layer:** The presentation layer transforms complex data into interactive timelines and heat maps. It integrates with Perfetto for kernel-level inspection and provides drill-down dashboards that highlight outliers across distributed GPU deployments. ### Automated Profiling and Data Capture * **Trigger Mechanisms:** To ensure data accuracy, Zoomer automatically triggers profiling for training workloads during stable states (typically around iteration 550) to avoid startup noise, while inference workloads use on-demand or benchmark-integrated triggers. * **Comprehensive Metrics:** The platform simultaneously collects GPU SM utilization, Tensor Core usage, memory bandwidth, and power consumption via DCGM. * **System-Level Telemetry:** Beyond the GPU, Zoomer captures host-level data including CPU utilization, storage access patterns, and network I/O through dyno telemetry. * **Distributed Communication:** For large-scale training, the tool analyzes NCCL collective operations and inter-node communication patterns to identify stragglers and network bottlenecks. ### Inference and Training Optimization * **Inference Performance:** Zoomer tracks request/response latency, GPU memory allocation patterns, and Thrift request-level profiling to identify bottlenecks in serving user requests at scale. * **Workflow Acceleration:** By correlating application-level annotations—such as forward/backward passes and optimizer steps—with hardware performance, developers can pinpoint exactly which part of a model's execution is inefficient. * **Operational Impact:** These insights have led to significant improvements in Queries Per Second (QPS) for recommendation models and reduced training times for generative AI features by eliminating resource waste. For organizations managing large-scale AI clusters, the Zoomer model suggests that the key to efficiency is moving away from manual, reactive debugging toward an "always-on" automated profiling system. Correlating high-level software phases with low-level hardware telemetry is essential for maximizing the return on investment for expensive GPU resources and maintaining rapid iteration cycles.

Code Quality Improvement Techniques Part 24: The Value of Legacy (opens in new tab)

The LY Corporation Review Committee advocates for simplifying code by avoiding unnecessary inheritance when differences between classes are limited to static data rather than dynamic logic. By replacing complex interfaces and subclasses with simple data models and specific instances, developers can reduce architectural overhead and improve code readability. This approach ensures that configurations, such as UI themes, remain predictable and easier to maintain without the baggage of a type hierarchy. ### Limitations of Inheritance-Based Configuration * The initial implementation used a `FooScreenThemeStrategy` interface to define UI elements like background colors, text colors, and icons. * Specific themes (Light and Dark) were implemented as separate classes that overridden the interface properties. * This pattern creates an unnecessary proliferation of types when the only difference between the themes is the specific value of the constants being returned. * Using inheritance for simple value changes makes the code harder to follow and can lead to over-engineering. ### Valid Scenarios for Inheritance * **Dynamic Logic:** When behavior needs to change dynamically at runtime via dynamic dispatch. * **Sum Types:** Implementing restricted class hierarchies, such as Kotlin `sealed` classes or Java's equivalent. * **Decoupling:** Separating interface from implementation to satisfy DI container requirements or to improve build speeds. * **Dependency Inversion:** Applying architectural patterns to resolve circular dependencies or to enforce one-way dependency flows. ### Transitioning to Data Models and Instantiation * Instead of an interface, a single "final" class or data class (e.g., `FooScreenThemeModel`) should be defined to hold the required properties. * Individual themes are created as simple instances of this model rather than unique subclasses. * In Kotlin, defining a class without the `open` keyword ensures that the properties are not dynamically altered and that no hidden, instance-specific logic is introduced. * This "instantiation over inheritance" strategy guarantees that properties remain static and the code remains concise. To maintain a clean codebase, prioritize data-driven instantiation over class-based inheritance whenever logic remains constant. This practice reduces the complexity of the type system and makes the code more resilient to unintended side effects.

Connecting Thousands of LY Corporation Services (opens in new tab)

LY Corporation developed a centralized control plane using Central Dogma to manage service-to-service communication across its vast, heterogeneous infrastructure of physical machines, virtual machines, and Kubernetes clusters. By adopting the industry-standard xDS protocol, the new system resolves the interoperability and scaling limitations of their legacy platform while providing a robust GitOps-based workflow. This architecture enables the company to connect thousands of services with high reliability and sophisticated traffic control capabilities. ## Limitations of the Legacy System The previous control plane environment faced several architectural bottlenecks that hindered developer productivity and system flexibility: * **Tight Coupling:** The system was heavily dependent on a specific internal project management tool (PMC), making it difficult to support modern containerized environments like Kubernetes. * **Proprietary Schemas:** Communication relied on custom message schemas, which created interoperability issues between different clients and versions. * **Lack of Dynamic Registration:** The legacy setup could not handle dynamic endpoint registration effectively, functioning more as a static registry than a functional service mesh control plane. * **Limited Traffic Control:** It lacked the ability to perform complex routing tasks, such as canary releases or advanced client-side load balancing, across diverse infrastructures. ## Central Dogma as a Control Plane To solve these issues, the team leveraged Central Dogma, a Git-based repository service for textual configuration, to act as the foundation for a new control plane: * **xDS Protocol Integration:** The new control plane implements the industry-standard xDS protocol, ensuring seamless compatibility with Envoy and other modern data plane proxies. * **GitOps Workflow:** By utilizing Central Dogma’s mirroring features, developers can manage service configurations and traffic policies safely through Pull Requests in external Git repositories. * **High Reliability:** The system inherits Central Dogma’s native strengths, including multi-datacenter replication, high availability, and a robust authorization system. * **Schema Evolution:** The control plane automatically transforms legacy metadata into standard xDS resources, allowing for a smooth transition from old infrastructure to the new service mesh. ## Dynamic Service Discovery and Registration The architecture provides automated ways to manage service endpoints across different environments: * **Kubernetes Endpoint Plugin:** A dedicated plugin watches for changes in Kubernetes services and automatically updates the xDS resource tree in Central Dogma. * **Automated API Registration:** The system provides gRPC and HTTP APIs (e.g., `RegisterLocalityLbEndpoint`) that allow services to register themselves dynamically during the startup process. * **Advanced Traffic Features:** The new control plane supports sophisticated features like zone-aware routing, circuit breakers, automatic retries, and "slow start" mechanisms for new endpoints. ## Evolution Toward Sidecar-less Service Mesh A major focus of the project is improving the developer experience by reducing the operational overhead of the data plane: * **Sidecar-less Options:** The team is working toward providing service mesh benefits without requiring a sidecar proxy for every pod, which reduces resource consumption and simplifies debugging. * **Unified Control:** Central Dogma acts as a single source of truth for both proxy-based and proxyless service mesh configurations, ensuring consistent policy enforcement across the entire organization. For organizations managing large-scale, heterogeneous infrastructure, transitioning to an xDS-compliant control plane backed by a reliable Git-based configuration store is highly recommended. This approach balances the need for high-speed dynamic updates with the safety and auditability of GitOps, ultimately allowing for a more scalable and developer-friendly service mesh.

Key Transparency Comes to Messenger - Engineering at Meta (opens in new tab)

Messenger has enhanced the security of its end-to-end encrypted chats by launching key transparency, a system that provides an automated, verifiable record of public encryption keys. By moving beyond manual key comparisons, this feature ensures that users can verify their contacts' identities without technical friction, even when those contacts use multiple devices. This implementation allows Messenger to provide a higher level of assurance that no third party, including Meta, has tampered with or swapped the keys used to secure a conversation. ## The Role of Key Transparency in Encrypted Messaging * Provides a verifiable and auditable record of public keys, ensuring that messages are always encrypted with the correct keys for the intended recipient. * Prevents "man-in-the-middle" attacks by a compromised server by making any unauthorized key changes visible to the system. * Simplifies the user experience by automating the verification process, which previously required users to manually compare long strings of characters across every device their contact owned. ## Architecture and Third-Party Auditing * Built upon the open-source Auditable Key Directory (AKD) library, which was previously used to implement similar security properties for WhatsApp. * Partners with Cloudflare to act as a third-party auditor, maintaining a public Key Transparency Dashboard that allows anyone to verify the integrity of the directory. * Leverages an "epoch" system where the directory is updated and published frequently to ensure that the global log of keys remains current and immutable. ## Scaling for Global Messenger Traffic * Manages a massive database that has already grown to billions of entries, reflecting the high volume of users and the fact that Messenger indexes keys for every individual device a user logs into. * Operates at a high frequency, publishing a new epoch approximately every two minutes, with each update containing hundreds of thousands of new key entries. * Optimized the algorithmic efficiency of the AKD library to ensure that cryptographic proof sizes remain small and manageable, even as the number of updates for a single key grows over time. ## Infrastructure Resilience and Recovery * Improved the system's ability to handle temporary outages and long delays in key sequencing, drawing on two years of operational data from the WhatsApp implementation. * Replaced older proof methods that grew linearly with the height of the transparency tree with more efficient operations to maintain high availability and real-time verification speeds. * Established a robust recovery process to ensure that the transparency log remains consistent even after infrastructure disruptions. By automating the verification of encryption keys through a transparent, audited directory, Messenger has made sophisticated cryptographic security accessible to billions of users. This rollout represents a significant shift in how trust is managed in digital communications, replacing manual user checks with a seamless, background-level guarantee of privacy.

Reducing EV range anxiety: How a simple AI model predicts port availability (opens in new tab)

Google Research has developed a lightweight AI model designed to predict the probability of EV charging port availability at specific future intervals, directly addressing the "range anxiety" experienced by electric vehicle drivers. By co-designing the model with deployment infrastructure, researchers found that a simple linear regression approach outperformed more complex architectures like neural networks and decision trees. The resulting system effectively predicts availability changes during high-turnover periods, providing more reliable navigation and planning data than traditional "no-change" assumptions. ### Model Architecture and Feature Selection * The development team prioritized a minimal feature set to ensure low-latency deployment and high speed in real-world navigational applications. * After testing various architectures, a straightforward linear regression model was selected for its robustness and superior performance in this specific predictive task. * The model was trained using real-time availability data from diverse geographical regions, specifically California and Germany, with an emphasis on larger charging stations that reflect high-traffic usage patterns. ### Temporal Feature Weights and Occupancy Trends * The model uses the hour of the day as a primary feature, treating each hour as an independent variable to capture specific daily cycles. * Learned numerical "weights" dictate the predicted rate of occupancy change: positive weights indicate ports are becoming occupied (e.g., during morning rush), while negative weights indicate ports are being freed up (e.g., during evening hours). * The system is designed to only deviate from the current occupancy state when the change rate is statistically significant or when a station's large size amplifies the likelihood of a status change. ### Performance Benchmarking and Validation * The model was evaluated against a "Keep Current State" baseline, which assumes future availability will be identical to the present status—a difficult baseline to beat since port status remains unchanged roughly 90% of the time over 30-minute windows. * Accuracy was measured using Mean Squared Error (MSE) and Mean Absolute Error (MAE) over 30-minute and 60-minute time horizons across 100 randomly selected stations. * Testing confirmed that the linear regression model provides its greatest value during infrequent but critical moments of high turnover, successfully identifying when a station is likely to become full or available. The success of this model demonstrates that sophisticated deep learning is not always the optimal solution for infrastructure challenges. By combining intuitive real-world logic—such as driver schedules and station capacity—with simple machine learning techniques, developers can create highly efficient tools that significantly improve the EV user experience without requiring massive computational overhead.

Things to know when using Kafka in a (opens in new tab)

The Apache Kafka ecosystem is undergoing a significant architectural shift with the introduction of Consumer Group Protocol v2, as outlined in KIP-848. This update addresses long-standing performance bottlenecks and stability issues inherent in the original client-side rebalancing logic by moving the responsibility of partition assignment to the broker. This change effectively eliminates the "stop-the-world" effect during rebalances and significantly improves the scalability of large-scale consumer groups. ### Limitations of the Legacy Consumer Group Protocol (v1) * **Heavy Client-Side Logic:** In v1, the "Group Leader" (a specific consumer instance) is responsible for calculating partition assignments, which creates a heavy burden on the client and leads to inconsistent behavior across different programming language implementations. * **Stop-the-World Rebalancing:** Whenever a member joins or leaves the group, all consumers must stop processing data until the new assignment is synchronized, leading to significant latency spikes. * **Sensitivity to Processing Delays:** Because heartbeats and data processing often share the same thread, a slow consumer can trigger a session timeout, causing an unnecessary and disruptive group rebalance. ### Architectural Improvements in Protocol v2 * **Server-Side Reconciliation:** The reconciliation logic is moved to the Group Coordinator on the broker, simplifying the client and ensuring that partition assignment is managed centrally and consistently. * **Incremental Rebalancing:** Unlike the "eager" rebalancing of v1, the new protocol allows consumers to keep their existing partitions while negotiating new ones, ensuring continuous data processing. * **Decoupled Heartbeats:** The heartbeat mechanism is separated from the main processing loop, preventing "zombie member" scenarios where a busy consumer is incorrectly marked as dead. ### Performance and Scalability Gains * **Reduced Rebalance Latency:** By offloading the assignment logic to the broker, the time required to stabilize a group after a membership change is reduced from seconds to milliseconds. * **Large-Scale Group Support:** The new protocol is designed to handle thousands of partitions and hundreds of consumers within a single group without the exponential performance degradation seen in v1. * **Stable Deployments:** During rolling restarts or deployments, the group remains stable and avoids the "rebalance storms" that typically occur when multiple instances cycle at once. ### Migration and Practical Implementation * **Configuration Requirements:** Users can opt-in to the new protocol by setting the `group.protocol` configuration to `consumer` (introduced as early access in Kafka 3.7 and standard in 4.0). * **Compatibility:** While the new protocol requires updated brokers and clients, it is designed to support a transition phase to allow organizations to migrate their workloads gradually. * **New Tooling:** Updated command-line tools and metrics are provided to monitor the server-side assignment process and track group state more granularly. Organizations experiencing frequent rebalance issues or managing high-throughput Kafka clusters should plan for a migration to Consumer Group Protocol v2. Transitioning to this server-side assignment model is highly recommended for stabilizing production environments and reducing the operational overhead associated with consumer group management.

Beyond the Side Effects of API- (opens in new tab)

JVM applications often suffer from initial latency spikes because the Just-In-Time (JIT) compiler requires a "warm-up" period to optimize frequently executed code into machine language. While traditional strategies rely on simulated API calls to trigger this optimization, these methods often introduce side effects like data pollution, log noise, and increased maintenance overhead. This new approach advocates for a library-centric warm-up that targets core execution paths and dependencies directly, ensuring high performance from the first real request without the risks of full-scale API simulation. ### Limitations of Traditional API-Based Warm-up * **Data and State Pollution:** Simulated API calls can inadvertently trigger database writes, send notifications, or pollute analytics data, requiring complex logic to bypass these side effects. * **Maintenance Burden:** As business logic and API signatures change, developers must constantly update the warm-up scripts or "dummy" requests to match the current application state. * **Operational Risk:** Relying on external dependencies or complex internal services during the warm-up phase can lead to deployment failures if the mock environment is not perfectly aligned with production. ### The Library-Centric Warm-up Strategy * **Targeted Optimization:** Instead of hitting the entry-point controllers, the focus shifts to warming up heavy third-party libraries and internal utility classes (e.g., JSON parsers, encryption modules, and DB drivers). * **Internal Execution Path:** By directly invoking methods within the application's service or infrastructure layer during the startup phase, the JIT compiler can reach "Tier 4" (C2) optimization for critical code blocks. * **Decoupled Logic:** Because the warm-up targets underlying libraries rather than specific business endpoints, the logic remains stable even when the high-level API changes. ### Implementation and Performance Verification * **Reflection and Hooks:** The implementation uses application startup hooks to execute intensive code paths, ensuring the JVM is "hot" before the load balancer begins directing traffic to the instance. * **JIT Compilation Monitoring:** Success is measured by tracking the number of JIT-compiled methods and the time taken to reach a stable state, specifically targeting the reduction of "cold" execution time. * **Latency Improvements:** Empirical data shows a significant reduction in P99 latency during the first few minutes of deployment, as the most CPU-intensive library functions are already pre-optimized. ### Advantages and Practical Constraints * **Safer Deployments:** Removing the need for simulated network requests makes the deployment process more robust and prevents accidental side effects in downstream systems. * **Granular Control:** Developers can selectively warm up only the most performance-sensitive parts of the application, saving startup time compared to a full-system simulation. * **Incomplete Path Coverage:** A primary limitation is that library-only warming may miss specific branch optimizations that occur only during full end-to-end request processing. To achieve the best balance between safety and performance, engineering teams should prioritize warming up shared infrastructure libraries and high-overhead utilities. While it may not cover 100% of the application's execution paths, a library-based approach provides a more maintainable and lower-risk foundation for JVM performance tuning than traditional request-based methods.

Were we solving the real (opens in new tab)

The POPM (Product Owner/Product Manager) training course at Kakao focuses on restructuring existing professional knowledge into a cohesive framework for solving real-world business problems. Rather than simply delivering new information, the program emphasizes aligning strategy with execution, transforming "strategy" from a vague concept into a practical set of decision-making criteria. The ultimate goal is to move teams away from a "release-only" mindset toward a cycle of continuous hypothesis verification and learning. ### Strategic Thinking and Metric Modeling * **Strategic Decision Criteria**: Strategy is redefined as the standard for team judgment, utilizing frameworks like MECE, MVP, and priority-setting models to align daily tasks with long-term goals. * **Metrics as Problem-Solving Language**: Key indicators such as Funnel, Retention, Cohort, and LTV are treated not just as data points, but as a language used to define and reveal underlying product issues. * **Context-Based Design**: UX design is approached through "context-based logic" rather than intuition, encouraging teams to ask which specific design fits the current user journey. ### Systematic Experimentation and A/B Testing * **The MASS Framework**: Experiments are designed and evaluated based on being Measurable, Attributable, Sensitive, and having a Short-term cycle. * **Failure Analysis Routines**: The curriculum emphasizes the importance of establishing a routine for interpreting failed experiments, ensuring that every test contributes to the team's institutional knowledge. * **Incremental Testing**: Encourages a culture of "starting small," giving teams the confidence to run experiments without requiring massive resource allocation. ### Building Repeatable Execution Loops * **Metric-Based Retrospectives**: Teams transition from simply finishing a release to a structured loop of "Problem Definition → Hypothesis → Metric → Verification → Retrospective." * **Formalizing Problem Definitions**: Using templates to 명문화 (formally document) the problem, expected behavior, and success metrics ensures that the entire team—not just the PO—understands the "why" behind every task. * **Operational Rhythms**: Teams are adopting fixed weekly or bi-weekly cycles for sharing insights and adjusting priorities, turning data-driven execution into a natural habit. The most critical takeaway for product teams is to constantly ask: "Is the work we are doing right now actually a solution to a defined problem, or are we just busy releasing features?" Success lies in moving beyond the sense of accomplishment from a launch and establishing a repeatable rhythm that validates whether those efforts truly move the needle.

How the POPM program became (opens in new tab)

Kakao developed its internal POPM (Product Owner/Product Manager) training program by treating the curriculum itself as an evolving product rather than a static lecture series. By applying agile methodologies such as data-driven prioritization and iterative versioning, the program successfully moved from a generic pilot to a structured framework that aligns teams through a shared language of problem-solving. This approach demonstrates that internal capability building is most effective when managed with the same rigor and experimentation used in software development. ## Strategic Motivation for POPM Training * Addressed the inherent ambiguity of the PO/PM role, where non-visible tasks often make it difficult for practitioners to define their own growth or impact. * Sought to resolve the disconnect between strategic problem definition (PO) and tactical execution (PM) within Kakao’s teams. * Prioritized the creation of a "common language" to allow cross-functional team members to define problems, analyze metrics, and design experiments under a unified structure. ## Iterative Design and Versioning * The program transitioned through multiple "versions," starting with an 8-session pilot that covered the entire lifecycle from bottleneck exploration to execution review. * Based on participant feedback regarding high fatigue and low efficiency in long presentations, the curriculum was condensed into 5 core modules: Strategy, Metrics, Experiment, Design, and Execution. * The instructional design shifted from "delivering information" to "designing a rhythm," utilizing a "one slide, one question, one example" rule to maintain engagement. ## Data-Driven Program Refinement * Applied a "Product Metaphor" to education by calculating "Opportunity Scores" using a matrix of Importance vs. Satisfaction for each session. * Identified "Data/Metrics" as the highest priority for redesign because it scored high in importance but low in satisfaction, indicating a structural gap in the teaching method. * Refined the "features" of the training by redesigning worksheets to focus on execution routines and converting mandatory practice tasks into selective, flexible modules. ## Structural Insights for Organizational Growth * Focused on accumulating "structure" rather than just training individuals, ensuring that even as participants change, the framework for defining problems remains consistent within the organization. * Designed practice sessions to function as "thinking structures" rather than "answer-seeking" exercises, encouraging teams to bring their training insights directly into actual team meetings. * Prioritized scalability and simplicity in the curriculum to ensure the structure can be adopted across different departments with varying product needs. To build effective internal capabilities, organizations should treat training as a product that requires constant maintenance and versioning. Instead of focusing on one-off lectures, leaders should design structural "rhythms" and feedback loops that allow the curriculum to evolve based on the actual pain points of the practitioners.

MTVi: A New Metric for (opens in new tab)

Toss has developed MTVi (Mid-term Value - incremental) to quantify the financial impact of specific services within its platform, moving beyond the limitations of traditional LifeTime Value (LTV). By focusing on the incremental value generated over a one-year period, the metric allows the company to justify services that may lose money individually but drive significant ecosystem-wide growth. This framework provides a data-driven standard for prioritizing features and setting marketing budgets based on actual financial contributions. ### Limitations of Traditional LTV * **Time Horizon Mismatch:** Traditional LTV projects value over 3 to 5 years, which is too slow for Toss’s rapid iteration cycles and fails to reflect the immediate impact of service improvements. * **Investment Recovery Gaps:** Standard LTV models often benchmark marketing costs (CAC) against long-term projections, making it difficult to evaluate the efficiency of short-term experiments. * **Lack of Incrementality:** LTV measures average user value but cannot isolate the specific "extra" value created by a single service, making it impossible to distinguish between a service's impact and natural user growth. ### Defining MTVi and DID Methodology * **Incremental Focus:** MTVi is defined as the net financial value generated over one year specifically because a user experienced a new service, rather than just the average revenue of a user. * **Quasi-Experimental Design:** Since A/B testing every service combination is impossible, Toss uses the Difference-in-Difference (DID) method to compare "Newly Activated Users" (NAU) against "Never" users. * **Segment-Based Analysis:** To prevent bias—such as highly active users naturally gravitating toward more services—Toss segments users by age and historical activity (e.g., app open frequency) to ensure "apples-to-apples" comparisons within identical cohorts. ### Organizational Impact and Strategy * **Unified Decision Metric:** MTVi provides a "common language" for different product teams (silos), allowing them to compare the value of disparate services—like pedometers versus remittances—on a single financial scale. * **Efficiency Benchmarking:** The metric establishes a hard ceiling for investment; for example, Customer Acquisition Cost (CAC) is strictly managed so it does not exceed the calculated MTVi. * **Platform-Wide Valuation:** By calculating both direct revenue and indirect spillover effects, Toss can prove the financial viability of "loss-leader" services that provide user benefits but increase overall app engagement and cross-service usage. For organizations operating complex multi-service platforms, adopting an incremental value metric like MTVi is essential for moving beyond isolated P&L statements. Data teams should prioritize quasi-experimental methods like DID and rigorous user segmentation to accurately map how individual features influence the broader financial health of the ecosystem.