Kakao

34 posts

tech.kakao.com

Filter by tag

kakao4 min readCurated summary

Experience Building and Operating a Personalized Airflow Testing Environment

Kakao’s data engineering team built AirZone to make Airflow DAG testing faster, easier, and safer across an ecosystem containing thousands of DAGs and multiple Hadoop clusters. Existing approaches required local setup, repeated Git synchronization, file copying, VPN access, or risky testing on production Airflow. AirZone instead creates an isolated, production-like Airflow environment for each pull request, managed through GitHub comments and Kubernetes automation. ## Limitations of Existing Testing Methods - **Local Airflow** - Requires configuring Airflow, Hadoop authentication, connections, and Docker locally. - Has a high initial setup cost and may differ from production. - **Development Airflow** - Requires committing and pushing every code change. - Git submodule updates and DAG parsing introduce long feedback delays. - **Test Airflow with SSH** - Allows files to be copied directly into a container. - Still requires copying files after every edit. - Access to production Hadoop requires connecting to a production VPN. - **Testing on production Airflow** - Heavy test DAGs consume shared scheduler, worker, and node resources. - A resource-intensive test can delay or interrupt unrelated projects. - Per-user isolation is therefore essential. ## AirZone Requirements - Provide an Airflow environment without requiring users to understand Kubernetes or Helm. - Allow code editing through a browser using Jupyter Notebook. - Execute DAGs against Hadoop and authentication mechanisms similar to production. - Create an independent environment for each pull request. - Prevent one user’s tests from affecting other workflows. ## PR-Based, Isolated Architecture - GitHub pull request comments serve as the user interface. - Users can create or delete an environment directly from a PR. - The resulting environment link is posted back to the PR. - Each PR receives a dedicated Kubernetes namespace based on the repository and PR number. - Airflow web server, scheduler, PostgreSQL, Jupyter, DAG volumes, and logs are isolated. - Multiple PRs can be tested simultaneously. - Cleanup is straightforward because the namespace defines the environment boundary. - A dedicated AirZone Helm chart packages the complete test environment. - Production-only components such as PGBouncer and external database connections are omitted where unnecessary. - Airflow, PostgreSQL, DAG storage, Jupyter, authentication, TLS, and logging are deployed together. ## Separating Requests from Deployment - `airzone-api` only validates requests: - Confirms that the PR exists and is open. - Checks branch information. - Prevents duplicate namespaces. - Kubernetes Jobs perform the long-running work: - Install the Helm release. - Run health checks. - Handle creation and deletion independently from the API process. - Job names include the operation and namespace, such as: - `create-airzone-{namespace}` - `delete-airzone-{namespace}` - Failed Jobs can be removed and recreated for retries. - Independent Job logs and status make deployment failures easier to diagnose. - A daily CronJob removes environments that remain after their PRs are closed. ## Building the Airflow Environment Each Helm deployment includes the components needed for a realistic test environment: - **Git integration:** Synchronizes the PR’s head repository and branch. - **DAG PVC:** Lets the scheduler and Jupyter use the same working directory. - **Airflow configuration:** Uses KubernetesExecutor and test-specific DAG scanning, logging, and Hadoop settings. - **Authentication:** Injects user and shared principals, keytabs, Jupyter tokens, and TLS certificates. - **Infrastructure placement:** Selects suitable node groups and a storage class in the same region. - **Centralized logging:** Connects Airflow logs to Elasticsearch and Kibana. - **Hadoop execution:** Existing infrastructure runs Hadoop tasks in dedicated pods using custom Hadoop images, Kerberos initialization, Spark, and Hive. ## Notifications and Security - KakaoWork sends: - An initial notification when a request is received. - A completion notification after deployment. - Operational error alerts. - Sensitive information, including Jupyter and Kubernetes namespace tokens, is not posted in public PR comments. - Tokens are delivered through KakaoWork instead, keeping authentication data separate from the broader PR audience. AirZone’s main recommendation is to make testing a disposable, reproducible environment tied to the pull request itself. By combining per-PR Kubernetes namespaces, Helm-based deployment, asynchronous Jobs, production-like Hadoop access, and automatic cleanup, teams can test DAGs quickly without burdening shared Airflow or production resources.

Read original(opens in new tab)
kakao1 min readCurated summary

Beyond AI That Speaks Well: Making Kanana-o Speak the Way Users Want

The provided content does not include the blog post’s body. It only contains the title, author links, navigation, and search interface, so the article’s technical argument and conclusions cannot be summarized reliably. ## Available Information - **Title:** “Beyond AI That Speaks Well: Making Kanana-o Speak the Way Users Want” - **Topic indicated by the title:** Improving Kanana-o’s voice-generation capabilities to produce speech according to user preferences. - **Authors:** martin.gale, abigail.r, and edwin.ai - **Missing:** The article’s main sections, implementation details, experiments, and conclusions. Please provide the full article text or its URL content for a detailed summary.

Read original(opens in new tab)
kakao4 min readCurated summary

From AI That Speaks Well to AI That Speaks Exactly as Desired: Advancing Kanana-o Voice Generation

Kanana-o’s latest speech-generation improvements target two goals: faster, more efficient synthesis and more precise adherence to user instructions. Kakao addresses these through LM-SPT, a speech tokenizer that separates semantic and acoustic information while reducing the token rate from 25 Hz to 12.5 Hz, and through online reinforcement learning. Together, these changes enable Kanana-o to generate natural speech more efficiently and control characteristics such as speed, pitch, tone, and volume more reliably. ## Goals for Kanana-o’s Speech Generation - Improve real-time performance by shortening speech-token sequences and simplifying decoding. - Move beyond merely natural speech toward speech that follows explicit user preferences. - Support control over: - Speaking speed - Voice quality and tone - Pitch and intonation - Volume - Emotional and conversational style - Combine better speech representation with instruction-following training. ## Limitations of the Original System - The original Kanana-o represented speech with 25 discrete tokens per second. - Its Voice Token LM generated these tokens sequentially based on text responses and conversation context. - Speech reconstruction required two stages: - **Token-to-Mel:** Convert speech tokens into a mel-spectrogram. - **Mel-to-Waveform:** Convert the mel-spectrogram into a waveform. - The tokenizer captured linguistic content effectively but did not explicitly represent acoustic properties such as voice timbre, pitch, intonation, speed, or emotion. - Long token sequences increased generation latency and computational cost. - The two-stage decoder, often involving iterative diffusion or flow-matching inference, made the pipeline difficult to optimize for real-time services. ## LM-SPT: A More Efficient Speech Tokenizer - LM-SPT stands for **LM-aligned SPeech Tokenizer**. - It compresses speech to 12.5 frames per second—half the original rate—reducing the number of sequential prediction steps. - It represents both: - **Semantic speech tokens:** The spoken content aligned with text and conversational context. - **Acoustic speech tokens:** Voice-specific details such as timbre, pitch, intonation, and speaking rate. - This separation allows the language model to generate content and acoustic characteristics more independently and controllably. - LM-SPT uses: - Two encoders for semantic and acoustic information - One semantic codebook - Multiple acoustic codebooks - A Split Residual Vector Quantization structure ## Semantic Speech-Resynthesis Distillation - Training only for waveform reconstruction does not guarantee that semantic and acoustic information remain separated. - Earlier systems commonly distilled representations from self-supervised models such as HuBERT or WavLM. - That approach can suffer from: - Misalignment between phonetic representations and higher-level language-model semantics - Loss of information when matching models with different frame rates - LM-SPT instead uses a **Semantic Speech-Resynthesis Distillation** method: - Reconstruct speech using only semantic tokens. - Compare the original and reconstructed speech with a pretrained speech encoder aligned to language-model representations. - Train the semantic tokens to preserve the same meaning without requiring exact frame-by-frame teacher alignment. - This approach helps retain meaningful content even at the lower 12.5 Hz token rate. ## Simplified Speech Decoding - During normal tokenization and reconstruction, the system does not require a heavy pretrained speech encoder. - A lightweight learned encoder is sufficient. - The final decoder uses semantic and acoustic tokens together to reconstruct the waveform directly. - This removes the intermediate mel-spectrogram stage and replaces the previous two-stage process with a lighter single-decoder structure. - As a result, the system reduces both language-model generation length and waveform reconstruction complexity. ## Instruction Following Through Online Reinforcement Learning - LM-SPT provides the representation needed to control acoustic features at the token level. - Kanana-o also applies online reinforcement learning to teach the speech-generation module to follow diverse vocal instructions. - The objective is to balance: - Accurate compliance with requested speaking styles - Natural and high-quality audio output Kakao’s approach combines a lower-rate, semantically and acoustically structured tokenizer with reinforcement learning for instruction adherence. The result is intended to make Kanana-o faster and more suitable for real-time use while allowing users to specify not only what the system says, but how it says it.

Read original(opens in new tab)
kakao4 min readCurated summary

Smaller and More Powerful Kanana SLM Development

Kanana-2 is Kakao’s second Small Language Model series, designed to deliver strong performance under the memory and compute constraints of on-device environments. It includes 3B, 1.3B, and 0.9B models, using improved pre-training, distillation, pruning, tokenizer efficiency, and Sliding Window Attention. Kakao reports that the resulting models outperform earlier Kanana models and compare favorably with similarly sized open-source models. ## Motivation and Model Lineup - On-device services require models that are small and fast because smartphones have limited memory and compute. - Kakao uses its own SLMs in the “Kanana in KakaoTalk” service. - The Kanana-2 series consists of: - Kanana-2-3B - Kanana-2-1.3B - Kanana-2-0.9B - Base and Instruct versions of the 3B and 1.3B models are being released. - The development builds on techniques from Kanana-2-30B-A3B and earlier Kanana Nano models. ## Efficiency Improvements - **Kanana-2 Tokenizer** - Improves Korean tokenization efficiency by more than 30% compared with the previous tokenizer. - Reduces the number of tokens required for Korean text, improving processing efficiency. - **Sliding Window Attention** - Reduces KV cache size during inference. - Improves memory efficiency and helps reduce decoding bottlenecks on devices. ## 3B Pre-Training ### TPU-Based Training from Scratch - Kanana-2-3B-Base was initially trained from scratch on a TPU v5e cluster. - Kakao used a MaxText-based internal training framework. - The team developed infrastructure allowing training to transition between TPU and GPU clusters: - Pre-training was completed on TPU. - Distillation was subsequently performed on GPUs using Megatron-LM. - Pre-training used two stages: - Stage 1: 7.5 trillion tokens - Stage 2: 2 trillion tokens - The Muon optimizer was used throughout pre-training. ### Learning-Rate Scaling - Directly searching learning rates at multi-trillion-token scale would be too expensive. - Kakao instead tested learning-rate candidates using a 100-billion-token proxy dataset while preserving the Stage 1 data distribution. - The selected learning rate was scaled to the full 7.5-trillion-token training run using the Token Horizon scaling rule: `LR_target ≈ LR_proxy × (D_target / D_proxy)^−β` - The experiments used: - `D_proxy = 100B` - `D_target = 7.5T` - `β = 0.32` - This approach enabled stable hyperparameter selection with a smaller exploration budget. ## Teacher-Based Distillation - The Kanana-2-30B-A3B-Instruct-2601 model was used as the teacher. - Kakao compared Base, Instruct, and Thinking versions of the teacher model. - The Instruct teacher consistently produced the strongest student-model performance. - The result supports recent findings that post-trained teachers can be especially effective for transferring mathematical and coding capabilities. ## Long-Context Training - The model’s context length was expanded from 4K to 32K using YaRN. - Additional mid-training data was introduced during the learning-rate decay phase. - The resulting Kanana-2-3B-Base reportedly surpassed earlier Kanana 3B models across Korean and English knowledge, mathematics, and coding. - It also exceeded many similarly sized open-source SOTA base models. ## Building the 1.3B and 0.9B Models - Kanana-2-1.3B-Base and Kanana-2-0.9B-Base were progressively derived from Kanana-2-3B-Base. - The process extended the Minitron-based structured pruning and knowledge-distillation approach used for Kanana Nano. - The smaller models were trained with SWA-aware long-context procedures to preserve efficiency in on-device decoding. ### Improved Hidden-Dimension Pruning - Traditional hidden-dimension pruning scores each dimension independently using activation statistics from calibration data. - This is efficient but may overlook information represented jointly across multiple dimensions. - Kanana-2 applies PCA-based pruning inspired by Ministral 3: - Collect activation statistics from Attention RMSNorm, MLP RMSNorm, and Final RMSNorm inputs. - Compute a global rotation matrix with PCA. - Apply the rotation consistently to token embeddings and attention/MLP projection weights. - Reduce hidden dimensions after rotation, aiming to preserve more meaningful shared representations. Kakao’s approach combines large-scale pre-training, teacher distillation, structured compression, better Korean tokenization, and attention-level inference optimization. For practical on-device deployment, the smaller Kanana-2 models are intended to offer a better balance of speed, memory usage, and capability than simply deploying a larger general-purpose LLM.

Read original(opens in new tab)
kakao3 min readCurated summary

Solving Social Problems with AI Beyond Development

The first “SSAFY X Kakao Tech Bootcamp AI Hackathon” brought together 90 trainees from 12 teams to use AI for solving real social problems. Rather than focusing only on coding competition, the event emphasized public value, practical service prototypes, expert feedback, and collaboration across different training programs. It demonstrated that future developers need both technical ability and the capacity to work with others on meaningful problems. ## Connecting Kakao and Samsung’s Developer Programs - Held June 13–14 at Kakao’s AI Campus in Yongin. - Organized jointly by Kakao Tech Bootcamp and Samsung’s SSAFY program. - Participants came from two major digital-training initiatives supported by Korea’s K-Digital Training program. - The event aimed to create opportunities for collaboration and growth among future AI developers. ## Applying AI to Everyday Social Problems - Teams selected challenges from the government’s “Top 10 AI Projects for People’s Livelihoods.” - Topics included: - Small-business support - Voice-phishing prevention - Child and youth protection - Maritime safety - Over two intensive, sleepless days, teams: - Defined a specific social problem - Designed solutions from the user’s perspective - Built AI-powered service prototypes - The hackathon stressed that AI’s value depends not only on technical advancement, but also on how effectively it improves society. ## Practical Mentoring from Government and Industry - Officials from agencies including the National Police Agency, Ministry of Justice, and Ministry of Gender Equality and Family provided policy and field expertise. - Kakao developers delivered lectures and technical mentoring based on real-world service development. - Teams refined their ideas through questions, feedback, and discussions with experts. - This allowed trainees to connect classroom learning with actual policy and operational challenges. ## Collaboration Across Different Backgrounds - Kakao Tech Bootcamp and SSAFY use different educational approaches, giving participants varied experiences and strengths. - Teams worked with people they had not previously met and actively discussed how to incorporate AI into their products. - Participants discovered new perspectives and solutions by sharing their knowledge. - Many came to recognize communication and teamwork as essential skills alongside technical competence. ## Projects and the Future Developer Ecosystem - Five teams received awards after the final presentations. - The Ministry of Employment and Labor award went to “Golden Time” for **DRIFT**, an AI service supporting maritime rescue when communications are unavailable. - Kakao’s CEO award went to “SSAIKA” for **Mindam**, an AI-based civil complaint intake and processing service. - Other awards were presented by Samsung Electronics, the Korea Chamber of Commerce and Industry, and the Korea Radio Promotion Association. - Although the total prize money was 15 million won, the article identifies hands-on experience solving social problems as the participants’ more important achievement. - Kakao has trained more than 660 digital professionals since joining the K-Digital Training initiative in 2022. The hackathon suggests that AI education should combine technical training with real-world projects, expert guidance, and cross-organizational collaboration. Kakao plans to expand these practical opportunities to support developers who can turn technology into social value.

Read original(opens in new tab)
kakao4 min readCurated summary

Automating KakaoTalk Recommendation Metric Analysis with an AI Agent

The post describes Kakao’s use of an AI agent to automate repetitive analysis of KakaoTalk recommendation metrics on an existing Hadoop environment. Rather than building a new platform or granting the model broad permissions, the team documented existing procedures, data definitions, and decision rules in Markdown-based agent skills and context files. The resulting system helps analysts produce draft reports and explore follow-up questions, while humans remain responsible for validating results and making final decisions. ## Repetitive Analysis Is an Ideal Automation Target - Recommendation analysis often begins with simple questions about CTR changes, experiments, or user-group anomalies. - Answering them typically requires: - Connecting to the analysis environment - Finding the right tables - Writing and executing queries - Interpreting results - Repeating the process across dimensions such as age, category, and time - Much of the effort lies in data preparation and extraction rather than interpretation. - The initial goal was for the AI to follow these steps and produce a first-pass analysis without requiring users to handle queries directly. ## Teaching the Agent to Use Hadoop - The team did not build a new analytics platform or add an MCP integration layer. - Existing Hadoop access scripts were sufficient; the missing component was documentation explaining how to use them. - These procedures were packaged as Agent Skills—Markdown files such as `SKILL.md` describing: - How to connect to Hadoop - How to submit queries - How to retrieve and organize results - The `hadoop-butler` plugin bundled these skills for internal use. - The main lesson was that existing infrastructure can often be extended by converting undocumented operational knowledge into instructions an agent can follow. ## Context Documents Improve Analytical Accuracy - Access to data does not guarantee correct analysis. - Context files such as `CLAUDE.md` or `AGENTS.md` documented: - Relevant tables and clusters - Feature definitions, such as `watch_length` and `valid_view` - User and session aggregation rules - Standard metric definitions - This prevented the agent from repeatedly guessing which tables, columns, or aggregation rules to use. - The documentation also captured institutional knowledge that could help new team members, not only AI systems. - Output quality was determined by the quality and precision of the available context. ## AI Produces Drafts; Analysts Continue the Investigation - Natural-language analysis was most useful for recurring tasks such as: - Detecting anomalies - Comparing experiments - Reviewing weekly performance - The agent’s first report helped identify areas for deeper investigation. - Analysts could then ask follow-up questions and refine the analysis conversationally. - AI-generated reports were treated as reviewable drafts, not final conclusions. - Query logic, selected columns, metric definitions, and interpretations still required human verification. ## Plausible but Incorrect Results The agent’s most dangerous errors were not syntax failures; they were queries that executed successfully but produced misleading results. - **Semantic errors** - To count users, the correct field was `user_id`. - The agent once selected the similarly named `session_user_id`, which represented a session-oriented identifier. - The query ran normally, but the resulting user count was wrong. - **Performance errors** - The agent combined several `COUNT(DISTINCT ...)` expressions in one Hive query. - Although valid SQL, this could force processing through a single reducer and make the query extremely slow. - The better approach was to split the calculations by column and run them in parallel. ## Documentation and Regression Testing - Explicit rules were added to context files and skills, including: - Which identifier to use for user-level aggregation - Wrapping column names in backticks - Splitting multiple `COUNT(DISTINCT)` operations into separate queries - Because natural-language instructions can break other behaviors when modified, the team tested them like software. - An MLflow-based end-to-end evaluation pipeline: - Defines expected behavior for each skill - Runs the agent headlessly with `claude -p` - Uses an LLM judge to evaluate tool-call order, execution traces, and final output - Runs regression scenarios before deployment - This made it possible to catch unintended behavior changes before release. The recommended architecture combines four elements: an AI model, precise domain context, an existing execution environment such as Hadoop, and a verification loop. Organizations should first document their established procedures and analytical definitions, then connect the agent to existing tools and test its behavior systematically.

Read original(opens in new tab)
kakao3 min readCurated summary

Is a Non-Developer Who Does Vibe Coding a Developer? (3)

AI coding agents have helped a non-developer move from creating small local HTML tools to building shared dashboards, integrations, automations, and repeatable workflows. The major change was not writing more code, but learning to define data, permissions, inputs, outputs, exceptions, and validation criteria. The author concludes that AI is making more kinds of work executable and structured, expanding questions about “development” beyond professional developers. ## From Local HTML to Shared Tools - Early tools were standalone HTML files used locally in a browser. - Sharing them introduced deployment, URLs, version updates, and maintenance concerns. - Once users needed persistent data and changing states, the problem expanded from UI design to: - Data storage - Access and edit permissions - Change history and rollback - Backups and operational responsibility - Google Sheets became a practical lightweight database because it already provided: - Collaboration and familiar interfaces - Permission management - Revision history - The author progressed from manually pasting Apps Script code to using `clasp` and Apps Script APIs for deployment and execution. - The key shift was learning to evaluate where data should live and which tools already work safely within an organization. ## Security as a Daily Habit - Connecting work tools through webhooks introduced the need to protect tokens and webhook URLs. - The author began using `.env` files and `.gitignore` to prevent secrets from entering source code or Git repositories. - Requests to AI agents increasingly included security requirements: - Read secrets from environment variables - Avoid printing sensitive values in logs - Use placeholders instead of real credentials - Small automations connected to external systems naturally required thinking about secret management, execution environments, and access control. ## Turning Manual Tasks into Workflows - Tasks such as copying files, organizing folders, converting documents, editing videos, and extracting audio or summaries were delegated to AI agents. - Delegating these tasks required explicit definitions of: - Input files - Output names and formats - Whether existing files may be overwritten - Failure conditions - Verification requirements - Informal, intuitive actions became structured work specifications. - The author found that understanding completion criteria and input/output formats was often more important than memorizing individual commands. ## Skills and Feedback Loops - Weekly meeting-note preparation revealed recurring editing patterns and implicit business rules. - These rules were encoded into Codex and Claude skills covering: - Note structure - Action-item extraction - PMO-related signals - Situations where the AI should ask questions instead of making assumptions - Skills functioned as stored decision criteria, not merely collections of prompts. - Comparing AI-generated drafts with the author’s final revisions enabled continuous refinement. - Deleting local data for security reasons accidentally removed useful conversation context, temporarily reducing skill quality and demonstrating the importance of preserving relevant operational knowledge. ## Using Outputs as New Inputs - Google Analytics reporting became more frequent and detailed through MCP-based API access and reusable reporting skills. - MCP provided the data connection, while skills preserved the recurring monthly report structure. - The valuable work remained interpreting changes between periods and deciding whether a change was meaningful. - Combining AI-generated metrics with the author’s contextual knowledge helped surface signals requiring further investigation in near real time. The author recommends focusing less on how much code AI can generate and more on how work can be clearly structured for AI execution. As agents become more capable, everyone—not only developers—will increasingly define inputs, outputs, permissions, security controls, repetition, and validation as part of everyday work.

Read original(opens in new tab)
kakao3 min readCurated summary

Key Players in the Agentic AI Ecosystem: MCP Player 10 Wraps Up, and What’s Next!

Kakao’s first MCP Player 10 competition showcased how developers are using Model Context Protocol (MCP) to build practical agentic AI services. More than 150 teams participated, and ten finalists were selected for solutions addressing childcare, startup support, culture, gaming, legal research, and safety. Kakao plans to expand this ecosystem through the upcoming Agentic Player 10 competition and deeper integration with Kakao Tools. ## The MCP Player 10 Competition - The competition ran from December 19, 2025, to January 18, 2026, on Kakao’s PlayMCP open platform. - It emphasized: - Creativity - Everyday usefulness - Technical stability - The goal was to encourage developers to create MCP servers that solve real-world problems with AI. - Ten teams were selected after internal evaluation and received a share of 21 million won in support funding, along with opportunities to collaborate with Kakao. ## Award-Winning MCP Services ### 어린이ZIP: AI Assistant for Childcare Teachers - Automates administrative work for daycare and kindergarten teachers. - Analyzes uploaded activity photos to generate drafts of parent notices and childcare journals. - Remembers child-specific details such as allergies and pickup arrangements. - Produces personalized responses in a warm, professional tone. ### SeedUp: Startup Support-Program Research - Collects and analyzes fragmented government startup-support announcements. - Summarizes eligibility requirements and relevant opportunities. - Helps founders develop application strategies. - Supports natural-language requests such as finding weekly deadlines or analyzing an uploaded announcement. ### Other Selected Services - **공유 비밀의 방:** An anonymous platform for sharing and empathizing with personal stories and AI conversations. - **바우만 16 안티에이징솔루션:** Recommends skincare routines using the Baumann 16 skin-type classification, cosmetic ingredient data, and skin pH analysis. - **아라드도우미:** A Dungeon & Fighter assistant using RAG and Vision AI to analyze patch notes, item trends, and optimized character builds. - **키즈허브:** Aggregates public data such as emergency-room availability, childcare waiting lists, and child-development information. - **택배추적기:** Combines package tracking with AI-based detection of smishing URLs in delivery-related messages. - **ArtBridge:** Recommends performances and exhibitions from approximately 200,000 records across nine cultural categories, using location, budget, and preferences. - **KidSafe:** Detects harmful language and emotional-crisis signals in children’s chatbot conversations, escalating serious cases to guardians or professional resources. - **LexiLink_ko:** Searches and organizes statutes, court precedents, and administrative interpretations through natural-language queries. All ten MCP servers are now officially available through the PlayMCP platform. ## PlayMCP’s Future Direction - PlayMCP will remain a developer-focused environment for building and distributing MCP servers. - Kakao Tools, available through ChatGPT for Kakao, will focus on helping general users experience MCP-based services. - Kakao plans to connect the two platforms more closely. - Kakao is considering managed infrastructure, including: - Kakao Cloud-based server support - Automated deployment - Greater operational responsibility for MCP service stability - PlayMCP may also support richer in-app interfaces through JSON-based widgets, similar to those already available in ChatGPT for Kakao. ## The Next Competition: Agentic Player 10 Kakao announced a second competition, Agentic Player 10, designed to connect developer-created agents with Kakao Tools and expose them to a broader audience. The program is positioned as an opportunity for startups and aspiring founders to test their services with real users and potentially bring their agents into KakaoTalk. Developers interested in building practical AI agents are encouraged to use PlayMCP and participate in Agentic Player 10 as the next step in Kakao’s expanding agentic AI ecosystem.

Read original(opens in new tab)
kakao3 min readCurated summary

Stress Testing Know-How for Messaging Servers and How AI Lightened the Load

Kakao’s messaging platform team uses a continuously available stress-testing environment to identify scalability limits, failure points, and recovery behavior before production incidents occur. The setup mirrors production hardware, generates realistic traffic patterns with Locust, and tests both routine and extreme scenarios. The central lesson is that performance testing must examine not only application throughput, but also observability, infrastructure, framework choices, and domain-specific traffic behavior. ## Continuous Stress-Testing Environment - The environment has two main components: - Target servers using the same JVM heap, CPU, memory, and network specifications as production. - Load-generating clients built primarily with Locust, with workers scaled to hundreds of pods when necessary. - Client capacity is deliberately oversized so that the load generators do not become the bottleneck. - JMH may also be used for focused benchmarking. - Traffic scenarios are maintained according to realistic production ratios rather than simply generating large volumes of identical requests. - Typical scenarios include: - Normal midday traffic. - New Year’s midnight bursts, when message sending increases sharply. - Scenarios are built from configurable settings, allowing new traffic patterns to be created without rewriting load-generation code. ## What the Team Stress-Tests ### Observability and Logging Infrastructure - New or modified components such as Logstash, Fluent Bit, OpenTelemetry, and Vector are tested under production-like load. - The team checks: - Application throughput and elapsed time. - CPU, memory, and network overhead. - Delays in metrics collection and alerting. - Previous increases in application load caused by metric collection intervals demonstrated why monitoring infrastructure must also be performance-tested. ### Protocol and Framework Benchmarks - Server protocols and frameworks are benchmarked before changing business logic. - Tests isolate I/O behavior and compare alternatives such as WebFlux or virtual threads using real worker-count changes and system metrics. - CPU-bound work and I/O wait are increased separately to understand how each affects: - Requests per second. - Latency. - CPU utilization and other system resources. - During the C++-to-Kotlin migration, stress tests exposed system-metric differences and supported additional garbage-collection tuning. ### Operating-System and Security Changes - Host OS migrations and the addition of antivirus, monitoring, or security agents are tested under high load. - Stress tests have revealed issues such as slab-memory leaks and resource spikes caused by security software. - Components that appear harmless under normal traffic can materially affect high-throughput applications. ### Domain-Specific User Scenarios - Messaging systems have distinctive worst-case patterns, including: - Many users writing simultaneously in one chat room. - Midnight message bursts. - Entering group chats with hundreds of members. - These cases are reproduced by adjusting configurable load settings. - New features are stress-tested to locate bottlenecks before launch. ## Interpreting Test Metrics ### Endpoint-Level Metrics - **RPS:** Increase workers gradually to find saturation, or hold worker count constant to verify that throughput remains stable. - Unexpectedly low saturation points or sharply fluctuating RPS indicate a problem requiring deeper investigation. - **Latency:** P50 represents typical user experience, while P95 and P99 expose worst-case behavior. - Sudden P95/P99 increases may indicate internal capacity limits. - A degraded P50 can signal broader performance regression. - **Error rate:** Analyze 5xx errors, timeouts, and business errors separately. - 5xx responses may indicate server capacity exhaustion. - Timeouts may result from insufficient client resources. - 400-level errors can indicate broken test data or business logic. - Nonlinear changes in RPS or latency, or any unexpected errors, are signals to investigate lower-level system metrics. ## Practical Recommendation Maintain a production-like, always-available stress-testing environment with configurable realistic scenarios. Validate every major application, framework, observability, infrastructure, and feature change under both normal and worst-case traffic, then diagnose problems from endpoint metrics down through system resources.

Read original(opens in new tab)
kakao3 min readCurated summary

Bringing a Voice AI Model to Production: The Journey of Optimizing Kanana-O Serving

Kanana-O is a multimodal model that understands text, images, and audio, then responds with text and speech. Deploying it for real-time voice conversations required solving problems that do not arise during model training, including low first-response latency, concurrent users, streaming across multiple models, and uneven GPU memory demands. Kakao built the specialized Kanana-Omni Server, achieving 1.6× the throughput of vLLM-Omni at 64 concurrent users. ## Kanana-O’s Three-Stage Architecture - **Thinker** processes multimodal inputs and generates text. - **Talker** converts Thinker’s text embeddings into sequential speech tokens. - **VoiceBox** combines speech tokens into audible audio waveforms. - In production, these components must operate concurrently rather than sequentially to deliver audio within hundreds of milliseconds. ## Why a Specialized Serving Server Was Needed - Thinker passes hidden-state embeddings directly to Talker rather than ordinary token IDs. - These high-dimensional tensors must be transferred continuously, making serialization or CPU copies too expensive. - Talker produces speech tokens step by step, while VoiceBox waits for enough tokens to form larger audio chunks. - Talker also combines speaker embeddings, Thinker outputs, and its own accumulated audio embeddings, creating an input structure unlike standard autoregressive decoding. - These constraints made a custom server more suitable than general-purpose frameworks. ## Zero-Copy Data Transfer - The server preallocates shared-memory blocks during startup. - Thinker writes tensors into an available block, while Talker receives only metadata such as the block identifier and byte size. - This avoids repeated allocation, copying, and serialization. - For GPU tensors on the same node, CUDA IPC transfers data directly between GPU processes, avoiding Device→Host→Device movement. ## Cascaded Streaming Pipeline - Thinker, Talker, and VoiceBox run as overlapping asynchronous stages. - Thinker can send its first output chunk while Talker processes earlier chunks and VoiceBox synthesizes audio from still earlier ones. - Talker buffers speech tokens until VoiceBox has enough data to create an audio chunk. - This pipelining significantly reduces the time before the user hears the first response. ## Process Isolation and Fault Containment - Thinker and Talker each run their own vLLM engine in separate processes. - This avoids conflicts between CUDA contexts, model memory, KV caches, and schedulers. - Processes are started with `spawn` rather than `fork`, preventing inherited CUDA state from causing corruption. - If one component fails, such as Thinker running out of memory, the other components and the API server can continue operating and be restarted independently. ## Continuous Batching with vLLM - Manually batching requests is difficult because multimodal inputs and accumulated Talker embeddings vary in size. - The server submits requests rapidly and delegates batch construction to vLLM’s continuous-batching scheduler. - Each request runs as an independent asynchronous generation task. - vLLM combines requests internally during forward passes, while request IDs ensure each task receives only its own streamed output. - This improves GPU utilization without requiring custom synchronization and padding logic. ## Single FastAPI Worker and Asynchronous Execution - Multiple Uvicorn workers would load separate copies of the vLLM engines, multiplying GPU memory usage and model-loading costs. - Therefore, the server uses `workers=1`. - Since a blocking operation would otherwise stall every connected user, the entire request path—from the API endpoint through final audio generation—is designed around `async`/`await`. - Keeping the pipeline non-blocking allows one worker to accept and progress many concurrent requests. Kakao’s main recommendation is to design serving infrastructure around the model’s actual dataflow rather than forcing it into a generic framework. For complex multimodal pipelines, zero-copy transfers, asynchronous cascaded streaming, process isolation, and engine-level continuous batching can be more important than simply scaling API workers.

Read original(opens in new tab)
kakao3 min readCurated summary

The Calendar We Designed for KakaoTalk Booking

KakaoTalk Reservation built a time-block calendar to help sellers understand inventory and bookings more easily than with a card list. The main challenge was arranging bookings with different start times and durations so that they remain readable and use space efficiently. The solution combines sorting rules, graph-based layout calculation, DFS, and additional expansion logic for edge cases. ## Product Requirements - The calendar is designed for the seller-facing reservation management center. - Each time slot can contain up to 10 bookings. - A booking can last from one to six hours. - Booking blocks should be displayed as clearly as possible without leaving unnecessary gaps. - The input data provides booking start and end times, while the frontend must calculate the visual layout. ## Booking Placement Rules - **Sort by earliest start time** - Earlier bookings are placed first to match the seller’s natural workflow. - This also supports the typical visual scanning order from the upper-left toward the lower-right. - **For bookings starting at the same time, sort by longest duration** - Longer bookings can block shorter bookings from expanding. - Placing them first gives them enough space and allows later bookings to occupy the remaining areas more effectively. ## Graph-Based Expansion - The bookings are modeled as nodes in a graph. - Each node stores relationships with preceding and following overlapping bookings. - A depth-first search calculates: - Each node’s depth, representing its horizontal position. - The maximum distance to the final booking in its connected path. - These values are used to calculate: - `left`: the node’s horizontal starting position. - `width`: how far the booking can expand across available space. - Nodes at the far-left edge of the graph are processed first, allowing the bookings to fill the calendar while respecting overlaps. ## Handling Layout Exceptions - The initial graph and DFS calculation did not always fill all available space. - Problems occurred when: - Multiple root nodes existed. - An upper root node had a longer path than a lower root node. - Connected nodes were constrained by earlier width calculations. - The implementation searches for unused gaps between neighboring nodes. - When multiple gaps exist, connected nodes are expanded by the smallest available amount needed to close the gaps. - A gap is detected when the next node’s `left` position is greater than the current node’s `left + width`. ## Lessons from the Implementation - A calendar that appears visually simple can require substantial algorithmic design. - Frontend developers are responsible not only for rendering data, but also for deciding how that data should be presented to users. - The calendar is treated as an evolving implementation that will be refined as new bugs, data patterns, and better algorithms are discovered. The practical approach is to begin with clear sorting rules, represent overlapping bookings as a graph, use DFS to determine layout constraints, and add targeted post-processing for unused space and edge cases.

Read original(opens in new tab)
kakao3 min readCurated summary

Kanana Scala 1st Seminar On-site Sketch

Kakao’s first Kanana Scholar seminar brought together seven leading AI professors and Kakao researchers to discuss the company’s independent AI strategy. Kakao presented its from-scratch Kanana foundation models, emphasizing data efficiency, Korean-language capability, and multimodal processing. The discussion concluded that Kakao should focus less on generic benchmark scores and more on technology sovereignty, personalized agents, and practical execution in real services. ## Kanana Foundation Models - Kakao is developing its own foundation-model lineup to strengthen competitiveness and reduce dependence on overseas providers. - Kanana reportedly achieved strong performance using 11 trillion training tokens, compared with 23 trillion tokens for a similarly sized global-target model. - Kakao attributed this efficiency to the quality and refinement of its training data. - The company also demonstrated **Kanana-o**, an omni model capable of processing text, images, and audio in real time. - The model handled emotional speech and multi-speaker conversations naturally, receiving praise for its Korean fluency. ## Technology Sovereignty and Customization - Kakao argued that proprietary models protect it from external risks such as changing licensing policies and closed technologies. - Owning the technology enables Kakao to build efficient, customized models optimized for its services. - Participating professors agreed that control over Korean cultural context and local issues is essential for technological sovereignty. - They viewed an independent model as a strategic asset for long-term service stability. ## Digital World Models and Personalized Agents - Kakao aims to understand users’ behavioral context within KakaoTalk and provide highly personalized assistance. - On-device AI could protect private conversations while allowing agents to respond immediately to user needs. - The professors suggested expanding the idea of “physical AI” into a **digital world model** that predicts interactions and causal relationships across a platform. - This direction could create an area of AI differentiation uniquely suited to Kakao’s ecosystem. ## Evaluating Practical Agentic Intelligence - Kakao is prioritizing AI systems that can create multi-step plans, call necessary tools, and complete tasks independently. - It plans to use an internally developed orchestration benchmark to evaluate real-world problem-solving ability. - The professors cited Claude as an example of how users perceive intelligence through successful completion of complex requests, not merely high benchmark scores. - They recommended competing through practical execution in real service environments rather than focusing only on text-generation performance. ## Industry-Academic Cooperation - Kakao plans to explore GPU support for university research labs and undergraduate AI clubs. - Possible support could include credits, project-based resources, and other forms of infrastructure assistance. - The seminar marked the beginning of a broader collaboration aimed at advancing Korea’s AI ecosystem and developing future talent. Kakao’s recommended path is to combine proprietary, efficient models with privacy-preserving personalization and strong agentic execution. Success will depend on how effectively Kanana turns technical depth into useful intelligence that users can experience in everyday services.

Read original(opens in new tab)
kakao4 min readCurated summary

Finding Real Threats Among Hundreds of Millions of Security Signals — Transforming the Security Monitoring Paradigm with AI

Kakao argues that monitoring hundreds of millions of daily security events cannot scale through human analysts and increasingly complex rules alone. Its solution is a hybrid AI pipeline that filters noise early, analyzes only high-value events with multiple models, and continuously improves through verified feedback. The goal is not to generate more alerts, but to understand context and identify threats worth investigating. ## The Scale Problem: Finding Threats in a Haystack - Endpoint activity such as process execution, network connections, file changes, and privilege escalation produces hundreds of millions of events. - The volume grows rapidly as services expand, while the proportion of genuine attacks remains very small. - Increasing the number of analysts alongside event volume is economically and operationally unsustainable. - AI is needed to correlate events, interpret behavior statistically and contextually, and dynamically distinguish normal activity from anomalies. ## Limitations of Rule-Based Monitoring - Rules can identify what happened, but not why, who initiated it, or whether it fits the environment. - Legitimate deployment commands can resemble backdoor installation, causing high false-positive rates. - Analysis quality varies by analyst experience, shift, and time of day. - Analysts must manually assemble host information, network sessions, process histories, and related logs into an incident narrative. - Expanding detection categories—behavior sequences, statistical anomalies, multi-source correlations, and rare events—makes manual rule maintenance impractical. - SIEM correlation improves on single-event rules but remains limited to predefined scenarios and struggles with unknown attack patterns. - As rule sets and event volumes grow, both maintenance costs and matching performance become problematic. ## A Funnel-Based Hybrid Architecture - Kakao filters events through multiple stages before using AI: - Rule-based filters remove obvious noise. - Learned normal patterns are automatically excluded. - AI performs detailed analysis only on the small remainder requiring judgment. - Rules handle clear, deterministic patterns quickly, while AI evaluates complex contextual situations. - The framework is designed to accommodate new threat types and detection categories without creating a separate system for each scenario. ## Multi-Model Verification and Operational Resilience - Multiple AI models independently analyze the same event and cross-check one another. - Disagreement is treated as an uncertainty signal that can trigger deeper analyst review. - Model diversity helps reduce bias, false positives, and missed detections. - It also provides resilience against model failures, API outages, and quality changes after model updates. - The design balances cost, processing speed, and accuracy rather than optimizing only for detection precision. ## Teaching AI the Environment’s Context - Generic LLMs initially misclassified legitimate activity because they lacked knowledge of Kakao’s infrastructure. - The system supplies structured context, including: - Host roles - Services running on each host - Accounts used for automation - Normal communication and operational patterns - This context allows the model to act more like an analyst familiar with the organization than a generic security classifier. ## Analyzing Complete Behavior Flows - Individual commands such as `curl`, `chmod`, and script execution can occur in both normal deployments and attacks. - Kakao therefore reconstructs activity at the host level, linking: - Process execution history - Network sessions - File changes - Temporal ordering - The same command can have different meanings depending on when, where, and in what sequence it occurred. - AI evaluates the complete sequence to distinguish routine operations from intrusion behavior. ## Translating Events into AI-Usable Data - Sending raw events directly to an LLM wastes tokens on irrelevant information and reduces accuracy. - Different detection tasks require different signals; statistical anomaly detection and sequence analysis cannot rely on one fixed format. - Kakao introduced: - A standardized event schema - Dynamic feature construction tailored to each detection type - This reduces token usage while improving the relevance and precision of AI analysis. ## WALT: A Self-Learning Detection Loop - Initially, analysts had to manually convert AI conclusions into new detection policies. - Kakao developed WALT, or **Whitelist-Assisted Learning and Tuning**, to automate this feedback process. - Repeatedly verified normal patterns are converted into exception policies. - Those policies filter future matching events before they reach the AI engine. - Thousands of detection policies are reportedly being generated and operated this way, allowing accuracy to improve over time. ## Cost and Performance Constraints - Sending every event to an AI model caused unsustainable costs and processing delays. - The funnel architecture addresses this by reserving expensive AI analysis for events that survive earlier filtering. - The overall system must continuously balance economic cost, response speed, detection accuracy, and reliability. Kakao’s practical recommendation is to treat AI as part of a carefully designed security pipeline—not as a replacement for rules or analysts. Effective large-scale monitoring combines deterministic filtering, contextual multi-model analysis, structured data, and a controlled feedback loop that learns from verified outcomes.

Read original(opens in new tab)
kakao4 min readCurated summary

From Student to Developer: Learning Rational Choices Over Right Answers—From DB and Security to AI

The onboarding of 40 new Kakao developers shifted their perspective from making features work to designing systems that survive real-world operations. Across databases, security, and AI, they learned that there is rarely one perfect answer; the best choice depends on scale, risk, maintainability, and business needs. The central lesson was to replace theoretical correctness with responsible, adaptable engineering judgment. ## Database: From Finding the Right Answer to Preparing for Change - Database design must be evaluated by whether it can withstand traffic, schema changes, and operational demands—not only by theoretical correctness. - Foreign keys are not automatically the best choice: - They can introduce locking, performance, and flexibility concerns. - Referential integrity can instead be managed at the application layer, provided testing and correction processes are strong. - Soft deletion, using fields such as `deleted_at`, supports auditability and recovery and is often an essential operational strategy. - Indexes should be selected according to the questions the database must answer: - B-tree, GIN, GiST, SP-GiST, and vector indexes serve different data and query patterns. - Execution plans reveal whether SQL uses indexes or performs full table scans, directly affecting I/O and response times. - Duplication is not always harmful: - Intentional denormalization can avoid expensive joins. - Snapshot data can simplify reads and preserve the information needed by a business workflow. - In MongoDB, embedding selected related data can make screen queries much simpler than relying exclusively on references. - Different database systems embody different trade-offs among performance, consistency, scalability, and operational cost. - The training covered MySQL high availability, PostgreSQL primary-key structures, cloud-native systems such as Neon, and the broader storage-to-analysis pipeline of Hadoop and Spark. - The resulting mindset favors designs that are safe to change and affordable to operate over designs that are theoretically perfect. ## Security and IT: From Someone Else’s Responsibility to a Personal Default - Security became a direct consequence of developers’ code rather than merely a compliance or infrastructure concern. - Everyday safeguards such as development/production separation, VPNs, and antivirus software demonstrate that safety often requires accepting some inconvenience. - DDoS defense is not only about blocking traffic: - It can be difficult to distinguish an attack from legitimate traffic spikes caused by a popular event. - Developers should apply basic controls such as rate limiting and escalate suspicious activity through established response channels. - Hands-on API exploitation made vulnerabilities concrete and encouraged developers to view security through an attacker’s perspective. - Security must be continuous: - AI is increasingly being used both to discover vulnerabilities and to strengthen attacks. - Social-engineering methods involving QR codes, app permissions, and human behavior require more than purely technical defenses. - Security checks should be integrated from the beginning of development, not performed only at the end. - Software quality also depends on people: - Code should remain understandable enough for another developer to take over quickly. - Strong engineering means choosing and communicating the most appropriate solution for the business context, not merely finding a technically possible one. ## AI: From Chatting with Models to Designing Systems - An AI agent is not simply a model; it is an architecture composed of tools, routing logic, error handling, and model calls. - Agent development applies familiar software-engineering practices to probabilistic models. - Because LLM outputs can vary, reliable systems need deliberate controls: - Prompt chaining breaks large tasks into smaller steps and limits context contamination. - Few-shot examples clarify required output formats. - Routing selects different prompts or workflows based on conditions. - Multi-agent systems divide responsibilities among specialized agents, echoing the modularity and scalability principles of microservices. - RAG reduces hallucinations structurally by: - Chunking documents. - Searching for semantically similar vectors. - Supplying retrieved information to the model as additional context. - MCP exposes internal systems and data as callable tools, effectively enabling remote function calling and connecting AI to enterprise capabilities. - Effective AI use shifted from criticizing poor answers to specifying clear objectives, formats, examples, context, and supporting data. - The goal is not merely to receive an intelligent response, but to design a system that consistently produces intelligent behavior. The training ultimately marked a transition from student-style problem solving to professional engineering. Developers should consider operational resilience, security, maintainability, and business value, then make and clearly explain the most reasonable choice for the circumstances.

Read original(opens in new tab)
kakao4 min readCurated summary

From Student to Developer: Learning Server Flow from Lotto Implementation to Legacy Improvement

The post describes Kakao’s 2026 server-engineering onboarding program, which turns uncertainty into practical understanding through structured implementation, testing, and refactoring. Rather than supplying fixed answers, the program repeatedly asks developers to explain their design decisions and assess what their tests protect. Its central lesson is that server development becomes manageable when engineers build clear reasoning, maintainable structures, and safe change processes. ## Onboarding Through Three Stages - The program follows a progression: 1. TDD- and OOP-based implementation 2. Acceptance testing for legacy code 3. Refactoring legacy code - The focus is not only on what to build, but on how to make engineering decisions. - Core goals include: - Designing maintainable structures - Analyzing and safely improving legacy systems - Collaborating effectively, including responsible AI usage - Although originally designed for server developers, the program expanded to frontend, Android, and iOS engineers because engineering principles apply across technology stacks. ## Learning Through Questions and Collaboration - Participants were repeatedly asked: - Why was this design chosen? - Does this object truly own this responsibility? - What behavior does this test protect? - Daily meetings, pair programming, troubleshooting discussions, and PR reviews made development a collaborative activity. - The program aimed to develop engineers who could explain and defend their designs, rather than merely produce working code. ## Mission 1: Building a Lottery Game with TDD and OOP - The first assignment implemented: - Automatic and manual lottery purchases - A fixed ticket price of 1,000 won - Winning-statistics calculations - Constraints encouraged better design: - One level of indentation - Methods limited to 10 lines - Primitive values wrapped in value objects - First-class collections - Avoiding `else` through early returns - TDD required tests to be written before implementation. ### Making Randomness Testable - Random lottery-number generation initially made tests unpredictable and tightly coupled to concrete implementations. - The solution was to: - Introduce a number-generation interface - Inject the generation strategy - Create a separate test generator - This made test results controllable and encouraged a more flexible design. ### Considering Value Objects and Caching - The team also questioned whether identical number values should always create new objects. - This led to discussions about caching and the difference between object identity and value equality. - The main lesson was to evaluate design decisions, not just make the feature work. ## Mission 2: Writing Acceptance Tests for Legacy Code - Participants first protected the existing system before modifying it. - Tests focused on externally observable behavior: - User actions - System responses - State changes - Strong assertions verified not merely that an operation succeeded, but that it produced the correct result. - Cucumber-based BDD expressed scenarios in a form understandable to non-developers, treating tests as shared specifications. ### Achieving Production Parity - To avoid “works on my machine” problems, the test environment was aligned with production: - PostgreSQL replaced H2 - Docker standardized execution environments - Gradle tasks automated test execution - Test-data isolation used: - Reverse-order foreign-key deletion - `TRUNCATE ... CASCADE` - Shared cleanup utilities - These measures ensured tests started from consistent, independent states. ## Mission 3: Refactoring Legacy Code Safely - The final mission treated refactoring as training in decision-making, not simply an exercise in clean code. - The central rule was to separate structural and behavioral changes: - Structural changes must preserve behavior. - Behavior changes must avoid unrelated structural modifications. - PR reviews helped identify unintended behavior changes and taught participants to predict and control the effects of modifications. - AI was used during refactoring to accelerate broad code changes, but large changes were difficult to verify, highlighting the need to control scope and validate changes carefully. The onboarding’s practical recommendation is to approach server development through small, explainable decisions: write controllable tests, protect legacy behavior before changing it, separate refactoring from feature changes, and use AI as an assistant rather than a substitute for engineering judgment.

Read original(opens in new tab)