LY Corporation runs the “Orchestration Development Workshop” to help engineers apply AI more effectively in real development work. The workshop focuses on connecting multiple AI tools and agents to amplify creativity through collaborative learning and creation. A related blog series will share workshop topics, beginning with an example of using AI to reduce PR review delays and improve review culture.
## Orchestration Development Workshop
- Targets engineers involved in development at LY Corporation.
- Emphasizes practical, workplace-oriented AI skills rather than purely theoretical knowledge.
- Uses “orchestrating multiple AIs to maximize creativity” as its central theme.
- Builds on the idea that organizational learning is essential for successfully adopting AI.
## Blog Series
- The series will publish workshop content incrementally.
- The article list will be updated as new posts are released.
- The listed update date is April 10, 2026.
## AI-Assisted PR Reviews
- The first installment addresses bottlenecks and delays in pull-request reviews.
- It explains how AI-assisted PR review support can help resolve review stagnation.
- It also describes an internal workshop designed to change team review practices and culture.
The series presents AI adoption as an organizational and collaborative practice, with PR review automation serving as an initial example of measurable workflow improvement.
LY Corporation is moving from simply adopting AI tools to building with AI as a collaborative development partner. Its new Orchestration Development Workshop teaches engineers to coordinate multiple AI systems across coding, testing, reviews, incident analysis, and other workflows. The initiative aims not only to improve efficiency but to free engineers from repetitive work so they can focus on more creative, high-value challenges.
## From AI Adoption to AI Collaboration
- AI-assisted development and operations are spreading rapidly across LY Corporation.
- Engineers use generative AI for code generation and testing, while combining it with non-generative AI for analysis and operational optimization.
- Despite broader adoption, employees differ significantly in how deeply they use AI in their daily work.
- The workshop was created to help the organization evolve from “using AI” to “creating alongside AI.”
## Orchestration: Coordinating Multiple AI Systems
- “Orchestration” refers to combining multiple AIs, along with human input, to produce a complete outcome.
- Example workflows include:
- Generating code automatically from a Jira ticket.
- Having AI run tests, conduct reviews, and create a pull request.
- Analyzing a Slack incident report, estimating the cause, and proposing a fix.
- The workshop turns these emerging practices into hands-on learning rather than passive demonstrations.
## A Hands-On, Interactive Learning Model
- Participants follow instructors in real time and perform the same tasks themselves.
- Zoom conversations and Slack questions create two-way communication during the session.
- Instructors and representative participants explore solutions to problems as they arise.
- The goal is for attendees to gain skills they can reproduce in their own projects, not merely acquire theoretical knowledge.
## Organization-Wide Support Through Guilds and DevRel
- The initiative is designed to avoid depending on individual enthusiasm.
- Three complementary functions support continuous growth:
- **DevRel:** Drives the program and promotes adoption.
- **Guilds:** Contribute practical insights from engineering teams.
- **TD:** Helps maintain quality and reproducibility.
- This structure supports consistent content quality and enables AI knowledge to spread across the company.
## Beyond Efficiency: Unlocking Engineering Creativity
- LY Corporation views AI as more than a way to complete tasks faster.
- By delegating repetitive work to AI, engineers can spend more time on creative and strategically valuable activities.
- The organization aims to move beyond a model where AI writes code and humans only review it.
- Instead, engineers should collaborate with AI from the design stage through implementation.
## Future Direction
- LY Corporation plans to share lessons from the workshops through external channels such as its technology blog.
- Future topics will include both generative and non-generative AI.
- The broader goal is to provide practical guidance for engineers building new workflows with AI.
The workshop represents a structured way to turn AI experimentation into repeatable organizational practice, helping engineers coordinate multiple AI tools while preserving human creativity and judgment.
LINE Plus replaced a full-dump ETL pipeline for product data with incremental processing using Apache Iceberg and Apache Flink. The previous HBase/Hive workflow rewrote hundreds of millions of rows for every update, causing high compute costs and delays that left data up to an hour out of date. With the new architecture, update intervals were reduced from 60 minutes to 5 minutes—roughly a 12× improvement—while preserving consistency and fault tolerance.
## Limitations of Full-Data ETL
- The existing HBase and Hive pipeline continuously collected CDC data in HDFS but had to merge it with existing data and rewrite the entire table before changes became queryable.
- This caused:
- High compute and storage costs
- Dependence on limited shared Hadoop resources
- Delayed updates and stale data
- Snapshot-based extraction provides consistency, but large snapshots can take hours and retain old versions through MVCC, increasing system overhead.
- Processing only the changed rows would reduce the workload from hundreds of millions of records to tens of thousands, separating update cost from total dataset size.
## Introducing Apache Iceberg
- Iceberg manages data through metadata and table snapshots rather than relying solely on directory structures like traditional Hive tables.
- It supports row-level `upsert` and `delete` operations.
- This allows incremental changes to be written without rewriting the entire table, making much shorter ETL intervals possible.
## Requirements for the Streaming Pipeline
The team evaluated Spark and Flink against three essential requirements:
- **Data freshness:** Late-arriving compensation or replay data must not overwrite newer records.
- **End-to-end exactly-once processing:** Iceberg updates and Kafka status messages must not partially succeed.
- **Fault tolerance and state management:** Processing state must survive failures and restarts.
A Kafka message indicating that all CDC data through a specific timestamp—such as 13:03—has been applied serves as the signal that a bulk extraction can safely begin. This requires complete confidence that the message accurately represents the Iceberg table’s committed state.
## Why Two-Phase Commit Was Necessary
- Iceberg and Kafka are independent systems, so writing to one while failing to write to the other could create inconsistent state.
- Two-phase commit (2PC) prevents partial success:
- Both systems prepare their writes.
- They commit only when all required operations succeed.
- Any failure causes the operation to roll back.
- Exactly-once processing also prevents duplicate or missing records during retries, network failures, or node restarts.
- Together, these guarantees make Kafka status messages a reliable representation of the Iceberg table’s state.
## Choosing Flink over Spark
- Spark Structured Streaming uses a micro-batch model, which makes fine-grained event-time and state control more difficult.
- Flink provides native event-by-event streaming and better support for the required consistency model.
- The team used Flink state to track each record’s `updatedate`:
- Older late-arriving events are ignored.
- Replayed historical data cannot overwrite newer values.
- Flink checkpoints:
- Persist streaming state externally.
- Enable recovery from the latest consistent point.
- Integrate with the Kafka sink’s 2PC mechanism.
- Kafka messages remain in a pre-commit state until the Iceberg write and checkpoint both succeed.
## Kubernetes Deployment Options
- The team compared:
- **Native Kubernetes:** Requires manually configuring roles, service accounts, services, routing, deployments, slots, and jobs.
- **Flink Kubernetes Operator:** Represents Flink infrastructure and jobs as custom resources, automating configuration such as routing and the web UI through Helm values.
- Although Flink has greater operational complexity and a steeper learning curve than Spark, it was selected because it was the only option that satisfied all three core requirements at the engine level.
The recommended architecture is an incremental Iceberg pipeline powered by Flink, with stateful processing, checkpoints, and two-phase commit between Iceberg and Kafka. This approach keeps data current, avoids expensive full-table rewrites, and provides reliable recovery and consistency at a five-minute update interval.
MessagingHub turns chat into a reusable platform rather than rebuilding it for each product domain. It separates domain-specific authentication and business context from common chat capabilities, allowing chatbot, customer-support, direct, and group conversations to share the same infrastructure. Its policy-driven design, modular architecture, and configurable metadata aim to reduce integration complexity while preserving flexibility.
## Why MessagingHub Was Introduced
- Chat requirements vary across chatbots, customer support, one-to-one conversations, and group chats.
- Building each implementation independently increases integration points, system complexity, development cost, and the impact of small changes.
- MessagingHub is designed as a domain-independent platform that can be adopted by multiple services.
- The platform focuses on chat itself while absorbing external requirements through generalized, reusable structures.
- It is currently used by a Japanese food-delivery service for users, drivers, customer-service agents, and restaurants.
## Supported Chat Types
- **Chatbots:** Delivered through a public web URL embedded in a partner service’s webview. Scenarios are created and deployed through an administrative console.
- **Inquiry chat:** A user is matched with a customer-service agent. The partner domain supplies contextual information such as user details and previous consultation history.
- The platform is also structured to support direct one-to-one and group conversations.
## Core Platform Policies
### Authentication and User Identification
- MessagingHub does not manage user accounts or domain authentication.
- Partner systems handle login, registration, permissions, and the decision of whether a user may access chat.
- After authenticating a user, the partner requests a connection token and passes it to the client.
- The client uses the token to establish a WebSocket connection; unauthenticated direct access is not allowed.
- A user is identified by a `client_id`, combining the partner domain identifier with the partner’s user identifier.
- Display names, profile images, and `pushToken` values are supplied and updated by the partner system.
### Service Contexts and Room Types
- A **service context** defines which roles may communicate, such as:
- `Driver2CS`
- `Consumer2CS`
- A **chat room type** defines the conversation structure, such as:
- `USER_DIRECT`
- `USER_GROUP`
- `INQUIRY_CHATBOT`
- `INQUIRY_CHAT`
- The combination of service context and room type controls room creation, participation, and message permissions.
### Room Lifecycle and Data Retention
- General room states progress from `WAIT` or `PENDING`, to `SERVICE`, and eventually to `DISABLE` or `BLOCK`, where sending messages is prohibited.
- Messages and potentially identifying data are encrypted at rest.
- Data can be deleted immediately when all participants leave a room.
- Partners can also configure retention periods for automatic deletion of older data.
## Modular Architecture
MessagingHub is not a monolithic chat server. Its components have clearly separated responsibilities and communicate through loosely coupled events.
- **`connection-manager`**
- Manages WebSocket connections and validates connection tokens.
- Tracks user connection status.
- Helps identify active chatbot scenario connections during `SOFT STOP` processing.
- **`chat-app`**
- Implements core chat logic, including message delivery, room creation, state transitions, and read status.
- Exposes functionality as commands that can be combined for different chat types.
- **`message-router`**
- Determines where recipients are connected.
- Routes messages from the chat server to the appropriate connection-management component.
- **`notification-app`**
- Sends push notifications when recipients are offline or the application is in the background.
- Uses partner-provided `pushToken` values and room-level notification settings.
- **`admin-hub`**
- Manages chatbot scenario editing and deployment.
- Handles agent accounts, roles, service contexts, events, webhooks, monitoring, and statistics.
## Command-Based Chat Flows
- Chat behavior is modeled as composable commands.
- Common commands provide functionality shared across chat types.
- Chatbot and inquiry-chat features add more specialized commands.
- This “building block” approach allows business requirements to be assembled without creating a separate chat implementation for every domain.
## Data Model
MessagingHub separates operational data from core chat data:
- **`chat` database:** Stores users, rooms, participants, metadata, and messages.
- **`chat_operation` database:** Stores operational and administrative information.
Important entities include:
- `chat_user`: Uniquely identifies users by `client_id`.
- `chat_room`: Represents rooms and enforces room uniqueness at the schema level.
- `chat_member`: Connects users to rooms.
- `chat_room_meta`: Stores participant-specific state, including read position, push settings, input restrictions, and room status.
- `chat_log`: Stores encrypted messages in a one-to-many relationship with rooms.
- `prev_chat_log_id` preserves message ordering.
- Room-level first and last message IDs, together with participant read positions, support unread-count calculation.
- Partner metadata such as `system_data`, `search_data`, `user_details`, and `descriptions` is stored as JSON. MessagingHub preserves and forwards it without interpreting its domain meaning.
- Scheduling, event, and webhook history are tracked through tables such as `chat_schedule`, `chat_event_record`, and `webhook_event_record`.
- `service_context`, `chat_event`, and `webhook` configure allowed role relationships, event-message policies, and webhook behavior.
## Chatbot Scenario Management
### Flexible Scenario Structure
- Administrators manage multiple chatbot scenarios through an editing tool.
- Scenarios define messages, selectable options, and answers.
- Webhooks can dynamically generate response content.
- The hierarchical data model supports a broad range of chatbot flows.
### Version Deployment and `SOFT STOP`
Chatbot scenarios transition through:
`WAIT → SERVICE → SOFT STOP → DISABLE`
- A newly deployed scenario becomes `SERVICE`.
- The previous scenario moves to `SOFT STOP`.
- Existing users can finish conversations using the previous version.
- New users are directed to the latest scenario.
- A scheduler periodically checks whether any users still have active connections to the old scenario.
- Connection information is collected from connection-management servers and stored in a shared resource.
- Once no active users remain, the old scenario is disabled and the scheduler stops.
- This provides backward compatibility without disrupting users during deployment.
## Inquiry Chat Metadata and Lifecycle
### Partner-Defined Metadata
Inquiry chat allows partner domains to provide information that helps agents handle cases effectively:
- Search data for finding conversations
- User details shown to agents
- Custom display data
- Event data for surveys or webhooks
- Basic consultation descriptions
- Room settings such as room names and push-notification titles
- Tracking data for identifying and mapping rooms in partner systems
### Room Lifecycle
Inquiry rooms generally move through:
`PENDING → SERVICE → DISABLE → BLOCK`
- `PENDING` represents the period while the user waits for an agent match.
- `SERVICE` is the active consultation period.
- `DISABLE` indicates that the consultation has ended.
- `BLOCK` prevents further messaging after closure.
MessagingHub’s overall approach is to keep the platform’s responsibilities narrow and reusable while allowing partner domains to own authentication, user meaning, and business-specific metadata. For organizations supporting multiple chat scenarios, a policy-driven, command-based platform with separated components and explicit data ownership can significantly reduce duplication and integration risk.
LINE Ads processes tens of billions of advertising events daily and nearly one hundred billion internal data records. As growing numbers of features increased computational demands, its Spark-on-YARN environment suffered from resource contention, inefficient scaling, and Hadoop dependencies. The team migrated to Spark on Kubernetes to achieve infrastructure independence, containerized execution, flexible scaling, and easier operational automation.
## Large-Scale LINE Ads Data Pipelines
- The data pipeline supports:
- Real-time advertising-event processing
- Abuse and validity checks
- Machine-learning systems and model training
- Analytics and system integration
- Advertiser reporting
- The platform must handle hundreds of billions of events per day and hundreds of thousands per second.
- It must provide low latency, elastic capacity, minimal service impact during failures, and rapid recovery.
- The most heavily used table grew to approximately 2.91 times its December 2022 size by December 2025 as more features were added.
## Limitations of Spark on YARN
- Hadoop’s storage and compute resources were colocated, causing Spark workloads to compete with HDFS and other Hadoop components.
- Scaling compute required adding Hadoop nodes, even when additional storage was unnecessary, increasing cost and wasting capacity.
- JVM and Spark versions were difficult to manage independently, limiting access to newer Spark features.
- Applications became tightly coupled to the Hadoop infrastructure.
## How Spark on Kubernetes Works
- Kubernetes replaces YARN as the cluster manager.
- Spark drivers and executors run as separate Kubernetes pods.
- In cluster mode:
- `spark-submit` requests a driver pod.
- Kubernetes schedules the driver on an appropriate node.
- The driver creates a `SparkContext`, builds the DAG, and requests executors.
- Executors run as independent pods with individually allocated CPU and memory.
- The driver divides the DAG into stages and distributes tasks to executors.
- Shuffle data is normally tied to executor-pod lifecycles unless an external shuffle service is configured.
## Advantages over YARN
- **Containerized execution:** Docker images package application dependencies, improving reproducibility and CI/CD integration.
- **Infrastructure independence:** Spark can use HDFS, S3, GCS, or other storage systems without requiring a Hadoop cluster.
- **Simpler autoscaling:** Kubernetes can scale pods and integrate with cloud VM autoscalers.
- **Unified platform:** Spark, Airflow, machine-learning workloads, and API servers can share a Kubernetes cluster.
- **Governance and isolation:** Namespaces, `ResourceQuota`, and RBAC provide flexible team-level controls.
- **Operational automation:** Helm, ArgoCD, GitOps, and rolling updates enable more automated application management.
## LINE Ads’ Kubernetes-Based System
The platform is organized into four layers:
- **Deployment layer**
- GitHub Actions runs CI workflows based on repository events.
- ArgoCD monitors desired and deployed states and supports easier rollback and synchronization.
- **Compute layer**
- Kubeflow’s Spark Operator deploys applications through the `SparkApplication` Kubernetes custom resource.
- Apache YuniKorn schedules batch jobs and supports resource coordination and gang scheduling.
- LogSender forwards pod logs to OpenSearch.
- ClusterMonitoring sends Prometheus metrics to the company’s monitoring system.
- **Storage layer**
- Kafka provides high-throughput, low-latency storage for real-time advertising actions.
- Hadoop remains available for large-scale, long-term analysis.
- **Monitoring layer**
- Kubernetes workers and Spark applications are monitored through exposed Prometheus metrics and centralized logging.
The migration to Spark on Kubernetes is recommended for organizations whose Spark workloads are outgrowing tightly coupled Hadoop environments. It separates compute from storage, improves deployment flexibility, and allows data applications to be managed as cloud-native workloads.
Image content moderation has evolved from simple rule-based filtering into an AI-powered decision system capable of handling visual context, text, and policy complexity. At large platforms, the challenge is not only accuracy but also latency, cost, scalability, and adaptability to changing policies. LY Corporation addresses these demands through optimized traditional ML models, a hybrid ML–multimodal LLM pipeline, and modular decision-making that combines OCR, visual analysis, and contextual reasoning.
## The Evolution of Content Moderation
- Early systems relied on keyword matching, rule-based filters, and predefined patterns.
- Machine learning enabled broader pattern recognition and detection of modified or less explicit violations.
- Modern systems combine:
- Deep learning for text and image classification
- Multimodal models for joint image–text understanding
- LLMs for context-sensitive judgments
- Separate prediction and policy layers for operational flexibility
- Despite these advances, image moderation remains difficult because images lack explicit structure and their meaning often depends on context.
## Why Image Moderation Is Difficult
- **Visual complexity:** Backgrounds, objects, people, colors, and composition interact in ways that simple object detection cannot fully interpret.
- **Context dependency:** Symbols, gestures, and imagery may have different meanings across cultures; embedded text can also determine whether an image is harmful.
- **Evasion and variation:** Memes, composites, partially obscured images, and AI-generated edits continually challenge existing detectors.
- **Scale requirements:** Platforms may receive millions or tens of millions of images daily, requiring high accuracy alongside low latency, reliability, and cost efficiency.
## LY Corporation’s Moderation API
- LY Corporation operates a monitoring platform designed to process large-scale traffic and enforce diverse content policies.
- Its image moderation API detects:
- Adult content
- Violent or graphic scenes
- Offensive or disturbing imagery
- Identity documents containing personal information
- Social media screenshots and other policy-sensitive images
- The system is designed to apply service-specific policies consistently while maintaining high throughput.
## Improving Accuracy, Speed, and Cost
### Traditional ML Model Optimization
- A PyTorch-based image classification model was selected with latency, cost, and throughput in mind.
- The model was converted to ONNX and optimized with FP16 precision.
- ONNX Runtime improved execution efficiency, while FP16 reduced memory usage and inference time.
- These changes increased throughput by up to **4.3 times**.
### Hybrid ML and Multimodal LLM Architecture
- The traditional classifier acts as a fast first-stage filter.
- Clear cases are resolved immediately by the image model.
- Ambiguous cases are sent to a multimodal LLM for deeper analysis.
- More than 90% of production data could be classified by the traditional model alone.
- Since multimodal LLM throughput was over 100 times lower than that of the traditional model, routing every image to the LLM would have significantly increased GPU usage and cost.
- The hybrid approach preserves high-quality reasoning where necessary while avoiding unnecessary LLM calls.
### vLLM-Based LLM Optimization
The team optimized multimodal LLM serving with vLLM, using characteristics such as repeated prompts, predictable token lengths, and prefill-heavy workloads.
- **`enable_prefix_caching`:** Reuses KV-cache blocks for repeated system prompts and templates, reducing prefill computation.
- **`max_model_len`:** Limits the maximum input-plus-output length to avoid excessive KV-cache allocation.
- **`max_num_seqs`:** Controls concurrent requests, balancing throughput against per-request latency and resource contention.
- **`max_num_batched_tokens`:** Sets the token budget per scheduling step; larger values can improve throughput for prefill-heavy workloads.
- Regularly updating vLLM is recommended because new releases add improvements such as asynchronous scheduling, CUDA graph support, and broader quantization options.
## Moving Beyond Single-Model Policy Prediction
- Earlier end-to-end vision models directly predicted final policy categories from images.
- This worked for visually obvious violations, such as detecting smoking, but struggled with complex behaviors such as tobacco sales.
- Sales-related judgments may require combining:
- Product presence
- Prices
- Sales language
- Contact information
- Encouragement to purchase
- Directly learning every combination of national regulations, service policies, and exceptions created overly complex output classes.
- It also made the model harder to extend and maintain, while limiting the use of text embedded in images.
## Hybrid Decision-Making with OCR and Multimodal Reasoning
- The redesigned system separates visual and textual information rather than forcing one model to learn every policy combination.
- OCR extracts text from images when relevant.
- Extracted text helps identify policy-violating behavior or intent.
- Visual signals and textual evidence are then combined with a multimodal LLM.
- This allows the system to reason about context and intent beyond simple object detection, while making policy logic more modular and adaptable.
The practical recommendation is to avoid routing all traffic through expensive general-purpose models. Use fast specialized models for clear cases, reserve multimodal LLMs for ambiguity, optimize serving according to workload characteristics, and separate content understanding from policy decisions so the system can evolve as requirements change.
LLM guardrails must detect prompt injection and jailbreak attempts without blocking legitimate requests that merely contain security-related keywords. The post argues that benchmark scores alone do not reflect production performance, especially false positives caused by missing input diversity. It presents a Codex-based, automated testing pipeline that generates categorized test data, evaluates the guardrail model, and analyzes failures reproducibly.
## The Gap Between Benchmark and Production Performance
- The initial guardrail model performed well on external benchmarks but produced unexpected false positives in production-like tests.
- Legitimate requests containing terms such as “ignore,” “bypass,” “override,” “system prompt,” or “jailbreak” were sometimes classified as attacks.
- Examples included:
- Development questions about temporarily bypassing authentication in a local test environment.
- Educational requests about jailbreak techniques and defensive guidelines.
- The core issue was insufficient representation of real-world input diversity, not simply poor model quality.
- This motivated an automated environment for repeatedly discovering and analyzing guardrail weaknesses.
## Using Codex as a Test Automation Tool
- The team adapted coding agents from software development tasks to complex, repeatable security testing.
- Codex was used through its CLI capabilities to:
- Read and create project files.
- Edit code.
- Execute evaluation scripts.
- The pipeline relies on three Codex concepts:
- **AGENTS.md:** Defines global rules, project conventions, commands, and security constraints.
- **Sub-agents:** Allow a main orchestrator to delegate independent category tests to parallel worker agents.
- **Skills:** Package repeatable procedures, input/output specifications, prompts, and scripts into reusable modules.
## Category-Based Experiments
- Instead of sending thousands of random samples, experiments are divided into vulnerability and false-positive categories.
- Example categories include:
- Normal development or IT requests containing security-related keywords.
- Educational or preventive requests involving sensitive topics such as jailbreaks or drug abuse prevention.
- Categorization improves:
- Root-cause analysis.
- Parallel execution through independent workers.
- Context clarity.
- Regression testing after model changes.
## Separate Generation and Evaluation Skills
### `synthetic-generator`
- Creates test queries according to each category’s specification.
- Enforces constraints such as:
- Attack type.
- Sentence length.
- Safe or dangerous target labels.
- Produces varied, realistic phrasing and stores the dataset as JSONL.
### `injection-classifier`
- Sends generated inputs to the guardrail model API through Python scripts.
- Compares predictions with ground-truth labels.
- Calculates false-positive and false-negative statistics.
- Stores the original text, labels, predictions, and metrics in a consolidated JSONL file.
Separating these procedures into skills provides intermediate artifacts for debugging, fixed input/output contracts for reproducibility, and independent maintenance of generation and evaluation logic.
## Pipeline Architecture
- A **main agent**:
- Reads `AGENTS.md` and `TEST_CATEGORY.md`.
- Determines categories, sample counts, and constraints.
- Creates and assigns work to category-specific workers.
- Collects completion reports and verifies the run.
- Each **category worker**:
- Generates `input.jsonl` using `synthetic-generator`.
- Evaluates the guardrail model using `injection-classifier`.
- Produces `result.jsonl` with predictions and metrics.
- Analyzes false positives and false negatives.
- Writes a Markdown analysis report.
- Stores outputs under `outputs/<run_id>/`, organized by category.
## Results and Practical Recommendation
The pipeline enables systematic, repeatable testing rather than isolated discovery of misclassifications. For production guardrails, teams should combine benchmark evaluation with categorized real-world simulations, modular generation and evaluation steps, parallel test agents, and preserved JSONL artifacts for debugging and regression analysis.
LINE Home DevOps created an SRE bot to reduce the repetitive work caused by growing services, Flava cloud migration, and increasing developer requests. By making Slack the central interface and automating Jira, Confluence, and workflow updates, the team reduced deployment-request handling from roughly 30 minutes to under one minute. The bot also improved tracking, consistency, and response speed, helping SREs move away from constant firefighting.
## Repetitive SRE Work and Its Costs
- Developers frequently asked how to inspect Flava pod logs, request permissions, interpret errors, and access staging environments.
- Deployment requests required manual movement between Slack, Confluence, and Jira:
- Finding release checklists
- Copying information into Jira
- Creating missing Fix Versions
- Linking Epics and active sprints
- Sharing ticket links and deployment documentation
- Each deployment request previously took about 30 minutes to an hour.
- Manual processing caused omissions and mistakes, especially during urgent releases.
- General requests were buried in Slack mentions, making ownership and completion status difficult to track.
- Measurement showed that each SRE spent nearly half a day per week on repetitive work.
## Slack-Centered Automation
The team adopted the principle that developers should only need Slack, while SREs should be able to manage work with a few clicks.
- **Slack as the single source of truth:** Requests begin and remain trackable in Slack.
- **Zero manual work:** Rule-based Jira and documentation tasks are automated.
- **Immediate visibility:** Status changes and results are posted to Slack in real time.
- **Permission control:** Only authorized SRE members can claim or complete requests.
## Key Technical Decisions
### Slack Workflows Instead of Slash Commands
- Slash commands are easy to implement but depend on users entering correctly formatted text.
- Slack Workflows provide structured forms with required-field validation.
- Because Workflows are native Slack functionality, the team avoided building a separate user interface.
- The lower usage barrier made adoption more likely.
### Asynchronous Processing
- Slack requires event responses within three seconds.
- Sequential calls to Jira, Confluence, and other APIs could exceed that limit.
- The bot immediately acknowledges the request, then performs external work in the background.
- Successes and failures are reported in the Slack thread, keeping processing transparent.
### Redis-Based State Management
- In-memory state would be lost whenever the bot restarted.
- Slack metadata APIs were considered too slow for real-time interactions such as emoji clicks.
- Redis was selected for sub-100-millisecond lookups and persistent state.
- A 30-day TTL limits stale data.
- Redis transactions using `WATCH/MULTI/EXEC` ensure consistent updates when multiple SREs interact simultaneously.
### Hexagonal Architecture
- The bot uses ports and adapters to isolate business logic from external systems.
- The architecture separates:
- Inbound Slack event adapters
- Application use cases and business logic
- Outbound Jira, Confluence, and Redis adapters
- External API or SDK changes can be handled without modifying core business logic.
- This structure also makes testing and future feature development easier.
## Automated Request Scenarios
### Deployment Requests
- Developers submit required project, release-version, checklist, and other details through a Slack Workflow.
- The bot automatically:
- Creates a missing Jira Fix Version
- Creates and configures the Jira ticket
- Links the Epic
- Adds the ticket to the active sprint
- Finds the relevant deployment manual
- Posts the result to the Slack thread
- An SRE can click 👀 to claim the work.
- Clicking ✅ completes the Jira ticket and posts a completion notification.
- SRE effort falls from about 30 minutes to under one minute, with minimal risk of missing required fields.
### Emergency Deployments
- Selecting an urgent request automatically sets Jira Priority to `Highest`.
- The bot immediately announces the request in Slack.
- An SRE can claim it with 👀, perform the deployment, and complete it with ✅.
- The process reduces delays from roughly 30–40 minutes to about one minute.
### General SRE Requests
- Requests such as production-access permissions are submitted through a structured Slack Workflow.
- The bot creates a Jira ticket, links the Epic, assigns the active sprint, and sets an appropriate priority.
- Slack retains the ticket link and status, eliminating the need to search through message history later.
- SREs claim and complete the request using the same emoji-based workflow.
The main recommendation is to automate repetitive, rule-based operations at the point where requests already occur. A Slack-centered, asynchronous bot with durable state and clean system boundaries can reduce manual effort while making ownership, progress, and completion visible to everyone.
The article describes how LINE Plus safely internalized black-box e-commerce systems without specifications or source code. The team built an automated equivalence-testing loop using Kafka, CDC, OpenSearch, and ksqlDB to compare legacy and new behavior at massive scale. By repeatedly identifying differences, fixing logic, and rechecking results, they could reduce discrepancies toward zero while also measuring performance and protecting production stability.
## Domain: Products, Catalogs, and Data Ingestion
- **Products** are individual seller-listed items, potentially with different prices and shipping conditions.
- **Catalogs** group products representing the same model or product type and provide derived value such as:
- Real-time lowest prices
- Unit-price metrics such as price per 100 ml
- **Ingestion** receives large product files from sellers, validates and transforms them into internal formats, and updates product and catalog data.
- Because the platform contains tens of millions of catalogs and hundreds of millions of products, small logic differences can affect the entire service.
## The Verification Loop
- The goal was not merely to find errors, but to help developers understand and correct them quickly.
- Inputs had to be identical for both systems, such as:
- The same IDs
- The same time-based snapshot
- The same product files
- Outputs were compared according to system type:
- API response objects
- Database update values
- Final registered product data
- The general loop consisted of:
- **Trigger:** Database changes, developer requests, or file arrivals
- **Execution:** Send identical inputs to legacy and new systems
- **Comparison:** Apply logic suited to reads, updates, or end-to-end flows
- **Processing:** Store detailed differences and produce real-time statistics
- **Action:** Developers inspect dashboards or Slack alerts, fix the implementation, and repeat
## Query Logic Verification
- The catalog API was difficult to reproduce because it had over 100 response fields, complex filters, undocumented defaults, and unknown sorting behavior.
- CDC streamed database binary-log changes into Kafka, allowing verification to begin from many real catalog states.
- The verifier made dual API calls and compared legacy and new responses field by field.
- Responses were converted into `Map<String, Object>` structures and compared recursively, avoiding the need to model every response class.
- If values differed only because list ordering varied, the verifier sorted serialized values and performed a second comparison.
- This helped distinguish real implementation defects from harmless ordering differences.
- Kafka isolated verification traffic from production services while handling large event volumes.
- Difference events were written to Kafka topics and indexed in OpenSearch for detailed investigation.
- ksqlDB aggregated streaming discrepancies and sent Slack notifications when abnormal patterns appeared.
- Rate limiting restricted repeated errors, such as those from the same field, to a manageable sample per minute.
- Because both APIs were called in parallel, the same pipeline also measured and compared their response times.
## Update Logic Verification
- The second case involved recalculating catalog statistics whenever product or catalog data changed.
- Unlike read verification, this process tested state transitions and asynchronous updates.
- When CDC detected a relevant change:
- The new statistics logic calculated an expected result.
- The verifier compared it with the result actually written by the legacy logic.
- Recursive Map-based comparison checked deeply nested statistics fields.
- To avoid wasting resources, verification was triggered only for updates related to the catalog-statistics module.
## Handling Asynchronous Lag
- Kafka-based processing caused timing gaps: the verifier could read the database before the legacy update had completed.
- The team introduced an **N-attempt retry queue**:
- Temporarily inconsistent events were requeued.
- Only differences that remained after several retries were treated as genuine defects.
- The verifier remained a separate process rather than being embedded in the production statistics stream.
- This avoided adding load or latency to the existing processing pipeline while preserving independent verification.
## ETL Batch Verification for Missing Triggers
- Real-time comparison could detect incorrect results, but not cases where an update should have happened and never occurred.
- During refactoring, a complex combination of product and catalog field changes contained a missing trigger condition.
- As a result, some statistics remained stale without generating any comparison event.
- To detect these silent omissions, the team designed a separate batch-verification process using ETL data alongside the real-time stream checks.
The practical recommendation is to treat system internalization as an evidence-building process: define identical inputs and observable outputs, compare legacy and replacement systems continuously, isolate verification through event streams, and supplement real-time checks with batch validation for silent or missing updates.
Repeated SLI/SLO adoption revealed a common process that could be standardized across services. The team turned that process into a reusable framework and built “LINE Status,” an internal tool that automatically presents service health according to user experience rather than raw alerts. Together, these initiatives create a shared organizational language for understanding reliability and its impact on users.
## A Reusable SLI/SLO Framework
After applying SLI/SLOs to several platforms and services, the SRE team identified recurring patterns independent of service type. They organized these patterns into a five-stage framework:
- **Select critical user journeys (CUJs) and define SLIs**
- Identify the experiences most important to users.
- Define measurable SLIs that represent those experiences.
- **Design instrumentation and metrics**
- Build or adapt metrics suitable for each CUJ.
- Use standardized naming based on Prometheus or OpenTelemetry.
- **Create dashboards and recording rules**
- Provide Grafana dashboards for quickly assessing SLO achievement.
- Precompute complex PromQL operations to improve query performance.
- **Set SLOs and alerts**
- Begin with flexible targets, such as 99.9% availability over a 28-day rolling window, allowing roughly 40 minutes of downtime.
- Define runbooks for responding to alerts.
- Refine targets after operational data and experience accumulate.
- **Establish error-budget governance**
- Balance release speed against reliability.
- Review objectives monthly or quarterly.
- Adjust SLOs and processes as needed.
The framework is currently distributed as a Confluence template containing guidance and FAQs, reducing the communication effort required from SREs during initial adoption.
## Moving from Alerts to User-Centered Service Status
As more services adopted SLI/SLOs, the team wanted a consistent way to understand the health of services they did not directly operate.
- The existing public LINE Status API page focused on external users and was updated manually during major incidents.
- The new internal tool was intended to:
- Represent the status of individual service components.
- Update automatically from SLI/SLO alerts and outage data.
- Show whether user experience was being affected.
- Rather than simply reflecting whether an alert or outage existed, status was based on CUJ-related SLI performance and SLO achievement.
- Only representative, high-value CUJs were exposed, avoiding unnecessary technical detail.
## LINE Status Architecture and Interface
LINE Status was designed as more than an alert list. It collects events through webhooks, stores them in a separate database, and uses that data to track both current status and historical changes.
- Technical SLI/SLO terms are translated into user-facing functions such as “Message Sending” or “Read Receipts.”
- Status colors provide an immediate overview:
- Green: normal
- Yellow: event detected
- Red: outage
- The main page provides:
- An overview of all services.
- CUJ status within each service card.
- AI-generated one-line summaries.
- Service detail pages provide:
- Recently affected items near the top.
- Timeline-based event displays.
- Monthly historical events.
- The history page shows:
- The scope of impact for each service during an event.
- Past events organized by month.
The initial implementation took about a month and was refined through colleague feedback. The author also used AI-assisted “vibe coding” for the frontend, emphasizing that clear, detailed requirements were more important than the development tool itself.
## Connecting the Framework and LINE Status
Once a service adopts SLI/SLOs through the framework, it can be registered in LINE Status. This connects the definition of reliability objectives with an organization-wide view of service health.
- Developers and operators can use the same CUJ-based standards.
- Teams can focus on whether users are affected instead of interpreting isolated alerts.
- During incidents, the tool helps identify impacted experiences quickly.
- Over time, the approach may improve decision-making speed and cross-team communication.
The team plans to refine CUJs, SLIs, and status-transition rules through continued operational experience.
The practical goal is to make SLI/SLOs a common language for describing service health, enabling reliability practices to scale without depending heavily on individual teams or specialists.
LINE’s “Service Configuration” system lets teams deploy features dynamically without waiting for LINE’s two-week app release cycle. As the iOS app grew to roughly 700 configuration keys across 60 modules, its monolithic design created dependency, usability, concurrency, testing, and QA problems. The article argues that the original design was reasonable at small scale but needed to evolve, beginning with lessons from Foundation’s type-safe `AttributedString` design.
## What Service Configuration Provides
- Service operators modify values through an administration page.
- The server notifies LINE clients, which fetch updated values.
- Values are selected based on factors such as:
- User region
- Device
- OS version
- The system supports:
- Feature flags
- Rollbacks
- A/B tests
- Error-reporting sample rates
- UI behavior policies
- Configuration is delivered as a string-to-string dictionary, for example:
- `"function.media.image_medium": "1280,70"`
- `"function.media.message.flow.v2.image": "Y"`
## Problems Caused by the Monolithic Design
The original implementation required every key to be declared in one roughly 7,000-line file. Although this was simple initially, growth in teams and modules made the structure increasingly costly.
### Circular Dependencies and Weak Typing
- Configuration values were exposed as raw strings because the configuration module could not depend on feature-specific modules.
- For example, `"1280,70"` represented image dimensions and JPEG quality, but callers had to parse it into an `ImageTransferQuality` value themselves.
- Defining `ImageTransferQuality` in the configuration module avoided repeated parsing but polluted unrelated modules with photo-specific types.
- Defining it in the photo module preserved separation of concerns but created an impossible reverse dependency.
### Incomplete and Confusing Abstractions
- Developers had to understand server-specific encoding rules and implementation details.
- Boolean values were sent as `"Y"` and `"N"`, requiring a custom `decodeBoolIfPresent(forKey:)` method.
- The custom decoder’s name resembled Swift’s standard decoding API, making incorrect implementations easy to write and review.
- Decoding failures could silently fall back to defaults, making the underlying problem difficult to diagnose.
- The same default value often had to be declared three times:
- A property-group default
- A decoding fallback
- A global `defaultConfiguration` entry
- These duplicated defaults served subtly different purposes, although the distinctions were generally unnecessary.
### Lack of Thread Safety
- Configuration groups were lazily decoded and replaced when new server values arrived.
- Multiple services could read configuration values concurrently on different threads.
- This caused use-after-free crashes— reportedly hundreds per day—leading to bug tickets and hotfix releases.
- As the number of services and concurrent operations increased, this became a systemic issue rather than an occasional edge case.
### No Built-in Debug Overrides
- QA frequently needed to temporarily change configuration values.
- Because the system had no override mechanism, each feature required custom:
- Persistent storage
- Debug-menu UI
- Value-display text
- Implementing this repeatedly required edits across several files and modules.
### Fragmented Test Doubles
- Since `LineConfigurationManager` was a singleton, modules created narrow protocols and custom mocks for the settings they used.
- This resulted in dozens of duplicated protocols and test doubles.
- These had to be updated alongside configuration keys and could fall out of sync.
- Differences between mocks and production behavior could allow bugs to escape tests or create false failures.
## Looking to Established Designs
The team first distilled the required properties of a replacement:
- Type-safe access to a large number of key-value pairs
- Independent key definitions by each module
- Safe behavior under concurrency
They identified Foundation’s `AttributedString` as a useful precedent because it manages many typed attributes while allowing UIKit, AppKit, SwiftUI, and other frameworks to define their own attributes independently. The article presents this as the starting point for redesigning Service Configuration around a more modular and type-safe architecture.
LINE is consolidating its two multi-person chat types—temporary “Rooms” and long-term “Groups”—into a single Group Chat model. The change aims to simplify the user experience, make all chat features available everywhere, and reduce duplicated server and client resources. A gradual migration strategy is being used to avoid disruption.
## Two Original Chat Models
- **Rooms** were designed for temporary conversations:
- No room name was required.
- Invited friends joined immediately without approval.
- Features such as albums and notes were unavailable.
- **Groups** were designed for long-term communities:
- They had names and supported features such as group albums and notes.
- Invitees had to accept or reject invitations before joining.
- Users often created Rooms without realizing their limitations, then created a new Group later when they needed additional features.
## Reasons for Unification
- Users found the distinction between Rooms and Groups difficult to understand.
- Existing conversations could not be converted from Rooms into Groups, forcing users to abandon their conversation history.
- Users frequently created multiple chats with the same members, causing:
- Cluttered conversation lists.
- Unnecessary data accumulation on servers.
- Increased client and server resource usage.
- The unified model standardizes behavior and features while retaining flexibility in how invitations work.
## Migrating Groups to Group Chats
- LINE introduced new Group Chat APIs and used **dual reads** to maintain compatibility with existing Group APIs and storage.
- The migration proceeded gradually:
1. The new API initially read Group data through a routing layer.
2. The number of Group Chats was progressively increased.
3. Eventually, only Group Chats were created.
- Batch processing migrated all existing Group data.
- After migration, LINE stopped dual reads and relied exclusively on the Group Chat model.
## Differences Between Rooms and Groups
### Invitation Mechanisms
- Groups required invitees to explicitly accept or reject an invitation.
- Rooms added people immediately when they were invited.
- The unified creation flow lets users choose whether invitees should join immediately or confirm participation first.
### Feature Availability
- Rooms lacked many Group features because they were intended to be temporary.
- The new model is based on the Group architecture, so all newly created conversations support the full feature set, including future features.
## Improving Conversation Discovery
- Users often created a new chat with the same participants instead of finding an older, inactive conversation in a long chat list.
- The new creation workflow displays a hint when an equivalent existing conversation is found.
- Users can then return to the existing conversation, reducing duplicate rooms and improving navigation.
## Migration Plans for Existing Rooms
- Conversations created in current LINE versions are already Group Chats.
- Groups created with older app versions are being converted server-side.
- The remaining objective is to migrate existing Rooms so their participants can use the complete set of Group Chat features.
The project is a long-term effort designed to minimize disruption while improving consistency and efficiency. Duplicate conversations with identical participants fell from 15% for Rooms to 0.78% for invitation-free Group Chats, demonstrating the practical impact of the consolidation.
LY Corporation is consolidating the former LINE “Verda” and Yahoo Japan “YNW” private clouds into Flava, a next-generation platform designed for large-scale, uninterrupted operations. Its approach assumes failures will occur, prioritizing stateless services, application-led availability, rapid IaC-based recovery, and extensive automation. Flava also restructures the architecture around shared resources, upstream OpenStack, default VPC networking, and user-driven cost optimization.
## Failure-Aware Design and Operations
- VM root disks are treated as temporary; persistent data is placed in external storage so instance failures have limited service impact.
- Availability is achieved through cooperation between infrastructure and applications rather than excessive infrastructure-side guarantees.
- Recovery focuses on maintaining service continuity, rebuilding environments quickly with infrastructure as code, and avoiding lengthy root-cause investigations during incidents.
- The company promotes KaaS and PaaS to help developers build resilient services without managing low-level infrastructure.
- OS configuration, package installation, networking, and other changes are managed as code through CI/CD.
- Deployments are performed by availability zone to limit the blast radius of failures.
## Observability from Fleet-Wide Trends to Root Causes
- Prometheus, Grafana, and custom dashboards monitor overall cloud health and long-term trends.
- When anomalies appear, engineers investigate at a deeper level using kernel traces, packet captures, and other low-level diagnostics.
- This combination of broad monitoring and detailed investigation allows teams to move between “forest” and “tree” perspectives.
- The operational model depends not only on tools but also on engineers capable of tracing problems down to their fundamental causes.
## OSS, Software-Defined Infrastructure, and Custom Development
- The platform relies heavily on OpenStack, Envoy, Linux kernel technologies such as eBPF/XDP, FRR, and Ceph.
- LY contributes patches and new capabilities upstream instead of maintaining long-lived private forks.
- It has developed SRv6 BGP functionality required for Flava’s VPCs and contributed related work to FRRouting and the Linux kernel.
- Compute, VPC, DNS, and load-balancing services run primarily on commodity x86 servers rather than specialized appliances.
- XDP-based data planes, hardware offload, and system tuning are used to achieve near-wire-speed throughput and low latency.
- Where OSS cannot meet internal requirements, LY builds systems from scratch, including the Dragon object store, SDN control-plane components, load-balancer health agents, and service discovery tools written in Rust, Go, and Python.
## Autonomous Hardware Operations
- With tens of thousands of hypervisors and petabyte-scale storage, hardware failures occur continuously.
- Failure detection, requests to data-center technicians, hardware replacement, and cluster reintegration are largely automated.
- Some exceptional cases still require engineers, but LY plans to use LLMs to automate more of these operational tasks.
## Flava’s Architectural Improvements
### Shared Resource Pools
- Older clouds used many dedicated clusters and resource pools, making capacity planning complex and reducing utilization.
- Flava consolidates most products and services into one large shared resource pool.
- This reduces planning variables, improves resource efficiency, and accelerates provisioning.
### Upstream-Compatible OpenStack
- Excessive customization in the legacy environment made upgrades difficult.
- Flava minimizes private patches, follows upstream OpenStack, and contributes necessary improvements back to the project.
- This enables regular upgrade cycles and keeps security fixes and features current.
### VPC by Default
- VPC networking is the standard security model for multi-tenant workloads.
- Logical isolation replaces many cases where dedicated VLANs or firewalls previously required months of preparation.
- Equivalent security environments can now be provisioned in minutes.
- The VPC data plane is being redesigned with XDP to support the reliability and performance required at company-wide scale.
### Built-In Cost Optimization
- Development environments require resource lifetimes, allowing unused “zombie” resources to be deleted automatically.
- Object storage offers bucket classes such as “High Performance” and “Scalable.”
- Users can change storage classes without changing endpoints, adapting cost and performance as access patterns evolve.
## Remaining Challenges
- Flava currently offers only a limited set of products and must expand its capabilities while addressing post-launch bugs and overlooked requirements.
- The largest challenge is migrating users from the legacy platforms.
- LY is working to provide transparent migration tools and reduce manual effort while shortening the period of duplicate investment in old and new infrastructure.
## Team and Engineering Culture
- The team includes specialists ranging from kernel developers to web-front-end engineers.
- Engineers are expected to understand and control infrastructure rather than treat it as a black box.
- Deep source-level expertise enables upstream OSS contributions and informed negotiations with commercial vendors.
- This culture of ownership and technical control is presented as a core reason the platform can evolve at LY’s scale.
LY’s experience demonstrates that large private clouds can combine OSS, custom software, commodity hardware, and rigorous automation effectively. The practical recommendation is to design for failure, keep infrastructure reproducible through IaC, contribute changes upstream where possible, and use custom development selectively for requirements that general-purpose platforms cannot satisfy.
NeurIPS 2025 research shows that AI safety is moving beyond simple post-training alignment and output filtering toward system-level, modular defenses. New approaches intervene in reasoning, multimodal interpretation, policy enforcement, and continuous evaluation to balance safety with latency and usefulness. The central conclusion is that deployable AI requires adaptable guardrails designed for real-world systems, not isolated attack benchmarks.
## The Shift Toward Practical AI Safety
- Guardrails protect AI services from harmful instructions, privacy leaks, confidential-data exposure, bias, prompt injection, and other failures.
- NeurIPS 2025 reflects a broader shift:
- From post-training safety tuning to intervention in reasoning mechanisms.
- From text-only LLMs to VLMs, RAG systems, and reasoning models.
- From laboratory attack scenarios to the practical balance between utility and safety.
- The article focuses on guardrail frameworks, multimodal moderation, prompt injection and jailbreaks, hallucinations, and over-refusal.
## Modular Guardrail Frameworks
**PRIME Guardrails: A General, Low-Latency Safety Framework for Generative AI** addresses the trade-off between rigorous safety checks and response latency through a modular architecture:
- **Policy specification:** Declarative, human-readable rules separate policies from model parameters, allowing legal or policy teams to control behavior.
- **Risk sensing and scoring:** Asynchronous detectors combine lexical rules, semantic similarity, and lightweight classifiers. Early exit blocks obvious attacks quickly while allowing domain-specific calibration.
- **Intervention router:** A deterministic controller chooses whether to allow, rewrite, or reject an interaction based on policies and risk scores.
- **Monitoring and memory:** Lightweight records preserve decisions and rejection reasons for predictability and auditing.
- **Evaluation and evolution:** Red-team recipes and automated vulnerability testing help the system adapt to new attack methods.
The framework supports defense in depth without running every expensive safety mechanism sequentially. Its modularity, auditing capabilities, and continuous-evaluation loop make it suitable for production environments.
## Turning Governance Policies into Code
**Policy-as-Prompt: Turning AI Governance Rules into Guardrails for AI Agents** converts informal organizational materials into runtime-enforceable controls.
- The framework analyzes sources such as PRDs, technical design documents, regulations, and source code.
- It builds a **source-linked policy tree** connecting individual rules to their original documents.
- The policies are compiled into lightweight prompt-based classifiers.
- When an agent rejects a request, the system can trace the decision back to its legal or organizational basis.
- The approach helps enforce:
- Least-privilege access.
- Data minimization.
- Restrictions on out-of-scope tasks.
- Protection against prompt injection.
- It may be especially valuable in regulated industries such as finance and healthcare, where frequently changing policies create substantial technical debt.
## Multimodal Safety and VLM Reasoning
Vision-language models create new safety challenges because harmful meaning can emerge from interactions between images and text.
**GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning** trains models to reason about combined modalities rather than classifying each input independently.
- It addresses cases where harmless text obscures harmful visual content, such as an image of a bloodied knife paired with “cooking.”
- Its GRPO-based training process includes:
- **Safety-aware data concatenation** to create difficult examples containing hidden or mixed harmful content.
- **Dynamic clipping** that encourages exploration early in training and tighter refinement later.
- **Length-aware safety rewards** that reward concise conclusions supported by reasoning.
- The method aims to detect subtle harms such as hate speech hidden in memes and visual metaphors.
## Hidden Vulnerabilities in Multimodal Training Data
**VLMs can Aggregate Scattered Training Patches** demonstrates that filtering training images may not be sufficient.
- A harmful image can be divided into individually innocuous patches and included in training.
- A VLM may reconstruct the harmful concept by associating patches that share the same text label.
- The paper calls this behavior **visual stitching**, related to cross-sample reasoning and inductive out-of-context reasoning.
- Text labels such as “safe” or “unsafe” can help the model connect fragmented visual information and infer the original image-level meaning.
- This suggests that safety evaluations must inspect not only final outputs but also:
- Input-processing pipelines.
- Cross-sample interactions.
- Internal or latent representations.
The available article ends while introducing research on distorted safety perception, so that section cannot be summarized further from the provided text. In practice, organizations should combine modular, low-latency enforcement with traceable policy management and multimodal evaluations that test hidden interactions—not just obvious harmful prompts or images.
FAA achieves a 96.1% response rate by favoring simple, maintainable techniques over complex AI architectures. Its design choices were to use RAG instead of knowledge-focused fine-tuning, retrieve complete documents before cutting them into question-relevant sections, and rely on a basic ReAct agent loop rather than elaborate workflows or multiple agents. The article concludes that improving documentation is more valuable than adding complexity when unanswered questions mainly result from missing source material.
## RAG Instead of Fine-Tuning
- Fine-tuning was rejected as the primary method for injecting enterprise knowledge.
- Research cited in the article found that fine-tuning was highly effective for changing a model’s style—about 97% success—but achieved only about 11% accuracy when teaching new factual knowledge.
- FAA’s experiment with approximately 40 examples showed that the model answered the exact training question correctly but failed when the wording changed slightly.
- Maintaining larger fine-tuning datasets would require experts to create, verify, and continuously update training examples whenever product documentation changes.
- RAG is better suited to frequently changing product information because only the source documents need to be updated.
- Fine-tuning may still be useful for domain-specific terminology or reasoning patterns, but not for keeping FAA’s product knowledge current.
## Retrieving Whole Documents Instead of Pre-Chunking
- Conventional RAG systems split documents into small chunks before embedding them, improving semantic search precision.
- Pre-chunking can remove essential context, especially when references such as “this case” or “the following settings” are separated from the text they depend on.
- FAA’s documents are generally short, well-structured, focused on one product and topic, making whole-document retrieval practical.
- Instead of chunking before search, FAA embeds and retrieves complete documents, then splits them after the relevant document is known.
- The post-split process has two stages:
- Split the document by Markdown headers into meaningful sections.
- Use a lightweight LLM to select only the sections relevant to the user’s question.
- For a question about creating and deleting a VM, the main model might receive only the “VM creation” and “VM deletion” sections.
- This extra filtering call remains inexpensive because the lightweight model outputs only section indexes rather than generating a full response.
- The key advantage is that splitting happens after the system understands the question, preserving context while delivering only the necessary information.
## ReAct Instead of Complex Agent Workflows
- FAA tested plan-and-execute workflows, in which the model first creates a multi-step plan and then carries it out.
- Planning and replanning increased system complexity without producing a noticeable improvement in answer quality.
- With well-designed tools and carefully filtered context, the model was able to determine tool order on its own.
- FAA therefore uses ReAct: the model reasons, takes an action, observes the result, and decides what to do next.
- This approach allowed the agent to handle troubleshooting questions without a separate planning layer.
## Rejecting Multi-Agent Architectures
- The team also tested specialized agents, such as separate VM and Kubernetes experts.
- Delegating questions and assembling the results required additional LLM calls, increasing response time from roughly 9 seconds to 14 seconds in one test.
- Multi-agent routing performed poorly for cross-domain questions, such as moving data from a VM to object storage.
- Specialists could miss information outside their assigned domain, whereas a single agent could maintain the complete context.
- FAA therefore kept one agent with access to progressively disclosed tools and relevant documentation.
## Documentation as the Main Bottleneck
- Analysis of unanswered questions showed that about 50% were caused by a documentation gap: no reference document existed.
- Other failures were mostly temporary API issues or questions outside FAA’s intended scope.
- This suggests the core retrieval and agent system performs well when documentation is available.
- The team shares missing questions with product teams, whose updated documents are then re-embedded and incorporated into future evaluations.
The practical recommendation is to start with the simplest architecture that fits the data: use RAG for changing knowledge, preserve document context during retrieval, and let a capable model operate through a ReAct loop. In enterprise systems, improving the underlying documentation may produce greater gains than adopting more sophisticated AI frameworks.
