Agile Development

2 posts

line4 min readCurated summary

Sharing the journey of LINE DEV

AI adoption at LY Corporation has moved beyond experimentation toward learning how to use these tools effectively in real work. The LINE DEV AI Reporters program connects scattered individual and team experiences through internal sharing sessions, helping practical lessons spread across the organization. Its central conclusion is that AI productivity depends not only on tools, but also on clear specifications, sound engineering practices, and a culture of continuous sharing. ## Turning Individual Experiments into Organizational Knowledge - AI enthusiasts across LY Corporation were independently experimenting with tools such as ChatGPT and Claude Code. - These experiences often remained limited to individuals or small teams. - AI Reporters brought together members of different roles and seniority who had experience sharing AI-related work. - Their goal was to turn personal trial and error into reusable organizational knowledge. ## Starting with Informal Personal Experiments - Early AI sharing sessions emphasized accessibility rather than polished success stories. - Multimedia Platform Dev’s Choi Jeong-min shared a “one service a day” vibe-coding experiment using Claude Code and Antigravity. - The experiment demonstrated that rapid implementation increases the importance of clearly defining what to build. - Sharing failures and unfinished experiments reduced the pressure to perform and encouraged more employees to try AI themselves. ## Applying AI to Real Development Work - As interest grew, discussions shifted from fun experiments to practical workplace applications. - Data Dev4’s Lee Yun-seong shared more than a month of project experience using Claude Code, project templates, and Vibe Kanban. - Developers spent more time on planning, design, review, and coordination while agents handled implementation. - Because the current codebase becomes the context for future agent work, poor architecture and coding styles can quickly be reproduced and amplified. - Continuous testing, refactoring, documentation, interface management, and architectural cleanup are therefore essential. - Skipping automated tests before commits led to increasing numbers of broken changes during later merges. - Humans remain responsible for ensuring that AI-generated code actually contributes to the project. - The most valuable skills increasingly involve task design, project management, system context, and meta-programming rather than implementation alone. - Developers can work in parallel with agents by planning the next task, researching requirements, and reviewing completed code while agents execute current work. ## Expanding from Teams to Organization-Wide Programs - Fintech Engineering organized a hands-on workshop covering the full path from idea to deployment. - Participants connected ChatGPT, Claude Code, and Stitch AI to plan, design, build, and complete a working service. - The integrated workflow helped participants understand how AI tools can support an entire product-development process, not just prototyping. - The GAI Study Group in the advertising organization broadened discussions to AI strategy, trends, agent behavior, developer workflows, and business applications. - Topics included: - AI agent reliability - Implementing interactions between PyTorch-based LLMs and MCP servers - Senior and junior developers’ vibe-coding workflows - NotebookLM-based RAG using wiki pages and Slack conversations - One session examined MCP internals by implementing JSON-RPC messaging and session-state management directly, revealing complexities hidden by libraries such as FastMCP. - Sessions were opened to participants and presenters from other teams, with some content published online for wider access. ## Building a Culture of Continuous Sharing - The most useful AI knowledge came from real workplace attempts, failures, and revisions—not only from polished documentation or external trends. - AI Reporters made existing but scattered experiences visible and connected them through presentations, Slack discussions, and monthly meetings. - Informal conversations such as “I tried this—how did it work for you?” helped normalize experimentation and learning from mistakes. - AI adoption is treated as an ongoing practice because tools and workflows continue to change. LY Corporation’s experience suggests that organizations should create lightweight, recurring forums where employees can share practical AI experiments. The combination of rapid experimentation, disciplined engineering, and open knowledge exchange allows individual discoveries to become lasting organizational capability.

Read original(opens in new tab)
datadog3 min readCurated summary

2023-03-08 incident: A deep dive into our incident response

Datadog’s March 8, 2023 global outage tested an incident-response process designed for large-scale failures. The company’s monitoring, on-call structure, training, and blameless culture enabled a coordinated response, but the incident also exposed challenges in diagnosing and managing a rapidly evolving, global outage. Datadog’s central lesson is that effective response depends less on rigid runbooks than on preparation, clear ownership, autonomous decision-making, and continuous learning. ## Datadog’s Incident Response Model - Datadog follows a “you build it, you own it” operating model. - Teams instrument their services extensively and configure monitors to detect problems around the clock. - Independent, out-of-band monitoring checks Datadog’s APIs from outside its infrastructure, ensuring that monitoring still works if Datadog itself becomes unavailable. - Slack channels are automatically created for incidents to provide shared situational awareness and enable additional engineers to contribute. ## Handling High-Severity Incidents - Senior engineers rotate on call for incidents involving substantial customer impact or multiple teams. - The first senior responder becomes the incident commander and retains overall responsibility. - A communications lead may manage internal updates and coordination. - For the most serious incidents, an engineering executive and customer-support manager join to provide leadership, business context, and customer-facing communication. - The incident commander remains accountable for coordinating the overall response. ## Preparation, Training, and Postmortems - Datadog uses a relatively low threshold for declaring incidents, giving engineers frequent practice with its response process. - Engineers complete incident-response training before joining an on-call rotation and repeat refresher training every six months. - Training covers on-call responsibilities, response roles, and blameless investigation practices. - Every high-severity incident receives a detailed postmortem focused on preventing recurrence. - Automation prompts responders to begin postmortems while the incident is still fresh. ## Autonomy and a Blameless Culture - Because large systems change constantly, detailed recovery procedures can quickly become outdated. - Datadog therefore gives engineers authority to choose the best response based on their knowledge of the affected services. - The company treats failures as weaknesses in systems rather than evidence of individual fault. - Blamelessness is intended to encourage creativity, honesty, and effective decision-making under pressure. ## The March 8 Outage - A systemd upgrade began around 06:00 UTC and ultimately triggered the outage. - Monitoring detected the problem within three minutes, and engineering teams were paged shortly afterward. - A high-severity incident was declared at 06:18, with an incident commander joining five minutes later. - The first public status update was posted at 06:31, and the outage was officially diagnosed as global at 06:32. - By 07:20, responders identified a Kubernetes failure and unhealthy intake systems as central problems. - Engineers confirmed by 08:00 that the Kubernetes failure was not spreading to additional or newly provisioned nodes. - A working mitigation for the EU1 region was found by 08:30. - Most US1 compute capacity recovered automatically by 11:00, while teams began organizing a longer recovery effort. - At 11:36, unattended upgrades were identified as the triggering event. - Compute capacity in EU1—the first step toward recovery—was restored by 12:05. ## Practical Lessons Datadog’s experience demonstrates the value of independent monitoring, practiced incident roles, rapid communication, and empowered responders. Organizations operating complex systems should regularly rehearse incident management, invest in resilient observability outside the primary platform, and use blameless postmortems to turn major outages into improvements.

Read original(opens in new tab)