The First Action That Determines the Success or
Woowa Brothers argues that incident impact depends not only on how quickly an outage is detected, but also on how quickly an effective First Action is executed. Analysis of more than 70 incidents showed that incidents beginning with a hotfix tended to last nearly twice as long as those beginning with a rollback. The company therefore established a standardized incident lifecycle and metrics system to make early response measurable and improve it through automation and process design. ## Why First Action Matters - Detection was generally fast in Woowa Brothers’ 2025 incidents, but customer impact often continued for a long time. - Rollbacks can immediately undo a problematic change, while hotfixes require diagnosis, coding, and redeployment. - During hotfix preparation, the service may remain impaired and customer impact continues to accumulate. - Predefined mechanical mitigations—such as rollback or scaling—are particularly effective because they can be executed without lengthy additional decisions. - First Action is defined by both: - **What** action was taken - **When** it was executed after the incident was recognized ## Standardizing Incident Response - Comparing First Actions across incidents was difficult because teams used different starting points: - The moment customer impact was recognized - The moment an incident was announced company-wide - Other team-specific interpretations - Woowa Brothers concluded that First Action could only be measured consistently if the entire incident timeline used shared definitions. - The lifecycle provides a common framework for understanding where an incident is, what action should happen next, and how long each stage takes. ## The Seven-Stage Incident Lifecycle ### Potential-Incident Lifecycle #### 1. Anomaly - A service or system shows abnormal behavior and the responsible team detects and acknowledges it. - Acknowledgment must leave objective evidence, such as: - An on-call alert acknowledgment - A customer-service response comment - An alert comment - If there is no customer impact, the process may end after internal handling. - If any part of the ordering process becomes unavailable, the event transitions into the Incident Lifecycle. ### Incident Lifecycle #### 2. Open - The service owner recognizes the event as an incident. - An incident-response channel is created and relevant technical and business organizations are invited to coordinate. #### 3. Investigating - Teams assess customer impact and investigate likely causes. - They first examine recent deployments, configuration changes, and failures in external dependencies. #### 4. Identified - Teams execute actions to reduce customer impact. - Rollbacks and scaling adjustments are prioritized as First Actions. - Multiple mitigation options may be evaluated and applied in parallel rather than waiting for a single definitive root cause. #### 5. Monitoring - Teams verify whether the applied actions are actually reducing customer impact. - If not, the process returns to investigation and mitigation for another response attempt. #### 6. Resolved - Customer impact has been eliminated and the incident is considered resolved. - Findings and actions are communicated across the organization according to an established process. #### 7. Closure Time - The team documents the root-cause analysis and tracks preventive follow-up work. - Woowa Brothers separates incident reporting from execution of corrective actions to ensure resolution leads to real operational improvement. ## Metrics for Measuring Response Performance The lifecycle is useful because it connects incident stages to time-based metrics. These metrics are intended to reveal bottlenecks and guide improvement, not serve as goals in themselves. ### MTTD: Mean Time to Detect - Measures the average time from incident occurrence until detection and acknowledgment. - Woowa Brothers includes acknowledgment evidence—similar to MTTA—because an alert alone does not prove that response has begun. - A high MTTD may indicate: - Inadequate monitoring coverage or thresholds - Excessive alert noise - Missing acknowledgment records ### MTTR: Mean Time to Repair - Measures the average time from incident acknowledgment until service recovery. - A high MTTR can indicate: - Poor First Action readiness - Insufficient service visibility - Complex recovery procedures - Slow decisions or communication bottlenecks - It points to structural improvements such as automation, standardized procedures, and clearer decision-making authority. ### MTTA: Mean Time to Action - Measures how quickly mitigation is initiated. - Rather than judging whether a response was “good,” it evaluates whether standardized response mechanisms activate promptly. - Woowa Brothers divides it into two more specific metrics. #### MTTFA: Mean Time to First Action - Measures the time until the first predefined mechanical mitigation, such as a rollback or scaling adjustment. - A high MTTFA may result from: - Complicated rollback paths - Manual scaling operations - Excessive decisions or preparation required before execution - The recommended solution is to automate and simplify First Action procedures. #### MTTEA: Mean Time to Effective Action - Measures the time from incident occurrence until abnormal metrics begin improving after an effective mitigation is applied. - It captures whether an action actually worked, not merely whether it was executed. - The relationship between MTTFA and MTTEA provides useful signals: - **MTTEA ≈ MTTFA:** the initial action was fast and immediately effective. - **MTTEA > MTTFA:** the First Action was insufficient and additional response was needed. - **MTTEA without MTTFA:** no mechanical initial action was possible, or responders relied directly on a hotfix or similar intervention. - **Increasing MTTEA:** standard scenarios or automation require improvement. The practical recommendation is to treat incident response as an organizational system rather than an individual skill. Define a shared lifecycle, prioritize fast and reversible mitigations such as rollback, automate their execution, and use metrics like MTTFA and MTTEA to continuously remove response bottlenecks.
Read original(opens in new tab)