lineApr 22, 2026

Implementing SLO/SLI for Improved Reliability Part 3 - Service Application Cases (opens in new tab)

SLI/SLO adoption is not merely a matter of choosing metrics; it requires redefining how a service is understood from the user’s perspective. LINE’s SRE team applies this approach by identifying critical user journeys, measuring reliability with clear criteria, and setting realistic objectives. The resulting data helps teams balance reliability, engineering costs, feature delivery, and incident response. ## The Mindset Behind SLI/SLO ### Understanding the Service and Users - Begin by identifying the services and features users depend on. - Map user journeys and select critical user journeys (CUJs) based on: - How frequently users use a feature - Whether the feature is essential to the service - Its relationship to business objectives - Aligning SLOs with business goals ensures that reliability work supports organizational priorities. ### Communication and Collaboration - SLI/SLOs should be defined and managed collaboratively rather than by a single team. - Product or service owners define CUJs because they understand the user experience best. - Infrastructure teams provide scalable systems for collecting and managing metrics. - SREs build the measurement tools and processes used to monitor and improve reliability. - Shared ownership is essential so SLOs can guide both daily operations and new feature launches. ## Implementing SLI/SLOs ### Analyze Critical User Journeys - List the services and functions provided to users. - Ask: - Which features are used most often? - Which features are indispensable? - LINE examples include: - Account registration - Sending and receiving messages - User authentication and encryption - LINE Login - Profile information - The goal is not to include every feature, but to select the most important ones from the user’s perspective. ### Define Service Level Indicators For each CUJ, determine: - **Measurement location:** Choose the point that best represents the user experience, such as a gateway, frontend, or backend. - **Measurement API:** Select a representative API to avoid unnecessarily complex calculations. - **Success criteria:** Establish clear boundaries between successful and failed requests. Common SLI criteria include: - **Latency:** Define a percentile, such as the 99.9th percentile, and the maximum acceptable response time. - **Success rate:** Define the required percentage of successful responses during the measurement period. For example, a messaging service might require 99.9% of requests to complete within 500 milliseconds and 99.999% of all requests to receive successful responses. If a CUJ cannot be measured reliably or its success criteria cannot be defined clearly, it may be excluded or supported with a dedicated measurement metric. ### Set SLO Targets - Define the reliability level the service must maintain over a specific period. - An example target is achieving the defined latency and success-rate criteria for 99.9% of a 28-day period. - Targets must be realistic: - Excessively high targets increase operational and infrastructure costs. - Excessively low targets can result in poor user experiences. - SLOs should balance reliability requirements with available resources. ### Visualize Reliability - Provide dashboards that allow all stakeholders to understand the current SLO status quickly. - Show overall SLO performance and error-budget consumption, with detailed dashboards for individual CUJs. - Keep dashboards simple and easy to scan rather than displaying excessive information. - Use visual indicators such as: - Green for healthy performance - Orange for warning conditions - Red for missed objectives ## How SLI/SLOs Are Used ### Quantifying Reliability - Replace vague descriptions such as “the service is slow” with measurable statements. - Teams can identify issues such as latency exceeding a 400-millisecond SLI threshold or success rates falling below 99.99%. - Dashboards also help correlate periods of poor performance with incidents or operational changes. ### Guiding Resource Allocation - SLOs show whether reliability targets are being met. - Error budgets indicate how much additional failure or downtime is acceptable. - When performance exceeds the SLO and the error budget is healthy, teams can invest more aggressively in: - New features - Faster release cycles - Product experimentation - When little error budget remains, resources can instead focus on prevention, remediation, and reliability improvements. ### Supporting On-Call Operations - LINE uses alerts triggered by changes in error-budget status to help on-call teams recognize and respond to service issues. - SLO reviews are also incorporated into regular meetings and preventive reliability work. SLI/SLO implementation works best as a shared, user-focused operating model. By combining clear CUJs, measurable criteria, realistic targets, and actionable dashboards, teams can make informed decisions about when to prioritize innovation and when to prioritize stability.

resource-allocationobservabilitysite-reliability-engineeringservice-level-indicator+4

pinterestFeb 17, 2026

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest (opens in new tab)

Pinterest developed **Auto Memory Retries** to reduce Spark out-of-memory failures without permanently assigning oversized executors to every task. The system detects OOM failures and retries affected tasks with progressively larger resource profiles, reducing both on-call incidents and wasted compute. Instead of tuning every job for its peak memory demand, Pinterest can size jobs around typical usage while handling exceptional tasks elastically. ## Pinterest’s Spark Environment - Pinterest processes more than **90,000 Spark jobs daily** across tens of thousands of nodes. - Its infrastructure includes: - Kubernetes clusters - Spark 3.2, with Spark 3.5 adoption underway - Apache Celeborn for shuffle - Apache YuniKorn for scheduling - Apache Gluten and Meta’s Velox for acceleration - Archer, Pinterest’s internal submission service - More than **4.6% of job failures** were caused by OOM errors. ## Why Manual Memory Tuning Was Insufficient - Pinterest’s clusters are memory-bound, so simply increasing executor sizes is expensive and difficult. - Automatic tuning generally reduces executor memory to match historical usage and improve resource efficiency. - Manual tuning can work, but requires substantial expertise because: - Different stages perform different operations. - Individual tasks may have very different memory needs because of data skew. - Configurations that work for most tasks may fail for a small number of high-memory tasks. - Auto Memory Retries allow jobs to target approximately their **P90 memory usage**, while automatically giving unusually demanding tasks more capacity. ## How Spark Executor Memory Works - An executor’s memory and CPU capacity determine how many tasks can run concurrently. - By default, each CPU core provides a task slot. - For example, with `spark.task.cpus=2`, an executor with two usable task slots and 8 GB of memory provides roughly 4 GB per task on average. - Memory is shared, so one task may temporarily use more than its average allocation if another uses less. - An OOM occurs when the combined memory usage of concurrent tasks exceeds the executor’s available memory. ## Auto Memory Retries Design Pinterest modified Spark’s scheduling loop so individual tasks can use resource profiles different from their parent `TaskSet`. - Each task can store an optional `taskRpId` identifying its retry resource profile. - Pinterest creates immutable retry profiles at **2x, 3x, and 4x** the base profile. - If off-heap memory is enabled, it is scaled as well. - Retries use a hybrid strategy: - **First retry:** Double `cpus per task`, allowing the task to run on an existing executor with fewer concurrent tasks. - **Later retry:** Launch a physically larger executor if the task still fails or already requires the entire executor. - The approach prioritizes reusing existing executors before provisioning larger ones. ## Changes to Spark Internals Pinterest extended core Spark components through Pinterest-specific subclasses rather than using a listener-only implementation. - **Task** - Stores the optional task resource profile ID. - **TaskSetManager** - Tracks tasks with non-default profiles. - Assigns the next larger retry profile after an OOM. - **TaskSchedulerImpl** - Allows tasks with increased CPU requirements to run on standard executors. - **ExecutorAllocationManager** - Tracks pending tasks by retry profile. - Requests larger executors when physical memory is required. - The feature-specific classes are loaded only when Auto Memory Retries is enabled. - The Spark UI was updated to display each task’s resource profile ID. ## Handling Tasks After an OOM - When a task fails on an executor with more than one core, its first retry doubles `spark.task.cpus`. - Other tasks in the same stage or future stages are unaffected. - Spark cannot reliably determine which concurrent task caused the executor-level OOM. - As a result, Pinterest treats **all tasks running on the terminated executor** as having failed due to OOM and routes them to retries that do not share the executor with other tasks. ## Practical Conclusion Pinterest’s approach makes executor sizing elastic at the task level: configure jobs for normal memory usage, then progressively increase resources only for tasks that need them. This can reduce OOM-related failures and operational load while avoiding the cost of running every task on oversized executors.

resource-allocationk8sapache-sparkmemory-management+4