techlist.io
KREN
pinterest

Drastically Reducing Out-of-Memory Errors in Apache Spark at Pinterest (opens in new tab)

k8sapache-sparkmemory-managementresource-allocationdistributed-computingtask-schedulingapache-glutenapache-celeborn

Pinterest developed Auto Memory Retries to reduce Spark out-of-memory failures without permanently assigning oversized executors to every task. The system detects OOM failures and retries affected tasks with progressively larger resource profiles, reducing both on-call incidents and wasted compute. Instead of tuning every job for its peak memory demand, Pinterest can size jobs around typical usage while handling exceptional tasks elastically.

Pinterest’s Spark Environment

  • Pinterest processes more than 90,000 Spark jobs daily across tens of thousands of nodes.
  • Its infrastructure includes:
    • Kubernetes clusters
    • Spark 3.2, with Spark 3.5 adoption underway
    • Apache Celeborn for shuffle
    • Apache YuniKorn for scheduling
    • Apache Gluten and Meta’s Velox for acceleration
    • Archer, Pinterest’s internal submission service
  • More than 4.6% of job failures were caused by OOM errors.

Why Manual Memory Tuning Was Insufficient

  • Pinterest’s clusters are memory-bound, so simply increasing executor sizes is expensive and difficult.
  • Automatic tuning generally reduces executor memory to match historical usage and improve resource efficiency.
  • Manual tuning can work, but requires substantial expertise because:
    • Different stages perform different operations.
    • Individual tasks may have very different memory needs because of data skew.
    • Configurations that work for most tasks may fail for a small number of high-memory tasks.
  • Auto Memory Retries allow jobs to target approximately their P90 memory usage, while automatically giving unusually demanding tasks more capacity.

How Spark Executor Memory Works

  • An executor’s memory and CPU capacity determine how many tasks can run concurrently.
  • By default, each CPU core provides a task slot.
  • For example, with spark.task.cpus=2, an executor with two usable task slots and 8 GB of memory provides roughly 4 GB per task on average.
  • Memory is shared, so one task may temporarily use more than its average allocation if another uses less.
  • An OOM occurs when the combined memory usage of concurrent tasks exceeds the executor’s available memory.

Auto Memory Retries Design

Pinterest modified Spark’s scheduling loop so individual tasks can use resource profiles different from their parent TaskSet.

  • Each task can store an optional taskRpId identifying its retry resource profile.
  • Pinterest creates immutable retry profiles at 2x, 3x, and 4x the base profile.
  • If off-heap memory is enabled, it is scaled as well.
  • Retries use a hybrid strategy:
    • First retry: Double cpus per task, allowing the task to run on an existing executor with fewer concurrent tasks.
    • Later retry: Launch a physically larger executor if the task still fails or already requires the entire executor.
  • The approach prioritizes reusing existing executors before provisioning larger ones.

Changes to Spark Internals

Pinterest extended core Spark components through Pinterest-specific subclasses rather than using a listener-only implementation.

  • Task
    • Stores the optional task resource profile ID.
  • TaskSetManager
    • Tracks tasks with non-default profiles.
    • Assigns the next larger retry profile after an OOM.
  • TaskSchedulerImpl
    • Allows tasks with increased CPU requirements to run on standard executors.
  • ExecutorAllocationManager
    • Tracks pending tasks by retry profile.
    • Requests larger executors when physical memory is required.
  • The feature-specific classes are loaded only when Auto Memory Retries is enabled.
  • The Spark UI was updated to display each task’s resource profile ID.

Handling Tasks After an OOM

  • When a task fails on an executor with more than one core, its first retry doubles spark.task.cpus.
  • Other tasks in the same stage or future stages are unaffected.
  • Spark cannot reliably determine which concurrent task caused the executor-level OOM.
  • As a result, Pinterest treats all tasks running on the terminated executor as having failed due to OOM and routes them to retries that do not share the executor with other tasks.

Practical Conclusion

Pinterest’s approach makes executor sizing elastic at the task level: configure jobs for normal memory usage, then progressively increase resources only for tasks that need them. This can reduce OOM-related failures and operational load while avoiding the cost of running every task on oversized executors.