How we optimized our Akka application using Datadog’s Continuous Profiler

Datadog discovered that an unexpected 20–30% CPU overhead came from Akka’s use of ForkJoinPool, not from the log-processing code they initially suspected. Profiling showed that an actor handling intermittent latency metrics repeatedly caused worker threads to park and unpark. Moving that actor to a busier, more stable dispatcher reduced CPU usage by about 30%.

How Profiling Revealed the Problem

• Datadog used Akka to parallelize log-event processing through actors and dispatchers.
• An optimization to log parsing produced little improvement, despite reducing parsing CPU time.
• Continuous Profiler flame graphs showed increased CPU time in:
  • ForkJoinPool.scan()
  • Unsafe.park()
• Thread-level analysis revealed that the default Akka dispatcher—not the expected dedicated work pool—was responsible.
• Many of the affected threads were executing a latency-reporting actor.

Why ForkJoinPool Was Consuming CPU

• ForkJoinPool dynamically manages worker threads:
  • It creates threads when work increases.
  • It suspends idle threads with Unsafe.park().
  • It resumes them with Unsafe.unpark().
  • It terminates idle workers after a default period.
• The latency actor received a few hundred events per second, processed them within milliseconds, and then remained idle until the next batch.
• Because the pool allowed up to 32 threads—matching the number of processor cores—it repeatedly activated and suspended many workers.
• These frequent parking and unparking operations created short CPU spikes and excessive time in ForkJoinPool.scan().

The Dispatcher Change

• The team moved the latency actor from Akka’s default dispatcher to the main work-dispatcher.
• The work dispatcher already handled a steadier stream of log-processing tasks, keeping its worker threads active.
• This required only a configuration change assigning the actor to work-dispatcher.
• CPU usage fell by roughly 30% across services.
• The default dispatcher also shrank from 32 threads to 2, confirming that unnecessary thread activation was the cause.

Recommendations

• Monitor CPU time spent in ForkJoinPool.scan(), especially when it exceeds roughly 10–15%.
• Limit the number of Akka actor instances.
• Set a suitable maximum thread count for each pool.
• Reduce the number of separate thread pools where practical.
• Use task queues to absorb frequent, short-lived workload spikes.
• Aim to keep the number of active ForkJoinPool workers relatively stable and avoid repeated parking and unparking.