How we optimized our Akka application using Datadog’s Continuous Profiler
Datadog discovered that an unexpected 20–30% CPU overhead came from Akka’s use of `ForkJoinPool`, not from the log-processing code they initially suspected. Profiling showed that an actor handling intermittent latency metrics repeatedly caused worker threads to park and unpark. Moving that actor to a busier, more stable dispatcher reduced CPU usage by about 30%. ## How Profiling Revealed the Problem - Datadog used Akka to parallelize log-event processing through actors and dispatchers. - An optimization to log parsing produced little improvement, despite reducing parsing CPU time. - Continuous Profiler flame graphs showed increased CPU time in: - `ForkJoinPool.scan()` - `Unsafe.park()` - Thread-level analysis revealed that the default Akka dispatcher—not the expected dedicated work pool—was responsible. - Many of the affected threads were executing a latency-reporting actor. ## Why `ForkJoinPool` Was Consuming CPU - `ForkJoinPool` dynamically manages worker threads: - It creates threads when work increases. - It suspends idle threads with `Unsafe.park()`. - It resumes them with `Unsafe.unpark()`. - It terminates idle workers after a default period. - The latency actor received a few hundred events per second, processed them within milliseconds, and then remained idle until the next batch. - Because the pool allowed up to 32 threads—matching the number of processor cores—it repeatedly activated and suspended many workers. - These frequent parking and unparking operations created short CPU spikes and excessive time in `ForkJoinPool.scan()`. ## The Dispatcher Change - The team moved the latency actor from Akka’s default dispatcher to the main `work-dispatcher`. - The work dispatcher already handled a steadier stream of log-processing tasks, keeping its worker threads active. - This required only a configuration change assigning the actor to `work-dispatcher`. - CPU usage fell by roughly 30% across services. - The default dispatcher also shrank from 32 threads to 2, confirming that unnecessary thread activation was the cause. ## Recommendations - Monitor CPU time spent in `ForkJoinPool.scan()`, especially when it exceeds roughly 10–15%. - Limit the number of Akka actor instances. - Set a suitable maximum thread count for each pool. - Reduce the number of separate thread pools where practical. - Use task queues to absorb frequent, short-lived workload spikes. - Aim to keep the number of active `ForkJoinPool` workers relatively stable and avoid repeated parking and unparking.
Read original(opens in new tab)