Spark Connect on Kubernetes #1: Building a Robust Spark Connect
Toss Securities operates Spark Connect as a production service on Kubernetes so analysts and engineers can use Spark without complex setup. Spark Connect replaces per-application Drivers with long-running servers, making clients lighter and sessions faster, but it also introduces shared-failure and resource-contention problems. The post argues that production reliability requires both reducing server-wide failure triggers and distributing sessions across multiple replicas. ## How Classic Spark Works - Spark consists of: - A **Driver**, which plans jobs, schedules tasks, and collects results. - **Executors**, which perform the distributed computations. - In Classic Spark: - **Client mode** runs the Driver inside the client process. - **Cluster mode** launches the Driver in the cluster for each submitted application. - Both modes assume that one application has one Driver and one workload. - Clients also need Spark libraries, JVM support, and configuration. ## What Spark Connect Changes - Spark Connect turns the Driver into a pre-started, long-running server. - Clients send unresolved logical plans encoded with Protocol Buffers over gRPC. - The server handles analysis, optimization, scheduling, and execution. - Results are streamed back using Arrow. - This resembles a database accessed through JDBC. ### Benefits - **Thin clients:** Clients do not need the full Spark runtime or JVM. - **Language and platform independence:** Notebooks, BI tools, SQL clients, and different programming languages can use the same server. - **Fast session creation:** Sessions connect to an already-running server. - **Better client-failure tolerance:** A disconnected notebook does not necessarily terminate server-side work. ## Problems Created by Shared Long-Running Servers Spark’s internal design often assumes “one application equals one workload.” Sharing one application across many users breaks that assumption. ### A Shared Driver Becomes a Single Point of Failure - Multiple sessions share one `SparkContext` and Driver JVM. - A Driver failure terminates all sessions, jobs, and caches attached to it. - Spark’s global `spark.executor.maxNumFailures` counter can shut down the entire application after enough executor failures. - Because all sessions contribute to the same counter, one user’s unstable or memory-intensive query can terminate unrelated users’ workloads. - The counter is global, persists over time, and is separate from per-query task-level fault tolerance such as `spark.task.maxFailures`. ### Resource Contention and Scheduling Limits - `newSession()` isolates SQL state and namespaces, but not CPU, memory, or executors. - Heavy workloads can occupy all task slots and delay smaller queries. - FIFO scheduling favors earlier jobs, and Spark does not preempt tasks already using slots. - Fair Scheduler pools can influence task-slot ordering, but cannot provide true CPU or memory isolation. - Spark Connect does not automatically propagate `spark.scheduler.pool` to the server-side execution thread, causing queries to fall into the default pool unless the server explicitly assigns pools. - Actual resource isolation must therefore be implemented outside Spark’s task scheduler. ### Fixed Server Capacity - A server’s image, Driver and Executor resources, and Spark configuration are fixed when it starts. - Dynamic Resource Allocation can adjust executor counts, but cannot change the server’s basic specification. - Flexible scaling and team-level isolation require creating or replacing servers, which is addressed in a later part of the series. ## Reducing Server-Wide Failures Before adding replicas, Toss Securities reduces the chance that one bad query can kill the shared server. - Set `spark.executor.maxNumFailures` effectively high enough to disable the global shutdown mechanism. - Use `spark.executor.failuresValidityInterval` to periodically clear accumulated failure records. - Rely on query-scoped controls: - `spark.task.maxFailures` stops tasks that repeatedly fail due to OOMs or exceptions. - `spark.stage.maxConsecutiveAttempts` stops jobs whose stages repeatedly fail, such as from shuffle-fetch errors. - These limits must be tuned carefully: overly aggressive values can cause healthy queries to fail during temporary infrastructure problems. - With this approach, executor failures terminate the problematic query rather than the entire Spark Connect server. ## Protecting Driver Memory from Large Results - Spark Connect streams query results through the Driver, so a large `collect()` can threaten Driver memory. - `spark.driver.maxResultSize` aborts an action when accumulated task results exceed the configured limit. - The limit is checked before large executor-side results are fetched into Driver memory. - The default 1 GB value assumes a single workload; in a multi-session server, it should be reduced or tuned based on the number of concurrent queries. ## Replicating Spark Connect Servers - Configuration alone cannot prevent Driver OOMs, node failures, or other catastrophic events. - The stronger isolation boundary is a separate SparkContext. - Multiple identical Spark Connect replicas are deployed: - Each replica has its own Driver, SparkContext, and Executors. - A failure affects only the sessions assigned to that replica. - Other replicas can continue accepting sessions. - Replica-based deployment reduces the blast radius from the entire Spark Connect service to an individual server instance. ## Practical Recommendation For a multi-user Spark Connect service, disable global executor-failure shutdown, enforce query-level failure limits, protect Driver memory with `spark.driver.maxResultSize`, and use multiple replicas to contain unavoidable Driver or node failures. Scheduler pools can improve ordering, but they should not be treated as true resource isolation.
Read original(opens in new tab)