toss·Jun 19, 2026·5 min readCurated summary

Spark Connect on Kubernetes #1: Building a Robust Spark Connect

Toss Securities operates Spark Connect as a production service on Kubernetes so analysts and engineers can use Spark without complex setup. Spark Connect replaces per-application Drivers with long-running servers, making clients lighter and sessions faster, but it also introduces shared-failure and resource-contention problems. The post argues that production reliability requires both reducing server-wide failure triggers and distributing sessions across multiple replicas. ## How Classic Spark Works - Spark consists of: - A **Driver**, which plans jobs, schedules tasks, and collects results. - **Executors**, which perform the distributed computations. - In Classic Spark: - **Client mode** runs the Driver inside the client process. - **Cluster mode** launches the Driver in the cluster for each submitted application. - Both modes assume that one application has one Driver and one workload. - Clients also need Spark libraries, JVM support, and configuration. ## What Spark Connect Changes - Spark Connect turns the Driver into a pre-started, long-running server. - Clients send unresolved logical plans encoded with Protocol Buffers over gRPC. - The server handles analysis, optimization, scheduling, and execution. - Results are streamed back using Arrow. - This resembles a database accessed through JDBC. ### Benefits - **Thin clients:** Clients do not need the full Spark runtime or JVM. - **Language and platform independence:** Notebooks, BI tools, SQL clients, and different programming languages can use the same server. - **Fast session creation:** Sessions connect to an already-running server. - **Better client-failure tolerance:** A disconnected notebook does not necessarily terminate server-side work. ## Problems Created by Shared Long-Running Servers Spark’s internal design often assumes “one application equals one workload.” Sharing one application across many users breaks that assumption. ### A Shared Driver Becomes a Single Point of Failure - Multiple sessions share one `SparkContext` and Driver JVM. - A Driver failure terminates all sessions, jobs, and caches attached to it. - Spark’s global `spark.executor.maxNumFailures` counter can shut down the entire application after enough executor failures. - Because all sessions contribute to the same counter, one user’s unstable or memory-intensive query can terminate unrelated users’ workloads. - The counter is global, persists over time, and is separate from per-query task-level fault tolerance such as `spark.task.maxFailures`. ### Resource Contention and Scheduling Limits - `newSession()` isolates SQL state and namespaces, but not CPU, memory, or executors. - Heavy workloads can occupy all task slots and delay smaller queries. - FIFO scheduling favors earlier jobs, and Spark does not preempt tasks already using slots. - Fair Scheduler pools can influence task-slot ordering, but cannot provide true CPU or memory isolation. - Spark Connect does not automatically propagate `spark.scheduler.pool` to the server-side execution thread, causing queries to fall into the default pool unless the server explicitly assigns pools. - Actual resource isolation must therefore be implemented outside Spark’s task scheduler. ### Fixed Server Capacity - A server’s image, Driver and Executor resources, and Spark configuration are fixed when it starts. - Dynamic Resource Allocation can adjust executor counts, but cannot change the server’s basic specification. - Flexible scaling and team-level isolation require creating or replacing servers, which is addressed in a later part of the series. ## Reducing Server-Wide Failures Before adding replicas, Toss Securities reduces the chance that one bad query can kill the shared server. - Set `spark.executor.maxNumFailures` effectively high enough to disable the global shutdown mechanism. - Use `spark.executor.failuresValidityInterval` to periodically clear accumulated failure records. - Rely on query-scoped controls: - `spark.task.maxFailures` stops tasks that repeatedly fail due to OOMs or exceptions. - `spark.stage.maxConsecutiveAttempts` stops jobs whose stages repeatedly fail, such as from shuffle-fetch errors. - These limits must be tuned carefully: overly aggressive values can cause healthy queries to fail during temporary infrastructure problems. - With this approach, executor failures terminate the problematic query rather than the entire Spark Connect server. ## Protecting Driver Memory from Large Results - Spark Connect streams query results through the Driver, so a large `collect()` can threaten Driver memory. - `spark.driver.maxResultSize` aborts an action when accumulated task results exceed the configured limit. - The limit is checked before large executor-side results are fetched into Driver memory. - The default 1 GB value assumes a single workload; in a multi-session server, it should be reduced or tuned based on the number of concurrent queries. ## Replicating Spark Connect Servers - Configuration alone cannot prevent Driver OOMs, node failures, or other catastrophic events. - The stronger isolation boundary is a separate SparkContext. - Multiple identical Spark Connect replicas are deployed: - Each replica has its own Driver, SparkContext, and Executors. - A failure affects only the sessions assigned to that replica. - Other replicas can continue accepting sessions. - Replica-based deployment reduces the blast radius from the entire Spark Connect service to an individual server instance. ## Practical Recommendation For a multi-user Spark Connect service, disable global executor-failure shutdown, enforce query-level failure limits, protect Driver memory with `spark.driver.maxResultSize`, and use multiple replicas to contain unavoidable Driver or node failures. Scheduler pools can improve ordering, but they should not be treated as true resource isolation.

Read original(opens in new tab)

KubernetesApache SparkGrpcFault Tolerance+4

toss·Mar 17, 2026·3 min readCurated summary

Embracing the Software 3.0 Era

Software 3.0 replaces hand-written rules with natural-language instructions to LLMs, but models alone cannot reliably perform real-world work. The missing piece is the harness: tools, context, and environments that connect an LLM to codebases, commands, databases, and users. Claude Code illustrates how familiar Software 1.0 architecture can guide agent design while adding a new capability—asking humans for judgment when uncertainty arises. ## From Software 1.0 to Software 3.0 - **Software 1.0:** Developers explicitly write logic using languages such as Python, Java, or C++. - **Software 2.0:** Data and training produce neural-network weights that function as the program. - **Software 3.0:** Prompts and natural-language instructions direct LLM behavior. - Karpathy’s central claim is that Software 3.0 is increasingly absorbing both traditional code and trained models. ## Harnesses Make LLMs Useful - A raw LLM cannot independently read a codebase, execute commands, modify files, or access databases. - A **harness** supplies the tools and environment needed to turn model capability into practical work. - Claude Code is presented as a harness for Claude: it transforms a language model into an agent capable of completing and shipping tasks. ## Mapping Agent Concepts to Layered Architecture The terminology of agent systems can be understood through familiar Software 1.0 design patterns: - **Slash commands → Controllers** - They serve as entry points for user requests, such as `/review` or `/refactor`. - **Sub-agents → Service layer** - They coordinate multiple skills to complete a workflow. - Each sub-agent has an independent context and acts as a self-contained unit of work. - **Skills → Domain components** - Each skill should have one focused responsibility, such as reviewing code, generating tests, or writing documentation. - **MCP → Infrastructure or adapters** - MCP provides abstraction boundaries for external systems such as APIs and databases. - **CLAUDE.md → Project constitution** - It records stable project information: technology choices, conventions, and build commands. - Frequently changing task details should be provided through the conversation or injected into an agent’s context instead. ## Agent Design Has Familiar Anti-Patterns Traditional code smells also apply to agent systems: - **Feature Envy:** A skill relies excessively on another skill’s data. - **Duplication:** Prompts are copied across multiple skills. - **Long Method:** A single sub-agent performs an overly long sequence of many skills. - Clear boundaries, single responsibility, and limited coupling remain valuable. ## The Difference: Agents Can Ask Humans Layered architecture generally requires every failure and edge case to be handled through predefined exceptions, policies, or branches. - Traditional code must decide what to do when an unusual case occurs. - An agent using human-in-the-loop interaction can pause and ask the user for clarification. - In this model, exceptions become questions, allowing the agent to continue after receiving a decision. Agents should ask when: - An action is difficult to reverse, such as deletion or deployment. - Several valid options exist without a clear best choice. - The decision has significant consequences. They should proceed automatically when: - The operation is safely repeatable. - Existing conventions provide a clear answer. - The action is easy to undo. ## What Carries Forward into Software 3.0 The new paradigm does not make established engineering practices irrelevant. - Move away from explicitly coding every possible rule and edge case. - Do not reduce LLMs to simple autocomplete tools. - Preserve layered design, single responsibility, abstraction, dependency management, and interface design. - Continue emphasizing testability, debugging, code review, and iterative improvement. The practical approach is to combine Software 3.0’s flexible reasoning with Software 1.0’s architecture and engineering discipline, while giving agents a clear way to involve humans when decisions require judgment.

Read original(opens in new tab)

KubernetesLarge Language ModelsReactTypeScript+4

toss·Feb 9, 2026Original article

From Perimeter Security to Zero (opens in new tab)

Toss Payments transformed its security infrastructure from a vulnerable, single-layered legacy system into a robust "Defense in Depth" architecture spanning hybrid IDC and AWS environments. By integrating advanced perimeter defense, internal server monitoring, and container runtime security, the team established a comprehensive framework that prioritizes visibility and continuous verification. This four-year journey demonstrates that modern security requires moving beyond simple boundary protection toward a proactive, multi-layered strategy that assumes breaches can occur. ### Perimeter Defense and SSL/TLS Visibility * Addressed the critical visibility gap in legacy systems by implementing dedicated SSL/TLS decryption tools, allowing the team to analyze encrypted traffic for hidden malicious payloads. * Established a hybrid security architecture using a combination of physical DDoS protection, IPS, and WAF in IDC environments, complemented by AWS WAF and AI-based GuardDuty in the cloud. * Developed a collaborative merchant response process that moves beyond simple IP blocking; the system automatically detects malicious traffic from partners and provides them with detailed vulnerability reports and remediation guides (e.g., specific SQL injection points). ### Internal Network Security and "Assume Breach" Monitoring * Implemented **Wazuh**, an open-source security platform, in IDC environments to monitor lateral movement, collect centralized logs, and perform file integrity checks across diverse operating systems. * Leveraged **AWS GuardDuty** for intelligent threat detection in the cloud, focusing on malware scanning for EC2 instances and monitoring for suspicious process activities. * Established automated detection for privilege escalation and unauthorized access to sensitive system files, such as tracking instances where root privileges are obtained to modify the `/etc/passwd` file. ### Container Runtime Security as the Final Defense * Adopted **Falco**, a CNCF-hosted runtime security tool, to protect Kubernetes environments by monitoring system calls (syscalls) in real-time. * Configured specific security rules to detect "container escape" attempts, unauthorized access to sensitive files like `/etc/shadow`, and the execution of new or suspicious binaries within running containers. * Integrated **Falco Sidekick** to manage security events efficiently, ensuring that anomalous behaviors at the container level are instantly routed to the security team for response. ### Zero Trust and Continuous Verification * Shifted toward a Zero Trust model for the internal work network to ensure that all users and devices are continuously verified regardless of their location. * Focused on implementing dynamic access control and the principle of least privilege to minimize the potential impact of credential theft or device compromise. Organizations operating in hybrid cloud environments should move away from relying on a single perimeter and instead adopt a multi-layered defense strategy. True security resilience is achieved by gaining deep visibility into encrypted traffic and maintaining granular monitoring at the server and container levels to intercept threats that inevitably bypass initial defenses.

KubernetesAWSZero TrustHybrid Cloud+4

toss·Jan 20, 2026Original article

The story of how I destroyed (opens in new tab)

Toss Payments modernized its inherited legacy infrastructure by building an OpenStack-based private cloud to operate alongside public cloud providers in an Active-Active hybrid configuration. By overcoming extreme technical debt—including servers burdened with nearly 2,000 manual routing entries—the team achieved a cloud-agnostic deployment environment that ensures high availability and cost efficiency. The transformation demonstrates how a small team can successfully implement complex open-source infrastructure through automation and the rigorous technical internalization of Cluster API and OpenStack. ### The Challenge of Legacy Networking - The inherited infrastructure relied on server-side routing rather than network equipment, meaning every server carried its own routing table. - Some legacy servers contained 1,997 individual routing entries, making manual management nearly impossible and preventing efficient scaling. - Initial attempts to solve this via public cloud (AWS) faced limitations, including rising costs due to exchange rates, lack of deep visibility for troubleshooting, and difficulties in disaster recovery (DR) configuration between public and on-premise environments. ### Scaling OpenStack with a Two-Person Team - Despite having only two engineers with no prior OpenStack experience, the team chose the open-source platform to maintain 100% control over the infrastructure. - The team internalized the technology by installing three different versions of OpenStack dozens of times and simulating various failure scenarios. - Automation was prioritized using Ansible and Terraform to manage the lifecycle of VMs and load balancers, enabling new instance creation in under 10 seconds. - Deep technical tuning was applied, such as modifying the source code of the Octavia load balancer to output custom log formats required for their specific monitoring needs. ### High Availability and Monitoring Strategy - To ensure reliability, the team built three independent OpenStack clusters operating in an Active-Active configuration. - This architecture allows for immediate traffic redirection if a specific cluster fails, minimizing the impact on service availability. - A comprehensive monitoring stack was implemented using Zabbix, Prometheus, Mimir, and Grafana to collect and visualize every essential metric across the private cloud. ### Managing Kubernetes with Cluster API - To replicate the convenience of Public Cloud PaaS (like EKS), the team implemented Cluster API to manage the Kubernetes lifecycle. - Cluster API treats Kubernetes clusters themselves as resources within a management cluster, allowing for standardized and rapid deployment across the private environment. - This approach ensures that developers can deploy applications without needing to distinguish between the underlying cloud providers, fulfilling the goal of "cloud-agnostic" infrastructure. ### Practical Recommendation For organizations dealing with massive technical debt or high public cloud costs, the Toss Payments model suggests that a "Private-First" hybrid approach is viable even with limited headcount. The key is to avoid proprietary black-box solutions and instead invest in the technical internalization of open-source tools like OpenStack and Cluster API, backed by a "code-as-infrastructure" philosophy to ensure scalability and reliability.

KubernetesDevOpsInfrastructure As CodeHigh Availability+4

toss·Jan 8, 2026Original article

Managing Thousands of API/ (opens in new tab)

Toss Payments manages thousands of API and batch server configurations that handle trillions of won in transactions, where a single typo in a JVM setting can lead to massive financial infrastructure failure. To solve the risks associated with manual "copy-paste" workflows and configuration duplication, the team developed a sophisticated system that treats configuration as code. By implementing layered architectures and dynamic templates, they created a testable, unified environment capable of managing complex hybrid cloud setups with minimal human error. ## Overlay Architecture for Hierarchical Control * The team implemented a layered configuration system consisting of `global`, `cluster`, `phase`, and `application` levels. * Settings are resolved by priority, where lower-level layers override higher-level defaults, allowing servers to inherit common settings while maintaining specific overrides. * This structure allows the team to control environment-specific behaviors, such as disabling canary deployments in development environments, from a single centralized directory. * The directory structure maps files 1:1 to their respective layers, ensuring that naming conventions drive the CI/CD application process. ## Solving Duplication with Template Patterns * Standard YAML overlays often fail when dealing with long strings or arrays, such as `JVM_OPTION`, because changing a single value usually requires redefining the entire block. * To prevent the proliferation of nearly identical environment variables, the team introduced a template pattern using placeholders like `{{MAX_HEAP}}`. * Developers can modify specific parameters at the application layer while the core string remains defined at the global layer, significantly reducing the risk of typos. * This approach ensures that critical settings, like G1GC parameters or heap region sizes, remain consistent across the infrastructure unless explicitly changed. ## Dynamic and Conditional Configuration Logic * The system allows for "evolutionary" configurations where Python scripts can be injected to generate dynamic values, such as random JMX ports or data fetched from remote APIs. * Advanced conditional logic was added to handle complex deployment scenarios, enabling environment variables to change their values automatically based on the target cluster name (e.g., different profiles for AWS vs. IDC). * By treating configuration as a living codebase, the team can adapt to new infrastructure requirements without abandoning their core architectural principles. ## Reliable Batch Processing through Simplicity * For batch operations handling massive settlement volumes, the team prioritized "appropriate technology" and simplicity to minimize failure points. * They chose Jenkins for its low learning curve and reliability, despite its lack of native GitOps support. * To address inconsistencies in manual UI entries and varying Java versions across machines, they standardized the batch infrastructure to ensure that high-stakes financial calculations are executed in a controlled, predictable environment. The most effective way to manage large-scale infrastructure is to transition from static, duplicated configuration files to a dynamic, code-centric system. By combining an overlay architecture for hierarchy and a template pattern for granular changes, organizations can achieve the flexibility needed for hybrid clouds while maintaining the strict safety standards required for financial systems.

KubernetesAWSInfrastructure As CodeGitops+3