Building Toss’s Device Farm (opens in new tab)
Nebula is Toss’s centralized device farm, turning real-device testing into a simple API call instead of a team-specific infrastructure project. It grew from 15 devices and one developer into a 24/7 platform with more than 100 devices, shared across the company. The team replaced Appium with a faster, stateless custom driver and invested heavily in real-time streaming, reliability, security, and compliance. ## From Team-Owned Farms to a Central Platform - Before Nebula, teams managed their own small farms using MacBooks or Mac minis. - Each team repeatedly solved the same problems: - Appium setup and maintenance - Device detection and recovery - OS-version compatibility - USB and hardware failures - Security and compliance - Most teams could manage only five to ten devices, and resources remained isolated. - Nebula centralized device operations so product teams could focus on testing rather than infrastructure. ## One API for Real Devices Nebula’s core goal is to let anyone control a physical device from anywhere through a single API. - Clients reserve a device and invoke actions such as clicks or text input. - Users do not need to know which host owns the device or configure ADB, Xcode, cables, or test servers. - The same API supports frontend tools, SDKs, CLIs, direct API clients, and automated systems. ## Four-Layer Architecture - **Clients:** Web interfaces, SDKs, CLIs, and direct API calls. - **Server:** Orchestrates device discovery, allocation, and test execution. - Kafka distributes execution requests. - Multiple runners consume work horizontally as demand grows. - `occupy`, `assign`, and `release` provide distributed locking so tests cannot interfere with one another. - **Agents:** Run on Android/Linux and iOS/Mac hosts, discover local devices, and forward server requests. - **Devices:** Each device has a controller server and runner that execute actions on the physical phone. ## Why Nebula Replaced Appium ### Faster execution - Nebula’s click and input operations were more than ten times faster than Appium in common cases. - Much of Appium’s latency comes from `waitForIdle`, which waits for the screen to stabilize before acting. - Disabling that wait narrows the difference to roughly two or three times. - Appium prioritizes robustness against changing screens, while Nebula prioritizes immediate interaction for live remote control. ### Stateless operation - Appium requires sessions that can take 15–40 seconds to initialize. - Session startup becomes increasingly fragile and difficult to manage at scale. - Nebula keeps device controllers pre-warmed and accepts stateless HTTP requests, eliminating session setup and reducing failure points. ### Company-specific customization Because Nebula owns its driver specification, it can provide: - A custom IME that preserves Korean and emoji input. - Toss-specific signal triggers. - App Center integration for installing pre-release builds. - Built-in enforcement of internal security policies. The driver uses Android ADB and UiAutomation, and Swift/XCTest on iOS. Its OpenAPI specification generates Go and TypeScript interfaces. ## Real-Time Interaction and Screen Mirroring Nebula needed users to watch and control remote devices simultaneously, not merely replay predefined test steps. ### Android - Instead of using `scrcpy` directly, the team adapted its encoding approach. - Android’s `SurfaceControl` creates a virtual display. - `MediaCodec` encodes the output as H.264. - A broadcaster distributes the stream to multiple browser viewers. ### iOS - iOS screen capture is constrained by USB access and cannot expose the display as freely as Android. - Existing approaches such as QVH and Appium MJPEG did not support simultaneous viewing and interaction. - Nebula developed a capture path inspired by QuickTime’s iOS capture mechanism without exclusively claiming the USB connection. - Both platforms now use H.264 streaming and broadcasting, allowing the entire device farm to be viewed in a browser. ## Security and Compliance - Centralization made it possible to apply consistent security requirements across every device. - The team worked with Toss’s security organization to define mobile-device compliance standards. - Policies are enforced at the platform and driver levels rather than relying on individual developers. - Teams can therefore test on devices that already meet the company’s security requirements. ## Operating Hundreds of Devices Nebula must keep hardware and software running continuously. - **Hardware operations** - USB cables, hubs, and power delivery had to be tested and designed for large-scale use. - Physical failures still require human intervention. - Redundancy is being introduced to reduce service interruptions. - **Software operations** - Controllers and mirroring processes are orchestrated across Mac mini and Linux hosts. - Dead processes are automatically recovered. - Server, agent, controller, and mirroring deployments are performed without interrupting user tests. - Monitoring and observability cover devices, processes, and server performance. ## An Internal Ecosystem Built on the API Nebula has evolved from a device provider into shared testing infrastructure. - Web console for no-code interaction and test-step creation. - SDKs for writing E2E tests. - CLI support for terminals, CI/CD, and local AI agents. - Automated log verification during device interaction. - AI agents that plan and execute tests dynamically. - Direct API access for teams with specialized needs. Reported benefits include faster Appium migration, lower barriers to regression testing, and reducing manual verification from 30–40 minutes to under 10 minutes. Nebula’s main lesson is that a stable, simple API can become the foundation for a much broader testing ecosystem. Centralizing device infrastructure, replacing unsuitable abstractions, and treating real-time operation, compliance, and reliability as first-class requirements enabled Toss to scale physical-device testing across the company.