airbnbFeb 11, 2026

My Journey to Airbnb — Anna Sulkina (opens in new tab)

Anna Sulkina’s career journey moved from hardware diagnostics and frontend development into backend infrastructure and engineering leadership. Her experiences at Twitter taught her to design distributed systems for failure and to build consensus around transformative technologies like GraphQL. She joined Airbnb in 2022 because it aligned her passion for travel with an opportunity to strengthen developer infrastructure, organizational strategy, and engineering collaboration. ## Discovering Technology in Post-Soviet Ukraine - Sulkina grew up in Eastern Ukraine as the Soviet Union collapsed. - Her older brother introduced her to computers by bringing home hardware components and assembling a machine that loaded programs from a cassette player. - Seeing how individual components formed a working system inspired her to pursue technology. ## Learning English While Building Technical Skills - She studied programming at a Ukrainian university before immigrating to the United States. - Although she understood written English and knew how to program, communicating in English was initially more difficult than learning programming languages. - She took ESL classes while studying C++ and Java through Berkeley Extension. - Her first job was in hardware diagnostics at a five-person company. - A language barrier caused her to run out of time on a technical interview, but an interviewer familiar with her Berkeley class gave her another opportunity. - She eventually transitioned from C++ to Java, which became her primary language for many years. ## Moving Down the Stack and Into Leadership - Sulkina’s career progressed from hardware diagnostics to frontend, backend, and infrastructure engineering. - At the same time, she increasingly took on leadership responsibilities. - At Caymas Systems, her manager recognized her leadership potential and showed her the difference effective leadership makes. - At Comcast, she moved from individual contributor to engineering manager. - Coaching engineers, building software collaboratively, and developing high-performing teams convinced her that leadership was the right path. ## Lessons from Twitter’s Distributed Systems - During nearly nine years at Twitter, Sulkina advanced from first-line manager to director. - She worked through major operational events, including the “fail whale” period and the tweetstorm surrounding Ellen DeGeneres’s viral selfie. - Twitter’s transition from a monolith to microservices taught her that failure is inevitable in complex systems. - Resilient distributed systems must be designed to handle failures rather than assuming failures can be prevented. - Her cultural lesson involved turning promising ideas into adopted technologies. - She helped bootstrap Twitter’s GraphQL API, replacing legacy REST services. - The effort required leadership support, cross-team consensus, and stakeholder alignment, but ultimately improved product teams’ development velocity. ## Choosing Airbnb - Airbnb contacted Sulkina in 2022, when she felt ready to move beyond a well-established organization at Twitter. - The company appealed to her because it combined her professional interests with her personal passion for travel; she had been an Airbnb guest since 2013. - Airbnb’s Developer Platform organization had strong work happening in separate silos but needed clearer strategy, direction, and trust across engineering. - Sulkina began by clarifying the organization’s purpose and future direction. - Her early priorities included strengthening the organization, coaching leaders, and creating alignment within the team and with the teams it supported. - Over the following years, this work produced a high-performing organization with clearer strategy, stronger execution, and a focus on delivering business value. Sulkina’s story emphasizes that technical growth, organizational leadership, and personal motivation can reinforce one another. Her experience suggests that successful engineering leaders design for failure, invest in alignment, and use clear strategy to turn fragmented efforts into meaningful platform-wide impact.

distributed-systemsmicroservicesjavagraphql+3

airbnbOct 9, 2025

From Static Rate Limiting to Adaptive Traffic Management in Airbnb’s Key-Value Store (opens in new tab)

Airbnb evolved Mussel’s QoS system from static, per-client QPS limits into adaptive traffic management designed to maximize goodput. The newer approach accounts for the actual cost of requests, prioritizes critical workloads under stress, and detects hot keys or attack traffic before they overwhelm storage. Together, resource-aware quotas and real-time load shedding provide stronger protection against traffic spikes, uneven workloads, and DDoS-like bursts. ## Why Static QPS Limits Fell Short - Mussel is a multi-tenant key-value store serving millions of point and range reads across Airbnb. - Its original Redis-backed limiter assigned each client a fixed requests-per-second quota. - Requests exceeding the quota received HTTP 429 responses. - This model worked when backend effort roughly matched request count. - As usage grew, it could not account for: - The difference between a cheap one-row lookup and a 100,000-row scan. - Hot keys accessed by many clients simultaneously. - Localized storage-shard overload that affected unrelated traffic. - Sudden events such as bot floods, DDoS attacks, or large uploads. ## Resource-Aware Rate Control - Mussel replaced raw request counting with request units (RU), which represent estimated backend work. - RU calculations incorporate: - Fixed per-request overhead. - Rows and payload bytes processed. - Request latency, which distinguishes cached operations from disk-heavy ones. - The system uses calibrated linear formulas for reads and writes, with weights based on compute, network, and disk-I/O measurements. - Dispatchers debit a local token bucket according to each request’s RU cost rather than charging every request equally. - Periodic RU refills preserve simple, static quotas while making them more proportional to actual resource consumption. - Requests are rejected with HTTP 419 when the RU bucket is exhausted. - Load shedding remains separate, allowing latency-based protection to react dynamically without changing the underlying quota-refill mechanism. ## Load Shedding Under Sudden Stress - RU rate limiting smooths normal traffic but may react too slowly to rapidly changing workloads. - Mussel adds a load-shedding layer based on: - Traffic criticality. - A real-time latency ratio. - A CoDel-inspired queue-management policy. - Each dispatcher compares long-term p95 latency with short-term p95 latency. - A ratio near 1.0 indicates stable performance; a drop toward 0.3 signals rapidly increasing latency. - When stress crosses the threshold: - The system raises the effective RU cost for a designated lower-priority client class. - That class’s token bucket drains faster, causing its traffic to back off. - If conditions worsen, the penalty expands to additional classes. - Critical workloads, such as customer support and trust-and-safety traffic, can remain responsive while less important traffic is reduced. - The latency estimate uses the constant-memory P² algorithm, avoiding raw sample storage and cross-node coordination. ## Hot-Key Detection and DDoS Protection - Client-level quotas cannot prevent overload when many clients request the same popular key. - Mussel therefore detects skewed access patterns in real time. - When duplicate requests target a hot key, the system can protect storage by: - Serving responses from cache. - Coalescing identical requests before they reach the backend. - This approach protects the underlying shard whether the traffic comes from legitimate popularity, automation, or a DDoS burst. Mussel’s experience suggests that mature multi-tenant services should move beyond fixed QPS limits. Combining resource-based accounting, priority-aware load shedding, and hot-key mitigation provides a more effective way to preserve reliability while maximizing useful work during unpredictable traffic conditions.

distributed-systemsk8sredisrate-limiting+4

airbnbSep 15, 2025

Viaduct, Five Years On: Modernizing the Data-Oriented Service Mesh (opens in new tab)

Viaduct, Airbnb’s data-oriented service mesh, has evolved substantially over five years while retaining its core model: a central schema, hosted business logic, and re-entrant composition through GraphQL. Its usage has grown eightfold, supporting more than 130 teams and over 1.5 million lines of production code, without increasing operational overhead. Viaduct Modern now aims to simplify its developer API and establish stronger architectural boundaries, alongside the project’s release as open source. ## Adoption and Evolution - Viaduct traffic has increased by a factor of eight since 2020. - More than 130 teams now host code in Viaduct, supported by hundreds of weekly active developers. - The hosted codebase has grown to over 1.5 million lines, with roughly the same amount of test code. - Operational overhead has remained constant, incident-minutes have been cut in half, and costs have grown linearly with QPS. - Viaduct is now available as open-source software. ## Core Principles That Remain - **Central schema:** Viaduct provides one integrated schema connecting domains across Airbnb. - More than 75% of requests are internal. - The schema is developed by many teams but exposed as a connected graph. - **Hosted business logic:** Teams run business logic directly in Viaduct rather than maintaining separate microservices. - This reduces operational overhead and can allow standalone services to be retired. - Viaduct provides a serverless environment so developers can focus on application logic. - **Re-entrancy:** Hosted logic composes with other hosted logic through GraphQL fragments and queries. - This supports modularity. - It helps avoid the tightly coupled structure and maintenance problems associated with traditional monoliths. ## Problems with the Earlier Design - Viaduct’s APIs evolved reactively in response to individual use cases. - Multiple mechanisms emerged for accomplishing similar tasks, creating confusion for developers. - Some capabilities were well supported while others were not. - The framework’s layers had loose, inconsistent interfaces. - The boundary between Viaduct and hosted application code was weak. - These issues made framework improvements increasingly risky because changes could disrupt existing users. ## Simplifying the Tenant API - Viaduct Modern overhauls the developer-facing API and execution engine. - The new Tenant API reduces the implementation choices to two mechanisms: - **Node resolvers** - **Field resolvers** - The choice is determined by the schema rather than by ad hoc behavioral distinctions. - Resolver APIs have been unified wherever possible. - The goal is a smaller, more consistent surface that preserves successful ideas from the old API while removing unnecessary alternatives. ## Tenant Modules and Re-Entrant Composition - Viaduct uses modules and re-entrancy to provide boundaries similar to service definitions and RPC APIs in microservice architectures. - A tenant module combines: - Schema owned by a team - The code implementing that schema - Modules can create rich connections in the shared graph, but direct code dependencies between teams are discouraged. - Instead, teams declare their data requirements through GraphQL fragments and queries. ### Example: Extending the `User` Type - A Core User team owns the base `User` type and resolves fields such as `firstName` and `lastName`. - A Messaging team can extend `User` with a `displayName` field. - Its resolver declares that it needs `firstName` and `lastName`. - Messaging does not depend directly on Core User’s implementation or need to know where those fields originate. - This declarative model lets teams collaborate through the schema while preserving ownership and modularity. ## Framework Modularity - Viaduct Modern also restructures the framework itself. - The system consists of: - The GraphQL execution engine - The Tenant API - Hosted application code - Historically, the interfaces between these layers were weak, making performance and reliability improvements difficult to introduce safely. - The redesign focuses on stronger abstraction boundaries so the framework can evolve independently of application code. Viaduct’s modernization is intended to preserve its centralized, data-oriented model while making development simpler and framework evolution safer. The open-source release provides an opportunity for other organizations to evaluate or adopt this approach to schema-driven, modular service composition.

distributed-systemsserverlessmicroservicesapi-design+3