Inside the feature store powering real-time AI in Dropbox Dash
Dropbox Dash’s ranking system depends on a hybrid feature store that can combine real-time user behavior with large-scale historical data. Because Dropbox operates across on-premises and cloud environments, and because each query can trigger thousands of feature lookups, off-the-shelf systems could not meet its latency, scale, and integration requirements. The resulting architecture uses Feast for orchestration, Spark for computation, Dynovault for low-latency storage, and a custom Go serving layer, achieving roughly 25–35 ms p95 latency while keeping features fresh. ## Goals and Requirements - Dash ranks documents, images, and conversations using behavioral, contextual, and real-time signals. - A single query can fan out into thousands of feature lookups across many candidate files. - The feature store needed to: - Support sub-100 ms search latency. - Reflect user actions within seconds or minutes. - Bridge Dropbox’s on-premises services and Spark-based cloud infrastructure. - Handle both streaming-style updates and batch computations. - Let engineers develop features without managing serving and orchestration details. ## Choosing a Hybrid Architecture - Dropbox evaluated Feast, Hopsworks, Featureform, Feathr, Databricks, and Tecton. - Feast was selected because: - It separates feature definitions from infrastructure concerns. - Engineers can focus on PySpark transformations. - Its modular adapter system supports existing Dropbox infrastructure. - Feast’s DynamoDB adapter enabled integration with Dynovault, Dropbox’s DynamoDB-compatible storage system. - The architecture combines: - Feast for orchestration and serving APIs. - Spark jobs for feature computation and ingestion. - Cloud storage for offline indexing and data management. - Dynovault for online, low-latency lookups. - A custom Go service replacing Feast’s Python online serving path. - Dynovault is colocated with inference workloads and provides approximately 20 ms client-side latency. - Monitoring covers job failures, feature freshness, and data lineage. ## Replacing Python with Go for Low Latency - The initial Feast-based Python service struggled under heavy concurrency. - CPU-bound JSON parsing and Python’s Global Interpreter Lock became bottlenecks. - A multi-process design helped temporarily but introduced coordination overhead. - The serving layer was rewritten in Go using: - Lightweight goroutines. - Shared memory. - Faster JSON parsing. - The Go service now handles thousands of requests per second. - It adds only about 5–10 ms beyond Dynovault latency and achieves roughly 25–35 ms p95 latency. ## Keeping Features Fresh - Fresh signals are essential for ranking quality; actions such as opening a document should influence subsequent searches quickly. - Fully real-time computation is impractical for features requiring large joins, aggregations, and historical context. - Dropbox therefore built a three-part ingestion strategy. - Batch ingestion handles complex, high-volume transformations using a medallion architecture. - Intelligent change detection updates only modified records rather than rewriting every feature. - This reduced online-store writes from hundreds of millions to fewer than one million per run and significantly shortened update time. ## Practical Takeaway The system demonstrates that a feature store does not need to be entirely off-the-shelf or entirely real-time. Combining a modular framework with custom serving, colocated storage, batch optimization, and freshness monitoring allowed Dropbox to meet demanding latency and scale requirements while keeping feature development manageable.
Read original(opens in new tab)