Scaling Security Insights: how we achieved a 10x increase in global scanning capacity (opens in new tab)
Security Insights needed a 10x throughput increase to scan all customers more frequently and detect risks sooner. The existing system was overwhelmed by Kafka backlogs, slow processing, database inefficiencies, and API timeouts. Cloudflare improved capacity by introducing parallel and lane-based processing, optimizing bulk database writes, and addressing regional latency between its API and database. ## Scaling Kafka Processing - Scans are scheduled and published to Apache Kafka. - Go-based checker services consume these messages, inspect accounts, zones, and DNS records, and send findings to an internal API. - Kafka’s partition ordering limits each consumer group to one active consumer per partition. - Slow messages could block all subsequent messages in the same partition. - Adding partitions was avoided because it would increase resource usage for shared Kafka brokers. ## Introducing Parallel Processing - Checkers were changed to consume messages in batches. - Each message in a batch is processed concurrently in its own goroutine. - This increased throughput without requiring additional Kafka partitions. - The trade-offs were higher memory usage and potentially more work to repeat after a process crash. ## Separating Slow and Fast Work - Some scans took seconds or milliseconds, while unusually large accounts or zones could take minutes or hours. - These slow messages caused head-of-line blocking for faster work. - Consumer groups and checkers were split into: - A fast lane for predictable, short-running scans - A slow lane for messages expected to require substantially more time - Fast-lane consumers skipped slow messages, allowing normal scans to continue without delay. ## Optimizing Postgres Writes - The API originally executed one insert/upsert transaction per insight. - A request containing up to 500,000 insights could therefore generate hundreds of thousands of database round trips. - Bulk insertion with `COPY` into a temporary table was tested but caused bloat in Postgres system tables. - The final hybrid approach used: - `UNNEST` for smaller batches - `COPY` for batches above a configured threshold - This delivered millisecond-level performance for small writes and completion within seconds for very large writes. ## Diagnosing API Timeouts - Client-side timeouts increased as scan volume grew. - Checkers sometimes spent 20–90% of their processing time waiting on a single API call. - Throughput initially rose but then deteriorated under heavy load. - The root cause was network latency: - Postgres was hosted in Portland, Oregon. - The API ran active-active in Portland and Amsterdam. - Requests routed to Amsterdam incurred roughly 50 milliseconds of network round-trip latency. - Amsterdam database queries held client connection-pool connections much longer—nearly three seconds on average versus about 10 milliseconds in Portland. - The connection pool became exhausted, causing requests to wait for available connections and creating uneven Kafka lag across partitions. Cloudflare’s results came from improving the full processing pipeline rather than relying on a single infrastructure change. Parallelize message handling, isolate slow workloads, batch database writes, and place latency-sensitive services close to their databases to achieve large throughput gains and more frequent security scanning.
