techlist.io
KREN
cloudflare

Evolving Cloudflare’s Threat Intelligence Platform: actionable, scalable, and ETL-less (opens in new tab)

database-designcloudflare-workersgraphqlsqlitethreat-intelligence-platformsecopsetl-lessreal-time-analysis

Cloudflare’s Threat Intelligence Platform (TIP) is designed to turn massive volumes of security telemetry into actionable intelligence without relying on traditional ETL pipelines. Its sharded, SQLite-backed architecture uses Durable Objects and edge-based GraphQL to provide near-real-time analysis across millions of events. By combining automated telemetry with analyst investigations, the platform aims to help security teams understand threats and block them proactively.

Motivation for Building the Platform

  • Cloudflare began developing the TIP after launching Cloudforce One in 2022 and discovering that existing tools could not adequately track adversary infrastructure.
  • The platform models the full threat lifecycle, connecting:
    • Threat actors to malware
    • Cases to indicators
    • Events to broader campaigns
  • It is designed for:
    • Multiple datasets and tenants
    • Group-based and tenant-to-tenant sharing
    • Extensibility and edge-scale performance
    • Visual analysis and automated response
  • Cloudflare Workers allow the platform to evolve with the runtime and support features such as Smart Placement, higher CPU limits, and Hyperdrive.

Beyond the SIEM

  • The TIP complements rather than replaces a SIEM:
    • SIEMs focus on real-time log aggregation and alerting.
    • The TIP provides long-term retention, specialized threat schemas, and historical context.
  • Analysts can enrich alerts with:
    • Indicator history
    • Known threat-actor associations
    • Campaign relationships
    • Risk scores and intelligence context
  • Findings from analysts feed new indicators of compromise back into the platform.
  • This feedback loop keeps intelligence current and helps organizations move from reactive investigation to proactive defense.

Sharded Storage Without ETL Bottlenecks

  • Cloudflare distributes Threat Events across many logical shards instead of using one centralized database.
  • Each shard is a Durable Object with a private SQLite database, providing transactional consistency and avoiding a single database bottleneck.
  • Cloudflare Queues handle asynchronous ingestion, helping absorb high-volume attack spikes.
  • R2 stores data for long-term retention, while SQLite maintains a hot index for fast access.
  • Because data is available directly in the platform’s operational store, complex ETL pipelines and synchronization delays are avoided.

Parallel Queries at the Edge

  • GraphQL runs in the same Worker-based system that powers the Threat Events platform, keeping data live from ingestion through querying.
  • Queries are fanned out to relevant Durable Objects in parallel rather than executed against one large table.
  • The platform first verifies permissions and excludes shards that cannot contain matching events, such as shards outside the requested date range.
  • Results from multiple shards are aggregated with Promise.all, enabling low-latency searches across global datasets.
  • Smart Placement positions query Workers near the Durable Objects they access, reducing tail latency.

Cloudflare’s approach combines edge-native storage, parallel execution, and analyst-driven enrichment to make threat intelligence both scalable and actionable. The practical goal is a unified system that explains not only what is malicious, but also why it matters and how to automatically prevent it.