cloudflare-r2

3 posts

cloudflare

Dogfooding at scale: migrating cdnjs to Cloudflare’s Developer Platform (opens in new tab)

cdnjs now runs entirely on Cloudflare’s Developer Platform after a migration intended to improve maintainability rather than performance. Despite the rise of bundlers and modern JavaScript tooling, cdnjs still serves about 9 billion requests per day because it is free, familiar, immutable, auditable, and widely used by both developers and AI coding assistants. The migration replaces a fragmented GCP, GitHub, VM, and Cloudflare setup with a unified architecture built around Workers, R2, Workflows, Queues, D1, KV, Cache, and Containers. ## cdnjs’s Scale and Continued Relevance - cdnjs serves roughly: - 108,000 requests per second - 9 billion requests per day - Traffic across more than 330 Cloudflare data centers - A 98.6% cache-hit rate - It is used by approximately 12% of websites and holds a 48.3% share of the JavaScript CDN market. - Its simple `<script>`-tag model remains popular because: - URLs and versions are consistent and immutable. - Libraries are available without accounts, API keys, or rate limits. - Files include Subresource Integrity hashes. - The project is open source and community-driven. - AI assistants frequently generate cdnjs URLs because they appear throughout years of tutorials, documentation, GitHub repositories, and Stack Overflow answers. ## Why the Existing Architecture Became a Problem - Cloudflare moved cdnjs file serving to Workers and KV in 2020, improving resilience and enabling pre-compressed Brotli and gzip assets. - The publishing pipeline remained on GCP because Cloudflare previously lacked suitable tools for: - Fetching large package archives - Running CPU-intensive processing - Coordinating multi-step jobs over hours - The old pipeline combined GCP Functions, Google Cloud Storage, Pub/Sub, a git-sync VM, GitHub, Workers KV, and a bare-metal origin. - New features and bug fixes required coordinating deployments across multiple platforms, while observability required manually stitching together unrelated logs. ## Problems with the Legacy Pipeline - **No shared tracing** - Package updates could pass through several systems without a common correlation ID. - Partial failures could leave KV updated while GitHub remained stale, with no alert indicating the divergence. - **Split-brain storage** - File content existed both in Workers KV and a GitHub repository. - Neither system was cleanly authoritative, making reconciliation difficult. - **Storage-driven orchestration** - GCP Cloud Functions triggered one another through object-created events. - Storage effectively acted as a message queue without dead-letter handling, backlog visibility, or reliable replay. - **Operational fragmentation** - npm polling required 26 separately deployed Cloud Functions, one for each alphabetic shard. - Health monitoring required checking all 26 deployments and their logs. - **An oversized GitHub repository** - The repository exceeded 1.1 TB of packed storage. - GitHub could no longer generate archive downloads reliably. - Cloning and forking became impractical. - A 274-entry `.gitignore` accumulated to exclude releases the pipeline could not reject properly. - **Security overhead** - Cloud Functions, a VM, container images, storage buckets, and service-account credentials all required patching, auditing, and protection. - Retiring these components reduced the attack surface and eliminated recently exposed vulnerabilities. ## The New Cloudflare-Based Architecture - The rebuilt system uses Cloudflare’s Developer Platform end to end. - **R2** becomes the single source of truth for file content. - It can store large assets that previously did not fit comfortably in KV, including source maps, large bundles, and font packages. - Its S3-compatible API makes the catalog accessible to external tools and mirrors. - The broader platform combines: - Workers for request handling - Workflows for orchestration - Queues for reliable asynchronous processing - R2 for durable object storage - D1, KV, Workers Cache, and Containers for supporting services - Centralizing the pipeline should make processing state observable, reduce deployment complexity, and eliminate inconsistencies between edge storage and the GitHub repository. ## Practical Conclusion The cdnjs migration demonstrates that a globally critical, high-volume open-source service can evolve from a collection of legacy systems into a unified serverless platform. Its continued value comes not only from speed, but from being free, predictable, immutable, and easy for both humans and automated tools to consume.

cloudflare

Improve global upload performance with R2 Local Uploads (opens in new tab)

R2 Local Uploads improves global upload performance by first writing object data near the client, then asynchronously copying it to the bucket’s region. Objects become immediately available and remain strongly consistent during replication. Cloudflare reports up to a 75% reduction in upload request duration for cross-region uploads. ## Faster Global Uploads - Local Uploads targets `PutObject` and `UploadPart` requests made far from the bucket’s location. - Synthetic tests showed median upload TTLB dropping from about 2 seconds to 500 milliseconds. - Tests used 5 MB objects uploaded from Western North America to an Asia-Pacific bucket at roughly 20 requests per second. - The feature is available in open beta and can be enabled in the Cloudflare Dashboard or with: ```bash npx wrangler r2 bucket local-uploads enable [BUCKET] ``` ## The Cross-Region Distance Problem - R2 requests enter through a globally distributed Gateway Worker, which handles authentication and routing. - Object metadata is managed by a distributed Durable Object Metadata Service. - Encrypted object data is stored in R2’s distributed storage infrastructure. - Without Local Uploads, streamed data must travel to the bucket’s region before the upload can complete. - Long-distance transfers can increase latency and introduce upload variability or reliability issues. ## How Local Uploads Works - If the client and bucket are in the same region, R2 uses its normal storage flow. - If they are in different regions: - Data is initially written to storage near the client. - Metadata is published in the bucket’s region. - The object becomes readable as soon as the local write completes. - Background replication later copies the data to the bucket’s primary region. - There is no read-unavailability window while replication is in progress. - Local Uploads is unavailable for jurisdiction-restricted buckets, including EU and FedRAMP buckets. ## When to Use It - Applications have users or devices distributed across multiple regions. - Upload speed and reliability are important. - You want faster writes without moving the bucket’s primary location. - R2’s Metrics page can help identify regional request patterns through the “Request Distribution by Region” graph. ## Replication Architecture - R2 represents the background copy operation as a replication task. - Cloudflare Queues process these tasks asynchronously. - Queues provide: - Rate control for replication. - Automatic retries. - Dead-letter queue support for failures. - Sharding across multiple queues for each storage region. - When publishing object metadata, R2 atomically: - Stores the object metadata. - Creates a pending-replica key describing unfinished replication work. - Creates a timestamp-based replication marker that determines when the task enters a queue. - The pending-replica record includes the replication plan, source and destination locations, mode, priority, and whether the source can be deleted after successful replication. Local Uploads is a strong fit for globally distributed upload-heavy workloads. Enable it when cross-region write latency matters, while keeping in mind the restriction on jurisdiction-constrained buckets.

cloudflare

Introducing Moltworker: a self-hosted personal AI agent, minus the minis (opens in new tab)

Moltworker adapts the self-hosted Moltbot personal AI assistant to run on Cloudflare without requiring users to buy a dedicated Mac mini. It combines a Cloudflare Worker, Sandbox SDK, Browser Rendering, R2, AI Gateway, and Cloudflare Access to provide a globally available, secured deployment. The result is a managed infrastructure layer around Moltbot’s standard Gateway runtime while preserving its integrations and persistent state. ## Running a Personal Agent on Cloudflare - Cloudflare Workers increasingly supports Node.js APIs natively, reducing the need for compatibility hacks and making it easier to run existing JavaScript and TypeScript packages. - An internal test of the 1,000 most popular NPM packages found that only 15 relevant packages failed to run in Workers. - Although much of Moltbot runs inside a container, improved Workers compatibility is useful for building agent logic closer to users. - Cloudflare’s Developer Platform provides the main infrastructure components: - **Sandboxes** for securely running untrusted code. - **Browser Rendering** for automated headless browser interactions. - **R2** for persistent object storage. - Cloudflare’s global network for scalability and security. ## Moltworker Architecture - Moltworker consists of: - An entrypoint Worker serving as an API router and proxy. - Cloudflare Access protecting the Worker and administration interface. - A Sandbox container running Moltbot’s standard Gateway and integrations. - R2 for persistent storage. - This structure separates the public API and administrative layer from the isolated environment where the agent executes. ## AI Gateway Integration - Cloudflare AI Gateway proxies requests between Moltbot and AI providers. - It provides: - Centralized request visibility. - Cost monitoring, logs, and analytics. - Provider and model switching without changing Moltbot code. - Fallback providers or models for improved reliability. - Secrets can be managed through: - **Bring Your Own Key (BYOK)**, where provider credentials are stored centrally. - **Unified Billing**, where users purchase credits and Cloudflare handles provider billing. - Integration requires creating an AI Gateway instance, enabling a provider such as Anthropic, and setting `ANTHROPIC_BASE_URL`; Moltbot itself does not need code changes. ## Sandbox-Based Execution - The Sandbox SDK runs agent code in isolated environments built on Cloudflare Containers. - It provides simplified APIs for: - Executing commands. - Managing files and directories. - Running background processes. - Exposing services. - Executing code in contexts such as Python. - The SDK abstracts container lifecycle, networking, filesystem, and process-management concerns behind TypeScript APIs. Moltworker offers a way to run a capable personal AI agent online with managed security, storage, browser automation, and model access—without maintaining dedicated hardware.