data-locality

1 posts

cloudflare

Improve global upload performance with R2 Local Uploads (opens in new tab)

R2 Local Uploads improves global upload performance by first writing object data near the client, then asynchronously copying it to the bucket’s region. Objects become immediately available and remain strongly consistent during replication. Cloudflare reports up to a 75% reduction in upload request duration for cross-region uploads. ## Faster Global Uploads - Local Uploads targets `PutObject` and `UploadPart` requests made far from the bucket’s location. - Synthetic tests showed median upload TTLB dropping from about 2 seconds to 500 milliseconds. - Tests used 5 MB objects uploaded from Western North America to an Asia-Pacific bucket at roughly 20 requests per second. - The feature is available in open beta and can be enabled in the Cloudflare Dashboard or with: ```bash npx wrangler r2 bucket local-uploads enable [BUCKET] ``` ## The Cross-Region Distance Problem - R2 requests enter through a globally distributed Gateway Worker, which handles authentication and routing. - Object metadata is managed by a distributed Durable Object Metadata Service. - Encrypted object data is stored in R2’s distributed storage infrastructure. - Without Local Uploads, streamed data must travel to the bucket’s region before the upload can complete. - Long-distance transfers can increase latency and introduce upload variability or reliability issues. ## How Local Uploads Works - If the client and bucket are in the same region, R2 uses its normal storage flow. - If they are in different regions: - Data is initially written to storage near the client. - Metadata is published in the bucket’s region. - The object becomes readable as soon as the local write completes. - Background replication later copies the data to the bucket’s primary region. - There is no read-unavailability window while replication is in progress. - Local Uploads is unavailable for jurisdiction-restricted buckets, including EU and FedRAMP buckets. ## When to Use It - Applications have users or devices distributed across multiple regions. - Upload speed and reliability are important. - You want faster writes without moving the bucket’s primary location. - R2’s Metrics page can help identify regional request patterns through the “Request Distribution by Region” graph. ## Replication Architecture - R2 represents the background copy operation as a replication task. - Cloudflare Queues process these tasks asynchronously. - Queues provide: - Rate control for replication. - Automatic retries. - Dead-letter queue support for failures. - Sharding across multiple queues for each storage region. - When publishing object metadata, R2 atomically: - Stores the object metadata. - Creates a pending-replica key describing unfinished replication work. - Creates a timestamp-based replication marker that determines when the task enters a queue. - The pending-replica record includes the replication plan, source and destination locations, mode, priority, and whether the source can be deleted after successful replication. Local Uploads is a strong fit for globally distributed upload-heavy workloads. Enable it when cross-region write latency matters, while keeping in mind the restriction on jurisdiction-constrained buckets.