cloudflareJun 22, 2026

How we found a bug in the hyper HTTP library (opens in new tab)

The Images binding’s migration to a local Unix-socket architecture exposed a rare race condition in Rust’s `hyper` HTTP library. Under slow-reader conditions, large image responses were truncated even though they returned `200 OK` and a full `Content-Length`, causing downstream processing or image decoding to fail. After six weeks of investigation, the issue was traced to premature socket shutdown and fixed with four lines of code. ## Images Bindings and the Request Path - Cloudflare’s Images service runs on Workers and uses `hyper` to manage HTTP connections. - The Images binding lets Workers send image data directly to the service, chain transformations, and receive the processed result as a stream. - The response path involved: - The Images service generating the complete encoded image. - `hyper` buffering the response. - Data moving through socket buffers managed by the kernel. - A client or intermediary reading the response. - If the reader was fast, `hyper` could flush the entire response and safely shut down the socket. - If the reader was slower, the socket’s outbound buffer filled, requiring `hyper` to pause and resume writing. ## Moving from FL to Local Unix Sockets - Initially, binding traffic passed through Cloudflare’s FL intermediary service. - In December 2025, the Images team replaced FL with an internal binding running on the same machine. - Unix sockets removed network and FL-processing overhead, including routing and DNS work. - The redesign improved performance and allowed the Images team to release binding changes independently. - The bug appeared within days of the rollout. ## Successful Responses with Truncated Bodies - The first report involved nested image-processing pipelines: - An inner Images binding composited large JPEG and PNG inputs from R2. - An outer URL-based pipeline resized, compressed, and transcoded the result. - The inner pipeline returned `200 OK` and a `Content-Length` for several megabytes, but delivered only a fraction of the body. - One response contained roughly 200 KB instead of the expected 3.3 MB. - The outer pipeline reported an end-of-file error because the body ended before the declared message length. - Depending on the image format, clients saw partially rendered images or completely broken images. ## Reproducing and Isolating the Race - Engineers recreated the nested setup, then removed layers until the failure occurred with the binding alone. - Batch testing produced failures reliably—for example, 19 of 25 requests in one run. - The amount of data received, approximately 200 KB, closely matched the production socket-buffer size. - This indicated that the failure was related to backpressure and socket-buffer exhaustion rather than the customer’s specific configuration. - Investigation eventually identified a race in `hyper` where the connection could be shut down before buffered response data had finished flushing. The incident demonstrates that HTTP success status codes do not guarantee complete response bodies when connection handling is incorrect. Systems streaming large payloads over sockets should test slow-reader and backpressure scenarios, and libraries should only close connections after all buffered data has been written.

race-conditioncloudflare-workersrusthttp+2

kakaoFeb 8, 2026

In Search of Lost Reports: Kakao (opens in new tab)

KIMS, Kakao’s internal SMS platform, experienced rare cases where vendors sent delivery reports successfully, yet messages remained stuck in `SENT` instead of becoming `REPORTED`. The cause was a race condition: a fast vendor’s report arrived before the API server had committed the message record. The investigation showed that an unnecessarily long transaction—especially for paid messages with billing-event processing—delayed persistence and allowed valid reports to be dropped. ## KIMS Message Processing Flow - KIMS processes roughly one million SMS messages per day across multiple IDC environments and external vendors. - The normal flow is: - Route the request to a suitable vendor. - Call the vendor and record the message as `SENT`. - Deliver the message to the recipient. - Receive the vendor’s delivery report. - Update the message to `REPORTED`. - These stages run asynchronously across separate services, so their execution order is not guaranteed. ## Discovering the Missing Reports - Some messages remained in `SENT` even though Report Server logs confirmed that delivery reports had arrived. - The issue affected only about `0.02%` of messages, making it difficult to reproduce in tests or local environments. - Two patterns emerged: - Missing reports were concentrated among messages sent through one particular vendor. - Paid messages were affected more often than free messages. ## The Race Condition - The problematic vendor returned reports unusually quickly: - Other vendors typically took more than one second. - This vendor averaged around 20 ms. - Missing-report cases averaged only about 8 ms. - The API server performed additional processing before committing the message record. - For paid messages, billing-event publication was included in the same `@Transactional` scope, making the transaction longer. - Consequently, the sequence could become: 1. API Server calls the vendor. 2. API Server performs billing-related processing. 3. The vendor delivers the message and immediately sends a report. 4. Report Server receives the report before the message row exists in the database. 5. Report Server treats the report as invalid and drops it. 6. API Server finally commits the message as `SENT`. - The report was not lost at the network or vendor level; it was discarded because the system’s write path had not completed. ## Reducing Transaction Scope - The first fix was to remove nonessential work from the main transaction. - Billing-event publication was moved to asynchronous processing using `@Async` and `@TransactionalEventListener`. - The transaction was reduced to the essential state change and database commit. - This advanced the average commit point by approximately 10 ms and significantly reduced report omissions. - It also avoided a dual-write anti-pattern in which an external Kafka event was published inside a database transaction that could later roll back. ## Reconsidering the Need for a Transaction The incident prompted a broader review of whether the transaction was needed at all. - **Atomicity:** The transaction contained only one database write, with no multi-table or cross-record operation requiring all-or-nothing rollback. - **Read isolation:** Metadata such as vendor quality metrics was updated only every few minutes, and using a slightly stale value was acceptable. The independently read tables did not require a single consistent snapshot. - **Write isolation:** JPA’s dirty checking kept the status change in the persistence context until transaction completion, delaying the actual database write. This delay was precisely what allowed the report to arrive first. The article therefore presents the transaction itself—not the vendor or report receiver—as a source of unnecessary latency and an architectural anti-pattern in this workflow. ## Practical Recommendation Use transactions only when their guarantees are required. Keep critical persistence paths short, move external events and nonessential processing after commit, and critically evaluate whether delayed commit semantics could allow asynchronous consumers to observe a missing record.

race-conditiondistributed-systemskafkamysql+4