Idempotency

2 posts

cloudflare3 min readCurated summary

How we built saga rollbacks for Cloudflare Workflows

Cloudflare Workflows now supports saga rollbacks, letting each durable step declare how to compensate for its side effects if a later operation fails. This addresses partial failures in multi-step processes, such as refunding a debit when a subsequent credit cannot complete. Rollbacks execute in reverse order and preserve Workflow durability, while requiring the same idempotency safeguards as normal steps. ## The Saga Problem - Durable Workflows can retry steps and persist state, but completed external operations cannot always be directly undone. - In a bank transfer: - Bank A debits the sender. - Bank B fails to credit the recipient. - The original debit must be reversed with a new credit operation. - The pairing of a forward action and its semantic compensation is known as the saga pattern. ## Manual Compensation Before Rollbacks - Developers had to track which steps completed and write centralized `try`/`catch` logic. - Compensation had to: - Run only for completed operations. - Execute in reverse order. - Continue even if one rollback fails. - Remain durable and retryable. - This approach becomes increasingly complex as workflows gain more steps. ## Rollback Functions on `step.do()` - Rollback logic is now declared directly in the step’s options: ```js await step.do("debit-bank-a", debitFn, { rollback: async ({ output }) => refundFn(output.id), }); ``` - Each step carries its own undo operation, making compensation easier to maintain. - Rollbacks can use the original step output, such as a payment or transaction ID. - If a later step fails, previously registered rollback handlers run automatically in reverse step-start order. ## Idempotency and Partial Failures - Rollback functions must be idempotent because they may be retried. - External operations should use idempotency keys to prevent duplicate refunds, credits, or inventory releases. - A step that fails may still need compensation: - It could have modified an external system before failing. - The operation may have succeeded even though Workflows never received its result. - Rollback handlers must therefore handle `output === undefined`. - If user code catches an error and the Workflow continues, rollback does not immediately start. However, if the Workflow later fails, previously registered handlers can still run. ## Practical Usage - Developers pass an options object with a `rollback` function as the final argument to `step.do()`. - Rollbacks can reverse payments, release resources, or perform other compensating actions. - This removes the need for growing manual catch blocks and explicit rollback ordering while retaining durable execution behavior. Cloudflare’s rollback support is best suited to workflows involving external side effects. Developers should define compensation alongside every reversible step and make both forward and rollback operations safely repeatable.

Read original(opens in new tab)
tossOriginal article

From Legacy Payment Ledger to Scalable System (opens in new tab)

Toss Payments successfully modernized a 20-year-old legacy payment ledger by transitioning to a decoupled, MySQL-based architecture designed for high scalability and consistency. By implementing strategies like INSERT-only immutability and event-driven domain isolation, they overcame structural limitations such as the inability to handle split payments. Ultimately, the project demonstrates that robust system design must be paired with resilient operational recovery mechanisms to manage the complexities of large-scale financial migrations. ### Legacy Ledger Challenges * **Inconsistent Schemas:** Different payment methods used entirely different table structures; for instance, a table named `REFUND` unexpectedly contained only account transfer data rather than all refund types. * **Domain Coupling:** Multiple domains (settlement, accounting, and payments) shared the same tables and columns, meaning a single schema change required impact analysis across several teams. * **Structural Limits:** A rigid 1:1 relationship between a payment and its method prevented the implementation of modern features like split payments or "Dutch pay" models. ### New Ledger Architecture * **Data Immutability:** The system shifted from updating existing rows to an **INSERT-only** principle, ensuring a reliable audit trail and preventing database deadlocks. * **Event-Driven Decoupling:** Instead of direct database access, the system uses Kafka to publish payment events, allowing independent domains to consume data without tight coupling. * **Payment-Approval Separation:** By separating the "Payment" (the transaction intent) from the "Approval" (the specific financial method), the system now supports multiple payment methods per transaction. ### Safe Migration and Data Integrity * **Asynchronous Mirroring:** To maintain zero downtime, data was initially written to the legacy system and then asynchronously loaded into the new MySQL ledger. * **Resource Tuning:** Developers used dedicated migration servers within the same AWS Availability Zone to minimize latency and implemented **Bulk Inserts** to handle hundreds of millions of rows efficiently. * **Verification Batches:** A separate batch process ran every five minutes against a Read-Only (RO) database to identify and correct any data gaps caused by asynchronous processing failures. ### Operational Resilience and Incident Response * **Query Optimization:** During a load spike, the MySQL optimizer chose "Full Scans" over indexes; the team resolved this by implementing SQL hints and utilizing a 5-version Docker image history for rapid rollbacks. * **Network Cancellation:** To handle timeouts between Toss and external card issuers, the system uses specific logic to automatically send cancellation requests and synchronize states. * **Timeout Standardization:** Discrepancies between microservices were resolved by calculating the maximum processing time of approval servers and aligning all upstream timeout settings to prevent merchant response mismatches. * **Reliable Event Delivery:** While using the **Outbox pattern** for events, the team added log-based recovery (Elasticsearch and local disk) and idempotency keys in event headers to handle both missing and duplicate messages. For organizations tackling significant technical debt, this transition highlights that initial design is only half the battle. True system reliability comes from building "self-healing" structures—such as automated correction batches and standardized timeout chains—that can survive the unpredictable nature of live production environments.