techlist.io
KREN
kakao

In Search of Lost Reports: Kakao (opens in new tab)

distributed-systemskafkamysqlrace-conditionspringjpatransaction-managementhibernate

KIMS, Kakao’s internal SMS platform, experienced rare cases where vendors sent delivery reports successfully, yet messages remained stuck in SENT instead of becoming REPORTED. The cause was a race condition: a fast vendor’s report arrived before the API server had committed the message record. The investigation showed that an unnecessarily long transaction—especially for paid messages with billing-event processing—delayed persistence and allowed valid reports to be dropped.

KIMS Message Processing Flow

  • KIMS processes roughly one million SMS messages per day across multiple IDC environments and external vendors.
  • The normal flow is:
    • Route the request to a suitable vendor.
    • Call the vendor and record the message as SENT.
    • Deliver the message to the recipient.
    • Receive the vendor’s delivery report.
    • Update the message to REPORTED.
  • These stages run asynchronously across separate services, so their execution order is not guaranteed.

Discovering the Missing Reports

  • Some messages remained in SENT even though Report Server logs confirmed that delivery reports had arrived.
  • The issue affected only about 0.02% of messages, making it difficult to reproduce in tests or local environments.
  • Two patterns emerged:
    • Missing reports were concentrated among messages sent through one particular vendor.
    • Paid messages were affected more often than free messages.

The Race Condition

  • The problematic vendor returned reports unusually quickly:
    • Other vendors typically took more than one second.
    • This vendor averaged around 20 ms.
    • Missing-report cases averaged only about 8 ms.
  • The API server performed additional processing before committing the message record.
  • For paid messages, billing-event publication was included in the same @Transactional scope, making the transaction longer.
  • Consequently, the sequence could become:
    1. API Server calls the vendor.
    2. API Server performs billing-related processing.
    3. The vendor delivers the message and immediately sends a report.
    4. Report Server receives the report before the message row exists in the database.
    5. Report Server treats the report as invalid and drops it.
    6. API Server finally commits the message as SENT.
  • The report was not lost at the network or vendor level; it was discarded because the system’s write path had not completed.

Reducing Transaction Scope

  • The first fix was to remove nonessential work from the main transaction.
  • Billing-event publication was moved to asynchronous processing using @Async and @TransactionalEventListener.
  • The transaction was reduced to the essential state change and database commit.
  • This advanced the average commit point by approximately 10 ms and significantly reduced report omissions.
  • It also avoided a dual-write anti-pattern in which an external Kafka event was published inside a database transaction that could later roll back.

Reconsidering the Need for a Transaction

The incident prompted a broader review of whether the transaction was needed at all.

  • Atomicity: The transaction contained only one database write, with no multi-table or cross-record operation requiring all-or-nothing rollback.
  • Read isolation: Metadata such as vendor quality metrics was updated only every few minutes, and using a slightly stale value was acceptable. The independently read tables did not require a single consistent snapshot.
  • Write isolation: JPA’s dirty checking kept the status change in the persistence context until transaction completion, delaying the actual database write. This delay was precisely what allowed the report to arrive first.

The article therefore presents the transaction itself—not the vendor or report receiver—as a source of unnecessary latency and an architectural anti-pattern in this workflow.

Practical Recommendation

Use transactions only when their guarantees are required. Keep critical persistence paths short, move external events and nonessential processing after commit, and critically evaluate whether delayed commit semantics could allow asynchronous consumers to observe a missing record.