One Million Events per Second: Implementing End-to-End Encryption with Apache Kafka in the LINE App
LINE handles billions of messages daily, including highly sensitive personal data. While Kafka already provides TLS, authentication, and authorization, those controls do not protect message contents stored in brokers from privileged access. LY Corporation therefore introduced Kafka client-to-client end-to-end encryption, keeping payloads encrypted from producers through consumers while supporting large-scale traffic, flexible consumers, and minimal overhead. ## Limits of Kafka’s Existing Security Model - TLS protects data in transit between clients and brokers. - SASL authenticates clients before they connect. - ACLs control which users or groups can publish to or consume from topics. - These mechanisms primarily control access and communication channels; broker-stored payloads may still exist in plaintext. - End-to-end encryption adds a defense-in-depth layer by encrypting data at production and decrypting it only at authorized consumers. ## Record-Level Encryption - LY Corporation chose record-level rather than batch-level encryption. - Batch encryption offers better compression and lower CPU overhead, but would require modifying Kafka client internals because standard extension points operate at the record level. - Record encryption works with Kafka interceptors, serializers, and deserializers without modifying existing Kafka clients. - Using standard APIs also improves compatibility with future Kafka upgrades, despite somewhat larger messages and reduced compression efficiency. ## DEK–KEK Key Architecture - Payloads are encrypted with a symmetric AES-GCM data encryption key (DEK). - The DEK is encrypted with an ECC-based key encryption key (KEK), using ECIES and the `secp521r1` curve. - KEKs are managed through a key management service (KMS). - Producers use the KEK’s public key, while authorized consumers obtain the private key from KMS. - This hybrid approach: - Avoids the high cost of encrypting large payloads with asymmetric cryptography. - Keeps message size effectively independent of the number of consumers. - Separates encryption and decryption permissions according to the least-privilege principle. ## Encrypted Kafka Message Structure - **Key:** The existing Kafka message key remains unchanged for partitioning. - **Header:** Contains the KEK identifier and the DEK encrypted with that KEK. - **Body:** Contains the payload encrypted with the DEK. - Embedding metadata directly in each message avoids dependencies on external databases or caches. - Consumers identify the appropriate KEK, decrypt the DEK, and then decrypt the payload. ## Producer and Consumer Architecture ### Producer Encryption - Interceptors generate or select the DEK and place the encrypted DEK in the message header. - A wrapper serializer encrypts the serialized payload with the DEK. - The interceptor and serializer share the DEK through `ThreadLocal`, since they run on the same thread. - DEKs are cached for a limited period rather than regenerated and re-encrypted for every message, reducing asymmetric cryptographic overhead. ### Consumer Decryption - Consumers retrieve authorized private KEKs from KMS. - The deserializer reads the encrypted DEK from the header, decrypts it with the private KEK, and decrypts the payload. - Consumers cache encrypted-DEK/plain-DEK pairs, allowing repeated messages from the same producer to bypass redundant DEK decryption. - The existing deserialization process is wrapped so decryption occurs before normal deserialization. ### KMS Operations - Topic owners generate and register KEK key pairs. - Producers retrieve public keys, while authorized consumers retrieve private keys. - New consumers must request access to the private key and receive approval from the topic owner. - KMS manages key distribution, access control, and key rotation. ## Scaling Optimizations ### Shared KEKs - Assigning a unique KEK to every consumer would cause message headers to grow with the consumer count. - This would reduce Kafka batch sizes and increase network, CPU, and memory usage, especially for topics reaching up to one million messages per second. - Multiple consumers therefore share a single KEK, keeping the header size constant. - The trade-off is reduced per-consumer key isolation, mitigated through: - KMS authorization controls. - Mandatory periodic key rotation. - Centralized key management by the topic owner. ### Zero-Downtime Migration - During migration, encrypted and plaintext messages must coexist. - The consumer deserializer checks whether encryption metadata exists: - If headers are present, it decrypts the message. - If headers are absent, it processes the message using the existing plaintext path. - The migration sequence is: - Deploy compatible consumers first. - Enable producer encryption after all consumers support both formats. - Monitor the plaintext-message ratio and complete the migration once it reaches zero. - Producer encryption is intended to be enabled progressively rather than switched to 100% immediately, reducing the risk of unexpected performance or cryptographic failures. ## Practical Conclusion Kafka’s built-in security controls should be supplemented with payload-level encryption when brokers handle highly sensitive data. A record-level AES-GCM design combined with DEK–KEK key wrapping, KMS authorization, caching, shared KEKs, fallback processing, and gradual rollout provides a practical balance between confidentiality, scalability, and operational continuity.
Read original(opens in new tab)