line Introducing a New A/B (opens in new tab)
LY Corporation has developed an advanced A/B testing system that moves beyond simple random assignment to support dynamic user segmentation. By integrating a dedicated targeting system with a high-performance experiment assigner, the platform allows for precise experiments tailored to specific user characteristics and behaviors. This architecture enables data-driven decisions that are more relevant to localized or specialized user groups rather than relying on broad averages. ## Limitations of Traditional A/B Testing * General A/B test systems typically rely on random assignment, such as applying a hash function to a user ID (`hash(id) % 2`), which is simple and cost-effective. * While random assignment reduces selection bias, it is insufficient for hypotheses that only apply to specific cohorts, such as "iOS users living in Osaka." * Advanced systems solve this by shifting from general testing across an entire user base to personalized testing for specific segments. ## Architecture of the Targeting System * The system processes massive datasets including user information, mobile device data, and application activity stored in HDFS. * Apache Spark is used to execute complex conditional operations—such as unions, intersections, and subtractions—to refine user segments. * Segment data is written to Object Storage and then cached in Redis using a `{user_id}-{segment_id}` key format to ensure low-latency lookups during live requests. ## A/B Test Management and Assignment * The system utilizes "Central Dogma" as a configuration repository where operators and administrators define experiment parameters. * A Test Group Assigner orchestrates the process: when a client makes a request, the assigner retrieves experiment info and checks the user's segment membership in Redis. * Once a user is assigned to a specific group (e.g., Test Group 1), the system serves the corresponding content and logs the event to a data store for dashboard visualization and analysis. ## Strategic Use Cases and Future Plans * **Content Recommendation:** Testing different Machine Learning models to see which performs better for a specific user demographic. * **Targeted Incentives:** Limiting shopping discount experiments to "light users," as coupons may not significantly change the behavior of "heavy users." * **Onboarding Optimization:** Restricting UI tests to new users only, ensuring that existing users' experiences remain uninterrupted. * **Platform Expansion:** Future goals include building a unified admin interface for the entire lifecycle of an experiment and expanding the system to cover all services within LY Corporation. For organizations looking to optimize user experience, transitioning from random assignment to dynamic segmentation is essential for high-precision product development. Ensuring that segment data is cached in a high-performance store like Redis is critical to maintaining low latency when serving experimental variations in real-time.