A B Testing

11 posts

figma2 min readCurated summary

Measuring Time Savings From Figma Make | Figma Blog

Figma’s Data Science team found that Figma Make reduced design-task completion time by 20% and made work 16% easier. Product managers benefited most, completing tasks 23% faster and reporting a 37% improvement in ease. Because ordinary A/B tests and observational analyses could not adequately control for task complexity and user experience, Figma used a randomized controlled trial (RCT) with 100 participants. ## Why Measuring AI Time Savings Is Difficult - Productivity is influenced by confounders such as: - Job tenure and career experience - Individual design ability - Task complexity - Without controlling for these factors, it is difficult to determine whether improvements come from AI or from differences among users and tasks. ## Limitations of Common Research Methods - **Online A/B testing** - Randomly assigning users to treatment and control groups helps balance user characteristics. - However, users may perform different tasks, making it difficult to ensure that task complexity is comparable. - **Causal inference using product logs** - Methods such as propensity score matching require all relevant confounders to be present in the data. - Anonymized logs cannot capture subjective factors such as a user’s design experience. - Instrumental-variable analysis requires a valid factor that influences AI usage without independently affecting task speed; Figma could not identify one. ## The Randomized Controlled Trial - RCTs were selected because they can control confounders before data collection begins. - The study combined: - Random assignment to Figma Make and control groups - Identical tasks for all participants - Moderation by trained researchers - The study focused only on Figma Make to avoid introducing variables from multiple AI tools. - Participants included 100 people: - 50 product designers - 50 product managers - The sample size was based on effect sizes from prior industry research, including GitHub Copilot RCTs, followed by a statistical power analysis. ## Findings - Overall, Figma Make: - Made design work **20% faster** - Made work **16% easier** - Product managers experienced the largest gains: - Tasks were **23% faster** - Tasks were **37% easier** The study suggests that a carefully controlled RCT is a more reliable way to measure AI’s productivity impact when task differences and user characteristics are difficult to capture in product data. Teams evaluating similar tools should standardize tasks, randomize participants, and moderate the study to separate genuine AI benefits from other sources of variation.

Read original(opens in new tab)
discord3 min readCurated summary

Measure Less to Learn More: Using Fewer, Higher-quality Metrics to Capture What Matters

Discord argues that experiments should measure fewer, higher-quality metrics rather than automatically collecting every potentially useful signal. Large metric sets increase compute and cognitive costs while creating a tradeoff between false positives and missed real effects. Multiple-testing corrections such as Benjamini–Hochberg reduce false discoveries but also lower recall, so the most effective solution is selecting metrics that represent distinct, important concepts. ## The Cost of Measuring Too Much - Discord’s “Default Metric List” gradually expanded as teams added metrics and rarely removed them. - More metrics create: - Higher compute costs - More difficult experiment readouts - Increased risk of false positives - With 100 metrics and an uncorrected significance threshold of 0.05, roughly five metrics may appear significant purely by chance. - Correcting for multiple comparisons reduces false alarms but makes genuine changes harder to detect. ## The Multiple Comparisons Problem - Discord uses the Benjamini–Hochberg (BH) procedure to control the false discovery rate at 5%. - BH ranks p-values and compares each one with a rank-specific threshold: `i × α / n` where `i` is the metric’s rank, `α` is 0.05, and `n` is the total number of metrics. - A metric with an unadjusted p-value of 0.038 might be significant without correction but fail after BH adjustment. - BH treats all metrics equally because it has no information about which ones are more likely to reflect a real effect. - The resulting tradeoff is: - Fewer false alarms - Lower recall for real changes - The article notes that Bayesian methods could potentially incorporate prior knowledge, but Discord’s default system is frequentist. ## Simulation Results - Discord simulated 50,000 experiments containing: - Twenty null metrics generated from `N(0, 1)` - One metric with a real effect centered at `z = 2.8` - The simulations tested how metric count affects: - Experiment-level false alarm rates - Recall of the metric with the known effect - Without correction, false alarm rates rose sharply as more metrics were added—approximately from 23% with five metrics to 93% with 50. - BH kept false alarm rates near 5%, but recall declined as the metric pool grew, falling from roughly 60% to 30% across the same range. - These results demonstrate that adding metrics makes statistical correction stricter and makes genuine effects harder to identify. ## Fewer, Higher-Quality Metrics - Reducing the metrics automatically included in experiments improves the balance between false alarms and recall. - Metrics should be selected for quality and conceptual distinctness rather than added “just to be safe.” - The article’s central conclusion is that no sophisticated statistical method eliminates the underlying tradeoff created by excessive measurement. Teams should maintain a focused default metric set, regularly remove low-value or redundant metrics, and reserve specialized metrics for experiments where they are genuinely relevant.

Read original(opens in new tab)
spotify3 min readCurated summary

Better Experiments with LLM Evals — A funnel, not a fork | Spotify Engineering

LLM evaluations and A/B experiments serve different purposes and should work as a funnel rather than alternatives. Evals verify whether a change meets qualitative standards and filter out weak ideas, while experiments validate whether it improves real user and business outcomes. Their greatest value comes from a feedback loop that continuously calibrates eval scores against online results. ## Evals as Verification, Experiments as Validation - LLM judges can assess relevance, coherence, tone, intent alignment, and other qualities at scale. - Evals identify promising candidates before they consume experiment capacity. - Experiments determine whether users actually respond better to the change. - An eval can uncover unexpected failure patterns, guide product fixes, and later verify that violations have decreased. - It cannot establish whether those fixes improve long-term outcomes such as engagement, trust, or retention. ## The Importance of Guardrails - Evals typically measure selected quality dimensions, not every possible business risk. - Spotify reports that roughly 42% of launched experiments are rolled back because of regressions in secondary metrics such as crashes, session length, or retention. - Online experiments expose production effects and monitor guardrail metrics that offline evals may miss. - Although only about 12% of Spotify’s A/B tests produce a shipped positive result, approximately 64% generate useful learning. ## Calibrating Evals Against Outcomes - Evals are proxies, so their scores must be validated against the outcomes they are intended to predict. - LLM judges introduce an additional calibration layer on top of conventional metrics such as precision, recall, and ranking scores. - Judges can reward superficial patterns or miss improvements that matter in real-world use. For example, Qodo’s coding evals did not detect improvements in Anthropic’s Opus 4.5 on longer tasks. - Both judges and quantitative metrics can drift, making offline-to-online comparison essential. - Continuous calibration clarifies when evals are reliable verification tools and whether they may eventually become strong enough to support validation. ## Closing the Evaluation Loop - Run evals early and frequently to select the strongest treatments. - Use experiments to confirm user impact, assess business outcomes, and detect unintended regressions. - Apply the same LLM evals to A/B test data to compare qualitative scores with actual user results. - A mismatch between eval scores and experiment outcomes is valuable diagnostic information: - Improved scores and improved outcomes validate the judge. - Improved scores without better outcomes show that the judge is measuring something that may not drive value. - Use lightweight experiments for iteration and learning, and more rigorous tests for launch decisions. The practical recommendation is to treat LLM evals as an upstream filter and a downstream calibration tool—not a replacement for experimentation. This funnel improves test efficiency while making both the judges and product decisions more trustworthy over time.

Read original(opens in new tab)
toss3 min readCurated summary

From Intern to Solo Designer: Growth

As a Toss Bank product design intern, Jeon Nuri designed experiments to improve non-member sign-up conversion. She prioritized the funnel using speed and impact, studied previous experiments, and learned that clear, narrowly defined hypotheses were more valuable than constantly generating new ideas. The experience showed that failed experiments can still guide better decisions when they produce actionable learning. ## Prioritizing the Right Funnel Stage - The largest drop-offs occurred in the intro, consent, and identity-verification screens. - Consent and identity verification were shared modules requiring legal and compliance review, making rapid iteration difficult. - The intro screen could be changed more quickly and had the potential to affect the greatest number of users. - Based on this speed-versus-impact assessment, she chose the intro screen as the starting point. ## Learning from Previous Experiments - Instead of immediately designing new concepts, she reviewed existing experiments, including both winners and unsuccessful variations. - She examined: - The problem each experiment addressed - The reasoning behind its hypothesis - How the test variation was designed - Experiments from unrelated screens were also useful because their problem definitions and hypothesis structures could be adapted. - The main lesson was that inexperienced experimenters benefit more from systematically analyzing existing learning than from rushing to create new ideas. ## First Experiment: A Counselor Concept - The first variation presented benefits as if they were being recommended by a counselor and offered a small number of choices. - The hypothesis was vague: fewer choices would increase conversion. - The result was negative: - Click-through rate fell by more than 10%. - Conversion rate fell by more than 3%. - The design actually introduced more choices than the original, which had only one CTA button. - The experiment also failed to consider why users had entered the screen and whether they needed recommendations. - This led her to analyze the existing screen and user context before creating a hypothesis. ## Identifying and Solving Concrete Problems - Rather than inventing an entirely new design, she identified two specific weaknesses in the existing version: - The copy did not clearly communicate benefits users cared about. - Images loaded slowly, taking two to three seconds on low-end devices. - Previous experiments showed that users responded well to messages about high interest rates and receiving interest daily. - She incorporated those themes into the copy and optimized the visuals with newer graphics and lower-weight image formats. - Both click-through rate and conversion rate increased, demonstrating that a hypothesis grounded in clear problems can provide a stable direction for design. ## Making Benefits Easier to Imagine - Building on the earlier results, she changed functional wording into language that helped users imagine a concrete situation and immediate benefit. - Instead of simply explaining that interest could be earned after depositing money for one day, the revised copy foregrounded the moment when users would experience the benefit. - Copy alone increased CTR by 5% and also produced a meaningful improvement in CVR. - The result reinforced that different expressions of the same information can create significantly different first impressions. ## Principles for Designing Experiments - Break the funnel into stages and prioritize opportunities by speed and potential impact. - Understand the existing context before defining the core problem. - Study previous experiments through their hypotheses and problem definitions, not just their numerical outcomes. - Establish a clear hypothesis and success metric before designing the variation. - Make sure the experiment visibly tests the stated hypothesis. - Treat failure as input for the next decision rather than as wasted effort. A practical starting point for new designers is to begin with a small, focused experiment—but make the hypothesis precise enough to guide both the design and the next iteration.

Read original(opens in new tab)
figma2 min readCurated summary

The New Business Case For Design Systems | Figma Blog

Design systems are no longer merely efficiency tools or static component libraries; they are strategic investments that can influence revenue, customer loyalty, global expansion, and product quality. Research from the Design Executive Council shows that organizations are increasingly measuring design-system value through customer and business outcomes, not just reduced rework or faster handoffs. The strongest business case connects design-system work to metrics executives already care about. ## Linking Design Systems to Customer Outcomes - Teams can measure design-system impact through adoption, retention, engagement, satisfaction, and support metrics. - Freshworks attributed its design system to: - A 28% reduction in customer service costs - Faster support-ticket resolution - SAP collects more than one million in-app user feedback data points to improve its design system. - Freshworks uses CSAT scores, A/B tests, and funnel diagnostics to identify onboarding friction and guide new components, patterns, and features. - These metrics help design teams demonstrate business value while creating a roadmap for improving customer experience and product “stickiness.” ## Scaling Company Values and Product Quality - Design systems can scale not only brand identity but also company principles and product standards. - Linear uses its design system to support a culture of craft and quality, which contributes to customer loyalty and net revenue retention. - The system is intentionally flexible and continuously updated rather than governed by rigid rules. - Its goal is to ensure that products feel thoughtfully crafted while allowing teams to adapt components as needed. ## Supporting Global Growth and Localization - Design systems help companies expand internationally while maintaining consistency, brand identity, and cultural relevance. - Hyundai Motor Group uses one system across more than 30 vehicle models and three brands—Hyundai, Kia, and Genesis—while preserving each brand’s distinct identity. - Grammarly built localization into its design-system strategy from the beginning by: - Employing in-house linguists - Accounting for cultural nuances - Treating right-to-left readability as a core design input - Distributed teams across North America, South Korea, and Poland use a shared foundation to handle different languages, hardware constraints, screen sizes, and cultural expectations. - Hyundai’s 42dot uses custom Figma plugins to test multilingual user experiences. The practical recommendation is to frame a design system around outcomes that business leaders already value—customer satisfaction, retention, revenue, global growth, and product quality. Productivity improvements remain useful, but the most persuasive evidence comes from showing how design-system decisions change customer and business performance.

Read original(opens in new tab)
tossOriginal article

Creating the worst experience at Toss (opens in new tab)

Toss designer Lee Hyeon-jeong argues that business goals and user experience are not mutually exclusive, even when integrating controversial elements like advertising. By identifying the intersection between monetization and usability, her team transformed intrusive ads into value-driven features that maintain user trust while driving significant revenue. The ultimate conclusion is that transparency and appropriate rewards can mitigate negative feedback and even increase user engagement. ### Reducing Friction through Predictability and Placement * Addressed "surprise" ads by introducing clear labeling, such as "Watch Ad" buttons or specifying ad durations (e.g., "30-second ad"), which reduced negative sentiment without decreasing revenue. * Discovered that when users are given a choice and clear expectations, their anxiety decreases and their willingness to engage with the content increases. * Eliminated "flow-breaking" ads that mimicked functional UI elements, such as banners placed inside transaction histories that users frequently mistook for personal bank records. * Established a design principle to place advertisements only in areas that do not interfere with information discovery or core user navigation tasks. ### Transforming Advertisements into User Benefits * Developed a dedicated B2B ad platform to scale the variety of available advertisements, ensuring that users receive ads relevant to their specific life stages, such as car insurance or new credit cards. * Shifted the internal perception of ads from "noise" to "benefits" by focusing on the right timing and high-quality matching between the advertiser and the user's needs. * Institutionalized regular "creative ideation sessions" to explore interactive formats, including advertisements that respond to phone movement (gyroscope), quizzes, and mini-games. * Leveraged long-term internal experiments to ensure that even if an idea cannot be implemented immediately, it remains in the team's "creative bank" for future product opportunities. ### Optimizing Value Exchange through Rewards * Conducted over a year of A/B testing on reward thresholds, comparing small cash amounts (1 KRW to 200 KRW), non-monetary items (gifticons), and high-stakes lottery-style prizes. * Analyzed the "labor intensity" of ads by adjusting lengths (10 to 30 seconds) to find the psychological tipping point where users felt the reward was worth their time. * Implemented a high-value lottery system within the Toss Pedometer service, which successfully transitioned a loss-making feature into a profitable revenue stream. * Maintained user activity and satisfaction levels despite the increased presence of ads by ensuring the "worst-case experience"—viewing ads for no gain—was entirely avoided. Product teams should stop viewing business requirements and UX as a zero-sum game. By focusing on user psychology—specifically transparency, non-disruption, and fair value exchange—it is possible to achieve aggressive business targets while maintaining a sustainable and trusted user environment.

figma2 min readCurated summary

Navigating the intersection of design and business: a conversation with Airbnb’s Brian Chesky | Figma Blog

Airbnb CEO Brian Chesky argues that design should shape business strategy, not merely product appearance. He describes how Airbnb drifted from its design-led origins as it grew, becoming fragmented, experiment-heavy, and less innovative. By reconnecting with the principles of Apple’s design renaissance, Chesky began reconsidering how the company should be organized and led. ## Pulling Airbnb Back from the Brink of Failure - Airbnb was founded by two designers and an engineer, challenging conventional views that designers do not start companies. - Its original philosophy was to bring design into the boardroom and create a product that felt distinctive and meaningful. - By late 2019, Chesky feared Airbnb had become unrecognizable: - The company had roughly 10 divisions, each with numerous subdivisions. - Product managers and A/B testing dominated decision-making. - More employees and projects produced fewer visible product changes while increasing costs. - Chesky believed he had lost the courage required for creative leadership and that conventional corporate practices had displaced Airbnb’s founding vision. - The timing was especially difficult because Airbnb was preparing for its IPO, making major organizational changes risky. ## Rediscovering Design-Led Leadership - During the 2019 holiday season, Chesky met Hiroki Asai, a former Apple creative director, and Jony Ive, Apple’s former design leader. - Their perspective reminded him of the design-centered culture Steve Jobs had established at Apple. - Chesky began exploring whether Airbnb could return to a similarly integrated model, where design and creative judgment played a central role in how the company operated. ## Design’s Place in Business - Chesky suggests that designers are rarely elevated to CEO roles because businesses tend to favor scientific, analytical, and operational methods. - Creative work requires conviction and risk-taking, qualities that can weaken as organizations become larger and more bureaucratic. - His experience led him to question whether standard corporate structures are compatible with maintaining a company’s original creative vision. The excerpt presents Chesky’s central lesson: sustained growth can dilute a company’s founding principles unless leaders deliberately preserve design, courage, and customer-centered decision-making at the heart of the business.

Read original(opens in new tab)
figma2 min readCurated summary

How we use data | Figma Blog

Figma uses data in two broad ways: functional data to provide its service and analytics data to improve the product. The company emphasizes collecting only the information necessary to operate Figma, while using aggregated usage insights and experiments to guide product decisions. Examples include improving file sharing and identifying performance problems in the iOS app. ## Functional Data - Functional data supports core account and product operations. - Figma collects a relatively small amount of information at signup: - Email address - Name - Role - This information enables usernames, password-reset messages, file creation, and collaboration. - Figma generally does not require sensitive information such as identity documents or verification. - Payment information for paid plans is collected and processed by Stripe. ## Analytics Data - Analytics data describes how users access and use Figma. - It helps teams understand: - Which features users adopt - Which features they ignore - Where users encounter difficulties - How usage varies across platforms - Data scientists analyze these signals alongside user research, product intuition, and direct feedback. ## Improving Features Through Experiments - A/B testing is a central part of Figma’s product-development process. - Experiments test hypotheses and measure how proposed changes affect user behavior. - In one study, Figma examined its file-sharing modal, where users invite collaborators, manage permissions, and publish work. - Research showed that: - Only 20% of users opened the share modal during their first month. - Only half of those users successfully shared a file. - Figma simplified the interface and moved secondary functions into separate tabs. - The change produced: - A 2% increase in users sending invitations - A 2% increase in users invited to each file - No observed decline in users publishing work to Figma Community ## Identifying Performance Issues - Figma uses data to monitor application performance across platforms and prioritize improvements. - After launching the beta iOS app, the data team analyzed crashes by platform, scenario, and timing. - The analysis found that prototypes were a major source of iOS crashes. - About 25% of prototype crashes occurred within the first 10 seconds of loading. Figma’s approach combines minimal functional data collection with analytics, experimentation, and performance monitoring. The practical goal is to use data selectively to improve usability, reliability, and collaboration while limiting the amount of sensitive information required from users.

Read original(opens in new tab)
figma2 min readCurated summary

From experiment to launch: how data shaped a new comments experience | Figma Blog

Figma used a series of data-driven experiments to redesign its comments experience and encourage collaboration among both editors and viewers. Although comments strongly predicted team retention and growth, they were underused because users struggled to discover them. Experiments confirmed that improving visibility increased comment creation, while an intuitive-looking relocation of the comments control unexpectedly reduced discoverability. ## Collaboration and Comments - Teams that collaborate during their first month are: - 1.75× more likely to be retained. - 6.5× more likely to become customers. - Figma identified comments as a potential “front door” to collaboration because both editors and viewers can use them. - Despite their value, comments were not widely used, prompting the data science team to investigate the gap between user needs and behavior. ## Testing Comment Discoverability - Users previously entered comment mode through an icon in the upper-left corner of the editor. - Research sessions suggested that comments were valuable but difficult to find. - Figma tested a prompt encouraging view-only developers to leave comments. - After two weeks in a 50/50 experiment: - Comment creation increased by 45% in the test group. - The rate of users returning to comments the following week did not change. - The result showed that simply making comments more visible could substantially increase usage. ## An Unexpected Result - Figma hypothesized that moving the comments entry point from the left side of the menu bar to the right would improve discoverability. - The reasoning was that the left side emphasized creation tools, while the right side contained collaboration and viewing features more relevant to cross-functional users. - Among new users, the change caused a 20% decrease in comment discovery within seven days of signup. - The failed experiment demonstrated that seemingly logical product changes can significantly harm user behavior. - It also reinforced the value of controlled experiments: product hypotheses are often wrong, and teams should expect testing to disprove many ideas. Figma’s experience illustrates that effective product development depends on combining user research with rigorous experimentation. Rather than relying on intuition about interface placement, teams should validate each change with real behavioral data and use unexpected results to guide subsequent iterations.

Read original(opens in new tab)
figma2 min readCurated summary

Test designs early and often with the new Maze integration | Figma Blog

Figma’s new Maze integration lets teams test interactive prototypes with real users before development, making research faster and less expensive than waiting for production A/B tests. By importing a Figma prototype link into Maze, teams can measure usability across individual tasks or complete user flows. The post also recommends practical research habits to ensure tests produce reliable, actionable insights. ## Earlier Testing with Figma and Maze - Maze provides quantitative UX insights from real users, supporting both small-scale interviews and tests involving thousands of participants. - Teams can copy and paste a Figma prototype link into Maze to begin testing during the design phase. - The integration helps product teams identify usability problems before committing to development. - At launch, Maze offered Figma users 30% off paid plans for three months with the code `MAZE_LOVES_FIGMA`. ## Define Measurable Goals - Establish what success means before building the prototype. - Use specific metrics, such as requiring 80% of participants to complete tasks successfully and fewer than 2% to misclick. - Tailor goals to the purpose and scope of each test. ## Pilot the Test First - Run the test internally with teammates, friends, or family before involving real users. - Confirm that tasks are understandable, the prototype works correctly, and the success criteria are meaningful. - A pilot functions like a beta release, revealing problems before testing at scale. ## Include Users and Non-Users - Existing customers can provide informed feedback and may be enthusiastic early adopters. - Target customer segments most likely to use the proposed feature. - Include people unfamiliar with the product to avoid bias and gain fresh perspectives. ## Write Neutral Tasks - Avoid leading instructions such as “click here,” “go to,” or “navigate to.” - Focus tasks on the outcome users should achieve rather than the specific interface steps. - Avoid internal product terminology so the test measures intuitiveness rather than participants’ ability to follow directions. ## Collaborate on Analysis - Reviewing results as a team helps build agreement around findings and next steps. - Maze supports collaboration through instantly generated reports and shareable URLs. - Involving the broader product team can make research insights easier to act on. Teams should use the Figma–Maze workflow to validate designs early, measure outcomes against clear goals, and involve collaborators in interpreting the results.

Read original(opens in new tab)
figma2 min readCurated summary

"The Decade of Design”: How 10 years transformed design’s role in tech | Figma Blog

The 2010s transformed design from a secondary concern into a central function of technology companies. The smartphone, especially the iPhone, made products more complex, generated vast amounts of behavioral data, and raised consumer expectations for polished experiences. As a result, companies began hiring designers at scale and treating design as a driver of product strategy and growth. ## The Mobile Wave ### More Complex Interactions - Smartphones introduced smaller, varied screens and touch-based gestures such as swiping, tapping, and holding. - Users interacted with devices in many contexts—while walking, commuting, or multitasking—forcing teams to design for real-world conditions. - Designers became essential to prioritizing functionality and simplifying complex technical systems. - Mobile effectively made the phone the primary computer people used throughout the day. ### The Rise of Data-Driven Design - Smartphones produced unprecedented volumes of behavioral data. - Product teams began continuously A/B testing interface details, including button colors, wording, and layouts. - These experiments measured outcomes such as sign-ups, purchases, and sharing. - “Growth design” emerged as a distinct discipline, with designers increasingly focused on optimizing user behavior and business metrics. - Smartphone ownership among U.S. adults rose from 35% in 2011 to 81% in 2019. ### Apple Raised the Standard - The iPhone changed expectations for how technology should look and feel, replacing the perception of software as merely functional and bland. - Apple’s design became a benchmark that competitors sought to imitate. - The iPhone’s broader mainstream appeal made high-quality industrial and interface design visible to ordinary consumers, not just creative professionals. - Products such as Tinder adopted mobile-first design conventions influenced by Apple’s operating system. - Even Google reportedly discarded an Android device it had been developing after reassessing its design direction. ## Design’s Expanding Role in Technology - The rapid growth of mobile products exposed the limitations of engineering-only approaches to product development. - Designers were particularly valuable for translating complicated systems into simple, intuitive interactions. - Technology companies that had previously operated with minimal design resources began hiring designers and acquiring design agencies. - Silicon Valley gradually shifted from primarily celebrating engineering expertise to recognizing design as a competitive advantage. The decade’s central lesson is that design became indispensable because technology moved into people’s pockets and daily lives. Companies that want to compete should treat designers as strategic product partners, not merely as people responsible for visual polish.

Read original(opens in new tab)