# 78% Wanted This AI Feature. Almost Nobody Used It - Supriya Subudhi, Bumble

## Executive summary

This talk details the complexities of A/B testing for Generative AI (GenAI) features, using the example of an AI photo feedback tool at Bumble. The core message is that a positive A/B test result is merely the starting point. Successful AI product deployment requires deep consideration of user adoption patterns, analyzing the 'long tail' of data rather than just averages, managing the escalating costs of advanced models, and ensuring global legal compliance.

## Key takeaways

- Openness does not predict usage: Users may express high interest in an AI feature (e.g., 78% wanting feedback), but actual adoption during A/B testing can be much lower (10-15%). To improve usage, the AI guidance must be baked into the user's existing workflow, not presented as a standalone feature. (2:38)
- The average win hides the tail: Relying solely on top-line metrics (the average win) is insufficient. Detailed analysis of the 'long tail' is necessary to identify issues, such as AI giving contradictory or unhelpful advice (e.g., telling already smiling people to smile more). (5:19)
- Cost vs. Quality (Green Dashboard, Red Bill): High-quality AI feedback often requires the best LLMs, which can be significantly more expensive (up to 10x). The cost of GenAI scales with usage, unlike traditional ML algorithms. It is critical to balance desired accuracy with scalability and budget. (7:54)
- Global Rollout and Compliance: AI features must account for international regulations and legal risks. A feature successful in one market (e.g., the US) may not be deployable elsewhere, requiring product teams to plan for compliance early in the development cycle. (11:49)

## Technical details

- A/B Testing for AI: A positive A/B test result only marks the beginning of the product journey. The evaluation must move beyond simple binary success/failure metrics. (0:30)
- AI Guardrails and Safety: When building AI features, strong guardrails must be set to prevent the model from suggesting changes that users cannot control (e.g., skin color, height). (5:19)
- Evaluation (Evals): Setting up rigorous evaluation tools (evals) is critical for analyzing real user outputs, ensuring the prompt is effective, and validating the model's performance across different user segments (e.g., gender, language). (6:30)
- LLM Cost Modeling: Unlike traditional ML algorithms, the incremental cost of GenAI increases directly with usage. Product teams must understand if the AI use case is for short-term revenue boosts or long-term brand equity. (7:54)

## Practical implications

- Always integrate AI features into existing user journeys rather than building them as standalone tools to boost adoption.
- Prioritize analyzing the 'long tail' of data and segment-specific outputs over overall average performance metrics.
- When budgeting for GenAI, model costs based on expected usage volume, as the cost scales directly with the number of interactions.
- Incorporate legal and regulatory compliance checks (e.g., regional market restrictions) at the earliest stages of product design.

## Topics

Generative AI (GenAI), LLMs, A/B Testing, Product Management, AI Ethics, Product Strategy, Bumble, AIE CODE summit

Source: https://www.youtube.com/watch?v=CvmZtAPmAW4