What it is
A controlled online experiment that randomly assigns users to versions of an experience and compares a predefined metric between groups. A/B tests compare two versions; multivariate tests vary several elements at once and need far more traffic. They can show that a change caused a difference in behavior, not why.
Why use it
Analytics cannot separate a design's effect from seasonality, marketing, or other changes; randomized comparison gives causal evidence about one change and limits exposure to a risky one.
Use this when
- A change can ship in two versions and a metric can capture its effect
- The team disagrees about which of two designs will perform better
- Traffic can detect a meaningful difference in reasonable time
- A risky change should reach a random subset, measured against a control
You may not need this when
- Traffic is too low for a reliable result in practical time
- The question is why users behave a certain way, which needs qualitative research
- The change is required, for example for accessibility or legal reasons, whatever the results
Dependencies — what you need
- A written hypothesis, a primary metric, and metrics that must not worsen
- A sample size and run time set beforehand, and no early stopping on interim results unless the method allows it
- Experimentation or feature-flag tooling, and effort to build both versions
- Analytics or data science support to analyze results correctly
People involved
- Design
- Product
- Engineering
- Analytics
- Data Science
Typical outputs
- A test plan: hypothesis, metrics, audience, duration
- Results with effect size and uncertainty
- A ship, iterate, or stop decision, with reasoning
- Questions for qualitative research on the reasons
Template
A/B Testing template
A fill-in structure for running this method: what to prepare, what to do, what to record, and how to read the result.
How AI can help
AI as augmentation — not replacement
- Help phrase a testable hypothesis and name metrics to watch for harm
- Draft a plan outline for the analyst to complete with sample size and duration
- Explain statistical terms in plain language
- Draft a summary from the analyst's figures; AI should not decide significance
AI prompt
A prompt for any AI assistant. It works only from material you paste in, keeps assumptions apart from findings, and leaves the decisions to you.