08 / Experiment analysis

A/B Test Statistical Significance Calculator.

Compare two conversion rates with a two-sided statistical significance check.

Private by design

Experiment totals

Example values included

Your numbers aren’t uploaded or saved.

Significance indicator

—

100 × (1 − two-sided p-value)

Use 95% as a common evidence threshold.

Conversion rate A
—
Conversion rate B
—
Relative lift of B
—
Z-score
—

Significance does not measure business value or guarantee the result will repeat. Avoid repeatedly checking and stopping a test early.

How it works

Compare signal with sampling noise

A pooled two-proportion z-test estimates how unusual the observed difference would be if both variations performed equally.

z = (rate B − rate A) ÷ pooled standard error

Worked example

10% versus 13%

With 1,000 visitors per variation, 100 conversions for A and 130 for B produces 30% relative lift and a two-sided p-value of approximately 0.035 (96.5% on the indicator).

Understand the result

What does A/B test significance tell you?

An A/B significance test assesses how unusual an observed conversion-rate difference would be if both variations had the same underlying rate. This calculator uses a pooled two-proportion z-test and a two-sided normal approximation. It also reports each conversion rate and the relative lift of variation B.

How to use this calculator

  1. Enter whole-number visitor and conversion counts for each randomized variation.
  2. Use the same conversion definition and count each visitor consistently.
  3. Review the evidence alongside sample size, business value and your preplanned stopping rule.

How to interpret your result

The displayed confidence is 100 × (1 − two-sided p-value), not the probability that B is better or that the result will repeat. The normal approximation can be unreliable with sparse successes or failures. It assumes independent observations and does not correct for repeated peeking or multiple comparisons.

Frequently asked questions

Can I stop as soon as the display reaches 95%?

Repeatedly checking a fixed-horizon test and stopping at the first significant result raises false-positive risk. Set the sample size and duration in advance or use a method designed for sequential testing.

Is relative lift the same as percentage-point change?

No. Moving from 10% to 13% is a three-percentage-point increase and a 30% relative lift. Consider the absolute change and its commercial impact as well as significance.

Published by Digital Ads Partner. Content updated . Free to use without an account; inputs are processed in your browser.