Experiments

How experiments are judged

The current version always runs alongside the challengers, so results are compared fairly.
A test is called once, at its planned end: a version wins only at 95% confidence or more, with at least 20 conversions in every version.
Before the planned end, a lead can still reverse, however strong it looks, so the test isn't called early.
If nothing clears the bar at the planned end, the current version stays.

Running now · 1

Running

Klaviyo · first-time buyers

Three short emails with a brew guide for the coffee they bought will get more first-time buyers to a second order than the old five-email sequence.

Day 5 of 14Oct 24 – Nov 6

Too early to call

Winner found

Klaviyo · follow-up email to first-time buyers

A follow-up email half the length will get more clicks to reorder.

Ran 10 daysOct 6 – Oct 15

The short email won: clicks to reorder more than doubled, from 3% to 7%, over 99% confident.