Conversion Rate likely between 1.6% and 2.6%
3,261 visitors·68 conversionsBaselineThe test is still running. The winner is already selling.
Turn on the multi-armed bandit and Mtrix sends nine in ten new shoppers to the variant that's converting best, once a day, while the others keep being measured. You earn from the leader during the test, not after it.
The Mtrix experiment report for lumière.co's test "Autumn drop hero", day 15, with the multi-armed bandit on. The chart shows unique visitors per day for each variant. From 3 to 4 September all four variants get about a thousand visitors a day. On 5 September the Gift-ready box leads and gets 4,112. From 6 September Lit at dusk leads and gets between 3,700 and 4,900 a day, while Control, Gift-ready box and Scent quiz fall to about 120 each.
An even split keeps paying
for the variants that lose.
A classic A/B test gives every variant the same share of shoppers until it ends. With four variants, three in four shoppers see one that isn't the best, every day of the test. Here are the same two weeks run both ways.
Conversion rate in this example
- Control 2.20%
- Lit at duskbest 2.74%
- Gift-ready box 2.31%
- Scent quizweakest 1.96%
Worked example over 14 days with 4,000 new shoppers a day. With an even split, each of the four variants gets 1,000 shoppers a day: 1,289 orders, and 14,000 shoppers see Scent quiz, the weakest version. With the multi-armed bandit, the split is even for two days, the gift box gets 92.5% on day 3, and Lit at dusk gets 92.5% from day 4: 1,463 orders, and 3,200 shoppers see Scent quiz. The bandit earns 174 more orders from the same 56,000 shoppers.
Day 14 of 14 · 16 Sep
+174 orders from the same 56,000 shoppers.
Scent quiz, the weakest version, reached 3,200 shoppers instead of 14,000.
Worked example: 4,000 new shoppers a day at the conversion rates in the key. The bandit found the leader on day 4, after backing the gift box for a day. Your store's numbers will be different.
Four rules decide
where tomorrow’s shoppers go.
Turn it on in Advanced settings when you create an experiment. From then on the bandit follows these four rules every day, and nothing else.
Multi armed bandit
Shifts traffic toward the leading arm while the test runs.
- Multi armed bandit — ε-greedy
Turn the switch off to see the rules without it.
-
Every variant starts equal.
New shoppers are split evenly until each variant has 100 visitors and 25 conversions. That’s the same minimum the report needs before it compares them.
3–4 Sep1,000 new shoppers to each variant
-
Then the leader gets nine in ten.
Once a day, 90% of new shoppers go to the variant with the best conversion rate so far. The other 10% is shared by all of them, so every variant keeps being measured.
From 6 Sep3,700 to Lit at dusk, 100 to each of the others
-
The lead can change hands.
If another variant overtakes, the next day’s shoppers follow it. A lucky first weekend doesn’t decide the test.
5 SepGift-ready box leads at 2.92%
6 SepLit at dusk overtakes at 2.67%
-
Returning shoppers keep their variant.
Someone who saw the gift box on Saturday sees it again on Tuesday. Only new shoppers are routed, so nobody’s page changes under them.
Sat 5 Sep First visit Gift-ready box Tue 8 Sep Comes back Gift-ready box8 SepNew shoppers that day mostly got Lit at dusk. This shopper still got the gift box.
See every move it makes,
and the numbers behind it.
The bandit works inside the same report as every Mtrix test. You can see today’s split, trace each change back to the conversion rates that caused it, and check what the verdict makes of it.
lumière.co’s experiment “Autumn drop hero” on day 15, in the Mtrix report. The All variants table shows the allocation the bandit set: Lit at dusk 92.5%, Control, Gift-ready box and Scent quiz 2.5% each. The Logs tab records that on 5 September the bandit moved 92.5% of new visitors to Gift-ready box, and on 6 September to Lit at dusk, which led at 2.67% against 2.45%. The Statistics view reads “Readable, but not decided yet — Trending, but keep it running”, notes the traffic split changed and the results are adjusted for it, and gives Lit at dusk a 93% chance of being the best, with every verdict still “Not enough to call yet”.
- Same bar for a winner A 95% chance of being best, low risk, 100 visitors and 25 conversions per variant, and seven days. The bandit doesn’t lower it.
- Adjusted for the moving split Variants are compared within each calendar day, so one doesn’t look better just because it got more traffic.
- Switch it off any time New shoppers go back to an even split. Everything collected so far stays in the report.
- Works on every test type A/B, multivariate, split URL and mask URL. On a multivariate test, every combination is a variant.
Turn it on when the window is short.
Leave it off when you need proof.
A bandit sells more while the test runs and takes longer to prove anything, because the weaker variants see fewer shoppers. So it’s a choice you make per test. This is how lumière.co would split a year.
Bandit on
Earning during the test matters most
Bandit off
You need to know the size of the win
-
Feb
-
Valentine’s gift guide headline
The page comes down on 15 February. There’s no later to roll a winner out to, so the best headline should get the traffic now.
-
Valentine’s gift guide headline
-
Mar
-
Spring candle drop hero
A drop does most of its selling in its first week. That’s when shoppers should be seeing the best version, not a quarter each.
-
Spring candle drop hero
-
Apr
-
Aurora candle at $38 or $42
A lower price usually converts better and can still earn less per visitor. The bandit follows conversion rate, so price tests stay even and you compare revenue per user in the report.
-
Aurora candle at $38 or $42
-
Jun
-
One-page checkout
You’ll keep it for years. Wait for a verdict on how much better it is before you commit.
-
One-page checkout
-
Aug
-
Gift card page layout
A few hundred visitors a week. There’s little traffic to move, and an even split reaches a verdict sooner.
-
Gift card page layout
-
Sep
-
Autumn drop hero
The test on this page. Lit at dusk has had nine in ten new shoppers since 6 September.
-
Autumn drop hero
-
Nov
-
Black Friday banner
Four days. An even split would still be sharing traffic five ways when the sale ended.
-
Black Friday banner
The longer comparison: multi-armed bandit vs A/B testing →