A/B tests: 6 statistical mistakes that invalidate results

A bad A/B test is worse than no test: it makes you take wrong decisions with false certainty. 70% of tests I see in SMEs have at least 2-3 statistical errors. Here are the 6 most common and how to avoid them.

1. Insufficient sample size

Test with 200 visits per variant says nothing statistically. Quick math: to detect +20% conversion rate change, you need ~3,500 visits per variant (at 5% base). Below, it's just noise.

2. Too short duration

Test stopped after 3 days "because we've already won". Wrong. Minimum 1 week to cover behavioural weekly variation (Mon vs Sat very different). Ideally 2-4 weeks.

3. Stop as soon as you have "statistical significance"

95% significance reached on day 4? Don't stop. Known that stopping at first win moment inflates false positive rate by 30-50% (peeking problem). Set duration upfront, respect it.

4. Testing 5 variants simultaneously

Multiple comparisons inflate false positives. If you test 5 variants, 23% chance at least one "wins" by pure chance. Fix: sequential tests (1 vs 1) or Bonferroni correction.

5. Changing the test mid-stream

"Let's add a third variant halfway". No. Reset everything and restart. Changing during test invalidates statistical base.

6. Ignoring practical significance

Statistically significant change (+1.2% conversion) but economically negligible. If it costs €5,000 to implement for €600/year gain, not worth it. Always calculate implementation ROI.

Serious A/B testing needs discipline and statistical tools. Recommended tools: Google Optimize is dead, free alternatives are VWO Free or A/B Tasty for SMEs. Tool included in the Conversion Site package.

Want to apply this to your SME?

If you'd like a concrete analysis of your situation, request the free audit from Gamerbit. In 48 hours we tell you exactly where you're losing leads, time or budget — no sales pitch.

For a complete system, the 2026 no-brainer offers are the most direct starting point.