Calculating the confidence of a split test helps you make a decision to choose a winning variation based on strong enough data. Here are some of the potential applications:
- PPC ad split tests
- Landing page split tests
- Email marketing tests
- … and virtually any other experiments you need to calculate statistical significance for
Hint: Bookmark this page (Ctrl+D) or share it with your colleagues.
VisitorsConversions
Treatment 1:
Interpreting the results
You will know you have a winner when a message dsiplays saying you have a statistically significant result. On the graph, you will see the range where the conversion rate is estimated to sit at 80% confidence (the large block) and 95% confidence (the thin lines). If there is no overlap between the large blocks, and chance to win for your treatment is >95%, you will have a winner.
Notes on statistical significance and validity
Significance and validity help make sure your test results can be generalisable. Here are a few general rules you may like to apply for your tests:
- Run your test for at least two weeks (weekends can really affect test results)
- Avoid testing during unusual periods (Christmas, stock market crash etc)
- Get at least 25 conversions per variation
- … and at least 100 visitors per variation
- If using gradual ramp-up, beware of Simpson’s Paradox
- Use a split testing tool that cookies visitors so they only see one variation
- Only compare data from the same data source (i.e. AdWords and Analytics don’t mix)
- Beware of robots and how they influence your split testing tools’ data (GA and GWO are pretty safe from robots)