A lot of websites advocate that you MUST TEST EVERYTHING. While this makes a lot of sense, it shouldn’t be followed blindly in all cases. Like many rules such as this, you get the best results when you know when to break them. Here’s why…

Your changes can’t be measured accurately

If you’re unable to easily measure the impact of what you’re testing, this may mean your experiment will be too costly or have rampant inaccuracies that plague the test results. For instance, if you’re measuring the impact of offline marketing on conversion rate, you’re going to be fighting an uphill battle.

Your customers keep demanding it (and it makes business sense)

If your customers struggle to find legal information and they keep demanding it, there is a good case to go ahead and implement. That is of course if hiding that information wasn’t part of some broader tactic.

You’ve tested something similar elsewhere - with conclusive results

Consider for example that on two separate occasions you tested adding reviews on two different product pages, each with a conclusive result and an 80% improvement in conversion rate. Testing it on another product page would be a waste of time and money. In many cases, you could deploy a change like this site-wide already.

It’s too costly to setup and run the test

All your evidence points to this one change being the crux of the issue, but testing it would require custom code that makes it possible to test dynamic content. That could cost a fortune to setup.

Best thing to do in this case is to look for different ways to test something, or back it up with hard data. Perhaps you may need data elsewhere, like from asking real customers, doing an eye-tracking study, or conducting usability studies.

The changes are too small and insignificant

If you want to track the impact of a different colour on a button with 1000 visits and just 200 conversions per month, be prepared for a long test…

When you think you’re willing to run a test for 3 months, think again. After time, events will occur that will affect your experiments’ validity - think Christmas, Black Friday, Easter, national holidays and the like. These things can all have an effect on the validity of your results and running your test longer increases their effect.