Ideas are powerful - you don’t need to watch Inception to know that, either. Results released by Microsoft, Amazon and Google indicate most of their ideas to improve their websites have failed to improve the metrics they were designed to lift.

Here are the results:

• Amazon - 50% of experiments fail to show improvement
• Microsoft - 1/3 ideas had a significant positive impact
• AppSumo - 1/8 tests produce results
• QualPro - 75% of important business decisions have no impact on, or actually hurt performance (these guys have done MVT for nearly 30 yrs)
• Google - Only 10% of the 12,000 experiments run in 2009 led to business changes
• Avinash Kaushik - 80% of the time we’re wrong about what the customer wants

That’s an average success rate of roughly 30%. That sounds pretty grim to me. In fact, it sounds as though it’s dangerous to implement ideas without testing them.

Test ideas from any old place might sound like a good idea, but sometimes it pays off to work for your ideas (through research).

Surely those results are on the low side?

I’d say Microsoft and Amazon are doing some pretty ground breaking stuff. Ultimately, that’s the point of testing. By pushing the envelope on counter-intuitive ideas, these guys are producing test results where they learn something and push the boundaries of what’s expected from website changes. They don’t have a lot issues on their sites because they already have specialist teams developing awesome sites.

How hard should you push the envelope?

If you’re seeing a high rate of success from your tests, perhaps you need to start looking outside the box and broadening your tests’ horizons. On the other hand if you push it too hard, you risk slowing down the results (or screwing things up), whilst potentially finding massive, game-changing ideas.

Alternatively, you could also do a bit of research and start to understand your visitors before you prescribe “solutions”.

Sources: Ron Kohavi, Visual Website Optimizer, Jim Manzi’s book Uncontrolled, Avinash Kaushik’s book Web Analytics: An Hour a Day.