Whether you are evaluating a new idea, or designing a new solution, or testing different versions of your product, there is always a chance you won’t improve on the existing design.

And that isn’t a bad thing.

Being rational isn’t as easy as it sounds.

Almost every time I have introduced the idea of A/B testing — launching several versions of a design to find the best one — I have seen apprehension in my client’s / boss’s / girlfriend’s eyes.

They were worried.

****

Many people look at experiments in a glass-half-empty way:

“You’re going to expose some users to something that is worse?”

“You’re going to spend time on proving that we suck?”

“We might not use the new designs at all?!”

Technically, the answer to all of those is “maybe.”

I have done tests where every new version is better than the current version.

And sometimes every new version is worse.

The thing is: you don’t know which versions are worse until you test them.

What if the worst version is the one every user is using right now?

****

Uncertainty is a part of life, and UX. Get comfortable with it.

We deal with a lot of subjective, “gray area” things, and those are the most valuable things to test. It is hard to guess which subjective thing your particular audience will prefer.

And that’s a good thing!

It is a mistake to think that you can “fail” when you do an experiment.

You can pick a favorite that doesn’t win. But that isn’t a failure.

You can have a hypothesis that turns out to be wrong. That isn’t a failure either.

And you might prove that all of your ideas are worse than the existing solution. Also not a failure.

How you feel about the designs you’re testing is irrelevant.

****

The purpose of an experiment is to find the truth.

Not to prove that you are right.

Huge difference.

No winners or losers. No right or wrong. No happy or sad.

Only true or false.

Does Version A perform better than Version B? Yes or no?

When your favorite design doesn’t “win” the test, you still know that the other version is better, which will benefit you in the long-run.

When your hypothesis is wrong, you still know not to follow that path, which will save you a lot of time and resources.

And when you prove that the existing solution is better than all your new ideas, you save yourself (and your company) from falling in love with something because it is shiny and new — not because it is better.

Even if all the versions are exactly equal, it probably means that whatever you’re testing isn’t the important part, which is extremely valuable to know.

****

The only way an experiment can fail is if the result teaches you nothing. (i.e. — a shitty experiment.)