Second last question! Another one about A/B testing, and you might be surprised how much I hear this one in real life.
Today we will answer:

“What if my favorite design doesn’t win the A/B test?”

Just starting the Crash Course? Start here!

****

Ok, so here’s the scenario: 

You have come up with a solid hypothesis for an A/B test. You have persuaded your boss/client that it could be important. You designed a couple versions of the page, set up the test, ran it, and… the new design loses.

Now you feel like an idiot, right? 

You’re doomed, and they will never trust you again, right?

Not exactly. 

So what do you do?

****

The stupid answer:

“You keep using the old design, obviously.”

Yes, that’s true. But not super helpful. 

****

The real answer:

Yes, obviously you shouldn’t use the losing version just because you like it more. If you were even considering that, stop it. Don’t be a knob.

The best version is the best version, whether you like it or not. Whether a design is your “favorite” or not is 100% irrelevant, and nobody cares. Print the design out, take it home, tape it on your bedroom wall, and give it a kiss before bed each night.

But for the love of everything holy, don’t use it. 

****

You should re-evaluate a couple things:

1) Was your hypothesis good? 

It’s possible that you just fell in love with an idea, or an assumption, and the test has revealed that your hypothesis sucked. Maybe you should look at your research again and try to think of other things that might be causing the problem you were testing.

I usually try to think of a follow-up A/B test before I do the first one, just to make sure I am seeing it properly. Assume you will lose!

2) Did your designs actually match the hypothesis?

You wouldn’t be the first person that used a good hypothesis as an excuse to add pink unicorns to the grocery store’s website, even though your hypothesis was something about milk and eggs. 

No, actually you were probably the first person to do that.

But make sure you’re designing things that actually test your hypothesis, and not just new designs you want to “try”.

****

Why this isn’t a stupid question:
Good experiments can’t fail. 

Humans are competitive. And we have pride. And we’re not rational. So we tend to look at an A/B test as a battle.

It isn’t.

If you did everything properly, you should learn something regardless of which design “wins” the test.

If your “favorite” doesn’t win, and you have done the test properly, now you know that your hypothesis is not the problem. That’s good! You didn’t know that before.

You’re investigating, not competing! Keep testing!

If you are thinking about the A/B test as “my awesome idea vs. that old shitty idea”, then you’re doing it wrong. That’s not science. That’s ego.

You will get wrapped up in whether you “win”, and you will start interpreting results as “pretty close” so “maybe we can use the cool one anyway” and then you’re running full-speed down the path to StupidTown.

If you run several tests and the existing design is hard to beat, rethink your approach. You might be testing the wrong stuff! Are you even sure there is a problem?!

Design your A/B tests so you learn something.
Science isn’t about winning.

****

Tomorrow, for our last lesson, we will conquer the smartest stupid question of them all: “What the hell is a multivariate test?”