This is it: the final question. And it’s a doozy. With all the talk of A/B tests in the UX community, it’s funny that you never hear about their more sophisticated cousin very often. So today we answer:
“What the hell is a multivariate test?”
Just starting the Crash Course? Start here!
****
An A/B test is not the only kind of test you can run on a design. And in my opinion multivariate tests are much more fun, because there is a sense that “anything can happen”.
****
The stupid answer:
“A multi-what-now? Never heard of it. Just do an A/B test instead.”
****
The real answer:
An A/B test usually tests one variable. That is, one specific change.
When you do an A/B test, you can be confident that the result is caused by that specific thing, because everything else should have stayed the same.
Everything else was controlled.
But what if you want to test a relationship between two things, so they are both changing?
*head explodes*
That’s a multivariate test. “Multiple variables”.
****
Why this isn’t a stupid question:
Design changes can affect each other.
When you change one thing on your page or site, it can affect how the users think about something else.
Let’s say you have 3 options for headlines:
Headline 1: “This is the greatest thing ever!”
Headline 2: “This is the worst thing ever!”
Headline 3: “This is kind of really ok, ish!”
But you also have three options for photos to go with those headlines!
Photo 1: A puppy.
Photo 2: A hamburger.
Photo 3: Your mom.
That doesn’t sound so bad. Seems like an A/B/C test. Right?
Nope.
Here’s the kicker:
Any of the headlines will work with any of the photos. But depending on the combination, the user might have different reactions.
Maybe some people want to see the greatest thing ever, but only if the photo shows something they like.
Maybe some people would love to see the worst thing ever, but only if the photos shows something they hate.
And maybe there is a “safe” option where the headline doesn’t create any expectations, and the photo will determine whether the user is interested or not.
Or maybe “the worst thing ever” creates the most curiosity when the photo is a puppy! Or maybe your mom makes every headline more effective!
Very subjective. Very complicated.
How would you know what combination to use!?
A multivariate test, that’s how.
****
Multivariate tests think harder than you can.
In this case there are 9 combinations of photos and headlines.
Headline 1 with photo 1, 2 or 3.
Headline 2 with photo 1, 2, or 3.
And headline 3 with photo 1, 2, or 3.
That is a lot of information to compare in your head. You will never be able to make a rational guess about which combination of headline and photo will be most popular.
So don’t!
Send a whole bunch of traffic to that page, let the software pick combinations of headlines and photos randomly, and at the end it will tell you that Headline 2 with Photo 3 got the highest percentage of clicks. Or Headline 1 and photo 1. Or whatever.
Multivariate tests need more traffic than an A/B test because there are more combinations to test, but they also produce answers that would be a pain in the ass using A/B testing.
****
CONGRATULATIONS! You’re reached the end of the UX Crash Course: 30 Stupid Questions.