The paradox at the heart of A/B testing

• HEART, Paradox

7 min. (1416 words)

the-paradox-at-the-heart-of-a/b-testing

A/B testing began with beer.

At the turn of the 20th century, William Sealy Gosset was exploring new ways of running experiments on his production line. Gosset was trying to improve the quality of Guinness’s signature stout but couldn’t afford to run large-scale experiments on all the variables in play. Fortunately, in addition to being an astute brewer, Gosset was a statistician; he had a hunch there was a way of studying his small snatches of data to uncover new insights.

Gosset took a year off from his work to study alongside another scientist, Karl Pearson. Together, they developed a new way of comparing small data sets to test hypotheses. They published their work in the leading statistics publication at the time, Biometrika. In “The Probable Error of a Mean,” the t-test, a cornerstone of modern statistical analysis, was born.

Gosset’s scientific approach was the foundation of a 38-year career with Guinness. He invented more ways of using statistics to make business decisions; founded the statistics department at Guinness; led brewing at Guinness’s newest plant in 1935; and finally, in 1937, became the head of all brewing at Guinness.

Since its early days as a tool of science (and beer), statistical decision-making has gone supernova. Today, it is used by every major tech company to make hundreds of thousands of decisions every year. Data-driven tests decide everything from the effectiveness of political ads to a link’s particular shade of blue. New methods like Fisher testing, multivariate testing, and multi-armed bandit testing are all descendants of Gosset’s early innovations. The most popular of these statistical tests is one of the oldest: A/B testing.

An A/B test is a measurement of what happens when you make a single, small, controlled change. In product design, this means changing something in the interface, like the color of a button or the placement of a headline. To run an A/B test, show an unchanged version of the interface (“A”) to a randomly-selected group of users. Show a changed version (“B”) to another randomly-selected group. Measure the difference in the behavior of the two groups using a t-test, and you can confidently predict how the changed version will perform when shown to all your users.

A/B tests are easy to understand, which explains their popularity in modern software development. But their simplicity is deceptive. The fundamental ideas of A/B tests contain a paradox that calls their value into question.

Fredkin’s paradox

In The Society Of Mind, Marvin Minsky explored a phenomenon that I experience every day as a designer: people often prefer one thing over another, even when they can’t explain their preference.

We often speak of this with mixtures of defensiveness and pride.

“Art for Art’s sake.”

“l find it aesthetically pleasing.”

“l just like it.”

“There’s no accounting for it.“

Why do we take refuge in such vague, defiant declarations? ”There’s no accounting for it” sounds like a guilty child who’s been told to keep accounts. And “I just like it” sounds like a person who is hiding reasons too unworthy to admit.

Minsky recognized that our capriciousness serves a few purposes: We tend to prefer familiar things over unfamiliar. We prefer consistency to avoid distraction. We prefer the convenience of order to the vulnerability of individualism. All of these explanations boil down to one observation, which Minsky attributes to Edward Fredkin:

Fredkin’s Paradox: The more equally attractive two alternatives seem, the harder it can be to choose between them—no matter that, to the same degree, the choice can only matter less.

Fredkin’s Paradox is about equally attractive options. Picking between a blue shirt and a black shirt is hard when they both look good on you. Choices can be hard when the options are extremely similar, too — see the previous link to Google’s infamous “50 shades of blue” experiment. The paradox is that you spend the most time deliberating when your choice makes no difference.

Parkinson’s law of triviality

In 1955, C. Northcote Parkinson wrote a book called Parkinson’s Law. It’s a satire of government bureaucracy, written when the British Colonial Office was expanding despite the British Empire itself shrinking. In one chapter, Parkinson describes a fictional 11-person meeting with two agenda items: the plans for a nuclear power plant, and the plans for an employee bike shed.

The power plant is estimated to cost $10,000,000. Due to the technical complexity involved, many experts have weighed in. Only two attendees have a full grasp of the budget’s accuracy. These two struggle to discuss the plans, since none of the other attendees can contribute. After a two-minute discussion, the budget is approved.

The group moves on to the bike shed, estimated to cost $2,350. Everyone in the meeting can understand how the bike shed is built. They debate the material the roof is made of — is aluminum too expensive? — and the need for a bike shed at all — what will the employees want next, a garage? — for forty-five minutes. This budget is also approved.

This allegory illustrates what’s called “Parkinson’s law of triviality”:

The time spent on any item of the agenda will be in inverse proportion to the sum [of money] involved.

We can generalize Parkinson’s law: The effort spent discussing a decision will be inversely proportional to the value of making that decision.

When faced with two similar alternatives, Fredkin’s paradox predicts you’ll have a hard time choosing. This is when A/B testing is most appealing: A/B tests settle debates with data instead of deliberation. But our generalization of Parkinson’s law of triviality says that this kind of A/B testing — testing as an alternative to difficult decisions — results in the least value.

Most of the time, A/B testing is worthless. The time spent designing, running, analyzing, and taking action on an A/B test will usually outweigh the value of picking the more desirable option.

Alternatives to A/B testing

Instead of A/B testing, I’ll offer two suggestions. Both are cheaper and more impactful.

Alternative 1: observe five users

Tom Landauer and Jakob Nielsen demonstrated in Why You Only Need to Test with 5 Users that insights about design happen logarithmically — that is, the first five users you study will reveal more than 75% of your usability issues. Doing a simple observation study with five users is an affordable way of understanding not just how to improve your design, but also why those improvements work. That knowledge will inform future decisions where a single A/B test can’t.

Alternative 2: A → B testing

The cheapest way to test a small change is to simply make that change and see what happens. Think of it like a really efficient A/B test: instead of showing a small percentage of visitors the variation and waiting patiently for the results to be statistically significant, you’re showing 100% of visitors the variation and getting the results immediately.

A → B testing does not have the statistical rigor that A/B testing claims. But when the changes are small, they can be easily reversed or iterated on. A → B testing embraces the uncertainty of design, and opens the door to faster learning.

When A/B testing is the right tool for the job

A/B testing is worthless most of the time, but there are a few situations where it can be the right tool to use.

  1. When you only have one shot. Sales, event-based websites and apps, or debuts are not the time to iterate. If you’re working against the clock, an A/B test can allow you to confidently make real-time decisions and resolve usability problems.
  2. When there’s a lot of money on the line. If Amazon A → B tested the placement of their checkout button, they could lose millions of dollars in a single minute. High-value user behaviors have slim margins of error. They benefit from the risk mitigation that A/B testing provides.

Conclusion

If there’s a lot on the line in the form of tight timelines or lots of revenue at stake, A/B testing can be useful. When settling a debate over which color of button is better for your email newsletter, leave A/B testing on the shelf. Don’t get caught by the one-two punch of Fredkin’s paradox and Parkinson’s law of triviality — avoid these counterintuitive tendencies by diversifying your testing toolkit.

112 comments

  1. What’s up Dear, are you genuinely visiting this web page regularly, if so then you will without doubt obtain nice know-how.

  2. I feel this is among the most important information for me.
    And i am glad reading your article. However wanna remark on few general issues,
    The site style is great, the articles is in reality excellent : D.
    Excellent task, cheers

  3. I am not sure where you are getting your information, but great
    topic. I needs to spend some time learning
    much more or understanding more. Thanks for fantastic information I
    was looking for this info for my mission.

    Feel free to surf to my web-site … http://www.ltlszc.com

  4. What i don’t realize is in fact how you are no
    longer actually a lot more neatly-favored than you might be now.
    You’re very intelligent. You recognize thus significantly
    with regards to this matter, made me individually consider it from so many varied angles.
    Its like men and women don’t seem to be involved except it’s
    something to do with Lady gaga! Your own stuffs outstanding.

    Always take care of it up!

    Also visit my website; https://bbs.cnction.com/home.php?mod=space&uid=276589&do=profile&from=space

  5. I think that is one of the so much vital info for me.
    And i’m glad reading your article. But should commentary on some basic things, The site taste is perfect, the articles is in point of fact nice : D.
    Excellent task, cheers

    Take a look at my web blog :: anapa-alrosa.com.ru

  8. Excellent blog you have here but I was wanting to know if you knew of any message boards that cover the same topics discussed here?
    I’d really love to be a part of online community where I
    can get advice from other experienced individuals that share the same interest.
    If you have any suggestions, please let me know. Cheers!

    Check out my page – http://www.lubertsi.net

  10. I’m pretty pleased to uncover this website. I wanted to thank you for your
    time due to this wonderful read!! I definitely appreciated
    every little bit of it and I have you saved to
    fav to look at new stuff on your blog.

    my website :: chengdian.cc

  13. Needed to compose you this little remark to be able to give
    thanks once again with your wonderful pointers you’ve shared on this page.
    It’s so wonderfully open-handed of you in giving publicly precisely what many individuals could have supplied for an e
    book to get some bucks for themselves, specifically considering the fact that you could have done
    it if you desired. Those inspiring ideas in addition served to become easy way to fully grasp that
    other people online have the identical eagerness
    similar to my own to understand a whole lot more when considering this problem.
    I’m certain there are thousands of more pleasant moments up front for individuals who see your website.

    Visit my homepage :: http://zhsk-845.ru/forum/profile.php?id=19558

  15. Hey There. I found your blog using msn. This is a really well
    written article. I’ll make sure to bookmark it and come back to
    read more of your useful information. Thanks for the
    post. I’ll certainly return.

    Feel free to visit my homepage; game slot lpe88

  16. Saya akan segera rebut rss feed Anda karena saya tidak bisa
    Ini waktu terbaik untuk membuat beberapa rencana untuk masa depan dan sekarang waktu untuk berbahagia.
    Saya telah membaca posting ini dan jika saya bisa, saya ingin menyarankan Anda beberapa
    hal menarik atau saran . Mungkin Anda bisa menulis artikel berikutnya yang
    mengacu pada artikel ini. Saya ingin membaca lebih
    banyak lagi hal tentang itu!

    Feel free to visit my page Joker123 Co

  20. Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates.

    I’ve been looking for a plug-in like this for quite some time and was
    hoping maybe you would have some experience with something like this.
    Please let me know if you run into anything. I truly enjoy reading
    your blog and I look forward to your new updates.

  22. My developer is trying to convince me to move to .net from PHP.
    I have always disliked the idea because of the costs.
    But he’s tryiong none the less. I’ve been using WordPress on several websites for about a year
    and am nervous about switching to another platform.
    I have heard excellent things about blogengine.net.
    Is there a way I can transfer all my wordpress content into it?
    Any help would be greatly appreciated!

    My blog post … greatwall99 android download (https://sg918kiss.com/greatwall99)

  24. Hmm it appears like your website ate my first comment (it
    was extremely long) so I guess I’ll just sum it up what I
    wrote and say, I’m thoroughly enjoying your blog.
    I too am an aspiring blog writer but I’m still new to everything.

    Do you have any tips and hints for newbie blog writers?
    I’d genuinely appreciate it.

    Look at my web page – http://www.voininatangra.org

  25. I enjoy you because of your entire efforts on this site.
    My aunt loves engaging in investigation and it’s easy to see why.

    My spouse and i know all about the dynamic mode you offer effective
    steps on the blog and even cause participation from
    other people on that area then our favorite princess has always been starting to learn a great
    deal. Take pleasure in the remaining portion of the year.

    You are always carrying out a glorious job.

    Here is my blog: http://usedtiresbrowardcounty.com/modules.php?name=Your_Account&op=userinfo&username=WanGeorge

  28. I like the valuable info you provide to your articles. I will bookmark your blog and test again right here frequently.

    I’m slightly sure I will be told a lot of new stuff proper right here!
    Best of luck for the next!

    Here is my website chengdian.cc

  30. Undeniably imagine that which you said. Your favourite reason appeared to be
    on the net the easiest thing to take note of. I say to you, I definitely get annoyed whilst folks
    consider concerns that they just do not understand about.
    You controlled to hit the nail upon the highest
    and defined out the whole thing with no need side-effects ,
    other folks could take a signal. Will probably be back to get more.
    Thanks

    Also visit my web site virtualchurchcamp.com

  31. I loved as much as you’ll receive carried out right here.
    The sketch is attractive, your authored material stylish.

    nonetheless, you command get bought an nervousness over that you wish
    be delivering the following. unwell unquestionably come further formerly again since exactly the same
    nearly a lot often inside case you shield this hike.

    Review my web blog :: 163.30.42.16

  35. Hey would you mind letting me know which webhost you’re utilizing?

    I’ve loaded your blog in 3 different web browsers and I must say this blog loads a lot faster then most.
    Can you suggest a good web hosting provider at a honest price?
    Cheers, I appreciate it!

    Here is my page http://www.jujumaow.com

  38. I together with my buddies came following the excellent tips
    from your website and so suddenly I had a horrible suspicion I
    never thanked the website owner for them. These men were
    passionate to study them and have now quite simply been making the most of them.

    Appreciation for turning out to be indeed accommodating and
    for having such cool tips most people are really desperate to understand about.
    My very own sincere regret for not saying thanks to earlier.

    Check out my webpage; http://shihan.com.ru/

  39. of course like your web-site however you need to take a look at the spelling on quite a few of your posts.
    Many of them are rife with spelling issues and I to find it very troublesome to
    inform the reality however I will definitely come back
    again.

    Feel free to visit my page; frun-test.sakura.ne.jp

  42. Hi there, just became aware of your blog through Google, and
    found that it’s truly informative. I am going to watch
    out for brussels. I will appreciate if you continue this
    in future. Many people will be benefited from your writing.

    Cheers!

    my web site kebe.top

  48. Undeniably imagine that which you stated.
    Your favourite reason seemed to be at the net the simplest thing to have in mind
    of. I say to you, I definitely get annoyed at the same time as
    folks consider concerns that they just do
    not recognise about. You managed to hit the nail upon the highest and
    outlined out the entire thing with no need side effect ,
    folks can take a signal. Will likely be again to get more.
    Thank you!

    Here is my blog post … kebe.top

  51. I became honored to receive a call from my friend immediately he found out the important recommendations shared on your own site.

    Studying your blog post is a real wonderful experience. Many thanks for taking into account readers like me, and I would
    like for you the best of achievements for a professional in this arena.

    my web site http://www.craksracing.com/

  52. I am not sure where you are getting your info, but good topic.
    I needs to spend some time learning much more or understanding more.
    Thanks for excellent information I was looking for this info for my mission.

  55. I enjoy you because of all your valuable labor on this site.

    Ellie take interest in doing investigation and it is simple to grasp why.
    Almost all learn all relating to the dynamic ways you create simple tactics by means of your website and even cause participation from
    website visitors on that article so our favorite simple princess is certainly learning
    a whole lot. Enjoy the remaining portion of the new year.
    You are conducting a stunning job.

    Here is my web-site :: http://chengdian.cc/forum.php?mod=viewthread&tid=141051

  56. I was just seeking this info for a while. After six hours of continuous Googleing,
    finally I got it in your site. I wonder what is the lack
    of Google strategy that do not rank this kind of informative
    sites in top of the list. Usually the top web sites are full of garbage.

    Also visit my blog post :: 1stanapa.ru

  59. What i do not realize is actually how you’re now not actually much more
    well-liked than you may be right now. You’re very intelligent.
    You know thus considerably when it comes to this subject,
    made me in my view consider it from a lot of numerous angles.

    Its like women and men aren’t fascinated unless it
    is something to do with Woman gaga! Your individual stuffs great.
    Always take care of it up!

    My blog post – chengdian.cc

  60. I have been surfing online greater than three hours lately, but
    I by no means found any interesting article like
    yours. It is pretty value enough for me. Personally, if all web owners
    and bloggers made excellent content material as you did, the web shall be
    a lot more helpful than ever before.

    Visit my page; http://www.craksracing.com/modules.php?name=Your_Account&op=userinfo&username=RoemerGenia

  61. Hey I know this is off topic but I was wondering if you knew
    of any widgets I could add to my blog that automatically
    tweet my newest twitter updates. I’ve been looking
    for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this.
    Please let me know if you run into anything. I truly enjoy reading your blog and I
    look forward to your new updates.

    Look into my web-site duna-anapa.net.ru

  64. When I initially left a comment I appear to have clicked the -Notify me when new comments are added- checkbox and from now on whenever a comment is
    added I receive 4 emails with the same comment. Is there a
    means you are able to remove me from that service? Many thanks!

    Have a look at my blog … anapa-alrosa.com.ru

  72. Hey! Quick question that’s totally off topic.
    Do you know how to make your site mobile friendly? My website looks weird when browsing from my iphone.
    I’m trying to find a theme or plugin that might be able to
    correct this issue. If you have any suggestions, please share.

    Many thanks!

    Feel free to visit my blog post; anapa-alrosa.com.ru

  74. I needed to thank you for this wonderful read!!
    I certainly enjoyed every little bit of it. I’ve got you saved
    as a favorite to check out new stuff you post?

    Feel free to visit my web site :: kebe.top

  80. hi!,I love your writing so much! proportion we be in contact extra approximately your post on AOL?
    I need an expert on this area to unravel my problem. Maybe that’s you!
    Taking a look ahead to peer you.

    My site – mpc-install.com

  85. I visited many web sites except the audio feature for audio songs current at this website is genuinely superb.

    Here is my site; Lucas

  87. With havin so much content do you ever run into any issues of plagorism or copyright infringement?
    My site has a lot of unique content I’ve either authored
    myself or outsourced but it appears a lot of
    it is popping it up all over the internet without my agreement.
    Do you know any methods to help reduce content from being stolen?
    I’d really appreciate it.

    my homepage :: Keto Premium Shot Review

  88. This is the right site for anybody who wishes to
    understand this topic. You understand so much its almost
    tough to argue with you (not that I actually will need to?HaHa).

    You definitely put a new spin on a topic that has been discussed for ages.

    Wonderful stuff, just excellent!

    Here is my web page :: http://shihan.com.ru

  89. Hello just wanted to give you a quick heads up. The text in your post seem
    to be running off the screen in Ie. I’m not sure if this is a format issue
    or something to do with internet browser compatibility but I thought
    I’d post to let you know. The design look great though!
    Hope you get the issue resolved soon. Cheers

    Also visit my website: mpc-install.com

  90. We are a gaggle of volunteers and starting a brand new scheme in our community.

    Your website provided us with valuable info to paintings on. You’ve performed an impressive activity and
    our whole group can be thankful to you.

    My homepage – mpc-install.com

  91. you’re really a just right webmaster. The site loading pace is amazing.
    It sort of feels that you’re doing any unique trick. Moreover,
    The contents are masterpiece. you’ve performed a fantastic
    activity in this subject!

    My web blog Tri-Bol Testo

  104. I just could not depart your web site before suggesting that I actually loved the standard information a person supply for your guests?
    Is gonna be again continuously to check out new posts.

    Also visit my homepage – Slimingo Keto

  105. Good day! I could have sworn I’ve been to this website before but after browsing through some of the post I
    realized it’s new to me. Anyways, I’m definitely glad
    I found it and I’ll be bookmarking and checking back frequently!

    Have a look at my website :: Max Brute

  106. Wow, incredible weblog structure! How long have you ever been running a blog for?

    you make running a blog look easy. The full glance of
    your site is magnificent, let alone the
    content![X-N-E-W-L-I-N-S-P-I-N-X]I just could not leave
    your site before suggesting that I actually enjoyed the standard info a person supply on your
    visitors? Is going to be again continuously to check out new posts.

    Look into my web-site … http://www.aniene.net

  111. Pretty section of content. I just stumbled upon your
    web site and in accession capital to assert that I get
    in fact enjoyed account your blog posts. Anyway I’ll be subscribing
    to your augment and even I achievement you access
    consistently quickly.

    Here is my web site :: vip5.moisait2021.ru

Leave a Reply

Your email address will not be published.