Lately I have been hearing a lot about how great A/B testing is and that it is the best thing since slice bread! Please understand I am a big fan of A/B testing but let us get real, this is not something new. The web like everything has an amplifying factor of a million, which is why it is like some hot stuff … just like BigData … it just did not happen over night and someone had some sort of epiphany and lo behold a phenomenon was born! Everything takes time to take hold and become a phenomenon …. and while it is taking time to take hold there are a series of A/B tests that are going on to make sure what takes hold and what does not. Practice that learn form the results survive and those who don’t well they perish!
I am from a software development back ground so oh BTW agile development, continuous delivery, DevOps or as I like calling it BizOps are all forms A/B testing.
A/B testing is so ingrained in our lives we just don’t realize it (just ask the hospitals and the Pharmaceutical folks, they have doing it for quite some time). I believe one of the best A/B tests that marketed well was the “Pepsi Challenge” by John Scully… to bad he messed it up for Apple though.
Why are A/B tests important ? Well because they help us build relationships (again us not realizing it). Everyday there are several A/B tests happening my house…. how can I make daughter do her home work ? or what can I do to improve my wife’s experience in being my wife everyday … (BTW sometimes I do get lazy … and pay the price). I would go even further to say A/B tests are so ingrained in our DNA, we just don’t realize it. I am going to share 2 old stories and an example of how A/B testing working even today outside of the web context … there are so many examples to choose from which makes it even harder….
Early human hunters
Imagine you are part of an hunter tribe in the early days of mankind, and you realize that you need to hunt something big to keep the village fed during the winter months. Back in the day when when did not have robots, the chief of the tribe would select one of their stronger men to go and hunt alone. The validation if the person came back alive with the hunt it was a successful expedition. Imagine you start realizing that the male population of the tribe is getting thinned out because of the lack of predictability of your hunt ( you realize sending one person for a hunt is your control group), this time you decide you are going to send 2 together with the explicit instruction that they work together and try to get a bigger animal. As luck would have it the coordination works now you figure out that sending one person alone does not make sense because sending 2 people for the hunt gets a better outcome for the tribe.. Next time you try sending out a larger hunting party to see what they can get. Hence you see humans have been at this A/B testing stuff for a long time… it is just that now we can do it a greater scale and with lot less loss of life.
Imagine you are part of an hunter tribe in the early days of mankind, and you realize that you need to hunt something big to keep the village fed during the winter months. Back in the day when when did not have robots, the chief of the tribe would select one of their stronger men to go and hunt alone. The validation if the person came back alive with the hunt it was a successful expedition. Imagine you start realizing that the male population of the tribe is getting thinned out because of the lack of predictability of your hunt ( you realize sending one person for a hunt is your control group), this time you decide you are going to send 2 together with the explicit instruction that they work together and try to get a bigger animal. As luck would have it the coordination works now you figure out that sending one person alone does not make sense because sending 2 people for the hunt gets a better outcome for the tribe.. Next time you try sending out a larger hunting party to see what they can get. Hence you see humans have been at this A/B testing stuff for a long time… it is just that now we can do it a greater scale and with lot less loss of life.
Alexander the great and the Persians
This one is an example of not only A/B testing but also of Big Data versus Smart Data. Imagine the small army of Alexander taking on the might of Persia and yet Alexander’s small army bought the behemoth Persian empire to its knees. Prior to the confrontation with the Persians, Alexander had a lot chance to experiment with other armies that the Greeks took down, he was able to implement strategic plays and work out ways to make his small army into a effective execution machine. I am not saying that the Persian Army lack discipline, they actually fought by the book (i.e. something that worked before should work the same way except now they have the bigger size). Persian Army did not challenge their conventional norms and just assumed that their army would just function as well with the larger contingent of soldiers (they did not A/B test) and in the context of Alexander they were so drunk with the size of their army that they did not take into account the terrain and it’s advantages offered to Alexander’s army. Plus the motivation that Alexander had made it very clear that they were moving forward not backwards actually helped the army to be more creative.
Another example of A/B testing at work and with some significant constraints too (actually all good A/B tests work with constraint, this is no different). Although it helped Alexander and team to win over the Persians … they eventually got as lazy with their power just like the Persians became
This one is an example of not only A/B testing but also of Big Data versus Smart Data. Imagine the small army of Alexander taking on the might of Persia and yet Alexander’s small army bought the behemoth Persian empire to its knees. Prior to the confrontation with the Persians, Alexander had a lot chance to experiment with other armies that the Greeks took down, he was able to implement strategic plays and work out ways to make his small army into a effective execution machine. I am not saying that the Persian Army lack discipline, they actually fought by the book (i.e. something that worked before should work the same way except now they have the bigger size). Persian Army did not challenge their conventional norms and just assumed that their army would just function as well with the larger contingent of soldiers (they did not A/B test) and in the context of Alexander they were so drunk with the size of their army that they did not take into account the terrain and it’s advantages offered to Alexander’s army. Plus the motivation that Alexander had made it very clear that they were moving forward not backwards actually helped the army to be more creative.
Another example of A/B testing at work and with some significant constraints too (actually all good A/B tests work with constraint, this is no different). Although it helped Alexander and team to win over the Persians … they eventually got as lazy with their power just like the Persians became
StarBucks !
I think starbucks is a great technical company. We all joke that there is a Starbucks every two blocks in the big cities. But what is going on here? Yes I know their menu across the all the a starbucks is the same but there is a subtlety. Have you noticed how starbucks promo’s are different in their different stores (even if the stores are just blocks away). Starbuck A/B tests so many different variants of coffee drink (just of the non-fat, soy based, vanilla bean latte that you drink next time). They know exactly which store has a demand for what type of variant of their coffee and it’s volume. Which helps them manage their inventory better…. which is why I believe this company may have it’s rough patches but it will come through …because they are always testing what take the Starbucks experience to the next level!
I know this was not one my regular types of Blog post…. But I wanted to make a point about A/B testing, DevOps all these new “Buzz Words” are great to push us forward. But we need to take a step back and look at this is in the grand scheme of things. What we are witnessing is the extreme scale of what we have always done without thinking twice and now we are applying in a more conscientious and effective ways to enrich our lives.
Let me know what you think and if you have any stories to share feel free to share your thoughts on with me on Twitter..
BTW A big shout out to my readers in Tanzania ! I hope you find my content useful !
Vital Cog 