Tag Archives: Metrics

A/B testing is not new!!! We are just realizing the importance of it and Starbucks a technical company

Leave a reply
Lately I have been hearing a lot about how great A/B testing is and that it is  the best thing since slice bread!  Please understand I am a big fan of A/B testing  but let us get real, this is not something new. The web like everything has an amplifying factor of a million, which is why it is like some hot stuff … just like BigData … it just did not happen over night and someone had some sort of epiphany and lo behold a phenomenon was born!  Everything takes time to take hold and become a phenomenon ….  and while it is taking time to take hold there are a series of A/B tests that are going on to make sure what takes hold and what does not. Practice that learn form the results survive and those who don’t well they perish!

I am from a software development back ground so oh BTW agile development, continuous delivery, DevOps or as I like calling it BizOps are all forms A/B testing.  
A/B testing is so ingrained in our lives we just don’t realize it (just ask the hospitals and the Pharmaceutical folks, they have doing it for quite some time).  I believe one of the best A/B tests that marketed well was the “Pepsi Challenge” by John Scully… to bad he messed it up for Apple though. 
Why are A/B tests important ? Well because they help us build relationships (again us not realizing it). Everyday there are several A/B tests happening my house…. how can I make daughter do her home work ? or what can I do to improve my wife’s experience in being my wife everyday … (BTW sometimes I do get lazy … and pay the price). I would go even further to say A/B tests are so ingrained in our DNA, we just don’t realize it. I am going to share 2 old stories and an example of how A/B testing working even today outside of the web context … there are so many examples to choose from which makes it even harder….
 Early human hunters
         Imagine you are part of an hunter tribe in the early days of mankind, and you realize that you need to hunt something big to keep the village fed during the winter months. Back in the day when when did not have robots, the chief of the tribe would select one of their stronger men to go and hunt alone. The validation if the person came back alive with the hunt it was a successful expedition. Imagine you start realizing that the male population of the tribe is getting thinned out because of the lack of predictability of your hunt ( you realize sending one person for a hunt is your control group), this time you decide you are going to send 2 together with the explicit instruction that they work together and try to get a bigger animal. As luck would have it the coordination works now you figure out that sending one person alone does not make sense because sending 2 people for the hunt gets a better outcome for the tribe.. Next time you try sending out a larger hunting party to see what they can get. Hence you see humans have been at this A/B testing stuff for a long time… it is just that now we can do it a greater scale and with lot less loss of life.

Alexander the great and the Persians
This one is an example of not only A/B testing but also of Big Data versus Smart Data. Imagine the small army of Alexander taking on the might of Persia and yet Alexander’s small army bought the behemoth Persian empire to its knees.  Prior to the confrontation with the Persians, Alexander had a lot chance to experiment with other armies that the Greeks took down, he was able to implement strategic plays and work out ways to make his small army into a effective execution machine. I am not saying that the Persian Army lack discipline, they actually fought by the book (i.e. something that worked before should work the same way except now they have the bigger size).  Persian Army did not challenge their conventional norms and just assumed that their army would just function as well with the larger contingent of soldiers (they did not A/B test) and in the context of Alexander they  were so drunk with the size of their army that they did not take into account the terrain and it’s advantages offered to Alexander’s army. Plus the motivation that Alexander had made it very clear that they were moving forward not backwards actually helped the army to be more creative.
Another example of A/B testing at work and with some significant constraints too (actually all good A/B tests work with constraint, this is no different). Although it helped Alexander and team to win over the Persians … they eventually got as lazy with their power just like the Persians became

StarBucks !
I think starbucks is a great technical company. We all joke that there is a Starbucks every two blocks in the big cities. But what is going on here? Yes I know their menu across the all the a starbucks is the same but there is a subtlety. Have you noticed how starbucks promo’s are different in their different stores (even if the stores are just blocks away). Starbuck A/B tests so many different variants of coffee drink (just of the non-fat, soy based, vanilla bean latte that you drink next time). They know exactly which store has a demand for what type of variant of their coffee and it’s volume. Which helps them manage their inventory better…. which is why I believe this company may have it’s rough patches but it will come through …because they are always testing what take the Starbucks experience to the next level!

I know this was not one my regular types of Blog post…. But I wanted to make a point about A/B testing, DevOps all these new “Buzz Words” are great to push us forward. But we need to take a step back and look at this is in the grand scheme of things. What we are witnessing is the extreme scale of what we have always done without thinking twice and now we are applying in a more conscientious and effective ways to enrich our lives.

Let me know what you think and if you have any stories to share feel free to share your thoughts on with me on Twitter..

BTW A big shout out to my readers in Tanzania ! I hope you find my content useful !

5 Web Metrics to track for online commerce for Newbies

Leave a reply

After my April blog on vanity metrics and real metrics (http://goo.gl/7KYj5) I have had a chance to meet several people,who are curious about web metrics but do not know where to start. I agree someof us a very adept at reading the googleanalytics blog and come up to speed to read Avinash Kaushik’s blog Occam’s Razor. Very few blogs cover how getstarted and what to focus on, over the last few years I have narrowed 5 keymetrics that work for me (BTW there is a certain level of comfort-ability oneneeds to have in order get these metrics). These metrics are multi-purpose aswell, they not only show you how your business is doing but also highlight theweb experience that a potential client might be going through via your website
The 5 metrics that have worked for me (especially if I want to track the buying funnel) are as follows:
  • · Exit Rates
  • · Churn Rates
  • · Cart Abandonment rates (this is needed if youhave a e-commerce site)
  • · Average Days to Purchase
  • · Average visits to Purchase

I am not inferring that metrics like Unique Visitors, Pageviews, Bounce Rates etc. are not relevant. They are,it just depends on the goal you want to accomplish. In the case of this blog the assumption is that there is a web site on the internet that is just starting and wants to do some level of commerce

Exit Rates:

I am big fan of exit rates because to me they can tell you which page of your web presence is leaking. Please do not confuse this with Bounce rates. Bounce rates are metrics that anyone who lands on your page and then just leaves. If your page has a clear call to action (CTA) and you have a high bounce rate then that is something to look into. Exit rates focus more on the page flow of the session i.e. which people did not just land on the page but more about how people navigate their way during the session
Here is an example of an exit rate
Monday
Page 1 -> Page 2-> Page 3
Tuesday
Page 2->Page 1-> Page 3
Wednesday
Page 2 -> Exit
Thursday
Page 1-> Page 3-> Exit
Friday
Page 2 -> Page 3 -> Exit
In the case above the exit rates would be calculated asfollows
· Page 1 would be 33% (3 of 5 sessions included Page 1)
· Page 2 would be 50% (4 of the 5 sessionsincluded Page 2)
· Page 3 would be 50% (4 of the 4 sessionsincluded Page 3)
Why are exit rates important? They give an idea of how youlead generation activities are performing especially in the context of thefunnel. In the case shown above I would want so see why Page 2 and Page 3 havehigh exit rates? You should ask yourself the question like are you call toactions not compelling enough?

Churn Rates

Churn rates deal with how are you retaining the people thatyou already have. This metric is borrowed from the Mobile operator industry. Ifyou are not seeing people renew (as inthe case of SaaS offerings) or even if you see that people that sign up for atrial once and do not return again (prospective customers). For the latterthere is a stronger content play especially if you selling downloadable onpremise solution. What you want is that after they download they come back toview more content and other key things back at your site. You can complementthis metric with some of the social media metrics especially for prospectingclients as in which user decided to comeback and contribute to forum or posteda comment on a blog post etc. You canalso track based on the cookies you have placed for a particular user etc.There are various ways to track churn via the web.

Abandonment Rate

If you have an ecommerce site then you should track thenumber people filled their shopping cart and then decided not to followthrough. If you are using Inside sales for closing your deals, then you need totrack the inside sales metric of loss percentages for every lead that came inthe funnel and the opportunity was lost.

Average Days to Purchase

This is about how long it takes some to purchase. This theweb term for the average sales cycle i.e. how long does it take to closebusiness. This metric can help you identify whether your offering hasself-evident value. Products are just tosimple and convert easily to sale others usually take a long time and that youwant to fix.

Average visits to purchase

This metric is about how many times a user visits beforethey purchase. This metric gives you an idea of how frequently someone iscoming to your site before they buy. This metric can help in making sure theright content is available to provide the right level of information to assistin the buying decision. This can also help optimize the web experience and helpyour clients go down the funnel you are guiding them too.
Like I said in the beginning of the blog, these are the metricsthat will help you get started. Not only will these metrics help you understandyour web business but also improve your web experience as well. I look forward to hearing your thoughts aboutmetrics that help you in your web business. You can contact me @ kkanakas on twitter with your comments

Web Strategy: What is a good bounce rate?

Leave a reply
I don’t know how many times I have heard that question, that is like asking how many times do get acknowledged when you say hello?  Sorry to sound to presumptuous here but it would be a wasted conversation because the person asking the question does not understand the web or even does not know what outcome they want to drive and hence ask for a “prescriptive” set of numbers so that they can measure against.

 Why is it hard for people to be more outcome driven? Because bounce rate although they are an ok metric, it is contingent upon other factors such as :
Location: If you are targeting a particular geography and you have high bounce rates from sources outside of the geography should you care ?
New vs Returning: This is another factor that you need to consider on the bounce rates. New users are great but if you have a higher rate of people returning then that is a good thing
Device: What are the devices people using to come to your website? If you find a high bounce rate because your site is not mobile optimized.Then you should do something about that
Medium: How are people coming to your site? SEO, Email campaigns, Social media, Paid Search, organic etc. If you break up your traffic this way you will be able to see what are some of the more effective mediums to reach out to your clientele.
All these divisions/ segments are interrelated I guess the net of this blog is that know you what your outcomes are and segment accordingly.  Bounce rates are great start if you don’t know web metrics but what you should be really looking at are exit rates (thanks to Avinash Kaushik: http://goo.gl/qjpn0). To me exit rates are more revealing about the leaks in your conversion funnel and what you need to fix. 
Another more important metric would be churn rates, similar to exit rates except churn rates focus on the customers you already have. What are you doing to take care of your existing customers and at what rate are they leaving (http://goo.gl/t1ADj).

So when some asks you about what is the ideal bounce rate? Please take the time to educate them and let them understand what outcomes are they trying to drive ( Increase registrations, downloads, donations, buy products etc,)

Caveat: If your company is obsessed  about bounce rates, then you have a good inkling that the organization has never thought of the web as a viable channel and you have long road to educate people.

You can contact me @ kkanakas on twitter with your comments

3 things that need to be considered when Job Mapping with Jobs to be done

Leave a reply

My first blog after a long summer break, I hope all of you had a safe and sound summer as well.  I did catch up on a lot well-deserved reading but somehow my reading list from amazon has not gotten shorter. I bet you if Amazon did a customer lifetime value analysis on me, I think Amazon has probably made quite a pretty penny off of me just on books.
When I wrote the “Jobs to be done” write up on my blog, I did get a few feedback from folks who read it.  Some of them mentioned that the “Jobs to be done” notion sounds great but it is mostly theoretical (Cleary some folks have not read Clay Christensen’s Innovator’s solution), because when have to jobs you want get done in real life there are constraints. To which I thought a little bit and realized, and a little inspiration from the book The Fortune at the Bottom of the Pyramid by C.K. Prahalad, that innovation is not limited by constraints but actually happens because of it. With realization I went through the process of retracing my steps on activities which forced me to improvise or bootstrap projects in the past, the reason I had to improvise were because constraints along with a few other things such as effort and risk. Considering these limits in mind I did proceed to execute with a series of actions.  I know this sounds trivial because this behavior is so ingrained in us that most of us don’t think twice.  So this blog is dedicated to those 3 things that me realize I had not done justice to my earlier blog on “Jobs to be done”.

Constraints

For innovation to happen constraints can come in various forms. These constraints could be the usual suspects like finance/budget, resources, business model at a micro level or these constraints can be imposed at a macro level such as Political, Environmental, Social, or Technological. There is a reason why such decision-making tools exist whether it is the SWOT analysis framework or the PEST analysis framework. These tools allow us to look at the constraint in an objective way and figure out the right course of action.  I mean think about if these constraints were not put in place how would we have gotten some of the greatest innovations of our time. These constraints become even more relevant as our economy moves on from a post-industrialized society to knowledge driven society. Just think about the Reinvent the toilet challenge held by the gates foundation with a very simple constraint “it should be useable by 2.5 billion people that do not have access simple, hygienic, and sustainable toilets”.

Effort

Beyond the constraint this is usually the top most thing in our mind. I know it because every time I have been given a project my initial request is always more resources and the answer usually is work with what you have.  Because of this limitation, I am forced to improvise and work around ways to get my job done. My usual formula is the get the maximum efficiency with the most minimum amount of effort but it does force me to think in ways that I would not thought in an environment that is more comfortable. That is the other thing comfortable surrounding means bad news to me it means you are going to go stale pretty quickly  (that is topic for a blog later)

Risk

This is the other big one. With everything you do there is an element of risk. The only question you have to ask yourself is what is the threshold you and are team can withstand.  I know there are plenty of books and magazines that romanticize the notion of risk taking, which sound a hell of lot cooler than saying that before a so called rock star CEO was about embark on a risky proposition he/she was sweating bullets. Which is why I am a big fan of things like continuous integration, delivery, A/B testing, and data driven analysis. All these paradigms help reducing the cone of uncertainity that we know as risk. But with all these tools that we have to help us make better decision a few complimentary tools help us in making the right decisions and they are “Observing people and what they do, and sometimes just plain listening”.  Both these non-technical elements that give you ample insight on what you need to do next to innovate (Yes, engaging LinkedIn groups can be a proxy provided that you actually engage in dialog in those groups)
I would like to add a fourth element to this as well (Yes, I know I said 3 things in the blog title, but you folks have not heard from me throughout the summer, so I feel compelled to share with you a little bit more).

Metric

What is the measure of success that people use for the job they want to get done? This dawned on me the other day that in the West, we don’t think about price when we know that a product has a high level of performance and reliability. But if you think about products in the emerging markets, performance is the least of the concerns, their threshold is so low that they want to be able to do basic things at reasonable price.  In both these cases you can observe that the metric of success for one constituent is performance whereas for the other constituent it is the price.  You can take a performance product and place it a price sensitive market but at best you will skim the market but if you listen to price person you might design a product that could not only work in the emerging markets but also in the rest of world there by creating a new market (this is part is a bonus).  The intent of this section to make sure success metric is not forgotten. Too often we get carried away by other things but really do not pay attention to actual metric that will define success for the target constituent. This is something we need to pay attention do and it might just be a statement like “Minimize the time to reinitialize after an emergency to eject thrusters”
Just take a look at what GEis doing on this whole notion of reverse innovation, it really shows that they looking into all 4 things, when it comes to the jobs to be done framework and it’s implementation.
As always I look forward to a very enriching discussion. You can contact me @ kkanakas on twitter with your comments