It is conventional wisdom that A/B testing (or in proper terms, randomized controlled experiments) is the gold standard for causal analysis, meaning if you run an A/B test, you know what caused an effect. In practice, this is not always true. Sometimes, the A/B test only provides a statistical understanding of causes but not an average Joe's understanding. Let's start with a hypothetical example in which both definitions are aligned.…
I answered some questions from KDnuggets. Here is Part 1 of the interview. They picked this image.
In Part 1, I covered the logic behind recent changes to the statistical analysis used in standard reports by Optimizely. In Part 2, I ponder what this change means for more sophisticated customers--those who are following the proper protocols for classical design of experiments, such as running tests of predetermined sample sizes, adjusting for multiple comparisons, and constructing and analyzing multivariate tests using regression with interactions. For this segment, the…
In my HBR article about A/B testing (link), I described one of the key managerial problems related to A/B testing--the surplus of “positive” results that don’t quite seem to add up. In particular, I mentioned this issue: When managers are reading hour-by-hour results, they will sometimes find large gaps between Groups A and B, and demand prompt reaction. Almost all such fluctuations result from temporary imbalance between the two groups,…
There have been few updates as I was working on things for other people. One of these things showed up today. Here is an excerpt from the beginning of my new article on HBR: For over 10 years and at three companies, I set up and ran A/B testing programs, in which we test a new offer with half a sample against a control group which doesn’t get a new…
Here are five amazing recommendations by Avinash Kaushik from a post about how to make Web analytics dashboards better by simplifying. Dashboards are not reports. Don't data puke. Include insights. Include recommendations for actions. Include business impact. NEVER leave data interpretation to the executives (let them opine on your recommendations for actions with benefit of their wisdom and awareness of business strategy). When it comes to key performance indicators, segments…
For those in Boston/Cambridge, I will be speaking at the Chief Data Scientist meetup on Wednesday night. See you there. *** Warning: this post may be hard to understand if you don't know SQL. SQL is one of the most fundamental tools in data science. It is used to manipulate data. Its simplicity is a big reason for its popularity. There are lots of things it can’t do but the…
It seems like Seth Kugel's article in the New York Times about "Crunching the Numbers to find the Best Airfare" is quite popular. In this article, he said things like this: The overall take on the best day to book tickets turns out to be somewhat underwhelming, if you look at the country as a whole. Hopper’s data shows it’s actually Thursday, but don’t expect that fact to save you…
I saw Joe N.'s tweet asking me about a study of how professors spend their time, reported by Lisa Wade at Sociological Images. This is an anthropological study, something that I am not at all familiar with although the people in the field seem to believe that they can make statistically valid observations. I'm glad the author of the study, John Ziker, wrote a (really) long article describing what he…
MailChimp, a major vendor that companies use to send marketing emails to customers, published an analysis of the effect of Gmail marketing tabs (link). How should you read such a study? I'd begin by clarifying what problem the analyst is solving. In May, Google rolled out to all Gmail users a tabbed interface, in which the inbox is split into three parts: the regular inbox, a "promotional" email box, and…
