At the start of the year, The Atlantic published a very nice, long article about Netflix's movie recommendation algorithm. You may remember this algorithm (internally known as Cinematch) received a $1 million makeover several years ago (the Netflix Prize), only that the prize-winning entry was deemed too complex--and does not generate sufficient incremental value--to be put into production. The reporter, Alexis Madrigal, noticed that Netflix has shifted attention from the…
Steroids continue to plague professional sports. The latest name to fall is Alex Rodriguez, the shortstop/3rd baseball superstar who currently plays for the Yankees. It wasn't long ago that he was considered a "good guy" of the sport. Now, he's a pariah. In the rush to make Rodriguez the villain, the media continues to miss these two important aspects to the steroids story: Anti-doping tests have a huge false-negative problem.…
When you hear about Big Data, you almost always hear about the supply side: Behold the data in un-pronounceable units of bytes! Admire the new science inspired by all the data! Missing from this narrative is the consumption side. A direct consequence of Big Data will be the explosion of data analyses—there will be more people producing more data analyses more quickly. This will be a world of confusing and…
The past week, you are unable to avoid, if you read the business news, mention of the "law of large numbers". We are led to believe that tech giants like Apple and Amazon are suffering from this statistical law. For example: Apple: Newest Victim of the Law of Large Numbers (CNet) New York Times Paywall Growth Slows (Columbia Journalism Review): "Much of this is due to the law of large…
Statistics is sometimes described as inverse probability. In a typical probability problem, one starts by positing that a certain quantity has some given probability distribution, say the number of people entering a bank branch follows a Poisson distribution, and then goes on to compute probabilities such as the chance that more than 100 people (max capacity) require service at the same time. In a typical statistical problem, one observes the…
