Kaiser Fung, founder of Principal Analytics Prep, discusses Seth Stephens-Davidowitz's new book, Everybody Lies
Kaiser Fung, founder of Principal Analytics Prep, discusses Seth Stephens-Davidowitz's new book, Everybody Lies
Most of my friends with families fret over school quality when deciding where to buy their homes. It's well known that good school districts are also associated with expensive houses. A feedback cycle is at work here: home prices surge where there are good schools; only richer people can afford to buy such homes; wealth brings other advantages, and so the schools tend to have better students, which leads to…
Last week, my Columbia students discussed this nice article in the New York Times called "The Most Detailed Map of Gay Marriages in America". (link) The center of the article is this map: I asked the students to identify the problem that this dataviz is supposed to address. Someone responded that it tells us where gay married couples are found geographically. I asked her what is the answer to the…
My co-columnist Andrew Gelman has been doing some fantastic work, digging behind that trendy news story that claims that middled-aged, non-Hispanic, white male Americans are dying at an abnormal rate. See, for example, this New York Times article that not only reports the statistical pattern but also in its headline, asserts that those additional deaths were due to suicide and substance abuse. It all bega n with the chart shown…
In the first two chapters of Numbersense, I discuss how people game statistics, and why gaming is inevitable. I have also written about the placebo effect before. Another article has appeared covering the same topic -- the industry doesn't like the fact that more and more drugs fail to clear the "placebo" hurdle; and the industry thinks the problem is that the placebo effect is mysteriously increasing over time. What…
It appears that Google Flu Trends (GFT) has slipped quietly into the night. In a short post to the Google Research blog, the team behind GFT announced that they are "no longer publishing" flu estimates, effectively ending the seven-year-old experiment. The GFT home page now links to some historical datasets. The post was dated August 15, and it appears that mainstream media completely missed it. GFT was one of the…
I am outsourcing this post to Aaron Carroll, whose Upshot column eviscerates the recent claim that eating meat will give you cancer, or that eating meat is the same as smoking cigarettes. While the media is partly culpable for spreading misinformation...
In our newest column, we take on the recent media obsession with companies who make robots that hire people. (link) As with most articles about data science, the journalists failed to dig up any evidence that these robots work, other than glowing quotes from the people who are selling these robots. We point out a number of challenges that such algorithms must overcome in order to generate proper predictions. We…
It is conventional wisdom that A/B testing (or in proper terms, randomized controlled experiments) is the gold standard for causal analysis, meaning if you run an A/B test, you know what caused an effect. In practice, this is not always true. Sometimes, the A/B test only provides a statistical understanding of causes but not an average Joe's understanding. Let's start with a hypothetical example in which both definitions are aligned.…
This is a supplement to the previous post about a new research paper on the effect of Alcoholics Anonymous, and an NY Times exposition that I commented on. A misreading of that article led me to complain about per-protocol analysis, which wasn't the methodology behind the Humphrey et. al. research. I will explain their methodology in this post (known as instrumental variables analysis). *** In the last post, I showed…
