Example showing how to diagnose bad data in data science models
What do the following items have in common? A. Motion-detection light switch in your office that shuts off while you're typing at your computer B. Automatic flush that turns on while you're seated C. Voice-recognition system that picks up ambient noise and asks you to repeat something you didn't say D. Auto-correct software that flips a correct spelling to the wrong one you didn't anti-auto-correct the last time E. Fuzzy-logic…
For me, 2016 is a year of water leaks. I was forced to move apartments during the summer. (Blame my old landlord for the lower frequency of posts this year!) That old apartment was overrun by water issues. In the past four years, there were two big leaks in addition to annual visible "seepage" in the ceiling. The first big leak ruined my first night back from Hurricane Sandy-induced evacuation.…
In class last week, I discussed this New York Times article with the students. One of the claims in the article is that the U.S. News ranking of colleges is under threat by newcomers whose rankings are more relevant because they more directly measure outcomes such as earnings of graduates. This specific claim in the article makes me head hurt: "If nothing else, earnings are objective and, as the database…
I am traveling so have to make this brief. I will likely come back to these stories in the future to give a longer version of these comments. I want to react to two news items that came out in the past couple of days. First, Ben Stiller said that prostate cancer screening (the infamous PSA test) "saved his life". (link) So he is out there singing the praises of…
I’ve had this book on pre-order since spring and it finally arrived on Friday. I subsequently devoured it over the weekend. The book lays out a clear and compelling case for how data-driven algorithms can become — in contrast to their promise of amoral objectivism — efficient means for reproducing and even exacerbating social inequalities […]
A very important article from the Times starts with the following sentence: Want to invisibly spy on 10 iPhone owners without their knowledge? Gather their every keystroke, sound, message and location? That will cost you $650,000, plus a $500,000 setup fee with an Israeli outfit called the NSO Group. In the U.S., there is a disconnect between a populace whose distrust of government is at an all-time high and the…
My friend John R. sent me this excellent Buzzfeed feature on music playlists. Here are some choice quotes to whet your appetite: In 2014, when Tim Cook explained Apple’s stunning $3 billion purchase of Beats by repeatedly invoking its “very rare and hard to find” team of music experts, he was talking about these guys. And their efforts since, which have pointed toward curated playlists (specifically, an industrial-scale trove of…
Well, it didn't take long but private investigators have found the next big thing: Big Data. Bloomberg reported on a company called IDI, who sells our data to private investigators. (link) Unfortunately, this article is short on details and long on sensationalized catchphrases ("Every move you make. Every click you take...") The CEO of IDI boasted that "We have data on that 21-year-old who’s living at home with mom and…
CNBC reports that Goldman Sachs flags employee emails based on a long list of "offending" phrases. If an employee types a profanity, apparently a window pops up to confirm that the person really truly wants to say that word. The other objective given is to detect fraudulent behavior. The list they published apparently came from 2008, so very aged, but I think it is a hoax. Many of the terms…
