I often need to work with many variables at a time in SAS, but I don’t like to type all of their names manually – not only is it messy to read, it also induces errors in transcription, even when copying and pasting. I recently learned of an elegant and efficient way to store multiple […]
This note describes a useful replyr tool we call a "join controller" (and is part of our "R and Big Data" series, please see here for the introduction, and here for one our big data courses). When working on real world predictive modeling tasks in production, the ability to join data and document how you … Continue reading Use a Join Controller to Document Your Work
replyr is an R package that contains extensions, adaptions, and work-arounds to make remote R dplyr data sources (including big data systems such as Spark) behave more like local data. This allows the analyst to more easily develop and debug procedures that simultaneously work on a variety of data services (in-memory data.frame, SQLite, PostgreSQL, and … Continue reading replyr: Get a Grip on Big Data in R
Consider the common following problem: compute for a data set (say the infamous iris example data set) per-group ranks. Suppose we want the rank of iris Sepal.Lengths on a per-Species basis. Frankly this is an “ugh” problem for many analysts: it involves all at the same time grouping, ordering, and window functions. It also is … Continue reading Organize your data manipulation in terms of “grouped ordered apply”
Statisticians and data scientists want a neat world where data is arranged in a table such that every row is an observation or instance, and every column is a variable or measurement. Getting to this state of “ready to model format” (often called a denormalized form by relational algebra types) often requires quite a bit … Continue reading The case for index-free data manipulation
Last Friday (August 22nd), I achieved a milestone on Stack Overflow, a computing question-and-answer site, by hitting 200,000 points. So far this year, I have also earned more points than anyone, a testament to my obsession with the site (you can...
Recently, I was trolling around the internet looking for some IP address data to play with. Fortunately, I stumbled across MaxMind's Geolite Database, which is available for free. All I have to do is include this notice:This product ...
SQL is a very powerful language, which could, of course, be made even more powerful and useful. This post discusses three features of the language -- ANSI standard features -- that seem not only unnecessary but downright detrimental. That i...
I happen to think that SQL is a very viable option for analyzing big data. I was thinking about this when I a book review recently:For instance, Siegel reports, people who buy small felt pads that adhere to the bottom of chair legs (to protect th...
Recently, I was approached by Vicky whom I'm working with at a client, to help with a particular problem. She wanted to calculate page view summaries for a random sample of visitors from a table containing about a billion page views. This i...
