Amateurs, dilettantes, hacks, cowboys, clones — Nick Cave

[Note from Dan 11Sept: I wanted to leave some clear air after the StanCon reminder, so I scheduled this post for tomorrow. Which means you get two posts (one from me, one from Andrew) on this in two days. That’s probably more than the gay face study deserves.]

I mostly stay away from the marketing aspect of machine learning / data science / artificial intelligence. I’ve learnt a lot from the methodology and theory aspects, but the marketing face makes me feel a little ill.  (Not unlike the marketing side of statistics. It’s a method, not a morality.)

But occasionally stories of the id and idiocy of that world wander past my face and I can’t ignore it any more. Actually, it’s happened twice this week, which—in a week of two different orientations, two different next-door frat parties until two different two A.M.s, and two grant applications that aren’t as finished as I want them to be—seems like two times too many to me.

The first one can be dispatched with quickly.  This idiocy came from IBM who are discovering that deep learning will not cure cancer.  Given the way that governments, health systems, private companies, and  research funding bodies are pouring money into the field, this type of snake oil pseudo-science is really quite alarming. Spectacularly well-funded hubris is still hubris and sucks money and oxygen from things that help more than the researcher’s CV (or bottom line).

In the end, the “marketing crisis” in data science is coming from the same place as the “replication crisis” in statistics.  The idea that you can reduce statistics and data analysis (or analytics) down to some small set of rules that can be automatically applied is more than silly, it’s dangerous. So just as the stats community has begun the job of clawing back the opportunities we’ve lost to p=0.05, it’s time for data scientists to control their tail.

Untouchable face

The second article occurred, as Marya Dmitriyevna (who is old-school) in Natasha, Pierre, and the Great Comet of 1812 shrieks after Anatole (who is hot) tries to elope with Natasha (who is young) , in my house. So I’m going to spend most of this post on that. [It was pointed out to me that the first sentence is long and convoluted, even for my terrible writing. What can I say? It’s a complicated Russian novel, everyone’s got nine different names.]

(Side note: The only reason I go to New York is Broadway and cabaret. It has nothing to do with statistics and the unrepeatable set of people in one place. [please don’t tell Andrew. He thinks I like statistics.])

It turns out that deep neural networks join the illustrious ranks of “men driving past in cars”, “angry men in bars”, “men on the internet”,  and “that one woman at a numerical analysis conference in far north Queensland in 2006” in feeling obliged to tell me that I’m gay.

Now obviously I rolled my eyes so hard at this that I almost pulled a muscle.  I marvelled at the type of mind who would decide “is ‘gay face’ real” would be a good research question. I also really couldn’t understand why you’d bother training a neural network. That’s what instagram is for. (Again, amateurs, dilettantes, hacks, cowboys, clones.)

But of course, I’d only read the headline.  I didn’t really feel the urge to read more deeply.  I sent a rolled eyes emoji to the friend who thought I’d want to know about the article and went back about my life.

Once, twice, three times a lady

But last night, as the frat party next door tore through 2am (an unpleasant side-effect of faculty housing it seems) it popped up again on my facebook feed. This time it had grown an attachment ot that IBM story.  I was pretty tired and thought “Right. I can blog on this”.

(Because that’s what Andrew was surely hoping for, three thousand words on whether “gay face” is a thing. In my defence, I did say “are you sure?” more than once and reminded him what happened that one time X let me post on his blog.)

(Side note: unlike odd, rambling posts about a papers we’d just written, posts about bad statistics published in psychology journals is right in Andrew’s wheelhouse.  So why aren’t I waiting until mid 2018 for a post on this that he may or may not have written to appear? [DS 11Sept: Ha!] There’s not much that I can be confident that I know a lot more about than Andrew does, but I am very confident I know a lot more about “gay face”.)

So I read the Guardian article, which (wonder of wonders!) actually linked to the original paper. Attached to the original paper was an authors’ note, and an authors’ response to the inevitable press release from GLAAD calling them irresponsible.

Shall we say that there’s some drift from the paper to the press reports. I’d be interested to find out if there was at any point a press release marketing the research.

The main part that has gone AWOL is a rather long, dystopian discussion that the authors have about how they felt morally obliged to publish this research because people could do this with “off the shelf” tools to find gay people in countries where its illegal.  (We will unpack that later)

But the most interesting part is the tension between footnote 10 of the paper (“The results reported in this paper were shared, in advance, with several leading international LGBTQ organizations”) and the GLAAD/HRC press release that says

 Stanford University and the researchers hosted a call with GLAAD and HRC several months ago in which we raised these myriad concerns and warned against overinflating the results or the significance of them. There was no follow-up after the concerns were shared and none of these flaws have been addressed.

With one look

If you strip away all of the coverage, the paper itself does some tings right. It has a detailed discussion of the limitations of the data and the method. (More later)  It argues that, because facial features can’t be taught, these findings provide evidence towards the prenatal hormone theory of sexual orientation (ie that we’re “born this way”).

(Side note: I’ve never like the “born this way” narrative. I think it’s limiting.  Also, getting this way took quite a lot of work. Baby gays think they can love Patti and Bernadette at the same time. They have to learn that you need to love them with different parts of your brain or else they fight. My view is more “We’re recruiting. Apply within”.)

So do I think that this work should be summarily dismissed? Well, I have questions.

(Actually, I have some serious doubts, but I live in Canada now so I’m trying to be polite. Is it working?)

Behind the red door

Male facial image brightness correlates 0.19 with the probability of being gay, as estimated by the DNN-based classifier. While the brightness of the facial image might be driven by many factors, previous research found that testosterone stimulates melanocyte structure and function leading to a darker skin. (Footnote 6)

Again, it’s called instagram.

In the Authors’ notes attached to the paper, the authors recognise that “[gay men] take better pictures”, in the process acknowledging that they themselves have gay friends [honestly, there are too many links for that] who also possess this power. (Once more for those in the back: they’re called filters. Straight people could use them if they want. Let no photo go unmolested.)

(Side note: In 1989 Liza Minelli released a fabulous collaboration with the Petshop Boys that was apparently called Results because that’s what Janet Street-Porter used to call one of her outfits. I do not claim to understand straight men [nor am I looking to], but I have seen my female friends’ tinder. Gentlemen: you could stand to be a little more Liza.)

Back on top

But enough about methodology, let’s talk about data. Probably the biggest criticism that I can make of this paper is that they do not identify the source of their data. (Actually, in the interview with The Economist that originally announced the study, Kosinski says that this is intentional to “discourage copycats”.)

Obviously this is bad science.

(Side note: I cannot imagine the dating site in question would like its users to know that it allows its data to be scraped and analysed. This is what happens when you don’t read the “terms of service”, which I imagine the authors (and the ethics committee at Stanford) read very closely.)

[I’m just assuming that, as a study that used identifiable information about people who could not consent to being studied and deals with a sensitive study, this would’ve gone though an ethics process.]

This failure to disclose the origin of the data means we cannot contextualise it within the Balkanisation of gay desire. Gay men (at least, I will not speak for lesbians and I have nothing to say about bisexuals except “I thank you for your service”) will tell you what they want (what they really really want). This perverted and wonderful version of “The Secret” has manifested in multiple dating platforms that cater to narrow subgroups of the gay community.

Withholding information about the dating platform prevents people from independently scrutinising how representative the sample is likely to be. This is bad science.

(Side note: How was this not picked up by peer review? I’d guess it was reviewed by straight people. Socially active gay men should be able to pick a hole in the data in three seconds flat.  That’s the key point about diversity in STEM. It’s not about ticking boxes or meeting quotas. It’s that you don’t know what a minority can add to your work if they’re not in the room to add it.)

If you look at figure 4, the composite “straight” man is a trucker, while the composite “gay” man is a twunk. This strongly suggests that this is not my personal favourite type of gay dating site: the ones that caters to those of us who look like truckers. It also suggests that the training sample is not representative of the population the authors are generalising to. This is bad science.

(Terminology note: “Twunk” is the past form of “twink”, which is gay slang for a young (18-early20something), skinny, hairless gay man.)

The reality of gay dating websites is that you tailor your photographs to the target audience. My facebook profile picture (we will get to that later) is different to my grindr profile picture is different to my scruff profile picture. In the spirit of Liza, these are chosen for results. In these photos, I run a big part of the gauntlet between those two “composite ideals” in Figure 4. (Not all the way, because never having been a twink, and hence I never twunk.)

(Side note: The last interaction I had on Scruff was a two hour conversation about Patti LuPone. This is an “off-label” usage that is not FDA approved.)

So probably my biggest problem with this study is that it the training sample is likely unrepresentative of the population at large. This means that any inferences drawn from a model trained on this sample will be completely unable to answer questions about whether gay face is real in Caucasian Americans.  By withholding critical information about the data, the authors make it impossible to assess the extent of the problem.

One way to assess this error would be to take the classifier trained on their secret data and use it to, for example, classify face pics from a site like Scruff. There is a problem with this (as mentioned in the GLAAD/HRC press release) that activity and identity are not interchangeable. So some of the men who have sex with men (MSM, itself a somewhat controversial designation) will identify as neither gay nor bisexual and this is not necessarily information that would be available to the researcher. Nevertheless, it is probably safe to assume that people who have a publicly showing face picture on an dating app mainly used by MSMs are not straight.

If the classifier worked on this sort of data, then there is at least a chance that the findings of the study will replicate. But, those of you who read the paper of the Author notes howl, the authors did test the classifier on a validation sample gathered from Facebook.

At this point, let us pause to look at stills from Derek Jarman films. 

Pictures of you

First, we used the Facebook Audience Insights platform to identify 50 Facebook Pages most popular among gay men, including Pages such as: “I love being Gay”, “Manhunt”, “Gay and Fabulous”, and “Gay Times Magazine”. Second, we used the “interested in” field of users’ Facebook profiles, which reveals the gender of the people that a given user is interested in. Males that indicated an interest in other males, and that liked at least two out of the predominantly gay Facebook Pages, were labeled as gay.

I beseech you, in the Bowels of Christ, think it possible that that your validation sample may be biased.

(I mean, really. That’s one hell of an inclusion criterion.)

Rebel Girl / Nancy Boy

So what of the other GLAAD/HRC problems? They are mainly interesting to show the difference between the priorities of an advocacy organisation and statistical priorities. For example, the study only considered caucasians, which the press release criticises. The paper points out that there was not enough data to include people of colour. Without presuming to speak for LGB (the study didn’t consider trans people, so I’ve dropped the T+ letters [they also didn’t actively consider bisexuals, but LG sells washing machines]) people of colour, I can’t imagine that they’re disappointed to be left out of this specific narrative. That being said, the paper suggests that these results will generalise to other races. I am very skeptical of this claim.

Those composite faces also suggest that fat gay men don’t exist. Again, I am very happy to be excluded from this narrative.

What about the lesbians? Neural networks apparently struggle with lesbians. Again, it could be an issue of sampling bias. It could also be that the mechanical turk gender verification stage (4 of 6 people needed to agree with the person’s stated gender for them to be included) is adding additional bias.  The most reliable way to verify a person’s gender is to ask them.  I am uncertain why the researchers deviated from this principle. Certain sub-cultures in the LGB community identify as lesbian or gay but have either androgynous style, or purposely femme or butch style.  Systematically excluding these groups (I’m thinking particularly of butch lesbians and femme gays) will bias the results.

Breaking the law

So what of the dystopian picture the authors paint of governments using this type of procedure to find and eliminate homosexuals? Meh.

Just as I’m opposed to marketing in machine learning, I’m opposed to “story time” in otherwise serious research.  This claim makes the paper really exciting to read—you get to marvel at their moral dilemma. But the reality is much more boring. This is literally why universities (and I’m including Stanford in that category just to be nice) have ethics committees. I assume this study went through the internal ethics approval procedure, so the moralising is mostly pantomime.

The other reason I’m not enormously concerned about this is that I am skeptical of the idea that there is enough high-quality training data to train a neural network in countries where homosexuality is illegal (which are not white caucasian majority countries). Now I may well be very wrong about this, but I think we can all agree that it would be much harder than in the Caucasian case. LGBT+ people in these countries have serious problems, but neural networks are not one of them.

Researchers claiming that gay men, and to a slightly lesser extent lesbians, are structurally different from straight people is a problem. (I think we all know how phrenology was used to perpetuate racist myths.)  The authors acknowledge this during their tortured descriptions of their moral struggles over whether to publish the article. Given that, I would’ve expected the claims to be better validated. I just don’t believe that either their data set or their  validation set gives a valid representation of gay people.

All science should be good science, but controversial science should be unimpeachable. This study is not.

Live and let die (Gerri Halliwell version)

Ok. That’s more than enough of my twaddle. I guess the key point of this is that marketing and story telling, as well as being bad for science, gets in the way of effectively disseminating information.

Deep learning / neural networks / AI are all perfectly effective tools that can help people solve problems. But just like it wasn’t a generic convolutional neural network that wins games of Go, this tool only works when deployed by extremely skilled people. The idea that IBM could diagnose cancer with deep learning is just silly. IBM knows bugger all about cancer.

In real studies, selection of the data makes the difference between a useful step forward in knowledge and “junk science”.  I think there are more than enough problems with the “gay face” study (yes I will insist on calling it that) to be immensely skeptical. Further work may indicate that the authors’ preliminary results were correct, but I wouldn’t bet the farm on it. (I wouldn’t even bet a farm I did not own on it.) (In fact, if this study replicates I’ll buy someone a drink. [That someone will likely be me.])

The post It seemed to me that most destruction was being done by those who could not choose between the two appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post It seemed to me that most destruction was being done by those who could not choose between the two appeared first on All About Statistics.

Brendan Nyhan writes:

Thought this might be of interest – new paper with Jacob Montgomery and Michelle Torres, How conditioning on post-treatment variables can ruin your experiment and what to do about it.

The post-treatment bias from dropout on Turk you just posted about is actually in my opinion a less severe problem than inadvertent experimenter-induced bias due to conditioning on post-treatment variables in determining the sample (attention/manipulation checks, etc.) and controlling for them/using them as moderators. We show how common these practices are in top journal articles, demonstrate the problem analytically, and reanalyze some published studies. Here’s the table on the extent of the problem:

Post-treatment bias is not new but it’s an important area where practice hasn’t improved as rapidly as in other areas.

I wish they’d round their numbers to the nearest percentage point.

The post “How conditioning on post-treatment variables can ruin your experiment and what to do about it” appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post “How conditioning on post-treatment variables can ruin your experiment and what to do about it” appeared first on All About Statistics.

Please comment on the article here: Big Data, Plainly Spoken (aka Numbers Rule Your World)

The post Surge protectors as teaching tools appeared first on All About Statistics.

It can be surprisingly easy to identify someone from data that’s not directly identifiable. One commonly cited result is that the combination of birth date, zip code, and sex is enough to identify most people. This post will look at how to quantify the amount of information contained in such data.

If the answer to a question has probability p, then it contains -log₂ p bits of information. Knowing someone’s sex gives you 1 bit of information because -log₂(1/2) = 1.

Knowing whether someone can roll their tongue could give you more or less information than knowing their sex. Estimates vary, but say 75% can roll their tongue. Then knowing that someone can roll their tongue gives you 0.415 bits of information, but knowing that they cannot roll their tongue gives you 2 bits of information.

On average, knowing someone’s tongue rolling ability gives you less information than knowing their sex. The average amount of information, or entropy, is

0.75(-log₂ 0.75) + 0.25(-log₂ 0.25) = 0.81.

Entropy is maximized when all outcomes equally likely. But for identifiability, we’re concerned with maximum information as well as average information.

Knowing someone’s zip code gives you a variable amount of information, less for densely populated zip codes and more for sparsely populated zip codes. An average zip code contains about 7,500 people. If we assume a US population of 326,000,000, this means a typical zip code would give us about 15.4 bits of information.

The Safe Harbor provisions of US HIPAA regulations let you use the first three digits of someone’s zip code except when this would represent less than 20,000 people, as it would in several sparsely populated areas. Knowing that an American lives in a region of 20,000 people would give you 14 bits of information about that person.

Birth dates are complicated because age distribution is uneven. Knowing that someone’s birth date was over a century ago is highly informative, much more so than knowing it was a couple decades ago. That’s why the Safe Harbor provisions do not allow including age, much less birth date, for people over 90.

Birthdays are simpler than birth dates. Birthdays are not perfectly evenly distributed throughout the year, but they’re close enough for our purposes. If we ignore leap years, a birthday contains -log₂(1/365) or about 8.5 bits of information. If we consider leap years, knowing someone was born on a leap day gives us two extra bits of information.

Independent information is additive. I don’t expect there’s much correlation between sex, geographical region, and birthday, so you could add up the bits from each of these information sources. So if you know someone’s sex, their zip code (assuming 7,500 people), and their birthday (not a leap day), then you have 25 bits of information, which may be enough to identify them.

This post didn’t consider correlated information. For example, suppose you know someone’s zip code and primary language. Those two pieces of information together don’t provide as much information the sum of the information they provide separately because language and location are correlated. I may discuss the information content of correlated information in a future post.

Related: HIPAA de-identification

Please comment on the article here: Statistics – John D. Cook

The post Quantifying the information content of personal data appeared first on All About Statistics.

Next open blog spots are in April but all these are topical so I thought I’d throw them down right now for ya.

1. Alex Durante writes:

I noticed that this study on how Trump supporters respond to racial cues is getting some media play, notably over at Vox. I was wondering if you have any thoughts on it. At first glance, it seems to me that its results are being way overhyped. Thanks for your time.

Here’s a table showing one of their analyses:

My reaction to this sort of thing is: (a) I won’t believe this particular claim until I see the preregistered replication. Too many forking paths. And (b) of course it’s true that “Supporters and Opponents of Donald Trump Respond Differently to Racial Cues” (that’s the title of the paper). How could that not be true, given that Trump and Clinton represent different political parties with way different positions on racial issues? So I don’t really know what’s gained by this sort of study that attempts to scientifically demonstrate a general claim that we already know, by making a very specific claim that I have no reason to think will replicate. Unfortunately, a lot of social science seems to work this way.

Just to clarify: I think the topic is important and I’m not opposed to this sort of experimental study. Indeed, it may well be that interesting things can be learned from the data from this experiment, and I hope the authors make their raw data available immediately. I’m just having trouble seeing what to do with these specific findings. Again, if the only point is that “Supporters and Opponents of Donald Trump Respond Differently to Racial Cues,” we didn’t need this sort of study in the first place. So the interest has to be in the details, and that’s where I’m having problems with the motivation and the analysis.

2. A couple people pointed me to this paper from 2006 by John “Mary Rosh” Lott, “Evidence of Voter Fraud and the Impact that Regulations to Reduce Fraud Have on Voter Participation Rates,” which is newsworthy because Lott has some connection to this voter commission that’s been in the news. Lott’s empirical analysis is essentially worthless because he’s trying to estimate causal effects from a small dataset by performing unregularized least squares with a zillion predictors. It’s the same problem as this notorious paper (not by Lott) on gun control that appeared in the Lancet last year. I think that if you were to take Lott’s dataset you could with little effort obtain just about any conclusion you wanted by just fiddling around with which variables go into the regression.

3. Andrew Jeon-Lee points us to this post by Philip Cohen regarding a recent paper by Yilun Wang and Michal Kosinski that uses a machine learning algorithm and reports, “Given a single facial image, a classifier could correctly distinguish between gay and heterosexual men in 81% of cases, and in 71% of cases for women. Human judges achieved much lower accuracy: 61% for men and 54% for women.”

Hey, whassup with that? I can get 97% accuracy by just guessing Straight for everybody.

Oh, it must depend on the population they’re studying! Let’s read the paper . . . they got data on 37000 men and 39000 women, approx 50/50 gay/straight. So I guess my classification rule won’t work.

More to the point, I’m guessing that the classification rule that will work will depend a lot on what population you’re using.

I had some deja vu on this one because last year there was a similar online discussion regarding a paper by Xiaolin Wu and Xi Zhang demonstrating an algorithmic classification of faces of people labeled as “criminals” and “noncriminals” (which I think makes even less sense than labeling everybody as straight or gay, but that’s another story). I could’ve sworn I blogged something on that paper but it didn’t show up in any search so I guess I didn’t bother (or maybe I did write something and it’s somewhere in the damn queue).

Anyway, I had the same problem with that paper from last year as I have with this recent effort: it’s fine as a classification exercise, and it can be interesting to see what happens to show up in the data (lesbians wear baseball caps!), but the interpretation is way over the top. It’s no surprise at all that two groups of people selected from different populations will differ from each other. That will be the case if you compare a group of people from a database of criminals to a group of people from a different database, or if you compare a group of people from a gay dating website to a group of people from a straight data website. And if you have samples from two different populations and a large number of cases, then you should be able to train an algorithm to distinguish them at some level of accuracy. Actually doing this is impressive (not necessarily an impressive job by these researchers, but it’s an impressive job by whoever wrote the algorithms that these people ran). It’s an interesting exercise, and the fact that the algorithms outperform unaided humans, that’s interesting too. But this kind of thing: like “The phenomenon is, clearly, troubling to those who hold privacy dear—especially if the technology is used by authoritarian regimes where even a suggestion of homosexuality or criminal intent may be viewed harshly.” That’s just silly, as it completely misses the point that the success of these algorithms entirely depends on the data used to train them.

Also Cohen in his post picks out this quote from the article in question:

[The results] provide strong support for the PHT [prenatal hormone theory], which argues that same-gender sexual orientation stems from the underexposure of male fetuses and overexposure of female fetuses to prenatal androgens responsible for the sexual differentiation of faces, preferences, and behavior.

Huh? That’s just nuts. I agree with Cohen that it would be better to say that the results are “not inconsistent” with the theory, just as they’re not inconsistent with other theories such as the idea that gay people are vampires (or, to be less heteronormative, the idea that straight people lack the vampirical gene).

Also some goofy stuff about the fact that gay men in this particular sample are less likely to have beards.

In all seriousness, I think the best next step here, for anyone who wants to continue research in this area, is to do a set of “placebo control” studies, as they say in econ, each time using the same computer program to classify people chosen from two different samples, for example college graduates and non-college graduates, or English people and French people, or driver’s license photos in state X and driver’s license photos in state Y, or students from college A and students from college B, or baseball players and football players, or people on straight dating site U and people on straight dating site V, or whatever. Do enough of these different groups and you might get some idea of what’s going on.

The post God, goons, and gays: 3 quick takes appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post God, goons, and gays: 3 quick takes appeared first on All About Statistics.

It was a great pleasure to talk to Christopher Salahub about Markov chains in the second episode of my new talk show, The Central Equilibrium!  Chris graduated from the University of Waterloo with a Bachelor of Mathematics degree in statistics.  He just finished an internship in data development at Environics Analytics, and he is starting a Master’s program in statistics at ETH Zurich in Switzerland.

Chris recommends “Introduction to Probability Models” by Sheldon Ross to learn more about probability theory and Markov chains.

The Central Equilibrium is my new talk show about math, science, and economics. It focuses on technical topics that involve explanations with formulas, equations, graphs, and diagrams.  Stay tuned for more episodes in the coming weeks!

You can watch all of my videos on my YouTube channel!

Please view this blog post beneath the fold to watch the video on this blog.  You can also watch it directly on YouTube.

Please comment on the article here: Statistics – The Chemical Statistician

The post Christopher Salahub on Markov Chains – The Central Equilibrium – Episode 2 appeared first on All About Statistics.

(In a stunning deviation from the norm, this post is not by Andrew or Dan, but Betancourt!)

Some important dates for StanCon2018 are rapidly approaching!

Contributed submissions are due September 16, 2017 5:00:00 AM GMT. That’s less than 6 days away!  We want to make sure we can review submissions early enough to get responses back to submitters in time for early registration.  For more details on the submission requirements and how to submit see the Submissions page.

Speaking of which, early registration ends Friday November 10, 2017 after which the registration cost significantly increases. That’s in just about two months!

Finally, just because I still can’t believe that we have such an amazing ensemble of invited speakers let me remind everyone that attendees will get to see talks from

• Susan Holmes (Department of Statistics, Stanford University)
• Frank Harrell (School of Medicine and Department of Biostatistics, Vanderbilt University)
• Sophia Rabe-Hesketh (Educational Statistics and Biostatistics, University of California, Berkeley)
• Sean Taylor and Ben Letham (Facebook Core Data Science)
• Manuel Rivas (Department of Biomedical Data Science, Stanford University)
• Talia Weiss (Department of Physics, Massachusetts Institute of Technology)

The post The StanCon Cometh appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post The StanCon Cometh appeared first on All About Statistics.

I recommend this discussion of how to summarize posterior distributions. I don’t recommend summarizing by the posterior probability that the new treatment is better than the old treatment, as that is not a bottom-line statement!

The post Looking for the bottom line appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Looking for the bottom line appeared first on All About Statistics.

Please comment on the article here: No Hesitations

The post 2017 NBER-NSF Time Series Meeting appeared first on All About Statistics.

Did you know that you can get SAS to compute symbolic (analytical) derivatives of simple functions, including applying the product rule, quotient rule, and chain rule? SAS can form the symbolic derivatives of single-variable functions and partial derivatives of multivariable functions. Furthermore, the derivatives are output in a form that can be pasted into a SAS program. The trick is to use PROC NLIN with the LIST option.

In SAS/IML, the nonlinear optimization routines will use analytical derivatives if they are provided; otherwise, they will automatically generate numerical finite-difference approximations to the derivatives. I rarely specify derivatives because it can be time-consuming and prone to error. But recently I realized that PROC NLIN and other SAS procedures automatically generate symbolic derivatives, and you can "trick" the procedure into displaying the derivatives so that they can be used in other applications. To get PROC NLIN to output symbolic derivatives, do the following:

• Create a data set to use for the DATA= option of PROC NLIN.
• List the variables that you want to take derivatives of on the PARMS statement. In open code, assign a value to constants that appear in the function.
• Specify the function to differentiate on the MODEL statement.

Symbolic derivatives of functions of one variable

Here is a one-variable example. Suppose you want to take the derivative of
f(x) = x³ + x sin(x) + exp( x² ).
The function and its derivative is shown to the right.

The following SAS statements create a "fake" data set that defines a variable named F. The call to PROC NLIN sets up the problem. The LIST option on the PROC NLIN statement generates the 'ProgList' table, which contains the derivative function:

data _dummy;
   f = 1;                                    /* name of function */
run;
 
proc nlin data=_dummy list;
   parms x=0;                                /* list variables */
   model f = x**3 + x*sin(x) + exp(x**2);    /* define function */
   ods select ProgList;                      /* output symbolic derivatives */
run;

The output shows the expressions for the function ("MODEL.f") and the derivative with respect to x. The derivative df/dx is written as "@MODEL.f/@x", where the symbol '@' is used for the 'd'. You can see that SAS took the simple derivative of the x³ term, applied the product rule to the x*sin(x) term, and applied the chain rule to the exp( x² ) term. You can also see that the expressions are written in SAS notation, with "**" indicating the power operator. SAS functions are written in upper case (SIN, COS, and EXP).

Symbolic partial derivatives of multivariate functions

In exactly the same way, you can obtain partial derivatives. In a partial derivative, all variables are held constant except one. Suppose you have the function for the normal density,
f(x) = 1/(sqrt(2 π) σ) exp( -(x - μ)**2 / (2 σ**2) )
Suppose that you want to compute the partial derivatives ∂f/∂μ and ∂f/∂σ. To compute these derivatives, list μ and σ on the PARMS statement and assign values to the other symbols (x and π) that appear in the function. It does not matter what value you assign to the x and π, you are just preventing PROC NLIN from complaining that there are unassigned variables. You could assign the value 1 to both variables, but I prefer to tell the procedure that π equals 3.14. In the following program, note that PROC NLIN reuses the fake data set that defines the F variable:

proc nlin data=_dummy list;
   parms mu=0 sigma=1;                        /* list variables */
   x=1; pi = constant('pi');                  /* assign arbitrary values to constants */
   model f = exp( -(x-mu)**2 / (2*sigma**2) ) / (sqrt(2*pi)*sigma); /* define function */
   ods select ProgList;                      /* output symbolic derivatives */
run;

The derivative with respect to σ requires using the chain rule and the quotient rule. Notice that the derivative includes a call to the original function ("MODEL.f"). Notice also that there is repetition of various expressions such as -(x - μ)**2 / (2 σ**2). If you copy and paste these expressions into a SAS program that computes the derivative, the computation will be slightly inefficient because the program will evaluate the same expression multiple times. One way to improve the efficiency is to collect sub-expressions into a temporary variable, as shown in the next section.

Sub-expressions and the chain rule

An experienced statistician will recognize that the expression z = (x - μ) / σ arises naturally as the standardized variable. Using such an expression might simplify the derivatives.

The following program computes the same derivatives as before, except this time the programmer defines the expression z = (x - μ) / σ and uses z in the function. When SAS takes the derivative, it computes dz/dμ and dz/dσ as part of the computation, as shown below:

proc nlin data=_dummy list;
   parms mu=0 sigma=1;                        /* list variables */
   x=1; pi = constant('pi');                  /* assign arbitrary values to constants */
   const = sqrt(2*pi);
   z = (x-mu) / sigma;                        /* intermediate expression */
   model f = exp(-z**2 / 2) / (const*sigma);  /* define function */
   ods select ProgList;                       /* output symbolic derivatives */
run;

Notice that the derivatives use several sub-expressions such as dz/dμ and dz/dσ. This computation is more efficient than the previous because it reuses previously computed quantities. For another example, define f1 = 1 / sqrt(2*pi*sigma) and f2 = exp(-z**2 / 2) and define the MODEL as f = f1*f2. You will see that each term in the derivative is efficiently evaluated in terms of previously expressions.

Discussion of symbolic derivatives

As I said previously, I do not usually specify analytical derivatives for my optimizations. I find that numerical derivatives (which are automatically computed) are fast and accurate for 99% of the optimizations that I perform.

Furthermore, some of the objective functions that I optimize do not have analytical derivatives. Others are matrix computations for which it would be difficult and time-consuming to write down the analytical derivative. (For an example, see the article on the derivative of the bivariate normal cumulative distribution.)

If you decide to specify an analytical derivative in SAS, make sure that the derivative is correct and efficient. The technique in this article will help to ensure that the derivative is correct. By strategically defining sub-expressions as in the previous section, you can help SAS generate derivatives that are computationally efficient.

Be aware that the derivatives are generated by applying the chain rule, product rule, and quotient rule without any algebraic simplifications of the result. Thus the result can be unnecessarily complicated. For an example, ask SAS to display the derivative of log( (1+x)/(1-x) ) / 2. The naive result is complicated. If you rewrite the function as log(1+x)/2 - log(1-x)/2, then the derivative is much simpler. However, in neither case will SAS simplify the derivative to obtain 1 / (1-x**2).

For more of my thoughts on using derivatives in SAS, see "Two hints for specifying derivatives." What about your thoughts? What do you think about using PROC NLIN to generate automatic derivatives of simple functions? Do you think this technique will be useful to you?

The post Symbolic derivatives in SAS appeared first on The DO Loop.

Please comment on the article here: The DO Loop

The post Symbolic derivatives in SAS appeared first on All About Statistics.