Richard Barker points us to an update on ipredict, the New Zealand political prediction market. From the news article by Hamish Rutherford:

The site, run by Victoria University of Wellington’s commercialisation arm, VicLink, issued a statement to its website and on Twitter on Thursday.

According to the iPredict statement, Associate Justice Minister Simon Bridges refused to grant it an exemption from the Anti-Money Laundering and Countering Financing of Terrorism Act, declaring that it was a “legitimate money laundering risk” because of the lack of customer due diligence. . . .

Geoff Todd, managing director of VicLink, said the website had been caught in a legal loophole which had caused problems globally.

“Predictions markets aren’t financial markets, and they’re not gambling, but the legislation is very binary. You’re either gambling or you’re a financial market.” . . .

The purpose of the website was to bring private information into the public domain, Todd said, meaning it was unreasonable to ask users to give their identities. A number of staff around Parliament, as well as MPs, are known to have used iPredict. . . .

The post “iPredict to close after Govt refuses anti-money laundering law exemption” appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post “iPredict to close after Govt refuses anti-money laundering law exemption” appeared first on All About Statistics.

Here’s the announcement:

Using Stan for variational inference, plus a couple lightning talks

Dustin Tran will give a talk on using Stan for variational inference, then we’ll have a couple lightening (5 minute-ish) talks on projects. David Sparks will talk, I will talk about some of my work and we’re looking for 1-2 more volunteers. If you have a project or idea of a project using Stan you’d like to talk about let me know!

Thanks to RStudio for sponsorship of the meeting.

Cool!

The post Boston Stan meetup 1 Dec appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Boston Stan meetup 1 Dec appeared first on All About Statistics.

Nick Carchedi is back visiting from DataCamp and for fun we came up with a dplyr Rubik's cube puzzle. Here is how it works. To solve the puzzle you have to make a 4 x 3 data frame that spells Thanksgiving like this:

View the code on Gist.

To solve the puzzle you need to pipe this data frame in 

View the code on Gist.

and pipe out the Thanksgiving data frame using only the dplyr commands arrange, mutate, slice, filter and select. For advanced users you can try our slightly more complicated puzzle:

View the code on Gist.

See if you can do it this fast. Post your solutions in the comments and Happy Thanksgiving!

Please comment on the article here: Simply Statistics

The post A thanksgiving dplyr Rubik’s cube puzzle for you appeared first on All About Statistics.

In response to some of the discussion that inspired yesterday’s post, Gary McClelland writes:

I remain convinced that discretizing a continuous variable, especially for multiple regression, is the road to perdition.

Here I explain my concerns. First, I don’t buy the motivation that discretized analyses are easier to explain to lay citizens and the press. Second, I believe there is an error in your logic for computing the relative efficiency for splitting into three groups. Third, and most importantly, dichotomizing or trichotomizing two or more continuous variables in a multiple regression is an especially bad idea. In such cases, the loss of efficiency is irrelevant because the discrete predictor variables have a different correlation than the continuous variables. As a consequence, the parameter estimates from the discrete analysis are biased. I’ll explain all three issues in some detail.

1. I just don’t buy the motivating issue—that the essence of a regression analysis can’t be explained to lay people. Explaining a regression result in terms of differences between averages is fine with me, but that doesn’t require a dichotomized analysis. We assume there is some true population difference between the average case in the top group (whether that be the top half, top third, top 27%, top quarter) and the average case in the bottom group. Let’s call those two population means muH and muL (for high and low). Our goal is to estimate that population mean difference. We, as statisticians, have two (at least) ways to estimate that mean difference muH – muL.

a. We do the split and compute the corresponding averages ybarH and ybarL and our estimate of muH – muL is ybarH – ybarL.

b. We regress y on x, as originally measured, to obtain yhat = b0 + b1 x. then we estimate muH – muL using (b0 + b1 xbarH) – (b0 + b1 xbarL) = b1(xbarH – xbarL).

Both are unbiased estimates of muH – muL and both can be described as “our data estimates the difference between the average person in the top group in the population and the average person in the bottom group in the population is …” The only difference between the two methods is that the variance of the estimate in (a) is greater than the variance of the estimate in (b). That implies that there will be many times when the estimate in (a) is either higher or lower than the estimate in (b). Hence, the two analyses will seldom agree on the magnitude of the raw effect. That gives researchers another degree of freedom to report the estimate that better fits their argument. We should use the the more precise regression estimate (b) and explain it in terms of a mean difference between high and low groups. If we are communicating to a lay group we should give them our best estimate and that is b1(xbarH – xbarL). We don’t need to explain to them how we got our estimate of muH – muL unless they ask. and even then the explanation isn’t that difficult. “We compared our prediction for a person with an average score in the top group to our prediction for a person with an average score in the low group.”

2. Using extreme groups in a three-way split of a single continuous predictor: The error in your mathematical analysis, I believe, is the assumption that the residual variance remains constant and is therefore the same in Eq. 4 and Eq. 5 in Gelman & Park (2008). It is easy to disprove that assumption. The residual variance is V(e) = V(Y)(1-r^2). Discretizing changes the correlation between X and Y. Furthermore, restricting the values of Y to cases that have extreme values of X will necessarily increase the V(Y). The exception is that when r = 0, V(Y) will be unchanged. Hence, your claims about the relative efficiency of extreme groups apply if and only if r = 0. In an attached Mathematica notebook (also included the pdf if you don’t use Mathematica) and an attached R simulation file, I did a detailed analysis of the relative efficiency for different values of b in the model Y = b X + e. This graph summarizes my results:

The curves represent the relative efficiency (ratio of the variances of the estimates of the slopes) for, top to bottom, slopes of b = 0, .25, 0.5, 1, 2, and 3. Given the assumption that V(e) = 1 in the full data, these correspond to correlations, respectively, of r = 0, 0.24, 0.45, 0.71, 0.89, and 0.95. The top curve corresponds to your efficiency curve for the normal distribution in your Figure 3. And, as you claim, using an extreme group split (whether the keep fraction is 0.2, 0.25, 0.27, or 0.333) is superior to a median split at all degrees of relationship between X and Y. However, relative efficiency declines as the strength of the X,Y relationship increases. Note also that the optimal fraction to keep shifts lower as the strength of the relationship increases.

Are these discrepancies important? For me and my colleagues in the social sciences, I decided the differences were of interest to geeky statisticians like you and me but probably not of practical importance. Within the range of most social science correlations (abs(r) < 0.5), the differences in the efficiency curves are trivial. And if a social scientist felt compelled for reasons of explainability to discretize the analysis, then I certainly agree that doing an extreme-groups analysis is preferable to doing a median split. However, if a researcher studying a physical industrial process (where the correlation is likely very high and high precision is desired) were tempted to do an extreme-groups analysis because it would be easier to explain to upper management, I would strongly advise against it. The relative efficiency is likely to be extremely low. On the right axis I’ve indexed the factor by which the sample size would need to be increased to compensate for the loss of efficiency. The price to be paid is extremely high. 3. When two or more correlated variables are analyzed via multiple regression, discretizing a continuous variable is a particularly bad idea not only because of reduced efficiency, but more importantly because discretizing changes the correlational structure of the predictors and that leads to bias in the parameter estimates. Most of the discussion in the set of median split papers in JCP concerned whether one could get away with splitting a single continuous variable which was to be analyzed in multiple regression with another continuous variable or as a covariate in an ANCOVA design. We thought both the considerable loss of power and the induced bias as a function of the predictor correlation were adequate reasons to reject such dichotomizations. I will be interested to see what your take is on that. However, I believe that doing an analysis with two discretized variables, whether by median splits or by “thirds” is a terrible idea because of the bias it induces. For median splits of two predictors with a bivariate normal distribution with correlation rho = 0.5, I can show analytically that the correlation between the dichotomized predictors will be 0.33, resulting in a confounding of the estimated slopes. Specifically, b1 = (5 b1 +b2)/6 and b2 = (b1 + 5 b2)/6. That is not good science. In the case of trichotomizing the two predictors and then using the extreme four corners of the 3 x 3 cells, I can show analytically that the predictor correlation INCREASES from 0.5 to 0.7. You can see why the correlation is enhanced in the bivariate distribution with correlation rho = 0.5 in this contour plot:

Using only the four extreme cells makes the correlation appear stronger.

I haven’t yet worked through analytically the bias this will cause, but I have experimented with simulations and observed that there is an enhancement bias for the coefficients. If one coefficient is larger than the other, then the value of the larger coefficient is unbiased but the value of the smaller coefficient is increased (i’ve been working with all positive coefficients). For example, when predictors x and z are from a bivariate normal distribution with correlation 0.5 and the model is y = x + 2 z + e, then the cut analysis yields coefficient estimates of 1.21 and 2.05. The 21% enhancement of the smaller coefficient isn’t just bad science, it isn’t science at all. The source of the problem can be seen in the comparison of two tables. The first table is the predicted means using the regression equation for the full model applied to the actual cell means for x and z.

-3.7 1.2
-1.2 3.7

The following table is the mean y values for each cell (equivalently, the model derived from the cut variables).

-4.0 1.0
-1.0 4.0

In other words, the cut analysis exaggerates the differences in the cell means. This arises because the cut analysis forces a false orthogonal design. This is confounding in the same sense that bad experimental designs confound effect estimates.

A particularly disturbing example is for the model y = x + 0*z + e, the coefficients for the cut analysis are 0.96 and 0.11, a spurious effect for z. This can be seen in the table of means for the four groups:

-1.3 -1.1
1.0 1.3

In fact, the columns should have been identical as in

-1.22 -1.23
1.22 1.21

consistent with the null effect for z. This spurious effect is akin to the problem of spurious effects due to dichotomizing two variables identified by Maxwell & Delaney (1993).

In short, discretizing two continuous predictors has no place in the responsible data analyst’s toolbox. At the end of section 2.8 you describe doing the appropriate full analysis as an option. I strongly disagree this is optional—it is a requirement.

You present bivariate analyses across years both for a continuous analysis (Figure 5) and an extreme-groups analysis (Figure 6). If both ways of estimating the effects were unbiased and equally efficient, we would expect the rank order of a given effect across years to remain the same as well as the rank order of the three effects for a given year to remain constant. Neither seems to be the case. The differences are not large relative to the standard error so perhaps these differences are just due to the increased variability of the discretized estimates. However, if religious attendance and income are correlated and especially if the degree of this correlation changes over the years, then I suspect that some of the differences between Figures 5 and 6 are due to bias induced by using discretized correlated predictors. I think the logits of Figure 5 transformed back to probability differences would have been more appropriate and no more difficult to explain.

I also am attaching a 5th paper in the JCP sequence—our effort at a rebuttal of their rebuttal that we posted on SSRN.

For the quick version, here’s McClelland’s note, which begins:

Gelman & Park (2008) argue that splitting a single continuous predictor into extreme groups and omitting the middle category produces an unbiased estimate of the difference and, although less efficient than using the continuous predictor, is less destructive than the popular median split. In this note I show that although their basic argument is essentially true, they overstate the efficiency of the extreme splits. Also their claims about optimal fractions for each distribution ignores a dependency of the optimal fraction on the magnitude of the correlation between X and Y.

In their Equations 4 and 5, Gelman & Park assume that the residual variance of Y is constant. It is easy to show that is not the case when discretizing a continuous variable, especially when using extreme groups. . . .

I don’t have time to look at this right now, but let me quickly say that I prefer to model the continuous data, and I consider the discretization to just be a convenience. I’ll have to look at McClelland’s notes more carefully to see what’s going on: is he right that we were overstating the efficiency of the comparison that uses the discretized variable? Stay tuned for updates.

P.S. I don’t want to make a big deal about this, but . . . this is the way to handle it when someone says you made a mistake in a published paper: you give them a fair hearing, you don’t dismiss their criticisms out of hand. And it’s more than that: if you have a reputation for listening to criticism, this motivates people to make such criticisms openly. Everybody wins.

The post Gary McClelland agrees with me that dichotomizing continuous variables is a bad idea. He also thinks my suggestion of dividing a variable into 3 parts is also a mistake. appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Gary McClelland agrees with me that dichotomizing continuous variables is a bad idea. He also thinks my suggestion of dividing a variable into 3 parts is also a mistake. appeared first on All About Statistics.

3 years ago…

MONTHLY MEMORY LANE: 3 years ago: November 2012. I mark in red three posts that seem most apt for general background on key issues in this blog.[1]. Please check out others that didn’t make the “bright red cut”. If you’re interested in the Likelihood Principle, check “Blogging Birnbaum” and “Likelihood Links”. If you think P-values are hard to explain, see how the “Bad News Bears” struggle to decipher Bayesian probability. (Some of the posts allude to seminars I was giving at the London School of Economics 3 years ago.)

November 2012

• (11/04) PhilStat: So you’re looking for a Ph.D. dissertation topic?
• (11/07) Seminars at the London School of Economics: Contemporary Problems in Philosophy of Statistics
• (11/10) Bad news bears: ‘Bayesian bear’ rejoinder – reblog
• (11/12) new rejected post: kvetch (and query)
• (11/14) continuing the comments. …
• (11/16) Philosophy of Science Association (PSA) 2012 Program
• (11/18) What is Bayesian/Frequentist Inference? (from the normal deviate)
• (11/18) New kvetch/PhilStock: Rapiscan Scam
• (11/19) Comments on Wasserman’s “what is Bayesian/frequentist inference?”
• (11/21) Irony and Bad Faith: Deconstructing Bayesians – reblog
• (11/23) Announcement: 28 November: My Seminar at the LSE (Contemporary PhilStat)
• (11/25) Likelihood Links [for 28 Nov. Seminar and Current U-Phil]
• (11/28) Blogging Birnbaum: on Statistical Methods in Scientific Inference
• (11/30) Error Statistics (brief overview)

[1] I exclude those reblogged fairly recently. Posts that are part of a “unit” or a group of “U-Phils” count as one. Monthly memory lanes began at the blog’s 3-year anniversary in Sept, 2014.

Please comment on the article here: Error Statistics Philosophy » Statistics

The post 3 YEARS AGO (NOVEMBER 2012): MEMORY LANE appeared first on All About Statistics.

Please comment on the article here: Realizations in Biostatistics

The post Even the tiniest error messages can indicate an invalid statistical analysis appeared first on All About Statistics.

A matrix is a convenient way to store an array of numbers. However, often you need to extract certain elements from a matrix. The SAS/IML language aupports two ways to extract elements: by using subscripts or by using indices. Use subscripts when you are extracting a rectangular portion of a matrix, such as a row, a column, or a submatrix. Use indices when you want to extract values from a non-rectangular pattern.

Extracting rows, columns, and submatrices

You can extract a submatrix by using subscripts to specify the rows and columns of a matrix. Use square brackets to specify subscripts. For example, if A is a SAS/IML matrix, the following are submatrices:

• The expression A[2,1] is a scalar that is formed from the second row and the first column of A.
• The expression A[2, ] specifies the second row of A. The column subscript is empty, which means “use all columns.”
• The expression A[ , {1 3}] specifies the first and third columns of A. The row subscript is empty, which means “use all rows.”
• The expression A[3:4, 1:2] specifies a 2 x 2 submatrix that contains the elements that are in the intersection of the third and fourth rows of A and the first and second columns of A.

The following SAS/IML statements demonstrate extracting submatrices:

proc iml;
A = shape(1:30, 5);     /* 5 x 6 matrix */
scalar = A[2,1];
row = A[2, ];           /* 2nd row */
cols = A[ , {3 1}];     /* 3rd and 1st columns */
matrix =  A[3:4, 1:2];  /* 2 x 2 matrix: (A[3,1] || A[3,2]) // 
                                         (A[4,1] || A[4,2]) */
print A, scalar, row, cols, matrix;

The previous examples were adapted from Wicklin (2013) "Getting Started with the SAS/IML Language", which I recommend for programmers who are starting to learn the SAS/IML language.

Extracting diagonals and triangular elements

Non-rectangular patterns are common in statistical programming. Examples include the matrix diagonal and the lower triangular portion of a square matrix. The SAS/IML provides special functions for extracting diagonal and triangular regions:

• The VECDIAG function extracts the diagonal of a matrix. The result is a column vector.
• Use the ROW function and COL function to extract an arbitrary diagonal or anti-diagonal pattern, such as elements from a banded matrix.
• Use the SYMSQR function to extract the lower-triangular elements in row-major order.
• Use the VECH function to extract the lower-triangular elements in column-major order.
• The SYMSQR and VECH function extract the diagonal elements, but you can use a trick to remove the diagonal elements and obtain only the elements below the diagonal.

For example, the following statements extract the main diagonal, the lower triangular elements in row-major order, and the lower triangular elements in column-major order:

proc iml;
S = shape(1:16, 4);    /* 5 x 5 matrix */
v = vecdiag(S);
L_row = symsqr(S);
L_col = vech(S);
print S, v, L_row, L_col;

Extracting arbitrary patterns of elements

For the extraction of arbitrary elements, you should use indices. SAS/IML software stores matrices in row-major order, which means the elements are enumerated as you move across the first row, then across the second row, and so forth. However, notice that you do not know the subscripts for A[3] unless you know the shape of A. If A is a 3 x 3 matrix, A[3] corresponds to A[1,3]. However, if A is a 2 x 2 matrix, A[3] corresponds to A[2,1].

The SUB2NDX function enables you to convert subscript information into the equivalent indices. For example, suppose that B is a 5 x 5 matrix and you want to extract the following elements: B[5,2], B[2,4], B[4,3], B[3,1], and B[1,5]. The following statements convert the subscripts into indices and extract the elements:

proc iml;
B = shape(1:25, 5);   /* 5 x 5 matrix */
subscripts = {5 2,  2 4,  4 3,  3 1,  1 5};  /* five (row,col) subscripts */
ndx = sub2ndx(dimension(B), subscripts);
vals = B[ndx];
print vals;

A powerful advantage of indices is that you can use them to assign values as well as to extract values. For example, if v is a five-element column vector, the expression B[ndx] = v assigns the values v to the elements of B. Notice that this is a vectorized operation. If you do not use indices, you would probably write a DO loop that iterates over the subscripts. In general, vector operations are more efficient than looping operations.

The ability to access elements in an arbitrary order is a big advantage for SAS/IML programmers. Whereas the DATA step processes one observation at a time, the SAS/IML language enables you to access SAS data in whatever order makes sense for the algorithm that you are writing.

The post Extracting elements from a matrix: rows, columns, submatrices, and indices appeared first on The DO Loop.

Please comment on the article here: The DO Loop

The post Extracting elements from a matrix: rows, columns, submatrices, and indices appeared first on All About Statistics.

One student of mine coded by mistake an independent Metropolis-Hastings algorithm with too small a variance in the proposal when compared with the target variance. Here is the R code of this implementation:

#target is N(0,1)
#proposal is N(0,.01)
T=1e5
prop=x=rnorm(T,sd=.01)
ratop=dnorm(prop,log=TRUE)-dnorm(prop,sd=.01,log=TRUE)
ratav=ratop[1]
logu=ratop-log(runif(T))
for (t in 2:T){
  if (logu[t]>ratav){
    x[t]=prop[t];ratav=ratop[t]}else{x[t]=x[t-1]}
  }

It produces outputs of the following shape
which is quite amazing because of the small variance. The reason for the lengthy freezes of the chain is the occurrence with positive probability of realisations from the proposal with very small proposal density values, as they induce very small Metropolis-Hastings acceptance probabilities and are almost “impossible” to leave. This is due to the lack of control of the target, which is flat over the domain of the proposal for all practical purposes. Obviously, in such a setting, the outcome is unrelated with the N(0,1) target!

It is also unrelated with the normal proposal in that switching to a t distribution with 3 degrees of freedom produces a similar outcome:

It is only when using a Cauchy proposal that the pattern vanishes:

Please comment on the article here: Xi'an's Og » R

The post a programming bug with weird consequences appeared first on All About Statistics.

1. In almost every country, younger people are less religious.
2. In most countries, people with more education are less religious.
3. In about half of the 34 countries, people with lower income are less religious.  In the rest, the effect (if any) is too small to be distinguished from noise with this sample size.
4. In most countries, people who watch more television are less religious.
5. In a fewer than half of the countries, people who listen to more radio are less religious.
6. The results for newspapers are similar: only a few countries show a negative effect, and in some countries the effect is positive.
7. In almost every country, people who use the Internet are less religious.
8. There is a weak relationship between the strength of the effect and the average degree of religiosity: the negative effect of Internet use on religion tends to be stronger in more religious countries.

Effect size

Quadratic age model

Model 1 results

Model 2 results

Please comment on the article here: Probably Overthinking It

The post Internet use and religion, part four appeared first on All About Statistics.

# Calculates Sparse Partial Correlation Matrix
sparse_matrix<-EBICglasso(cor(ratings), n=1000)
round(sparse_matrix,2)
 
# Plots results
gr<-list(1:4,5:8,9:12,13:15)
node_color<-c("lightgoldenrod","lightgreen","lightpink","cyan")
qgraph(sparse_matrix, fade = FALSE, layout="spring", groups=gr, 
       color=node_color,labels=names(ratings), label.scale=FALSE, 
       label.cex=1, node.width=.5, edge.width=.25, minimum=.05)

Please comment on the article here: Engaging Market Research

The post Statistical Models That Support Design Thinking: Driver Analysis vs. Partial Correlation Networks appeared first on All About Statistics.