In words, R² is a measure of how much of the variance in y is explained by the model, f.

Under “general conditions”, as Wikipedia says,
R² is also the square of the correlation between the actual and predicted outcomes:

I prefer the “squared correlation” definition, as it gets more directly at what is usually my primary concern: prediction. If R² is close to one, then the model’s predictions mirror true outcome, tightly. If R² is low, then either the model does not mirror true outcome, or it only mirrors it loosely: a “cloud” that — hopefully — is oriented in the right direction. Of course, looking at the graph always helps:

The question we will address here is : how do you get from R² to correlation?

If you look at the two equations for correlation and R², you can see that the relationship between them does not hold for general f and y. In particular, correlation is far more invariant to scaling. For correlation, all of the following relations are true:

But only the last relation is true for R². So in general, the two cannot be functions of each other.

However, we are making a specific assumption about f: it is the output of a predictive model. In fact, we are actually making several specific assumptions;

1. f is the model that minimizes squared-error loss
2. Because it is the optimum (in the sense of item 1), there is no shift of f that will improve the fit.
3. Because it is the optimum (in the sense of item 1), there is no scaling of f that will improve the fit.

We can express the above assumptions as follows:

If we express the last line as

Then loss is optimized at g(1,0).

Since g(1,0) is the optimum, then the derivatives of g are zero here:

From the partial with respect to a, we get that

and from the partial with respect to b, we get that

(since the mean is just the normalized sum).

Now, let’s shift the coordinate system so that y (and f) are equal to zero. This makes the equations much simpler, and doesn’t affect the generality of the result.

The equation for R² is now

And the equation for correlation is now

And we are done.

Notice that this result is true for any model fit that meets the assumptions that we outlined above (squared-error loss, optimality under shifting and scaling). Linear regression (with an intercept) fits this criterion, but so can other model-fitting techniques — generalized additive models, polynomial fits, decision (regression) trees, ensemble methods — if the proper loss function is used.

To repeat: for optimal models (under squared-error loss, shift and scale invariance), R² is the square of the correlation between the true and predicted outcomes. This relationship is not true for general f and y.

Related posts:

1. The Simpler Derivation of Logistic Regression
2. The equivalence of logistic regression and maximum entropy models
3. Living in A Lognormal World

We adapt these presentation ideas to make explicit the well known equivalence of logistic regression and maximum entropy models.In our new writeup: The equivalence of logistic regression and maximum entropy models we move to multi-category modeling and demonstrate how one invents something as remarkable as logistic regression.

Related posts:

1. The Simpler Derivation of Logistic Regression
2. Learn Logistic Regression (and beyond)
3. Large Data Logistic Regression (with example Hadoop code)

While you don’t have to know how to derive logistic regression or how to implement it in order to use it, the details of its derivation give important insights into interpreting and troubleshooting the resulting models. Unfortunately, most derivations (like the ones in [Agresti, 1990] or [Hastie, et.al, 2009]) are too terse for easy comprehension. Here, we give a derivation that is less terse (and less general than Agresti’s), and we’ll take the time to point out some details and useful facts that sometimes get lost in the discussion.

To make the discussion easier, we will focus on the binary response case. We assume that the case of interest (or “true”) is coded to 1, and the alternative case (or “false”) is coded to 0.

The logistic regression model assumes that the log-odds of an observation y can be expressed as a linear function of the K input variables x:

Here, we add the constant term b₀, by setting x₀ = 1. This gives us K+1 parameters. The left hand side of the above equation is called the logit of P (hence, the name logistic regression).

Let’s take the exponent of both sides of the logit equation.

This immediately tells us that logistic models are multiplicative in their inputs (rather than additive, like a linear model), and it gives us a way to interpret the coefficients. The value exp(b_j) tells us how the odds of the response being “true” increase (or decrease) as x_j increases by one unit, all other things being equal. For example, suppose b_j = 0.693. Then exp(b_j) = 2. If x_j is a numerical variable (say, age in years), then every year’s increase in age doubles the odds of the response being true — all other things being equal. If x_j is a binary variable (say, sex, with female coded as 1 and male as 0), then if the subject is female, then the response is two times more likely to be true than if the subject is male, all other things being equal.

We can also invert the logit equation to get a new expression for P(x):

The right hand side of the top equation is the sigmoid of z, which maps the real line to the interval (0, 1), and is approximately linear near the origin. A useful fact about P(z) is that the derivative P’(z) = P(z) (1 – P(z)). Here’s the derivation:

Later, we will want to take the gradient of P with respect to the set of coefficients b, rather than z. In that case, P’(z) = P(z) (1 – P(z))z‘, where ‘ is the gradient taken with respect to b.

The solution to a Logistic Regression problem is the set of parameters b that maximizes the likelihood of the data, which is expressed as the product of the predicted probabilities of the N individual observations.

(X, y) is the set of observations; X is a K+1 by N matrix of inputs, where each column corresponds to an observation, and the first row is 1; y is an N-dimensional vector of responses; and (x_i, y_i) are the individual observations.

It’s generally easier to work with the log of this expression, known (of course) as the log-likelihood.

Maximizing the log-likelihood will maximize the likelihood. As a side note, the quantity −2*log-likelihood is called the deviance of the model. It is analogous to the residual sum of squares (RSS) of a linear model. Ordinary least squares minimizes RSS; logistic regression minimizes deviance. A useful goodness-of-fit heuristic for a logistic regression model is to compare the deviance of the model with the so-called null deviance: the deviance of the constant model that returns only the global response probability for every data point. One minus the ratio of deviance to null deviance is sometimes called pseudo-R², and is used the way one would use R² to evaluate a linear model.

Traditional derivations of Logistic Regression tend to start by substituting the logit function directly into the log-likelihood equations, and expanding from there. The derivation is much simpler if we don’t plug the logit function in immediately. To maximize the log-likelihood, we take its gradient with respect to b:

where P_i is shorthand for P(x_i). The maximum occurs where the gradient is zero.

We can expand this equation further, when we remember that P’ = P(1-P):

The last line merges the two cases (y_i = 1 and y_i = 0) into a single sum. We can now cancel terms and set the gradient to zero. This gives us the set of simultaneous equations that are true at the optimum:

Notice that the equations to be solved are in terms of the probabilities P (which are a function of b), not directly in terms of the coefficients b themselves. This means that logistic models are coordinate-free: for a given set of input variables, the probabilities returned by the model will be the same even if the variables are shifted, combined, or rescaled. Only the values of the coefficients will change.

The other thing to notice from the above equations is that the sum of probability mass across each coordinate of the x_i vectors is equal to the count of observations with that coordinate value for which the response was true. For example, suppose the jth input variable is 1 if the subject is female, 0 if the subject is male. Then

In other words, the summed probability mass for the female subjects equals the count of female subjects with the response “true”. It is also true that the sum of all the probability mass over the entire training set will equal the number of “true” responses in the training set. This is what we mean when we say that logistic regression preserves the marginal probabilities of the training data.

Solving for the Coefficients

The most straightforward way to solve for the coefficients b is Newton’s method. The Fisher scoring method that is used in most off-the-shelf implementations is a more general variation of Newton’s method; it works on the same principles. We will describe solving for the coefficients using Newton’s method.

Suppose you have a vector valued function f: y = f(b). You want to find the value b_opt such that f(b)_opt = 0. Assuming that we start with an initial guess b₀, we can take the Taylor expansion of f around b₀:

Here, f‘ is a matrix; it is the Jacobean of first derivatives of f with respect to b. Setting the left hand side to zero, we can solve for Δ as

We then update our estimate for b:

and iterate until convergence.

In our case, f is the gradient of the log-likelihood, and its Jacobean is the Hessian (the matrix of second derivatives) of the log-likelihood function.

where W is a diagonal matrix of the derivatives P’_i, and the ith column of X corresponds to the ith observation. So we can solve for Δ at each iteration as

where W is the current matrix of derivatives, y is the vector of observed responses, and P_k is the vector of probabilities as calculated by the current estimate of b.

Compare this to the solution of a linear regression:

Comparing the two, we can see that at each iteration, Δ is the solution of a weighted least square problem, where the “response” is the difference between the observed response and its current estimated probability of being true. This is why the technique for solving logistic regression problems is sometimes referred to as iteratively re-weighted least squares. Generally, the method does not take long to converge (about 6 or so iterations).

Thinking of logistic regression as a weighted least squares problem immediately tells you a few things that can go wrong, and how. For example, if some of the input variables are correlated, then the Hessian H will be ill-conditioned, or even singular. This will result in large error bars (or “loss of significance”) around the estimates of certain coefficients. It can also result in coefficients with excessively large magnitudes, and often the wrong sign. If an input perfectly predicts the response for some subset of the data (but not all), then the term P_i (1 – P_i) will be driven to zero for that subset, which will drive the coefficient for that input to infinity (if the input perfectly predicted all the data, then the residual (y – P_k) has already gone to zero, which means that you are already at the optimum).

On the other hand, the least squares analogy also gives us the solution to these problems: regularized regression, such as lasso or ridge. Regularized regression penalizes excessively large coefficients, and keeps them bounded. If you are implementing your own logistic regression procedure, rather than using a package, then it is straightforward to implement a regularized least squares for the iteration step (as Win-Vector has done). But even if you are using an off-the-shelf implementation, the above discussion will help give you a sense of how to interpret the coefficients of your model, and how to recognize and troubleshoot some issues that might arise.

Conclusion

Here is what you should now know from going through the derivation of logistic regression step by step:

- Logistic regression models are multiplicative in their inputs.

- The exponent of each coefficient tells you how a unit change in that input variable affects the odds ratio of the response being true.

- Logistic regression is coordinate-free: translations, rotations, and rescaling of the input variables will not affect the resulting probabilities.

- Logistic regression preserves the marginal probabilities of the training data.

- Overly large coefficient magnitudes, overly large error bars on the coefficient estimates, and the wrong sign on a coefficient could be indications of correlated inputs.

- Coefficients that tend to infinity could be a sign that an input is perfectly correlated with a subset of your responses. Or put another way, it could be a sign that this input is only really useful on a subset of your data, so perhaps it is time to segment the data.

- Pseudo-R² is a useful goodness-of-fit heuristic.

References

[Agresti, 1990] Agresti, A. (1990). Categorical Data Analysis.

[Hastie, et.al, 2009] Hastie, T., R. Tibshirani, and J. Friedman (2009). The Elements of Statistical Learning, 2nd Edition.

Related posts:

1. The equivalence of logistic regression and maximum entropy models
2. Learn Logistic Regression (and beyond)
3. Large Data Logistic Regression (with example Hadoop code)

There is no final “right shape.” In fact even your data is never right. You will always be called to re-do your analysis (new variables, new data, corrections) so you should always understand you are on your “penultimate analysis” (always one more to come). This is why we insist on using general methods and scripted techniques, as these methods are much much easier to reliably reapply on new data than GUI/WYSWYG techniques.

In this article we will work a small example and call out some R tools that make reshaping your data much easier. The idea is to think in terms of “relational algebra” (like SQL) and transform your data towards your tools (and not to attempt to adapt your tools towards the data in an ad-hoc manner).Take a simple example where you are designing a new score called “score2” to predict or track an already known value called “score1.” The typical situation is score1 is a future outcome (such as the number of dollars profit on a transaction) and score2 is a prediction (such as the estimated profit before the transaction is attempted). Training data is usually assembled by performing a large number of transactions, recording what was known before the transaction and then aligning or joining this data with measured results when they become available. For this example we are not interested in the inputs driving the model (a rare situation, but we are trying to make our example as simple as possible) but only examining the quality of score2 (which is defined as how well it tracks score1).

All of this example will be in R, but the principles are chosen apply more generally. First let us enter some example data:



> d <- data.frame(id=c(1,2,3,1,2,3),score1=c(17,5,6,10,13,7),score2=c(13,10,5,13,10,5))

> d


This gives us our example data. Each row is numbered (1 through 6) has an id and both our scores:

  id score1 score2
1  1     17     13
2  2      5     10
3  3      6      5
4  1     10     13
5  2     13     10
6  3      7      5

We said our only task was to characterize how well score2 works at predicting score1 (or how good a substitute score2 is for score1). We could compute correlation, RMS error, info-gain or some such. But instead lets look at this graphically. We will prepare a graph showing how well score1 is represented by score2. For this we choose to place score1 on the y-axis (as it is the outcome) and score2 on the x-axis (as it is the driver).



> library(ggplot2)

> ggplot(d) + geom_point(aes(x=score2,y=score1))


Figure 1: score1 as a function of score2.

This does not look good. We would liked to have seen all of the dots falling on the line “y=x.” This plot shows score2 is not predicting score1 very well. Part of this is that we missed an important feature of the data (and because we missed it the feature becomes a problem): the ids repeat. First we re-order by id to make this more obvious.



> dsort <- d[order(d$id),]

> dsort


  id score1 score2
1  1     17     13
4  1     10     13
2  2      5     10
5  2     13     10
3  3      6      5
6  3      7      5

This is a very common situation. The original score is not completely a function of the known inputs. We are using “id” to abstract represent all of the inputs, two rows in our example have the same id if and only if all known inputs are exactly the same. The repeating ids are the same experiment run at different times (a good idea) and the variation in score1 could be the effect of an un-modeled input that changed value or something simple like a “noise term” (a random un-modeled effect). Notice that score2 is behaving as a function of id- all rows with the same id have the same value for score2. If score2 is a model then it has to be a function of the inputs (or more precisely if it is not a function of the inputs you have done something wrong). So any variation of score1 between rows with identical id is “unexplainable variation” (unexplainable from the point of view of currently tracked inputs). You should know about, characterize and report this variation (why it is good to have some repeated experiments). But this variation is not the model’s fault, if we want to know how good a job we did constructing the model (which we now see can be a slightly different question than how well the model works at prediction) we need to see how much of the explainable variation the model accounts for.

If we assume (as is traditional) the unexplained variation is from a “unbiased noise source” then we can lessen the impact of the noise source by replacing score1 with a value averaged over rows with the same id. This assumption is traditional because an unbiased noise source is present in many problems and assuming anything more requires more research into the problem domain. You would eventually fold such research into your model- so your goal is always have all effects or biases in your model and hope what is left over is unbiased. This is usually not strictly true, but not accounting for the unexplained variation at all is in many cases even worse than modeling the unexplained variation as being bias-free.

And now we find our data is the “wrong shape.” To replace score1 with the appropriate averages we need to do some significant data manipulation. We need to group sets of rows and add new columns. We could do this imperatively (write some loops and design some variables to track and manipulate state) or declaratively (find a path of operations from what you have to what you need through R’s data manipulation algebra). Even though the declarative method is more work the first time (you could often write the code in less time than it takes to read the manuals) it is the right way to go (as it is more versatile and powerful in the end).

Luckily we don’t have to use raw R. There are a number of remarkable packages (all by Hadley Wickham who is also the author of the ggplot2 package we use to prepare our figures) that really improve R’s ability to coherently manage data. The easiest (on us) way do fix up our data is to make the computer work hard and use the powerful melt/cast technique. These functions are found in the libraries reshape and plyr (which were automatically loaded with we loaded ggplot2 library).

melt is a bit abstract. What it does convert your data into a “narrow” format where rows are split into many rows each carrying just one result column of the original row. For example we can melt our data by id as follows:



> dmelt <- melt(d,id.vars=c('id'))

> dmelt


Which yields the following:

   id variable value
1   1   score1    17
2   1   score1    10
3   2   score1     5
4   2   score1    13
5   3   score1     6
6   3   score1     7
7   1   score2    13
8   1   score2    13
9   2   score2    10
10  2   score2    10
11  3   score2     5
12  3   score2     5

Each of the two facts (score1, score2) from our original row is split into its own row. The id column plus the new variable column are now considered to be keys. This format is not used directly but used because it is easy to express important data transformations in terms of it. For instance we wanted our table to have duplicate rows collected and score1 replaced by its average (to attempt to remove the unexplainable variation). That is now easy:



> dmean <- cast(dmelt,fun.aggregate=mean)

> dmean


  id score1 score2
1  1   13.5     13
2  2    9.0     10
3  3    6.5      5

We used cast() in its default mode, where it assumes all columns not equal to “value” are the keyset. It then collects all rows with identical keying and combines them back into wide rows using mean or average as the function to deal with duplicates. Notice score1 is now the desired average, and score2 is as before (as it was a function of the keys or inputs, so it is not affected by averaging). With this new smaller data set we can re-try our original graph:



> ggplot(dmean) + geom_point(aes(x=score2,y=score1))


Figure 2: mean(score1) as a function of score2.

This doesn’t look so bad. A lot of the error or variation in the first plot was unexplainable variation. score2 isn’t bad given its inputs. If you wanted to do better than score2 you would be advised to find more modeling inputs (versus trying more exotic modeling techniques).

Of course a client or user is not interested if score2 is “best possible.” They want to know if it is any good. To do this we should show them (either by graph or by quantitative summary statistics like we mentioned earlier) at least 3 things:

1. How well the model predicts overall (the very first graph we presented).
2. How much of the explainable variation the model predicts (the second graph).
3. The nature of the unexplained variation (which we will explore next).

We said earlier we are hoping the unexplained variation is noise (or if it is not noise it would be nice if it is a clue to new important modeling features). So the unexplained variation must not go unexamined. We will finish by showing how to characterize the unexplained variation. As before will will just make a graph, but the data preparation steps would be exactly the same if we were using a quantitive summary (like correlation, or any other). And, of course, our data is still not the right shape for this step. Luckily there is another tool ready to fix this: join().



> djoin <- join(dsort,dsort,'id')

> fixnames <- function(cn) {
n <- length(cn);
for(i in 2:((n+1)/2)) { cn[i] <- paste('a',cn[i],sep='') };
for(i in ((n+3)/2):n) { cn[i] <- paste('b',cn[i],sep='') };
cn
}

> colnames(djoin) <- fixnames(colnames(djoin))

> djoin


which produces:

   id ascore1 ascore2 bscore1 bscore2
1   1      17      13      17      13
2   1      17      13      10      13
3   1      10      13      17      13
4   1      10      13      10      13
5   2       5      10       5      10
6   2       5      10      13      10
7   2      13      10       5      10
8   2      13      10      13      10
9   3       6       5       6       5
10  3       6       5       7       5
11  3       7       5       6       5
12  3       7       5       7       5

All of the work was done by the single line “djoin <- join(dsort,dsort,'id')” the rest was just fixing the column names (as self-join is not the central use case of join). What we have now is a table that is exactly right for studying unexplained variation. For each id we have each row with the same id matched. This blows every id from having 2 rows in dsort to 4 rows in djoin. Notices this gives us every pair of score1 values seen for the same id (which will let us examine unexplained variation) and score2 is still constant over all rows with the same id (as it has always been throughout our analysis). With this table we can now plot how score1 varies for rows with the same id:



> ggplot(djoin) + geom_point(aes(x=ascore1,y=bscore1))


Figure 3: score1 as a function of score1.

And we can see, as we expected, the unexplained variation in score1 is about as large as the mismatch between score1 and score2 in our original plot. The important thing is this is all about score1 (score2 is now literally out of the picture). The analyst’s job would now be to try and tie bits of the unexplained variation to new inputs (that can be folded into a new score2) and/or characterize the noise term (so the customer knows how close they should expect repeated experiments to be).

What we are trying to encourage with the use of “big hammer tools” is an ability and willingness to look at and transform your data in meaningful steps. It often seems easier and more efficient to build one more piece of data tubing, but a lot of data tubes become an unmanageable collection of spaghetti code. The analyst should, in some sense, always be looking at data and not looking at coding details. For these sort of analyses we encourage analysts to think in terms of “data shape” and transforms. This discipline leaves more of the analysts energy and attention to think productively about the data and actual problem domain.

For the third plot showing the variation of score1 across different rows (but same ids) it may be appropriate to use a slightly more complicated join() procedure than we showed. The join shown produced rows of artificial agreement where both values of score1 came from the same row (thus had no chance of being different, so in some sense deserve no credit). This is also the only way any non-duplicated evaluations could make it to the plot. To eliminate these uninteresting agreements from the plot do the following:



> d$rowNumber <- 1:(dim(d)[1])

> djoin <- join(d,d,'id')

> colnames(djoin) <- fixnames(colnames(djoin))

> djoin <- djoin[djoin$arowNumber!=djoin$browNumber,]

> djoin


This gives us a table that shows only values of score1 from different rows:

   id ascore1 ascore2 arowNumber bscore1 bscore2 browNumber
2   1      17      13          1      10      13          4
4   2       5      10          2      13      10          5
6   3       6       5          3       7       5          6
7   1      10      13          4      17      13          1
9   2      13      10          5       5      10          2
11  3       7       5          6       6       5          3

And only plots points on the diagonal if “you have really earned them”:

So while the direct join() may not be the immediate perfect answer it is still a good intermediate to form as what you want is only simple data transformation away from it.

Related posts:

1. A Demonstration of Data Mining
2. Good Graphs: Graphical Perception and Data Visualization
3. The Data Enrichment Method

Logistic regression was invented in the late 1930s by statisticians Ronald Fisher and Frank Yates. The definitive resource on this (and other generalized linear models) is Alan Agresti “Categorical Data Analysis” 2002, New York, Wiley-Interscience. Logistic regression is a “categorical” tool in that it is used to predict categories (fraud/not-fraud, good/bad …) instead of numeric scores (like standard regression). For example: consider the “car” data set from the UCI machine learning database ( data, description ). The data is taken from a consumer review of cars. Each car is summarized by 6 attributes ( price, maintenance costs, doors, storage size, seating and safety ); there is also a conclusion column that contains the final overall recommendation (unacceptable, acceptable, good, very good). The machine learning problem is to infer the reviewer’s relative importance or weight of each feature. This could be used to influence a cost constrained design of a future car. This dataset was originally used to demonstrate hierarchical and decision tree based expert systems. But logistic regression can quickly derive interesting results.

Let us perform a complete analysis together (at least in our imaginations if not with our actual computers). First download and install the excellent free analysts workbench called “R”. This software package is an implementation of John Chambers’ S language (a statistical language designed to allow for self-service statistics to relieve some of Chambers’ consulting responsibilities) and a near relative of the SPlus system. This system is powerful and has a number of very good references (our current favorite being Joseph Adler “R in a nutshell” 2009, O’Reilly Media). Using R we can build an interesting model in 2 lines of code.

After installing R start the program and copy the following line into the R command shell.

CarData <- read.table(url('http://archive.ics.uci.edu/ml/machine-learning-databases/car/car.data'),
    sep=',',col.names=c('buying','maintenance','doors','persons','lug_boot','safety','rating'))

This has downloaded the car data directly from the UCI database and added a header line so we can refer to variables by name. To see roll-ups or important summaries of our data we could just type in "summary(CarData)." But we will move on with the promised modeling. Now type in the following line:

logisticModel <- glm(rating!='unacc' ~ buying + maintenance + doors + persons + lug_boot +safety,
    family=binomial(link = "logit"),data=CarData)

We now have a complete logistic model. To examine this model we type "summary(logisticModel)". And see the following (rather intimidating) summary:

Coefficients:
                  Estimate Std. Error z value Pr(>|z|)
(Intercept)       -28.4255  1257.5255  -0.023    0.982
buyinglow           5.0481     0.5670   8.904  < 2e-16 ***
buyingmed           3.9218     0.4842   8.100 5.49e-16 ***
buyingvhigh        -2.0662     0.3747  -5.515 3.49e-08 ***
maintenancelow      3.4064     0.4692   7.261 3.86e-13 ***
maintenancemed      3.4064     0.4692   7.261 3.86e-13 ***
maintenancevhigh   -2.8254     0.4145  -6.816 9.36e-12 ***
doors3              1.8556     0.4042   4.591 4.41e-06 ***
doors4              2.4816     0.4278   5.800 6.62e-09 ***
doors5more          2.4816     0.4278   5.800 6.62e-09 ***
persons4           29.9652  1257.5256   0.024    0.981
personsmore        29.5843  1257.5255   0.024    0.981
lug_bootmed        -1.5172     0.3758  -4.037 5.40e-05 ***
lug_bootsmall      -4.4476     0.4750  -9.363  < 2e-16 ***
safetylow         -30.5045  1300.3428  -0.023    0.981
safetymed          -3.0044     0.3577  -8.400  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

We also saw the cryptic warning message "glm.fit: fitted probabilities numerically 0 or 1 occurred" which we will discuss later. Returning to our result we see a left column that is formed by concatenating variable names and variable values (values are called "levels" when they are strings). For example the label "buyinglow" is a combination of "buying" and "low" meaning a low purchase price. The next column (and the last one we will dig into) is the score associated with this combination of variable and level. The interpretation is that a care that has "buyinglow" is given a 5.0481 point score bonus. Whereas a car with "safetylow" is given a -30.5045 scoring penalty. In fact the complete prediction procedure for a new car is to look the levels specified for all 6 variables and add up the correct scores (plus the "(Intercept)" score of -28.4255 which is used as an initial score). Any value not found is assumed to be zero. This summed-up score is called the "link" and is essentially the model prediction. Positive link values are associated with acceptable cars and negative link values are associated with unacceptable cars. For example the first car in our data set is:

  buying maintenance doors persons lug_boot safety rating
   vhigh       vhigh     2       2    small    low  unacc

According to the columns: we see above our scoring procedure assigns a very bad score of -68 to this car- correctly predicting the "unacc" rating. We can examine the error rates of our model with the single line:

table(CarData$rating,predict(logisticModel,type='response')>=0.5)

While yields the result:

        FALSE TRUE
  acc      32  352
  good      0   69
  unacc  1166   44
  vgood     0   65

This diagram is called a "contingency table" and is a very powerful summary. The rows are labeled with the ratings assigned at training time (unacceptable, acceptable, good and very good). The columns FALSE and TRUE denote the model predicted the car was unacceptable or at least acceptable. From the row "unacc" we see that 1166 of the 1166+44 unacceptable cars were correctly predicted FALSE (or not at least acceptable). Also notice the only face negatives are the 32 FALSEs in the "acc" row- none of the good or very good cars were predicted to be unacceptable. We can also look at this graphically:

library(lattice)
densityplot(predict(logisticModel,type='link'),groups=CarData$rating!='unacc',auto.key=T)

Which yields the following graph:

This is an area density chart. Each car that was defined as being unacceptable adds a single blue circle to the bottom of the chart. Each car that was defined as being acceptable adds a single magenta circle to the bottom of the chart. The left-right position of each circle is the link score the model assigned to the circle. There are so many circles that they start to overlap into solid smudges. To help with this charting software adds the density curves above the circles. Density curves are a lot like histograms- the height of the curve indicates what fraction of the circles of the same color are under that region of the curve. So we can see most of the blue circles are in 3 clusters centered at -55, -30 and -5 while the magenta circles are largely clustered around +5. From a chart like this you can see that a decision procedure of saying a link score above zero is good and below zero is bad would be pretty accurate (most of the blue is to the left and most of the magenta is to the right). In fact this rule would be over 95% accurate (though accuracy is not a be-all measure, see: Accuracy Measures).

So far so good. We have built a model that accurately predicts conclusions based on inputs (so it could be used to generate new ratings) and furthermore our model has visible "Estimate" coefficients that report the relative strength of each feature (which we could use prescriptively in valuing trade-offs in designing a new car). Except we need to go back to the warning message: "glm.fit: fitted probabilities numerically 0 or 1 occurred." For a logistic regression the only way the model fitter can encounter "probabilities numerically 0 or 1" is if the link score was out of a reasonable range of zero (say +-20). Remember we saw link scores as low as -68. With a link score of -68 the probability of the car in question being acceptable is around 2*10^-16. This from a training set that only included 1728 items (so really can not be expected to see events much rarer than one in a thousand. We are deliberately confusing two different types of probabilities here- but it is a good way to think).

What is the cause of these extreme link scores? Extreme coefficient estimates. The +29.96 preference for "persons4" (cars that seat 4) is a huge vote that swamps out effects from purchase price and maintenance cost. The model has over fit and made some dangerous extreme assumptions. What causes this are variable and level combinations that have no falsification in the data set. For example: suppose only one car had the variable level "person4" and that car happened to be acceptable. The logistic modeling package could always raise the link score of that single car by putting a bit more weight on the "person4" estimate. Since this variable level shows up only in positive examples (and in this case only one example) there is absolutely no penalty for increasing the coefficient. Logistic regression models are built by an optimizer. And when an optimizer finds a situation with no penalty- it abuses the situation to no end. This is what the warning was reporting. All link numbers map to probabilities between zero and one; only ridiculously large link values map to probabilities near one (and only ridiculously small values map to probabilities near zero). The optimizer was caught trying to make some of the coefficients "run away" or "run to infinity." There are some additional diagnostic signs (such as the large coefficients, large standard errors and low significant levels), but there is no advice offered by the system in how to deal with this. The standard methods are to suppress problem variables and levels (or suppress data with the problem variables and levels present) from the model. But this is inefficient in that the only way we have of preventing a variable from appearing to be too useful is not to use it. These are exactly the variables we do not want to eliminate from the model, but they are unsafe to keep in the model (their presence can cause unreliable predictions on new data not seen during training).

What can we do to fix this? We need to ensure that running a coefficient to infinity is not without cost. One way to achieve this would be something like Laplace smoothing where we enter two made-up data items: one that has every level set on and is acceptable and one that has every level set on and is unacceptable. Unfortunately there is no easy way to do this from the user-layer in R. For example each datum can only have one value set for each categorical variable- so we can't define a datum that has all features on. Another way to fix this would be to directly penalize large coefficients (like Tychonoff regularization in linear algebra). Explicit regularization is a good idea and very much in the current style. Again, unfortunately, the R user layer does not expose a regularization control to the user. But one of the advantages of logistic regression is that it is relatively easy to implement (harder than Naive Bayes or standard regression in that it needs and optimizer, but easier than SVM in that the optimizer is fairly trivial).

The logistic regression optimizer works to find a model of probabilities p() that maximizes the sum:

Or in english: assign large probabilities to examples known to be positive and small probabilities to examples known to be negative. Now the logistic model assigns probabilities using a function of the form:

The beta is the model parameters and the x is the data associated with a given example. The dot product of beta and x is the link score we saw earlier. The rest of the function is called the sigmoid (also used by neural nets) and its inverse is called the "logit" which is where logistic regression gets its name. Now this particular function (and others have the so-called "canonical link") has the property that the gradient (the vector of derivative directions indicating the direction of most rapid score improvement) is particularly beautiful. The gradient vector is:

This quantity is a vector because it is a weighted sum over the data_i (which are all vectors of feature values and value/level indicators). As we expect the gradient to be zero at an optimal point we now have a set of equations we expect to be simultaneously satisfied at the optimal model parameters. In fact these equations are enough to essentially determine the model- find parameter values that satisfy all of this vector equation and you have found the model (this is usually done by the Newton–Raphson method or by Fisher Scoring). As an aside this is also the set of equations that the maximum entropy method must specify; which is why for probability problems maximum entropy and logistic regression models are essentially identical.

If we directly add a penalty for large coefficients to our original scoring function as below:

Beta is (again) our model weights (laid out in the same order as the per-datum variables) and epsilon (>=0) is the new user control for regularization. A small positive epsilon will cause regularization without great damage to model performance (for our example we used epsilon = 0.1). Now our optimal gradient equations (or set of conditions we must meet) become:

Or- instead of the model having to reproduce all know feature summaries (if a feature is on in 30% of the positive training data then it must be on for 30% of the modeled positive probabilities) we now have a slop term of 2 epsilon beta. To the extent a coefficient is large its matching summary is allowed slack (making it harder for the summary to drive the coefficient to infinity). This system of equations is as easy to solve as the original system (a slightly different update is used in the Newton-Raphson method) and we get a solution as below:

variable	kind		level	value
		Intercept		-2.071592578277024
buying		Categorical	high	-1.8456895489650629
buying		Categorical	low	2.024816396087424
buying		Categorical	med	1.257553912038549
buying		Categorical	vhigh	-3.508273337437926
doors		Categorical	2	-1.8414721595612646
doors		Categorical	3	-0.391932359146582
doors		Categorical	4	0.08090597021546056
doors		Categorical	5more	0.08090597021546052
lug_boot	Categorical	big	0.8368739895290729
lug_boot	Categorical	med	-0.2858686820997001
lug_boot	Categorical	small	-2.62259788570632
maintenance	Categorical	high	-1.274387434600252
maintenance	Categorical	low	1.3677275941857292
maintenance	Categorical	med	1.3677275941857292
maintenance	Categorical	vhigh	-3.5326603320481342
persons		Categorical	2	-8.360699258777752
persons		Categorical	4	3.2937880724973065
persons		Categorical	more	2.9953186080035197
safety		Categorical	high	4.160122946359636
safety		Categorical	low	-8.028169713081502
safety		Categorical	med	1.7964541884449603

This model has essentially identical performance and much smaller coefficients. From a performance point of view this is essentially the same model. What has changed is the model no longer is able to pick up a small bonus by "piling on" a coefficient. For example moving a link value to infinity moves the probabilities from 0.999 to 1.0 which in turn moves the data penalty (assuming the data point is positive) from log(0.999) to log(1.0) or from -0.001 to 0.0. The approximate 1/1000 score improvement is offset by a penalty proportional to the size of the coefficient- making the useless "adding of nines" no longer worth it. Or, as has become famous with large margin classifiers, it is important to improve what the model does on probability estimates near 0.5 not estimates already near 0 or 1.

As expected: the coefficients are significantly different than the standard logistic regression. For example the original model has 3 variables with extreme levels (the intercept, number of passengers and safety) while the new model sees only extreme values (but much smaller) for number of persons and safety (which are likely correlated). Also consider the difference between the buying levels low and very high in the original model (5 - -2 = 7) and in the new model (2 - -3.5 = 5.5) differ by 1.5 or around 3 of the reported standard deviations (indicating the significance summaries are not enough to certify the location of model parameters). It is not just that all of the coefficients have shifted, many of the differences are smaller (and others are not- changing the emphasis of the model). We don not want to overstate differences- we are not so much looking for something better than standard logistic regression as adding an automatic safety that saves us both the effort and loss of predictive power found in fixing models by suppressing unusually distributed (but useful) variables and levels.

An analyst is well served to have logistic regression (and the density plots plus contingency table summaries) as ready tools. These methods will take you quite far. And if you start hitting the limits of these tools you can, as we do, bring in custom tools that allow for explicit regularization yielding effective and reliable results.

Related posts:

1. The equivalence of logistic regression and maximum entropy models
2. The Simpler Derivation of Logistic Regression
3. Large Data Logistic Regression (with example Hadoop code)

This client already tracked the net profit and loss generated by every customer who used his services, and had begun to analyze his customers by group. He was especially interested in Customer Type A; his gut instinct told him that Type A customers were quite profitable compared to the others (Type B) and he wanted to back up this feeling with numbers.

He found that, on average, Type A customers generate about $92 profit per month, and Type B customers average about $115 per month (The data and figures that we are using in this discussion aren’t actual client data, of course, but a notional example). He also found that while Type A customers make up about 4% of the customer base, they generate less than 4% of the net profit per month. So Type A customers actually seem to be less profitable than Type B customers. Apparently, our client was mistaken.

Or was he?

A little more elementary statistics revealed that the median profit generated by Type A customers is $65 — e.g., half the customers from group A generate more than $65 profit per month. The median for Type B customers is about $4.80 — so half the customers from group B generate less than five dollars profit every month. Maybe our client’s instincts aren’t completely off-base.

Let’s look at the distribution of net profit across both customer populations:

Figure 1: Distribution of net profit for Type A customers (blue) and Type B customers (red). The x-axis gives the net profit or loss, and the y-axis gives the fraction of the population that generates a given net profit.

This pattern is typical among the customers of many businesses. The majority of customers generate relatively moderate profit (or loss); but there is an important minority of large-profit and large-loss customers out on both tails. In this case, the monthly customer value actually ranges from losses in the tens of thousands to profits of several hundred thousands (I clipped the graph, for “clarity”).

I hesitate to call these large magnitude customers “outliers” because that term implies anomalous, possibly erroneous, data. In this case, the “outliers” are relatively rare, but important, customers who can potentially make the difference between a company that is in the black or in the red. Still, they are the exception and their behavior doesn’t necessarily tell you anything about the behavior of your typical customer. Knowing the mean profitability of a given customer group is important, of course, but the estimate will be dominated by your exceptionally profitable or lossy customers in that group, and as we’ve seen, that hides information about the majority of your customers.

You might remember from our Good Graphs article that if you have positive skewed data with a wide dynamic range, graphing the data on a log scale helps you see phenomena across the entire range of data that you might miss on the ordinary graph. Unfortunately, we have data here in the positive and negative range. So let’s split the customers into three groups: profitable, unprofitable, and break-even. About 5-6% of the customer base is break-even, roughly the same proportion in Groups A and B; we’ll ignore them for now, and look at the profitable customers first (over 80% of the customers, in both groups).

Figure 2: Distribution of profit from profitable Type A customers (blue) and Type B customers (red). The x-axis gives net profit on a log 10 scale, so every labelled tick corresponds to a change by a factor of 100 (eg. 10^0 = $1, 10^2 = $100, and so on). The y-axis represents the fraction of the profitable customer base that generates a given profit.

Now we can clearly see that (among profitable customers) the typical Type A customer is in fact more profitable than the typical Type B customer. The mean profit from profitable Type A customers is about $227, and the median profit is about $93 (shown by the dashed blue line). About 2/3 of the profitable Type A customers generate between $21 and $400 in profit, and over 95% of them generate between $5 and $1721 in profit. We can call that 95% the set of “typical” profitable Type A customers. That’s not a standard definition, but it’s an intuitive one, and useful for this discussion.

Approximately 2.5% of Type A customers generate profits greater than $1721; let’s call them the Type A “best-customers,” some of whom generate profits in the tens of thousands. They are responsible for 30% of the profit that comes from profitable Type A customers, and 3% of the profit that comes from all profitable customers (even though they only make up 0.2% of that population).

Profitable Type B customers generate $148 mean profit, and about $7.67 median profit (the red dashed line). A typical profitable Type B customer generates between six cents and $1031 in profit — a lower range than what the typical Type A customer generates, although the very highest-performing Type B customers are competitive with the highest-performing Type A customers (about 130 Type B customers outperform all the Type A customers).

Unfortunately, when Type A customers are unprofitable, they are typically more unprofitable than those of Type B. This is another reason why the mean profit from Type A customers overall was so low. Our client correctly perceived that Type A customers are typically quite profitable, but there is a small population of real clunkers in the group, too.

Figure 3: Distribution of loss from unprofitable Type A customers (blue) and Type B customers (red). The x-axis gives loss on a log 10 scale; further to the right on the graph means a larger loss. An unprofitable Type A customer loses a median of $137 a month, and a mean of $1180. Unprofitable Type B customers lose a median of $4.80, and a mean of $210.

We can do a similar analysis for the entire base of profitable customers. We would find that the typical profitable customer generates between six cents and $1200 in profit every month (median $8.65, mean $153), and that the 2.5% of best-customers generate over 60% of the profits.

The Lognormal Distribution

Figure 4: (Left) Distribution of profitable customers (graph clipped at $10,000). The x-axis gives the net profit, and the y-axis gives the fraction of the population that generates a given net profit. (Right) Distribution of profitable customers plotted on a log scale.

The distribution of highly skewed positive data, like the value of profitable customers, incomes, sales, or stock prices, can often be modelled as a lognormal distribution: that is, the log of the data is distributed in a bell-shaped curve centered (in log space) at the median of the data (remember, for a normal curve, the median and the mean are the same). In our case, both the profits (seen above, in Figure 4) and the losses are distributed approximately lognormally. For lognormal populations, the mean is generally much higher than the median, and the bulk of the contribution towards the mean will be made by a small population of highest-valued data points. If you use the mean as a stand-in for value, you will overstate the value of most of your customers.

If your customer value data is distributed approximately lognormally, then you can quickly estimate the range of values that 95% of your customers will fall into. About 95% of normally distributed data will fall within plus/minus two standard deviations of the mean, and taking logarithms converts multiplication into addition. So: if sd is the standard deviation of the natural log of your customer value data, M is the median profit, and k = exp(sd), then 95% of your customers will fall in the value range (M/(k*k), M*k*k). The 2.5% of customers who generate more than M*k*k profit are your best-customers, who often drive a majority of your profit.

Long Tail Theory

The distribution of customers above sounds a lot like Chris Anderson’s Long Tail Theory of consumer goods. Most of the revenue of (for example) a bookseller or a music store comes from a few “hits”, or blockbusters, with the rest of the merchant’s inventory out along the tail of Figure 5, moving a relatively small volume per title.

Figure 5: (Top) A notional long tail curve. The y-axis represents sales volume, and the x-axis represents goods ranked from most to least popular. The highest selling goods are to the left. Note that this figure represents the sales curve differently from how the distribution of customer value is represented on the left side of Figure 4. (Bottom) The customer value data (top 10,000 customers) from Figure 4, plotted in the style above. The y-axis has been limited to $50,000 for clarity.

Anderson generally assumes that sales of such goods are distributed as a power law distribution, rather than a lognormal; the log of power law data isn’t distributed symmetrically, but actually has a longer tail to the right. This means that even for the log of the data, the mean is higher than the median. In fact, in some cases, the mean of a power law distribution can be infinite. If sales volume is power law distributed, then top-selling hits are responsible for an even larger percentage of total sales volume than would be the case with a lognormal.

The Pareto Distribution, which is one form of a power distribution, has been proposed as an alternative to the lognormal for modelling income distribution and other similar phenomena. Researchers have debated whether lognormal or Pareto is a better model for income distribution since at least the 1950s. Qualitatively, the two distributions have similar behavior. There are certain estimation and forecasting tasks where it does make a difference if your data follows a power law rather than a lognormal, but for the purposes of this discussion, it doesn’t really matter. For those who are interested, Michael Mitzenmacher has a fairly approachable discussion about the difference between power laws and lognormal distributions.

Back to Long Tail Theory. Historically, merchants tend to concentrate on high-volume items, due to space limitations and the cost of holding inventory. Overall, however, the sum total of tail-product sales will add up to a respectable volume, especially for web retailers who have unlimited “floor space” — or so the Long Tail theory goes. A retailer must then decide whether to follow the traditional “hits-oriented” strategy, or a more “tail-oriented” strategy that caters to the numerous niche markets.

If we draw an analogy with customer value, then best-customers are “hits.” Obviously, our client would like to “fire” his unprofitable customers while retaining his best-performing customers, and even attract more customers like them. But what about his little customers — the 95% of customers in the typical range? If his retention and growth strategy focuses primarily on attracting and retaining big customers, he is following a hits-oriented strategy. If his campaign also includes reaching out to little customers, then he is following something analogous to a tail strategy.

Not all business works like a music or book seller; the appropriate strategy will vary. Still, we can think of a few reasons why keeping little customers happy is a good idea.

For one thing, big customers are not only rare, but they are the ones that your competitors covet the most. Little customers, meanwhile, can still add up to a respectable chunk of change (close to 40% of net profit in our example above). A solid cushion of smaller customers may soften the blow to your profit margin, should a few of your bigger customers defect.

Consider computer sales. Microsoft and Dell serve both the corporate and consumer markets. To judge from their past marketing practices, they consider business customers to be the more valuable segment (see here for a rant somewhat related to this topic). But business IT sales have declined in the current moribund economic climate; analysts attribute the growth in computer sales for the last quarter of 2009 primarily to consumer spending. Dell’s market growth for that last quarter was much lower than that of HP, Acer, and Apple, which are more consumer-oriented companies. It’s also worth noting that Microsoft saw a 14% decline in revenue for the quarter ending September 30, 2009, compared to the year-ago quarter (and their earnings were in large part due to sales of the Xbox, a consumer product), while at the same time, consumer-oriented Apple saw a 24% increase in revenue from its year-ago quarter.

Your pool of little customers is also a pool of potential future best-customers. And you can’t always guess which ones. So a wise strategy might be to allocate part of your retention and growth campaign to providing loyalty incentives to smaller customers, and educating them about how your higher-end services or products might benefit them. Those little customers who have the means or opportunity to move on to the next level might very well appreciate your efforts, and stay with you, rather than defecting to a competitor.

Optimizing Sales vs. Optimizing Customers

One last thought about retail hits and high-value customers. McPhee’s Theory of Exposure, which is cited by Anita Elberse in her Harvard Business Review article “Should You Invest in the Long Tail?”, states that the popularity of music, film, TV or books is largely driven by “marginal audience participants” — the casual, or light, consumer. Casual consumers gravitate to already popular products because they have limited exposure to alternatives, and hence limited knowledge of them. Consumers of more obscure products, on the other hand, tend to be heavy (and knowledgable) consumers: voracious readers, dedicated music or film buffs, or enthusiasts of specific genres, like science-fiction or horror.

McPhee’s research was done in 1963, using subjects who had a fairly small range of choices, compared to internet scale. Elberse found, however, that the phenomena McPhee described still held for the internet merchants that she studied. She uses this observation (along with McPhee’s companion theory of Double Jeopardy) to argue that retailers should not substantially alter their traditional hits-based strategies. There is an alternative interpretation:

If your business follows McPhee’s theory, then hit products disproportionately attract low-value (low-volume) customers, and vice-versa.

So an overly hits-oriented strategy will skew you towards a base of low-value customers. Indeed, Seth Godin argues that iTunes and Amazon, who are in a better position to implement a more tail-oriented strategy, are thriving at the expense of physical stores exactly because they have been able to steal the quality (high-volume) customers away.

The moral is that both sales and customer value live in a lognormal world, where blockbuster products are marketed to a large cloud of low revenue customers, and high revenue best-customers are supported by large catalogues of low volume products. Fail to serve one side of this relationship, and you risk losing the other side.

Related posts:

1. Statistics to English Translation, Part 2b: Calculating Significance
2. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
3. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures

As in the last article, we will concentrate on situations where we want to test the difference of means. You should read that previous article first, so you are familiar with the terminology that we use in this one.

A pdf version of this current article can be found here.

How is Significance Determined?

Generally speaking, we calculate significance by computing a test statistic from the data. If we assume a specific null hypothesis, then we know that this test statistic will be distributed in a certain way. We can then compute how likely it is to observe our value of the test statistic, if we assume that the null hypothesis is true.

We’ll explain the use of a test statistic with our Sneetch example from the last installment.

The t-test for Difference of Means

Suppose that the test scores for both Star-Bellies and Plain-Bellies are normally distributed, with the means and standard deviations as given in the table below.

	(number of subjects)	(mean score)	(standard error)
Star-Bellies	50	78	7
Plain-Bellies	40	74	8

Remember from the previous installment that we can estimate the true population means and as normally distributed around the empirical population means and respectively, with variances
and
. This is shown in Figure 1. Informally speaking, there is no significant difference in the two populations if the shaded overlap area in Figure 1 is large.

Figure 1: The estimates of the means for two populations

Calculating this area is somewhat involved. Instead, we calculate the t-statistic:

(1)

where is called the pooled variance of the two populations.

(2)

For our Sneetch example, , and , or the negative of that, depending on which group is Group 1. There are
degrees of freedom.

If the null hypothesis is true, and the two populations are identical, then is distributed according to Student’s distribution with
degrees of freedom. Student’s distribution is sort of a “stretched out” bell curve; as the degrees of freedom increase (
), Student’s distribution approaches the standard normal distribution, ¹.

In other words, if the null hypothesis is true, should be near zero. The probability of seeing a of a certain magnitude or greater under the null hypothesis is given by the area under the tails of Student’s distribution:

Figure 2: The area under the tails for a given

This area is . For the Sneetch example, .

The further out on the tails is, the stronger the evidence that you should reject the null hypothesis. If you know for some reason that the mean of one population will be greater than or equal to the other, than you can use the one-tailed test:

Figure 3: The one-tailed test for a given

This test halves the p-value as compared to the two-tailed test, making a given value twice as significant. When in doubt about which to use, the two-tailed test is more conservative against false positives².

In discussions of t-tests, you will often see statements of the form:

The t-test meets the hypothesis that two means are equal if

for a two-tailed test, or

for a (right-sided) one-tailed test.

The quantities on the right hand side of the two equations above are called the critical values for a given significance level (usually,
) and degrees of freedom. The critical values are the values for which the area of the right hand tail is equal to .

Figure 4: Critical value for a one-tailed test. Reject the null hypothesis if

For a two-tailed test, you must halve the area under a single tail.

Figure 5: Critical value for a two-tailed test. Reject the null hypothesis if

This convention dates back to the time when computational resources were scarce, and researchers had to use pre-computed tables of critical values, rather than calculating directly. Today, general statistical packages such as R or Matlab can compute the CDFs of any number of standard distributions; once you can compute the CDF, directly computing (the area under the tails) is straightforward. Despite this, many tutorials of the t-test (and of the F-test, and other significance tests) still adhere to the convention of comparing test statistics to critical values. This tends to needlessly ritualize the whole process, and make it seem more complicated and mysterious than it actually is, at least in my opinion.

David Freedman was very much against the continued practice of using critical values, rather than reporting the actual p-value. The last chapter of Freedman, Pisani and Purves [FPP07] is worth reading for its discussion of this, and other potential pitfalls of significance tests.

Some standard packages for evaluating t-tests, F-tests, or the ANOVA also present analysis results in terms of critical values. Most of them do usually print the actual p value as well, along with the value of the test statistic and the degrees of freedom. Most researchers rightfully report the test statistics along with the actual significance levels: “we conclude that there is a significant difference in mathematical performance (t(88) = 2.499, p = 0.014)… .” Here, 88 gives the degrees of freedom, is the value of the t-statistic, and is of course the p-value.

Similar comments apply to the F-test, discussed in more detail below.

Assumptions

Strictly speaking, the t-test is only valid for normally distributed data where both populations have equal variance. However, the test is fairly robust to non-normal data [Box53]. You can verify that the sample variances are “equal enough” – that is, they could plausibly both be sampled observations from populations with the same variance, by using the F-test. The F-statistic

is distributed according to the F distribution with
degrees of freedom

Figure 6: The F distribution

In practice, the larger variance is usually put in the numerator, so . The test should still be two-tailed, so you should double the area under the right-hand tail³. In this situation, you want to check if you ƒshould accept the null hypothesis (that
) at a given significance level. If so, then you can go ahead and apply the t-test.

There is a variation of the t-tests for distributions of unequal variance, called Welch’s t-test [Wikc]. In this case, you are only checking if the means are equal, not that the distributions are the same.

The F-test for Analysis of Variance (ANOVA)

ANOVA is an extension of the difference of means test above to the casae of more than two populations. The null hypothesis in this case is that all the sample means are equal – or more strictly, that all the treatment groups are drawn from the same population.

The simplest version of the ANOVA is the one-way ANOVA, where there are treatment groups (populations) with subjects (or repetitions, or replications) each, for a total of subjects. Each population corresponds to a different single factor (a treatment or a condition: for example, a type of medicine, or a Star-Bellied Sneetch vs. a Plain-Bellied Sneetch vs. a Grinch). Two- or three- way ANOVAs correspond to varying two or three different factors combinatorially. For example, we could do a two-way ANOVA of Sneetch math performance by considering both the belly type and the gender of the Sneetchs.

Figure 7: Table for a Two-way ANOVA of Sneetch math performance

We will only discuss one-way ANOVA in this article, since that covers all the relevant ideas about calculating significance.

For a one-way ANOVA, we have the population means and variances . We can also calculate the overall mean , over the entire aggregate population.

The between-groups mean sum of squares, which is an estimate of the between-groups variance, is given by

(3)

is sometimes designated It is a measure of how the population means vary with respect to the grand mean.

The within-group mean sum of squares is an estimate of the within-group variance:

(4)

is sometimes designated . It is a measure of the “average population variance”.

Figure 8: Within-group and between-group variance

If the null hypothesis is true, then

is distributed according to the F distribution wiht
degrees of freedom.

Figure 9: p-value for the one-tailed F-test

That is, under the null hypothesis, the within-group and between-group variances should be about equal:
. If , then some of the treatment groups overlap other groups substantially, so practically speaking, one might as well accept the null hypothesis. Hence, a one-sided F test is good enough. As with the t-test, research papers usually give the value of the F statistic, the degrees of freedom, and the p-value: “
”. In this example, the test statistic value is 6.6, and it was evaluated against the F distribution with (2, 864) degrees of freedom, which means that
. The p-value is 0.0014.

Assumptions

Like the t-test, ANOVA assumes that the data is normally distributed with equal variances. According to Box [Box53], ANOVA is fairly robust to unequal variances when the population sizes are about the same, but you might want to check anyway. If all the populations are the same size (all the are the same), the easiest way to check for equality of variances is an F-test of the statistic
with degrees of freedom[Sac84]. In other cases, you can use Bartlett’s Test [Wika] or Levene’s Test [Wikb]. Bartlett’s test uses a test statistic that is distributed as the distribution, and Levene’s test uses one that is distributed as the F distribution. Levene’s test does not assume normally distributed data.

If the data are not normally distributed, or have unequal variance, often they can be transformed to a form that is closer to obeying the assumptions of ANOVA. The following table of transformations is based on [Sac84, p. 517], and other sources [Hor].

Figure 10: Table of Transformations

Jim Deacon from the University of Edinburgh lists some suggestions as well [Dea]. He also reminds us that running ANOVA on the transformed data will identify significant differences in the transformed data. This is not the same as saying there are significant differences in the original data!

Once the Null Hypothesis is Rejected

If you are able to reject the ANOVA null hypothesis, you will usually want to know which population means are significantly different from the rest. Often, in fact, you are primarily interested in which population had the highest mean. For example, if you are comparing the efficacy of a new medicine A against existing medicines B and C, you are probably not too concerned about whether B and C perform significantly differently from each other, only about whether A is significantly better than both.

If all you care about is whether the highest mean is significantly higher than the others, you can simply test where the statistic

falls on the Student-t distribution with degrees of freedom. Here, is the within-group variance, as calculated in Equation 4, and are the highest and second highest population means, is the total number of samples (
), and is the number of treatment groups.

This test is usually written

where
is the (two-sided) critical value for significance level and is the number of degrees of freedom to use. This quantity is called the least significant difference (LSD) between the highest and second highest means, and the test is usually called the LSD test.

If you want to test all the population differences for significance, (or test the highest value against all of the others explicitly) then you need to take some care with the LSD test. Remember that a significance level of means that with probability you will make a false positive error. To test all possible population differences is = ( choose ) comparisons, or comparisons, if you sort all the means in descending order and compare adjacent ones. Testing the highest mean against all the lower values is also comparisons. This means you have a
probability of making a false positive error. So if you want the overall significance level to be , each individual comparison should use a stricter significance threshold
.

A preferred way to compare multiple means for significance (once the ANOVA null hypothesis has been rejected) is to use a multiple range test [Dea] or Tukey’s method [oST06], rather than the LSD test. Tukey’s method tests all pairwise comparison simultaneously, and the multiple range test starts with the broadest range (the highest and the lowest means), and works its way in until significance is lost.

Conclusion

We’ve skimmed over many complications in this discussion. Hopefully, though, what we have gone over is enough to demystify much of the statistical discussion in research papers. Perhaps, it will demystify the output of standard ANOVA and t-test packages for you, as well.

Chong-ho Yu’s site [hY] gives a brief discussion of some of the issues that I’ve skimmed over. It also lists a few common non-parametric tests. These are tests that do not make assumptions about how the data is distributed, and so they may be more appropriate for data that is very non-normal, or for discrete data. They tend to have less power than parametric tests (that is, they have a lower true positive rate); so if the data is at all normal-like, parametric tests are preferred.

Significance tests are used in other applications beyond testing the difference in means or variances. They are used for testing whether events follow an expected distribution, for testing if there is a correlation between two variables, and for evaluating the coefficients of a regression analysis. We hope to cover some of these applications in future installments of this series.

Bibliography

Box53

G.E.P. Box, Non-normality and tests on variances, Biometrika 40 (1953), no. 3/4, 318-335.

Dea

Jim Deacon, A multiple range test for comparing means in an analysis of variance, http://www.biology.ed.ac.uk/research/groups/jdeacon/statistics/tress7.html.

FPP07

David Freedman, Robert Pisani, and Roger Purves, Statistics, 4th ed., W. W. Norton & Company, New York, 2007.

Hor

Rich Horsley, Transformations, http://www.ndsu.nodak.edu/ndsu/horsley/Transfrm.pdf, Class notes, Plant Sciences 724, North Dakota State University.

hY

Chong ho Yu, Parametric tests, http://www.creative-wisdom.com/teaching/WBI/parametric_test.shtml.

oST06

National Institute of Standards and Technology, Tukey’s method, NIST/SEMATECH e-Handbook of Statistical Methods, 2006, http://itl.nist.gov/div898/handbook/prc/section4/prc471.htm.

Sac84

Lothar Sachs, Applied statistics: A handbook of techniques, 2nd ed., Springer-Verlag, New York, 1984.

Wika

Wikipedia, Bartlett’s test, http://en.wikipedia.org/wiki/Bartlett's_test.

Wikb

—–, Levene’s test, http://en.wikipedia.org/wiki/Levene's_test.

Wikc

—–, Welch’s t test, http://en.wikipedia.org/wiki/Welch's_t_test.

Footnotes

…¹

Remember from the last installment that when you are estimating the mean of a distribution with unknown mean and unknown variance , the 95% confidence interval around your estimate is
. Intuitively speaking, Student’s distribution is what you get if you calculate confidence intervals using the estimated variance instead of the true but unknown variance . The distribution is stretched out compared to the normal distribution to reflect this increased uncertainty.

… positives²

In his textbook Statistics, Freedman tells an anecdote about a study that was published in the Journal of the AMA, claiming to demonstrate that cholesterol causes heart attacks. The treatment group that took a cholesterol reducing drug had “significantly fewer” heart attacks than the control group (
). A closer reading revealed that the researchers used a one-tailed test, which is equivalent to assuming that the treatment group was going to have fewer heart attacks. What if the drug had increased the risk of heart attack? The proper two-tailed significance of their results would have been
, which is higher than JAMA‘s strict significance threshold of 0.05. [FPP07, p. 550]

… tail³

The area to the right of with degrees of freedom is equal to the area to the left of , with degrees of freedom.

Related posts:

1. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
2. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
3. Living in A Lognormal World

As always, a pdf version of this article is available as well.

Does too much salt cause high blood pressure, or doesn’t it? That debate has raged for decades, with a slew of studies finding “yes” and a slew of others finding “no.” Two new studies out today in the journal Hypertension tip the scales in favor of reducing sodium – particularly for those 1 in 4 Americans who have high blood pressure. One study found that reducing salt intake from 9,700 milligrams a day to 6,500 milligrams decreased blood pressure significantly in blacks, Asians, and whites who had untreated mild hypertension. Another study found that switching to a lower-salt diet helped lower blood pressure in folks with treatment-resistant hypertension.
- “10 salt shockers that could make hypertension worse,” U.S. News & World Report [Kot09]

“Great!” you think. “Who needs to spend money on high-blood pressure meds? I can just cut down my salt!” Well, maybe so, maybe not. To come to that conclusion, you need more information than you were given in that paragraph. What was the “significant” decrease in blood pressure? What was the “before” and the “after”? Does “significant” mean important, or useful? And why has there been so much controversy over this?

Let’s discuss the important points with an example.

Suppose that we wanted to test for a difference in intelligence between two groups, say Star-Bellied Sneetches and Plain-Bellied Sneetches². We take a group of 50 Star-Bellies and a group of 40 Plain-Bellies, and give them both a series of tests designed to measure their mathematical, linguistic, and problem-solving abilities. After evaluating the data, we conclude that there is “a significant difference in mathematical performance (t(88) = 2.499, p = 0.014) between the two groups”. The mean mathematics score of the Star-Bellies is 78, with a standard deviation of 7, and the mean mathematics score of the Plain-Bellies is 74, with a standard deviation of 8, for a difference of 4 points³.

Should we interpret this result to mean that Star-Bellied Sneetches are better than Plain-Bellied ones at math? It depends.

How Hypothesis Tests Work

The Sneetch example above and the blood-pressure study cited earlier are both examples of hypothesis tests. In hypothesis testing, researchers set their proposed hypothesis (that there is an effect or a relationship) against the null hypothesis that there is no effect or relationship. In this article, we consider proposed relationships of the form

The mean value of X measured for group A is different from the mean value of X measured for group B.

In this case, the null hypothesis is

The mean value of X is the same for groups A and B, and any difference observed in the data is only by observational chance.

In fact, we are actually testing the stricter null hypothesis:

The distribution of X is the same for groups A and B, and any difference observed is only by observational chance.

A and B are sometimes called treatment groups; this terminology comes from the original applications of hypothesis testing procedures, in agriculture and medicine. In the blood pressure study above, the treatment is daily salt intake. One group ingests about 9,700 milligrams of sodium a day, the other group about 6,500 milligrams a day. The question of interest is: does the difference in sodium intake make a difference in the average blood pressure of the two groups? The null hypothesis is “No.”

Significance

We call an observed difference significant – meaning that a difference as large as we observed is probably not by chance – if the the value is “high enough.” In the Sneetch example, is the significance level of the result. To interpret the p-value, suppose the null hypothesis is true: there is truly no difference between Star-Bellied math scores and Plain-Bellied math scores. If this is so, then there is only a 0.014 (1.4%) chance that the difference in the average scores of the two groups will be 4 points or larger. In other words, if the null hypothesis is true, and we administer this same test to different groups of 50 Star-Bellies and 40 Plain-Bellies a hundred times, then the difference in scores will be 4 points or more only about once or twice.

We interpret the fact that we have seen a difference that should be rare to be evidence that the null hypothesis isn’t true. So we reject the null hypothesis and say that there is a “significant difference” in the performance of the two groups. Alternatively, we could say that Star-Bellied Sneetches performed “significantly better” than Plain-Bellied Sneetches on the math test.

Effect Size

Four points (or about a 5% difference) is the effect size of the comparison. The effect size represents what might be called the “practical significance” of the result. In general, the larger the effect size, the better. In this example, Star-Bellies might truly outperform Plain-Bellies by about four points on average, but if we were to examine the relationship between math scores and real-life math performance (say, how well college-attending Sneetches do in their math and science courses), we might discover that it takes a test score difference of ten points or more to reliably predict which Sneetches will do better. In that case, a four point average difference would not be a practical difference.

Evaluating a Result

When evaluating a result, you should look both for its significance and its effect size. In practice, researchers usually consider a finding to be significant if
. This is actually a pretty large ; it means even if the null hypothesis is true, you would still observe a difference as large as the one that you observed about five times out of every one hundred trials. In fact, Sachs noted that
used to be the commonly used threshold for significance ([Sac84, p. 114]).

Sometimes results are reported using an asterisk convention: (*) means
, (**) means
, and (***) means
. Hopefully, the actual significance level is reported (it isn’t always), as well as the actual effect size (it isn’t always).

The effect size in medical studies is often reported in the popular press with statements like “those who abstained from coffee had triple the risk of contracting colon cancer compared to those who drank three or more cups a day.” Does that mean that all confirmed Lapsang Souchong drinkers and the uncaffeinated should run out and learn to embrace Starbucks? Well, no. First of all, ask yourself: what is the baseline risk of colon cancer? If abstaining from coffee triples the risk from 0.01% to 0.03%, well, it probably isn’t worth worrying about. On the other hand, if the risk triples from 5% to 15%, perhaps that is a reason to take up espressos. You should also see who were the subjects of the study, and how similar they are to you. Suppose the study was done on Caucasian males in the U.S., ages 55-65, with no family history of colon cancer. If you are a young white American male, it’s possible that this study says something about your future health. If you are female or non-Caucasian or not living in the U.S., the finding may or may not be relevant to you. It depends on the mechanism that drives the relationship, and whether or not it applies to you as well as to the subjects of the study.

“Significant” is not the same as “Important”

With a large sample, even a small difference can be “statistically significant”… . This doesn’t necessarily make it important. Conversely, an important difference may not be statistically significant if the sample size is too small.
- Freedman, Pisani and Purves, Statistics [FPP07, p. 550]

The ability of a study to detect a significant difference depends almost entirely on its size. When a researcher designs a study, she has to decide how much risk of error – and what type of error – she is willing to tolerate.

How big a risk [of inventing a difference] between two indistinguishable treatments are we willing to put up with? This risk is known as the significance level . [Sac84, p. 214]

is the probability of rejecting a null hypothesis that should be accepted. This is a Type I error (a false positive). enters the design of the study as the threshold for p-values that the researcher will accept as significant.

How big a risk do we allow of missing a substantial difference between two treatments? … This risk is called . [Sac84, p. 214]

is the probability of accepting a null hypothesis that should have been rejected. This is a Type II error (a false negative). The quantity is known as the power of the test: the probability that the test will correctly reject the null hypothesis when the alternative hypothesis is true.

How small a difference should still be recognized as significant? This difference is called . [Sac84, p. 214]

is the minimum effect size that we are willing to consider “practically significant.”

It is important to consider all three of , , and when determining an appropriate sample size for a trial. The power of a test and the significance of a result both increase as the sample size increases. So if is not specified, any difference can appear significant, with a large enough , even if the difference is really by chance.

The Central Limit Theorem

To see why the above statement is true, we need a few more facts about estimating the mean. Suppose we have a random variable that is normally (or nearly normally) distributed, with a true mean and (unknown) variance . You want to estimate by drawing samples; the sample mean gives you an estimate of . According to the Central Limit Theorem, if you were to repeat this experiment over and over again, you would see that the estimated has a normal distribution, with mean and variance
. So is a good estimate of , one that improves with a larger sample size .

Another fact about normal distributions is that a little over 95% of the probability mass is within standard deviations of the mean. So, for a single experiment, we can reason that the true mean is in the interval
with 95% probability⁴.

Figure 1: Confidence bounds on the estimate of for different values of

So, as gets larger, we zoom in on ⁵.

Now, back to the problem of checking for the difference of means. We’ll take samples from population and from population . Let’s assume for now that the variances are equal.

Figure 2: Confidence bounds overlap; means may not be truly different

With 95% probability,
, and
. If
is small compared to
, then the two confidence intervals overlap substantially, and we cannot reject the null hypothesis that
.

If, on the other hand,
is wide compared to
:

Figure 3: Confidence bounds don’t overlap; means are significantly different

then the confidence intervals are well separated, and we can reject the null hypothesis.

So , the minimum significant distance – the “resolution” of the experiment – is about the distance when the two confidence intervals touch:
, if our desired significance level is 0.05.

Figure 4: Minimum significant distance for a given sample size

If is too large, the experiment may be unable to detect important differences because the confidence intervals overlap too soon. This means that the sample size was too small (the test didn’t have enough power), and the experiment should be repeated with a larger test population.

If is too small, then the experiment will potentially detect statistically significant differences that are, for all practical intents and purposes, meaningless. To go back to the Sneetch example, if the math exam has one hundred questions, then an effect size of two points would correspond to one group answering two additional questions correctly, on average. Practically speaking, that’s probably not a very big difference. But if we made the experiment big enough, about 250 Sneetches in each group, it would be a statistically significant difference, to the 0.05 level. In theory, we could even make a difference of less than one point statistically significant! That is why knowing the effect size of a significant result is important.

“Significant” is not the same as “True”

The power and significance level of a test play similar roles to the sensitivity and specificity of a diagnostic test. You’ll remember from Part 1 of this series⁶that sensitivity and specificity are properties of the test, not how the test performs in a given population. To know the practical accuracy of a screening test, you must know the underlying prevalence of the condition that it is screening for. If it is crucial that the screening not miss any positive cases, then the test will be designed to be highly sensitive, possibly at the cost of specificity. In that case, the test will tend to have a high false positive rate if the condition is relatively rare. And yet, this same screening test will have a lower overall false positive rate when used in a population where the condition is more prevalent.

The same is true for hypothesis tests. The probability that a statistically significant result is actually true depends on the underlying probability that results “of that type” tend to be true in the domain of study. It also depends on whether the researcher was trying to minimize the chance of a false positive error, or a false negative error.

You should also be careful interpreting the results of exploratory work, where the researchers have run a series of several different studies, but only highlight the “significant” ones. Running twenty experiments and having one of them return a significant result to the level is actually not significant at all.

John Ioannides discusses these points (and a few others) in his 2005 essay “Why Most Published Research Findings are False”[Ioa05]. The essay made a few waves at the time of its publication, and it is still available online. We recommend that you read it, along with the 2007 followup article by Moonesinghe, et.al [MKJ07]. Now that you’ve read the first two installments of the Statistics to English translation, both essays should be a breeze!

Some Points to Remember

• “Significant” is a statistical statement that an observed relationship is unlikely to be by chance. It is not an necessarily a statement about the magnitude or the importance (or the truth!) of the relationship.
• Knowing the effect size of a significant result will help you decide if the relationship is “practically significant.”
• With a large enough sample size, any difference in means can appear significant, even when it is by chance.

You now have a general idea what a “statistically significant result” is. The next installment will go into a little more technical detail of how significance is calculated. You should read that installment if you want to decipher statements in research papers like “
” — or if you are simply curious.

Bibliography

FPP07

David Freedman, Robert Pisani, and Roger Purves, Statistics, 4th ed., W. W. Norton & Company, New York, 2007.

Ioa05

John P. A. Ioannidis, Why most published research findings are false, PLoS Med 2 (2005), no. 8, e124, Available as http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124.

Kot09

Deborah Kotz, 10 salt shockers that could make hypertension worse, U.S. News & World Report (2009), Online as http://health.usnews.com/articles/health/heart/2009/07/20/10-salt-shockers-that-could-make-hypertension-worse.html.

MKJ07

Ramal Moonesinghe, Muin J Khoury, and A. Cecile J. W Janssens, Most published research findings are false — but a little replication goes a long way, PLoS Med 4 (2007), no. 2, e28, Available as http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0040028.

Sac84

Lothar Sachs, Applied statistics: A handbook of techniques, 2nd ed., Springer-Verlag, New York, 1984.

SS99

Murray R. Spiegel and Larry J. Stephens, Schaum’s outline of statistics, 4th ed., McGraw-Hill, New York, 1999.

Footnotes

… series¹

http://www.win-vector.com/blog/category/statistics-to-english-translation/

… Sneetches²

“The Sneetchs,” from The Sneetches and Other Stories by Dr. Seuss.
http://www.youtube.com/watch?v=Ln3V0HgW4eM
and http://www.youtube.com/watch?v=s0LgMpfLD1Y

… points³

This example is based on Exercise 10.17 in [SS99]; the original exercise did not, unfortunately, involve Sneetches.

… probability⁴

The correct way to state this is that for a given (unknown) , the estimate falls in the interval
just over 95% of the time. This gets awkward to reason about. Luckily, symmetry arguments let us center the appropriate confidence interval around instead.

…⁵

Of course, we don’t actually know , so we don’t know exactly how fast we zoom in. That doesn’t affect our argument, though, since only changes

… series⁶

http://www.win-vector.com/blog/2009/11/i-dont-think-that-means-what-you-think-it-means-statistics-to-english-translation-part-1-accuracy-measures/

Related posts:

1. Statistics to English Translation, Part 2b: Calculating Significance
2. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
3. Why you can not to use statistics to dispute magic

The “Statistics to English Translation” series is a new set of articles that we will be posting from time to time, as an attempt to bridge the language gaps. Our goal is to increase statistical literacy: we hope that you will find it easier to read and understand the statistical results in research papers, even if you can’t replicate the analyses. We also hope that you will be able to read popular media accounts of statistical and scientific results more critically, and to recognize common misunderstandings when they occur.

The first installment discusses some different accuracy measures that are commonly used in various research communities, and how they are related to each other. There is also a more legible PDF version of the article here.

The Basics

In informal language and in popular press articles, “accuracy” is often discussed as if it were a one-dimensional property of a diagnostic test or a classifier.

In general though, a single number is not enough. A test or classifier should detect what’s interesting, and ignore what’s not. How well it accomplishes these two tasks is related to the two kinds of mistakes that a test or classifier can make: false negatives, and false positives.

For a classification task, positive means that an instance is labeled as belonging to the class of interest: we may want to automatically gather all news articles about Microsoft out of a news feed, or identify fraudulent credit card transactions. For a screening test, positive means that the test detects whatever it was designed to look for: an HIV test detects the presence of human immunodeficiency virus, for example, while an allergy test detects the presence of an allergic reaction. A negative is obviously the opposite of a positive.

A false positive is concluding that something is positive when it is not. False positives are sometimes called Type I errors. A false negative is concluding that something is negative when it is not. False negatives are sometimes called Type II errors. The terms “Type I error” and “Type II error” are not terribly mnemonic, but they are commonly used, and therefore worth knowing.

For binary classification or binary test procedures, the False Positive Rate, , is the fraction of negative instances that are erroneously misclassified as positive.

(1)

Likewise, the False Negative Rate, , is the fraction of positive instances that are erroneously misclassified as negative.

(2)

The True Positive Rate, , is the fraction of positive instances that are correctly identified as such. It follows from the Definition 2 above that
.

The True Negative Rate, , is the fraction of negative instances that are correctly identified as such. It follows from the Definition 1 above that
.

Sensitivity and Specificity

The terms sensitivity and specificity generally refer to diagnostic or screening procedures, such as an HIV or allergy tests. The sensitivity of a test is its true positive rate; the specificity is its true negative rate, although it can be more intuitive to think of specificity as the complement of the false positive rate: Specificity =
.

The Wikipedia entry on Sensitivity and Specificity [Wik] uses a nice example to illustrate the difference: think of a drug-sniffing dog as a screening test for illicit drugs. If the dog’s nose is highly sensitive to the smell of drugs, then it will detect all the hidden packets of drugs; if it is less sensitive, then it will fail to detect some of the packets. At the same time, the dog should react specifically to drugs, and not, say, jambalaya or doggie biscuits. If the dog is highly specific in its reactions, it will only react to drugs; if it is less specific, then it will react to the occasional care package of yummy home cooking from Mom.

Screening tests may trade off specificity for sensitivity (and vice-versa). To go back to our drug-sniffing example, we might treat every suitcase and bag that comes through the airport as if it contained drugs; this procedure is perfectly sensitive (it will detect every packet of drugs, for sure), but not specific at all. Or, we might assume that no one is carrying drugs. This is perfectly specific (we will never make a false accusation), but not sensitive at all.

A more realistic example, inspired by a discussion of mandatory AIDS testing by Joshua Rosenau [Ros06], is the use of the ELISA screening test to detect HIV-infected blood donations. The ELISA test is designed to be very sensitive: it detects 99.7% of the cases of HIV-infection, which gives a false negative rate of
. On the other hand, it is not very specific: it has a 1.9% false positive rate¹. If you assume that the incidence of HIV-positive individuals in the general population is about 448 out of every 100,000 people [Hig08], then a positive test result is correct only about 19% of the time: one case of true infection out of every five positives. This error rate may be appropriate for screening blood donations, since it is better to discard four perfectly good pints of blood, “just in case”, than to allow a pint of HIV-infected blood into the blood bank. But it is not appropriate to assume that all five of those poor blood donors are HIV-positive, without followup tests to increase the specificity of the screening procedure.

Sensitivity, Specificity, and Prevalence

The example above brings up an important point. Sensitivity and specificity are properties of the test itself, not how the test performs in a given population. The absolute accuracy (as the term is commonly understood) of a screening test will change, depending on the prevalence of the condition that the test is screening for.

Let’s imagine the ELISA test described above as an HIV-screening daemon, who uses two coins to generate uncertainty. When the daemon is shown a pint of infected blood, she flips an unfair quarter. If the quarter comes up heads (which it does 3 times out of every 1000 flips), then she lies and says the blood is uninfected, otherwise she tells the truth. When the daemon is shown a pint of uninfected blood, she flips a silver dollar that comes up heads about 2 times out of every 100 flips. If the silver dollar comes up heads, she lies and says the blood is infected, or else she tells the truth. The quarter and the silver dollar encode ELISA’s sensitivity and specificity, respectively.

Figure 1: The ELISA daemon screening an uninfected pint of blood

Suppose ELISA looks at the blood of 1000 people a day, drawn from the general population. We can expect that about 5 of them are truly infected. That means that ELISA flips her silver dollar 995 times; it will come up heads about 20 times. That’s about twenty false positives a day. She’ll flip the quarter about 5 times, and with high probability, won’t ever see a head. That’s near zero false negatives a day. In total, ELISA will read positive for about 25 pints of blood every day, and she will be wrong for 80% of those cases.

But suppose ELISA looks at the blood of 1000 people from a high-risk population, where one out of four people are infected. Then ELISA flips her silver dollar about 750 times, and it will come up heads about 15 times: 15 false positives. She’ll also flip the quarter 250 times; the coin just might come up heads one time. Let’s say it does. Then ELISA will read positive for 249+15 = 264 pints of blood, and she’ll be wrong for only about 6% of those cases – plus that case of infection that she missed.

Same test, same sensitivity and specificity, but different overall accuracy. The percentage of positives that are actually true positives in a given population is called the positive predictive value ( ) of the test within that population; it is the probability for that population that a positive test result correctly predicts a positive instance.

(3)

where is the probability of a positive instance, or in other words the prevalence of the condition in the population. is the probability of a negative instance, and of course
.²

Likelihood Ratios

Likelihood ratios are another measure of diagnostic test accuracy. The positive likelihood ratio is the true positive rate over the false positive rate:
. For our example ELISA test, the positive likelihood ratio is 0.997/0.019 = 52.47. The negative likelihood ratio is the false negative rate over the true negative rate,
. For our ELISA example, the negative likelihood ratio is 0.003/.981 = 0.003058.

Likelihood ratios are a property of the screening test, independent of the prevalence of the condition in the population. If you know the odds of infection for the population of interest,
, then you can calculate the posterior odds of infection for someone who has tested positive:

and the posterior odds of infection for someone who has tested negative:

It’s been argued that likelihood ratios make it easier for non-statistically-minded practitioners to interpret the results of a test than sensitivity and specificity do [JGS94]. It’s also been argued the other way [PSBtR05]. Which framework makes more sense depends on if you prefer thinking in odds or probabilities. In either case you should be leery of “guidelines” of the sort: “ indicates large and often conclusive increase in the likelihood of the disease.” There is certainly a large increase in the posterior likelihood of infection if , but as the ELISA coin-flipping example should have made clear, this posterior likelihood can still be quite small, if the disease is sufficiently rare.

I occasionally see something called the diagnostic odds ratio. It was developed as “a single indicator of test performance,” and I’ve seen it described as “the odds of the true positive rate divided by the odds of the false positive rate” [HC07]. I could give you the actual formula here, but frankly – it makes no sense. The whole point of having two measures for accuracy is that one is not enough, and if you absolutely must have one number, you are better off using something like the measure that we describe in the next section.

Precision and Recall

Precision and recall are similar (but not identical) to sensitivity and specificity. The measures are popular in the information retrieval and machine learning communities.

Recall is the same as sensitivity, or the true positive rate: the number of true examples correctly classified as such. Precision is defined as the fraction of instances classified as positive that really are positive:

   precision

This is not the same as specificity; it is the same as the positive predictive value, and is a joint property of the classifier and the population that it was evaluated on.

Information retrieval research concerns itself with efficient discovery of relevant documents from document collections, and that domain motivates the definitions of precision and recall. A library patron queries the library catalog for books on a given topic; the catalog’s search engine should return all of the books relevant to her query, and only those books. Recall is a measure of how well the search delivers “all of the relevant books”, and precision is a measure of how well it delivers “only the relevant books”. If recall is poor, then the patron will miss finding many relevant books; if precision is poor, then she will be inundated with a bunch of book suggestions that have nothing to do with her search.

As with diagnostic procedures, classifiers may trade precision for recall, and vice-versa. Suppose our library patron is looking for novels about vampires. She could request all novels with the word “vampire” in the title. This search would have almost perfect precision, since presumably a novel with the word “vampire” in the title is going to be about vampires. It would not have perfect recall, since many novels about vampires – like Dracula, or the books from the Twilight series – don’t announce themselves quite that blatantly. Now suppose she is only interested in novels from Anne Rice’s Vampire Chronicles series. She could request all novels authored by Anne Rice. This search would have perfect recall, but not perfect precision, since Ms. Rice did in fact write several novels that are not about vampires.

These examples show that neither high precision nor high recall guarantee a useful classifier. It is the tension between achieving high precision and high recall that leads to good classifiers.

As we discussed above, the primary difference between precision and specificity is that precision is a property of the algorithm and the population. One could argue that precision is a more appropriate measure than specificity for many classification and machine learning tasks, especially those related to text or natural language. The fundamental assumption, after all, is that such algorithms are trained on data that is representative of the population that the classifier will be deployed in.

If you insist: Single Score Measures

There is another measure called , the harmonic mean of precision and recall:

is near one when both precision and recall are high, and near zero when they are both low. It is a convenient single score to characterize overall accuracy, especially for comparing the performance of different classifiers.

Using to compare classifiers assumes that precision and recall are equally important for the application. If one criterion is more important than the other, then one can also use the weighted geometric mean:

precision   recall   precision + recall

describes how much more important recall is than precision: use if recall is twice as important as precision, if precision is twice as important as recall.

It is still better to have separate target goals for precision and recall that a candidate classifier must meet. Still, and are found in the literature, so they are presented here.

ROC Curves

Not all diagnostic tests or classifiers return a simple “yes-or-no” answer. In fact, most probably don’t. Generally, a classification or diagnostic procedure will return a score along a continuum; ideally, the positive instances score towards one end of the scale, and the negative examples towards the other end. It is up to the scientist or the analyst to set a threshold on that score that separates what is considered a positive result from what is considered a negative result. The Receiver Operating Characteristic Curve, or ROC Curve, is a tool that helps set the best threshold.

Figure 2: Plot of score distributions for positive and negative instances (Class 10 is positive)

Suppose we are trying to classify a set of instances into one of two classes, positive and negative³. We’ve gathered a test set of representative samples, and we’ve developed a scoring procedure to try to separate them. Positives tend to score on the high end of the scale, negatives toward the low end. We want to pick a threshold value.

Figure 2 shows what happens when score the test set. We can see that the scores of the positive instances (class 10) are in a cluster centered just above 7, and the scores of the negatives (class 0) are in a cluster centered near 5. Still, there is an interval where the two clusters overlap substantially. If we pick a threshold to the right of that interval (say, ), almost everything that scores greater than will be truly positive (high precision/specificity), but we miss a lot of positives, too (low recall/sensitivity). If we pick a threshold to the left of that interval (say ), we will catch almost all the positives (high recall/sensitivity), but we will also pick up a lot of negatives (low specificity/precision). So we want the threshold to be somewhere in the overlap interval, but where?

Figure 3: ROC Curve corresponding to Figure 2. Selected thresholds are marked on the curve.

ROC curves plot the false positive rate on the x-axis and the true positive rate on the y-axis, as we vary the threshold. The point corresponds to rejecting everything; the point corresponds to accepting everything. The ideal point is : accept all positive instances and reject all negative instances. The line corresponds to random guessing: that is, a procedure that assigns each instance a score uniformly drawn from (in this example) the interval without even checking if the instance is positive or negative.

The ROC curve represents the tradeoff between true positives and false positives that we make as we increase the threshold from accepting everything to rejecting everything. Figure 3 gives the ROC curve for our example, with a few example thresholds marked on the curve.

The area between the ROC curve and the line can be considered a measure of accuracy; the smaller that area, the more the scoring procedure is like random guessing. The larger the area, the better separated the two classes are. We can use the curve to help us decide how to set a threshold that will give us the most acceptable tradeoff between true positives and false positives. For this example, we would probably want to select a threshold somewhere between and .

In Conclusion

Some points to remember:

• Classifier and diagnostic test performance are not one-dimensional.
• Different fields use different (but related) measures of accuracy.
• Classifier and diagnostic test performance depend on the relative cost of Type I and Type II errors, as well as on the proportion of positive and negative instances in the population of interest.

Bibliography

HC07

Childrens Mercy Hospitals and Clinics, Stats: Meta-analysis for a diagnostic test, http://www.childrens-mercy.org/stats/model/diagnostic.asp, 2007.

Hig08

Liz Highleyman, CDC updates estimates of HIV prevalence in the United States, http://www.hivandhepatitis.com/recent/2008/100708_a.html, 2008.

JGS94

R. Jaeschke, GH Guyatt, and DL Sackett, Users’ guides to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients? The evidence-based medicine working group, JAMA 271 (1994), no. 6, 703-707.

Org04

World Health Organization, HIV assays: Operational characteristics (phase 1); Report 15 antigen/antibody ELISAs, http://whqlibdoc.who.int/publications/2004/9241592370.pdf, 2004.

PSBtR05

MA Puhan, J. Steurer, LM Bachmann, and G. ter Riet, “A randomized trial of ways to describe test accuracy: the effect on physicians’ post-test probability estimates”, Annals of Internal Medicine 143 (2005), no. 3, 184-âÄì189.

Ros06

Joshua Rosenau, AIDS testing, http://scienceblogs.com/tfk/2006/08/aids_testing.php, 2006.

Wik

Wikipedia, Sensitivity and specificity, http://en.wikipedia.org/wiki/Sensitivity_and_specificity).

Appendix: Glossary of Accuracy Terms

Basic Terms

Accuracy

Type I error

False Positive: to conclude something is a positive instance when it is not.

Type II error

False Negative: to conclude something is a negative instance when it is not.

False Positive Rate ( )

The fraction of negative instances that are erroneously misclassified as positive.

False Negative Rate ( )

The fraction of positive instances that are erroneously misclassified as negative.

True Positive Rate ( )

The fraction of positive instances that are correctly identified as such.
.

True Negative Rate ( )

The fraction of negative instances that are correctly identified as such.
.

Prevalence

The proportion of positive instances in the population, or the probability that someone drawn from the population at random in a positive instance. is the probability of drawing a negative instance from the population at random. The odds of a positive is the ratio of to .

Accuracy Terms

Terms to describe the accuracy of diagnostic tests are conventionally given in terms of sensitivity and specificity. They have been rephrased here in terms of true positive rate, false positive rate, etc., for clarity.

Single score measure of accuracy. The harmonic mean of precision and recall.

is near 1 for high accuracy, near 0 for low accuracy. Also see Precision, Recall.

Single score measure of accuracy when recall is twice as important as precision. Also see , Precision, Recall.

Single score measure of accuracy when precision is twice as important as recall. Also see , Precision, Recall.

Likelihood-ratio (negative)

. In diagnostic screening, used for calculating the posterior odds of a true positive for a subject who has tested positive:

Likelihood-ratio (positive)

. In diagnostic screening, used for calculating the posterior odds of a positive for a subject who has tested negative:

Positive Predictive Value

Probability (with respect to a specific assumed prevalence rate) that a positive result from a diagnostic or screening procedure is a true positive. Same as precision.

Also see Precision

Precision

In information retrieval, the fraction of returned documents that are actually relevant to the query. In classification, the fraction of all instances classified as class A that are truly in class A. The same as Positive Predictive Value.

   precision

Also see Positive Predictive Value

Recall

In information retrieval, the fraction of relevant documents that are returned by the query. In classification, the fraction of all true instances of class A that are classified into class A. The same as sensitivity.

   recall

Also see Sensitivity

ROC Curve

Plot of true positive rate versus false positive rate for a diagnostic test or binary classifier, as the decision threshold is varied.

Sensitivity

The true positive rate of a diagnostic or screening procedure. Also see Recall.

Specificity

The true negative rate
(or the complement of the false positive rate) of a diagnostic or screening procedure.

Footnotes

… rate¹

The ELISA sensitivity and specificity numbers are from WHO’s report on the operational characteristics of HIV Assays [Org04, p. 18], using the lower bounds of the confidence interval. They are slightly different from the numbers Rosenau uses

….²

The definition of is conventionally given in terms of sensitivity and specificity (similarly for the likelihood ratios discussed in the following section). The definitions in this article are given in terms of false positive rate, etc., since that is clearer for people reading outside their discipline.

… negative³

We are using a classifier in our example, but a diagnostic test would work the same way.

Related posts:

1. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
2. Statistics to English Translation, Part 2b: Calculating Significance
3. Living in A Lognormal World

This is a repost of an older paper- but a few people have pointed out they were put off by the incredibly uninformative title of the original post “New Paper.”

Related posts:

1. YAYGDA (Yet Another Yahoo Google Deal Article)
2. Should your mom use Google search?
3. New Paper