The important criterion for a graph is not simply how fast we can see a result; rather it is whether through the use of the graph we can see something that would have been harder to see otherwise or that could not have been seen at all.

– William Cleveland, The Elements of Graphing Data, Chapter 2

In this article, I will discuss some graphs that I find extremely useful in my day-to-day work as a data scientist. While all of them are helpful (to me) for statistical visualization during the analysis process, not all of them will necessarily be useful for presentation of final results, especially to non-technical audiences.

I tend to follow Cleveland’s philosophy, quoted above; these graphs show me — and hopefully you — aspects of data and models that I might not otherwise see. Some of them, however, are non-standard, and tend to require explanation. My purpose here is to share with our readers some ideas for graphical analysis that are either useful to you directly, or will give you some ideas of your own.

The graphs are all produced in R, using the ggplot2 package. While ggplot2 has a fairly high learning curve, it is the most flexible of the R graphing packages that I have encountered, and I’ve been able to quickly create rich graphics more easily than I would be able to with the R base graphics, or with other graphics packages.

Let’s start with some exploratory analysis. We will use the AdultUCI dataset that is included in the arules package.

library(arules)
data("AdultUCI")
dframe = AdultUCI[, c("education", "hours-per-week")]
colnames(dframe) = c("education", "hours_per_week")
         # get rid of the annoying minus signs in the column names

We want to compare the distribution of work-week length to education, using a box-and-whisker plot that is overlaid on a jittered scatterplot of the data.

library(ggplot2)
ggplot(dframe, aes(x=education, y=hours_per_week)) +
          geom_point(colour="lightblue", alpha=0.1, position="jitter") +
          geom_boxplot(outlier.size=0, alpha=0.2) + coord_flip()

The outlier.size=0 argument to geom_boxplot turns off the outlier plotting, and coord_flip switches the coordinate axes (because there are a lot of education levels).

The resulting graph:

Recall that the box of a box-and-whisker plot covers the central 50% of the data distribution; the line in the center marks the median. In this case, the work-week length concentrates so strongly at 40 hours (except for PhDs and those with professional degrees; they are doomed to work longer hours, typically) that most of the boxes appear one-sided; it’s easier to see what is happening with both the scatterplot and box-and-whisker superimposed, than it might be with the box-and-whisker alone. We can also see the relative concentration of the subjects along each educational level.

I’ve found that this superimposed graph is fairly easy to explain in a presentation (easier than a plain box-and-whisker, actually). The primary disadvantage that the scatterplot can get illegible for high volume datasets (this set has about 49 thousand rows). In this case, we have to return to the box-and-whisker plot alone.

Beyond exploratory analysis, we also want plots to evaluate the models that we fit. Win-Vector’s bread-and-butter recently has been logistic regression, so we will start with some visualizations for evaluating binary logistic regression models. We’ll use the heart disease dataset that Hastie, et.al, used in the Elements of Statistical Learning.

path = "http://www-stat.stanford.edu/~tibs/ElemStatLearn/datasets/SAheart.data"
saheart = read.table(path, sep=",",head=T,row.names=1)
fmla = "chd ~ sbp + tobacco + ldl + adiposity + famhist + typea + obesity"
model = glm(fmla, data=saheart, family=binomial(link="logit"),
             na.action=na.exclude)

We will make a data frame of chd (the true response, coronary heart disease), and the score from the model.

dframe = data.frame(chd=as.factor(saheart$chd),
                    prediction=predict(model, type="response"))

The standard diagnostic plot for logistic models is the ROC curve, which is fine, but personally, I don’t get a visceral feel for the model from looking at the ROC. Also, if you are interested in setting a score threshold on the model for classification purposes, the ROC adds an additional level of indirection, since it essentially integrates the score away. I used to plot the distribution of score (prediction) versus true response, like so:

ggplot(dframe, aes(x=prediction, colour=chd)) + geom_density()

This visualization tells me whether or not the model scores actually separate the response — in this case, the model identifies negative cases (no coronary heart disease) better than positive cases. The graph is hard to explain to a non-technical audience, and it has the disadvantage that both distributions are separately normalized to have unit area, so you get no sense of the relative proportion of positive and negative cases (in this case, about 35% of the population have coronary heart disease).

Here’s an alternate graph:

ggplot(dframe, aes(x=prediction, fill=chd)) +
               geom_histogram(position="identity", binwidth=0.05, alpha=0.5)

This is two semi-transparent histograms; the blue histogram for chd=1 is “in front” of the the red histogram. Because they are histograms, rather than density plots, we can more clearly see the relative distribution of positive to negative cases, and we have a better sense of how well (or not) the model separates the positive cases from the negative ones. Clearly, for most score thresholds, the model will have a fairly high false positive rate. I use this visualization all the time, but it is also fairly hard to explain, the transparency in particular.

We can also use our friend the box-and-whisker scatterplot.

ggplot(dframe, aes(x=chd, y=prediction)) +
               geom_point(position="jitter", alpha=0.2) +
               geom_boxplot(outlier.size=0, alpha=0.5)

The median score for the coronary heart disease cases is pulled away from the median score of the healthy subjects, but the central 50% of the two distributions still overlap.

Finally, let’s look at visualizations for linear regression. We’ll use the prostate cancer data from Elements of Statistical Learning.

fmla = "lpsa ~ lcavol + lweight + age + lbph + svi + lcp + gleason + pgg45"
model = lm(fmla, data=prostate.data)

We can just plot(model) for some diagnostic graphs:

par(mfrow = c(2, 2), oma = c(0, 0, 2, 0))
plot(model)

These diagnostics are useful to determine whether or not a linear model is suitable, and to identify outliers; but again, I personally don't get a visceral feel for the model. I prefer to directly plot prediction against true response:

dframe = data.frame(lpsa=prostate.data$lpsa, prediction=predict(model))

title = sprintf("Prostate Cancer model\n R-squared = %1.3f",
                summary(model)$r.squared)
ggplot(dframe, aes(x=lpsa, y=prediction)) +
               geom_point(alpha=0.2) +
               geom_line(aes(y=lpsa), colour="blue") +
               opts(title=title)

This graph gives you the same information as the Residuals vs. Fitted plot, and the Q-Q plot -- in particular, whether there is systematic over- or under-prediction in specific ranges of the data. It will expose outliers, and it is intuitive to explain when presenting your results. Furthermore, it can be used to evaluate other models that predict a continuous response, such as regression trees or polynomial fits.

Which graphs do you find especially useful for your day-to-day work?

Related posts:

1. The cranky guide to trying R packages
2. Good Graphs: Graphical Perception and Data Visualization
3. Learn Logistic Regression (and beyond)

Therefore Win-Vector LLC is granting the right to reformat and redistribute (with attribution and link) our blog‘s R content in the R-bloggers site and feeds.

We hope to see our R content shared through this network.

Related posts:

1. Kernel Methods and Support Vector Machines de-Mystified
2. Should your mom use Google search?

public class SomeBigProject1 {
	public static double logStirlingApproximation(final int n) {
		return n*(Math.log(n)-1) + 0.5*Math.log(2*Math.PI*n);
	}

	public static double logFactorial(final int n) {
		double r = 0.0;
		for(int i=n;i>1;--i) {
			r += Math.log(i);
		}
		return r;
	}

	public static void main(final String[] args) {
		int nbad = 0;
		for(int n=1000;n<10000;++n) {
			if(Math.abs(logFactorial(n)-logStirlingApproximation(n))>=1.0e-5) {
				++nbad;
			}
		}
		System.out.println("nbad: " + nbad);
	}
}

Imagine that this is some humongous project to use Stirling’s Approximation as a replacement for factorial. All the code up until main is great. But the unfortunate developer has hard-coded an acceptance test into main(). If they run their big project all they get out is:

nbad: 7334

The developer needs to re-code and re-build to diagnose the failure, tweak their acceptance criteria or add more measurements.

I strongly recommend a different work pattern. Instead of bringing criteria into the code, bring the data out:

public class SomeBigProject2 {
	public static void main(final String[] args) {
		System.out.println("n"
				+ "\t" + "logFactorial"
				+ "\t" + "logStirlingApproximation");
		for(int n=1000;n<10000;++n) {
			System.out.println(String.valueOf(n)
					+ "\t" + SomeBigProject1.logFactorial(n)
					+ "\t" + SomeBigProject1.logStirlingApproximation(n));
		}
	}
}

Capture this output in a file named “data.tsv” and both Microsoft Excel and R can open it. Naturally I prefer to use R (so that is what I will demonstrate). To read the results into R you start up an R and type in a command like the following:

 > d <- read.table('data.tsv',
        header=T,sep='\t',quote='',as.is=T,
        stringsAsFactors=F,comment.char='',allowEscapes=F)

Most of the arguments controlling the style of file R is to expected (what the field separator is, weather to expect escapes and quotes and so on). The settings I suggest here are the “ultra hardened” settings. If you make sure none of your fields have a tab or line-break in them when you print then it is guaranteed R can read the data (not matter what whacky symbols are in it). On the java side that usually means making sure any varying text fields are run through .replaceAll("\\s+"," ") “just in case.” At this point you can already look at your data with the summary() command:

 > summary(d)

       n         logFactorial   logStirlingApproximation
 Min.   :1000   Min.   : 5912   Min.   : 5912
 1st Qu.:3250   1st Qu.:23034   1st Qu.:23034
 Median :5500   Median :41870   Median :41870
 Mean   :5500   Mean   :42536   Mean   :42536
 3rd Qu.:7749   3rd Qu.:61653   3rd Qu.:61653
 Max.   :9999   Max.   :82100   Max.   :82100

This immediately hints that you should have been thinking in terms of relative error instead of absolute error (since insisting on high absolute accuracy on large results does not always make sense).

You also have access to standard statistical measures of agreement like correlation:

 > with(d,cor(logFactorial,logStirlingApproximation))

result: 1

You can see where your failures were:

 > library(ggplot2)
 > d$bad <- with(d,abs(logFactorial-logStirlingApproximation)>=1.0e-5)
 > ggplot(d) + geom_point(aes(x=n,y=bad))

Yields the graph:

You can see all your failures are in the initial interval. You can then drill in:

 > ggplot(d) + geom_point(aes(x=n,y=logFactorial-logStirlingApproximation))
                + scale_y_log10()

And here we see some things (that are in general true for Stirling’s approximation):

1. It is very accurate.
2. It is always an under estimate.
3. It gets better as n gets larger.

Essentially by poking around with graphs in R you can figure out the nature of your errors (telling you what to fix) and generate findings that tell you how to fix your criteria (perhaps your code is working- but your test wasn’t sensible). The “dump everything and then use R” technique is also particularly good for generating reports on code timings using either geom_histogram or geom_density.

For example, if we had data with a field runTimeMS then it is a simple one-liner to get plot like the following:

 > ggplot(t) + geom_density(aes(x=runTimeMS))

From this graph we can immediately see:

1. Most of our run-times are very fast.
2. We have a heavy right-tail (evidence of “contagion” or one slow-down causing others, like CPU or IO contention).
3. Data is truncated at 100MS (could be something “censoring” the measurement, an exception being thrown or an abort).
4. There is a spike at 30MS (something is true and slow for some subset of the data that isn’t present in the majority).

This is a lot more that would be seen in a mean-only or mean and standard deviation summary. We may even being seeings signs of two different bugs (the truncation and the spike).

In all cases the key is to dump a lot of data in machine readable form and then come back to to analyze. This is far more flexible than hoping to code in the right summaries and then further hoping the summaries don’t miss something important (or that you at least get a chance to notice if they do miss something). Being able to do exploratory statistics on dumps from your code (both results and timing) gives you incredible measurement, tuning and debugging powers. The scriptability of R means any later analysis is as easy as cut and paste.

Related posts:

1. Automatic Differentiation with Scala
2. R examine objects tutorial
3. Survive R

There is no final “right shape.” In fact even your data is never right. You will always be called to re-do your analysis (new variables, new data, corrections) so you should always understand you are on your “penultimate analysis” (always one more to come). This is why we insist on using general methods and scripted techniques, as these methods are much much easier to reliably reapply on new data than GUI/WYSWYG techniques.

In this article we will work a small example and call out some R tools that make reshaping your data much easier. The idea is to think in terms of “relational algebra” (like SQL) and transform your data towards your tools (and not to attempt to adapt your tools towards the data in an ad-hoc manner).Take a simple example where you are designing a new score called “score2” to predict or track an already known value called “score1.” The typical situation is score1 is a future outcome (such as the number of dollars profit on a transaction) and score2 is a prediction (such as the estimated profit before the transaction is attempted). Training data is usually assembled by performing a large number of transactions, recording what was known before the transaction and then aligning or joining this data with measured results when they become available. For this example we are not interested in the inputs driving the model (a rare situation, but we are trying to make our example as simple as possible) but only examining the quality of score2 (which is defined as how well it tracks score1).

All of this example will be in R, but the principles are chosen apply more generally. First let us enter some example data:



> d <- data.frame(id=c(1,2,3,1,2,3),score1=c(17,5,6,10,13,7),score2=c(13,10,5,13,10,5))

> d


This gives us our example data. Each row is numbered (1 through 6) has an id and both our scores:

  id score1 score2
1  1     17     13
2  2      5     10
3  3      6      5
4  1     10     13
5  2     13     10
6  3      7      5

We said our only task was to characterize how well score2 works at predicting score1 (or how good a substitute score2 is for score1). We could compute correlation, RMS error, info-gain or some such. But instead lets look at this graphically. We will prepare a graph showing how well score1 is represented by score2. For this we choose to place score1 on the y-axis (as it is the outcome) and score2 on the x-axis (as it is the driver).



> library(ggplot2)

> ggplot(d) + geom_point(aes(x=score2,y=score1))


Figure 1: score1 as a function of score2.

This does not look good. We would liked to have seen all of the dots falling on the line “y=x.” This plot shows score2 is not predicting score1 very well. Part of this is that we missed an important feature of the data (and because we missed it the feature becomes a problem): the ids repeat. First we re-order by id to make this more obvious.



> dsort <- d[order(d$id),]

> dsort


  id score1 score2
1  1     17     13
4  1     10     13
2  2      5     10
5  2     13     10
3  3      6      5
6  3      7      5

This is a very common situation. The original score is not completely a function of the known inputs. We are using “id” to abstract represent all of the inputs, two rows in our example have the same id if and only if all known inputs are exactly the same. The repeating ids are the same experiment run at different times (a good idea) and the variation in score1 could be the effect of an un-modeled input that changed value or something simple like a “noise term” (a random un-modeled effect). Notice that score2 is behaving as a function of id- all rows with the same id have the same value for score2. If score2 is a model then it has to be a function of the inputs (or more precisely if it is not a function of the inputs you have done something wrong). So any variation of score1 between rows with identical id is “unexplainable variation” (unexplainable from the point of view of currently tracked inputs). You should know about, characterize and report this variation (why it is good to have some repeated experiments). But this variation is not the model’s fault, if we want to know how good a job we did constructing the model (which we now see can be a slightly different question than how well the model works at prediction) we need to see how much of the explainable variation the model accounts for.

If we assume (as is traditional) the unexplained variation is from a “unbiased noise source” then we can lessen the impact of the noise source by replacing score1 with a value averaged over rows with the same id. This assumption is traditional because an unbiased noise source is present in many problems and assuming anything more requires more research into the problem domain. You would eventually fold such research into your model- so your goal is always have all effects or biases in your model and hope what is left over is unbiased. This is usually not strictly true, but not accounting for the unexplained variation at all is in many cases even worse than modeling the unexplained variation as being bias-free.

And now we find our data is the “wrong shape.” To replace score1 with the appropriate averages we need to do some significant data manipulation. We need to group sets of rows and add new columns. We could do this imperatively (write some loops and design some variables to track and manipulate state) or declaratively (find a path of operations from what you have to what you need through R’s data manipulation algebra). Even though the declarative method is more work the first time (you could often write the code in less time than it takes to read the manuals) it is the right way to go (as it is more versatile and powerful in the end).

Luckily we don’t have to use raw R. There are a number of remarkable packages (all by Hadley Wickham who is also the author of the ggplot2 package we use to prepare our figures) that really improve R’s ability to coherently manage data. The easiest (on us) way do fix up our data is to make the computer work hard and use the powerful melt/cast technique. These functions are found in the libraries reshape and plyr (which were automatically loaded with we loaded ggplot2 library).

melt is a bit abstract. What it does convert your data into a “narrow” format where rows are split into many rows each carrying just one result column of the original row. For example we can melt our data by id as follows:



> dmelt <- melt(d,id.vars=c('id'))

> dmelt


Which yields the following:

   id variable value
1   1   score1    17
2   1   score1    10
3   2   score1     5
4   2   score1    13
5   3   score1     6
6   3   score1     7
7   1   score2    13
8   1   score2    13
9   2   score2    10
10  2   score2    10
11  3   score2     5
12  3   score2     5

Each of the two facts (score1, score2) from our original row is split into its own row. The id column plus the new variable column are now considered to be keys. This format is not used directly but used because it is easy to express important data transformations in terms of it. For instance we wanted our table to have duplicate rows collected and score1 replaced by its average (to attempt to remove the unexplainable variation). That is now easy:



> dmean <- cast(dmelt,fun.aggregate=mean)

> dmean


  id score1 score2
1  1   13.5     13
2  2    9.0     10
3  3    6.5      5

We used cast() in its default mode, where it assumes all columns not equal to “value” are the keyset. It then collects all rows with identical keying and combines them back into wide rows using mean or average as the function to deal with duplicates. Notice score1 is now the desired average, and score2 is as before (as it was a function of the keys or inputs, so it is not affected by averaging). With this new smaller data set we can re-try our original graph:



> ggplot(dmean) + geom_point(aes(x=score2,y=score1))


Figure 2: mean(score1) as a function of score2.

This doesn’t look so bad. A lot of the error or variation in the first plot was unexplainable variation. score2 isn’t bad given its inputs. If you wanted to do better than score2 you would be advised to find more modeling inputs (versus trying more exotic modeling techniques).

Of course a client or user is not interested if score2 is “best possible.” They want to know if it is any good. To do this we should show them (either by graph or by quantitative summary statistics like we mentioned earlier) at least 3 things:

1. How well the model predicts overall (the very first graph we presented).
2. How much of the explainable variation the model predicts (the second graph).
3. The nature of the unexplained variation (which we will explore next).

We said earlier we are hoping the unexplained variation is noise (or if it is not noise it would be nice if it is a clue to new important modeling features). So the unexplained variation must not go unexamined. We will finish by showing how to characterize the unexplained variation. As before will will just make a graph, but the data preparation steps would be exactly the same if we were using a quantitive summary (like correlation, or any other). And, of course, our data is still not the right shape for this step. Luckily there is another tool ready to fix this: join().



> djoin <- join(dsort,dsort,'id')

> fixnames <- function(cn) {
n <- length(cn);
for(i in 2:((n+1)/2)) { cn[i] <- paste('a',cn[i],sep='') };
for(i in ((n+3)/2):n) { cn[i] <- paste('b',cn[i],sep='') };
cn
}

> colnames(djoin) <- fixnames(colnames(djoin))

> djoin


which produces:

   id ascore1 ascore2 bscore1 bscore2
1   1      17      13      17      13
2   1      17      13      10      13
3   1      10      13      17      13
4   1      10      13      10      13
5   2       5      10       5      10
6   2       5      10      13      10
7   2      13      10       5      10
8   2      13      10      13      10
9   3       6       5       6       5
10  3       6       5       7       5
11  3       7       5       6       5
12  3       7       5       7       5

All of the work was done by the single line “djoin <- join(dsort,dsort,'id')” the rest was just fixing the column names (as self-join is not the central use case of join). What we have now is a table that is exactly right for studying unexplained variation. For each id we have each row with the same id matched. This blows every id from having 2 rows in dsort to 4 rows in djoin. Notices this gives us every pair of score1 values seen for the same id (which will let us examine unexplained variation) and score2 is still constant over all rows with the same id (as it has always been throughout our analysis). With this table we can now plot how score1 varies for rows with the same id:



> ggplot(djoin) + geom_point(aes(x=ascore1,y=bscore1))


Figure 3: score1 as a function of score1.

And we can see, as we expected, the unexplained variation in score1 is about as large as the mismatch between score1 and score2 in our original plot. The important thing is this is all about score1 (score2 is now literally out of the picture). The analyst’s job would now be to try and tie bits of the unexplained variation to new inputs (that can be folded into a new score2) and/or characterize the noise term (so the customer knows how close they should expect repeated experiments to be).

What we are trying to encourage with the use of “big hammer tools” is an ability and willingness to look at and transform your data in meaningful steps. It often seems easier and more efficient to build one more piece of data tubing, but a lot of data tubes become an unmanageable collection of spaghetti code. The analyst should, in some sense, always be looking at data and not looking at coding details. For these sort of analyses we encourage analysts to think in terms of “data shape” and transforms. This discipline leaves more of the analysts energy and attention to think productively about the data and actual problem domain.

For the third plot showing the variation of score1 across different rows (but same ids) it may be appropriate to use a slightly more complicated join() procedure than we showed. The join shown produced rows of artificial agreement where both values of score1 came from the same row (thus had no chance of being different, so in some sense deserve no credit). This is also the only way any non-duplicated evaluations could make it to the plot. To eliminate these uninteresting agreements from the plot do the following:



> d$rowNumber <- 1:(dim(d)[1])

> djoin <- join(d,d,'id')

> colnames(djoin) <- fixnames(colnames(djoin))

> djoin <- djoin[djoin$arowNumber!=djoin$browNumber,]

> djoin


This gives us a table that shows only values of score1 from different rows:

   id ascore1 ascore2 arowNumber bscore1 bscore2 browNumber
2   1      17      13          1      10      13          4
4   2       5      10          2      13      10          5
6   3       6       5          3       7       5          6
7   1      10      13          4      17      13          1
9   2      13      10          5       5      10          2
11  3       7       5          6       6       5          3

And only plots points on the diagonal if “you have really earned them”:

So while the direct join() may not be the immediate perfect answer it is still a good intermediate to form as what you want is only simple data transformation away from it.

Related posts:

1. A Demonstration of Data Mining
2. Good Graphs: Graphical Perception and Data Visualization
3. The Data Enrichment Method

Suppose you want to try out a novel statistical technique? A good fraction of the time R is your best bet for a first trial. Take as an example general additive models (“Generalized Additive Models,” Trevor J Hastie, Robert Tibshirani, Statistical Science (1986) vol. 1 (3) pp. 297-318); R has a package named “gam” written by Trevor Hastie himself. But, like most R packages, trying the package from the supplied documentation brings in unfamiliar data and concerns. It is best to start small and quickly test if the package itself is suitable to your needs. We give a quick outline of how to learn such a package and quickly find out if the package is for you.

To start, install and activate the package in R:

install.packages('gam')
library(gam)
help(gam)

From the help we see gam fits in much the same way lm() and glm() do. So we need some data to try it out. I suggest not using the package example data or a real problem- use deliberately trivial data so you are diagnosing the new package (not diagnosing something else). I like to create a quick data frame as follows:

d <- data.frame(
   x1=rnorm(100),
   x2=sample(100),
   x3=0*(1:100),
   x4=sample(c('a','b','c'),size=100,replace=T),
   x5=as.factor(sample(c('d','e','f'),size=100,replace=T)),
   x6=sample(c(F,T),size=100,replace=T),
   x7=NA + 1:100,
 stringsAsFactors=F)

Right now d is a data frame with a lot of variable types we are likely to see in practice (we have left out ordered factors):

• Nicely distributed continuous values (x1)
• Integer values (x2)
• Stuck or constant varible (x3)
• String values (x4)
• Factor values (x5)
• Logical values (x6)
• A heap of missing values (x7)

We now augment our data frame with one more input (a duplicated variable) and an output (to be predicted):

d$x8 = d$x1
d$y = rnorm(100)
  + with(d,20*exp(x1) + x2
  + 7*as.integer(as.factor(x4))
  + 9*as.integer(x5) + 10*as.integer(x6))

This is our simple test data set. Our standard method of fitting a linear model for y (before trying the gam package) is either to (falsely) assume x1′s contribution is linear or to (amazingly) guess the exact transformation required is exp(x1).

In the first case the model looks like this (using the ggplot2 package):

(As a side note- statisticians usually cluck if you ask for a graph showing “truth” as a function of “prediction” (or y ~ f(x)). They say you should be looking at residuals (or (y-f(x) ~ f(x)) instead (and usually only supply functions that produce residual plots). But in their own publications, such as the paper we started with, when they want to make a point: they actually plot truth as a function of prediction.)

m1 <- lm(y ~ x1 + x2 + x3 + x4 + x5 + x6 + x8, data=d)
ggplot(d,aes(predict(m1),y)) + geom_point(shape=1)
   + geom_abline(slope=1) + opts(aspect.ratio=1)

And in the second case the model looks like this:

m2 <- lm(y ~ exp(x1) + x2 + x3 + x4 + x5 + x6 + x8, data=d)
ggplot(d,aes(predict(m2),y)) + geom_point(shape=1)
   + geom_abline(slope=1) + opts(aspect.ratio=1)

(In both cases we had to leave out x7, which was all NAs.) Notice how knowing the functional form of x1 moves us from a good fit to an extraordinary fit. We would like to automatically learn the form of x1′s contribution from data and get this better fit automatically. This is in fact the point of the gam package. To perform the gam fit we add the
smoothing symbol s() to each variable we want to try and learn the possibly non-linear shape of contribution of.

mG <- gam(y ~ s(x1) + s(x2) + x3 + x4 + x5 + x6 + s(x8), data=d)
ggplot(d,aes(predict(mG),y)) + geom_point(shape=1)
   + geom_abline(slope=1) + opts(aspect.ratio=1)

We could not add the s() symbol to any of the un-ordered factors, strings, logicals, the NAs or constant columns. Except for that the gam package is performing as robustly as the built in lm() package and produces a fit essentially as good as knowing the shape of x1′s contribution ahead of time:

That is the good part. But it just wouldn’t be an R-package without some bad parts.

The first problem is: at first glance the gam package plot() appears as if it is broken (or at least not a compatible extension of plot.lm or plot.glm, classes even though gam claims to extend both of those classes). Traditionally when you call plot() on a model it steps you through a bunch of arcane graphs that statisticians swear are more important than examining the fit directly. But plot(mG) seems to step through all of a different family of graphs without waiting for user input, and we are left only with the graph that presumably shows the shape adjustment of variable x8. Only if you anticipate plot() is different for gam than for other models (or dump the function code for plot.gam) do you learn you need to add the argument ask=T. plot(mG,ask=T) enters an interactive mode where you can see the inferred shape of each variable. That is, there is an argument that defaults to a value you don’t want (or as we say in industry: “you forgot to set the don’t lose flag.”). gam is actually a very high quality package, but these sort of “poison defaults” are something you have to be in the habit of looking out for in R.

The second problem is intrinsic to the method: we are not guaranteed that s(x1) or s(x8) either look much like exp(x1). It is only guaranteed that some linear of them does (as that was how they were used in the model). We can get direct access to the learned reshapings by calling predict() to ask for term contributions and see how the model is linear in the transformed coordinates (essentially with all coefficients 1).

pG = predict(mG,type='terms')
summary(lm(d$y ~ pG))

Call:
lm(formula = d$y ~ pG)
Residuals:
    Min      1Q  Median      3Q     Max
-3.5690 -0.7121 -0.0288  0.6924  5.3883
Coefficients: (1 not defined because of singularities)
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.125e+02  1.363e-01  824.92   <2e-16 ***
pGs(x1)     9.994e-01  6.486e-03  154.09   <2e-16 ***
pGs(x2)     1.002e+00  4.905e-03  204.19   <2e-16 ***
pGx3               NA         NA      NA       NA
pGx4        9.973e-01  2.548e-02   39.14   <2e-16 ***
pGx5        1.004e+00  1.926e-02   52.10   <2e-16 ***
pGx6        9.967e-01  2.796e-02   35.64   <2e-16 ***
pGs(x8)     1.057e+00  2.520e-02   41.96   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

We can examine the transforms (like s(x8) = f(x8)) by plotting:

ggplot(d) + geom_point(aes(x8,pG[,'s(x8)']))

Even though s(x8) has a funny shape it plus s(x1) are an excellent approximation of exp(x1) (with commensurate magnitudes):

> lm <- lm(exp(d$x1)~pG[,'s(x1)']+pG[,'s(x8)'])
> summary(lm)

Call:
lm(formula = exp(d$x1) ~ pG[, "s(x1)"] + pG[, "s(x8)"])
Residuals:
      Min        1Q    Median        3Q       Max
-0.118086 -0.014125  0.002899  0.023389  0.217152
Coefficients:
               Estimate Std. Error t value Pr(>|t|)
(Intercept)   1.2536112  0.0046410  270.12   <2e-16 ***
pG[, "s(x1)"] 0.0500079  0.0002126  235.25   <2e-16 ***
pG[, "s(x8)"] 0.0534411  0.0008349   64.01   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.04641 on 97 degrees of freedom
Multiple R-squared: 0.9987,	Adjusted R-squared: 0.9986
F-statistic: 3.588e+04 on 2 and 97 DF,  p-value: < 2.2e-16

The functions do have different variances (s(x1) is doing most of the work):

> var(pG[,'s(x1)'])
[1] 514.8578
> var(pG[,'s(x8)'])
[1] 33.37209

Yet the coefficients of mG seem to be gibberish (notice the 0 on s(x8)):

> mG$coefficients
(Intercept)       s(x1)       s(x2)          x3         x4b         x4c         x5e
  45.580747   22.749160    1.007585    0.000000    7.048177   13.346365    8.740460
        x5f      x6TRUE       s(x8)
   18.003981   10.102217    0.000000

So by poking around we have learned not to look at this slot of the returned model (and it is much cheaper to learn this through this cranky poking around on a trivial example than to learn it while trying to analyze real data or blundering through R’s overly operational documentation).

The third (and last) problem is one of attitude (and one of the barriers to learning statistics). There is not a lot of support for exporting the derived gam smoothers (the transforms on the input variables) from R. The original paper suggests that you should think of the non-parametric smoothers as graphs and user linear interpolation between your data points. You can do this by calling “predict(mG,type=’terms’)” as we did above. Or you can try to switch to parametric splines and then run into the same problem that the splines package is not really export friendly. Or you can ask around. The R community is generally quite bright and friendly- but every once in a while you get a whiff of statistics territorialism (or perhaps a defensiveness, where if you are correct but not fully general you fear you will look shallow). Sensible requirements, like wanting to export usable model parameters to another system, are considered naive. A favorite example of mine: in this help thread somebody who is asking how to configure and export explicit spline transforms to meet an external requirement to get their paper published is advised: “I think that the referee is being unreasonable here. There are many perfectly respectable ways of estimating GAMs for which no explicit expression for the estimated smooth terms is available (See Hastie and Tibshirani’s GAM book).” And then offered additional references to help educate the referee (instead of recognizing that an explicit sharable solution in hand could, in some cases, be more useful than a maximally general solution that can’t be communicated succinctly). It is like lecturing a drowning man on how important water is to fish.

In summary- never first try a new R package on real data. R packages are often realization of very deep concepts from the literature that bring in their own terminology, trade-offs and attitudes. You need time to absorb these things in isolation. Expect and forgive non-essential errors (many important and valuable packages have them). Approach new packages with a cranky inquisitiveness about the package, otherwise you may actually fall into a non-productive state of frustration.

Also, gam methods are amazing.

(note: we also recommend trying the mgcv package for gam modeling. It represents a different set of tradeoffs.)

Related posts:

1. My Favorite Graphs
2. R examine objects tutorial
3. Your Data is Never the Right Shape

Logistic regression was invented in the late 1930s by statisticians Ronald Fisher and Frank Yates. The definitive resource on this (and other generalized linear models) is Alan Agresti “Categorical Data Analysis” 2002, New York, Wiley-Interscience. Logistic regression is a “categorical” tool in that it is used to predict categories (fraud/not-fraud, good/bad …) instead of numeric scores (like standard regression). For example: consider the “car” data set from the UCI machine learning database ( data, description ). The data is taken from a consumer review of cars. Each car is summarized by 6 attributes ( price, maintenance costs, doors, storage size, seating and safety ); there is also a conclusion column that contains the final overall recommendation (unacceptable, acceptable, good, very good). The machine learning problem is to infer the reviewer’s relative importance or weight of each feature. This could be used to influence a cost constrained design of a future car. This dataset was originally used to demonstrate hierarchical and decision tree based expert systems. But logistic regression can quickly derive interesting results.

Let us perform a complete analysis together (at least in our imaginations if not with our actual computers). First download and install the excellent free analysts workbench called “R”. This software package is an implementation of John Chambers’ S language (a statistical language designed to allow for self-service statistics to relieve some of Chambers’ consulting responsibilities) and a near relative of the SPlus system. This system is powerful and has a number of very good references (our current favorite being Joseph Adler “R in a nutshell” 2009, O’Reilly Media). Using R we can build an interesting model in 2 lines of code.

After installing R start the program and copy the following line into the R command shell.

CarData <- read.table(url('http://archive.ics.uci.edu/ml/machine-learning-databases/car/car.data'),
    sep=',',col.names=c('buying','maintenance','doors','persons','lug_boot','safety','rating'))

This has downloaded the car data directly from the UCI database and added a header line so we can refer to variables by name. To see roll-ups or important summaries of our data we could just type in "summary(CarData)." But we will move on with the promised modeling. Now type in the following line:

logisticModel <- glm(rating!='unacc' ~ buying + maintenance + doors + persons + lug_boot +safety,
    family=binomial(link = "logit"),data=CarData)

We now have a complete logistic model. To examine this model we type "summary(logisticModel)". And see the following (rather intimidating) summary:

Coefficients:
                  Estimate Std. Error z value Pr(>|z|)
(Intercept)       -28.4255  1257.5255  -0.023    0.982
buyinglow           5.0481     0.5670   8.904  < 2e-16 ***
buyingmed           3.9218     0.4842   8.100 5.49e-16 ***
buyingvhigh        -2.0662     0.3747  -5.515 3.49e-08 ***
maintenancelow      3.4064     0.4692   7.261 3.86e-13 ***
maintenancemed      3.4064     0.4692   7.261 3.86e-13 ***
maintenancevhigh   -2.8254     0.4145  -6.816 9.36e-12 ***
doors3              1.8556     0.4042   4.591 4.41e-06 ***
doors4              2.4816     0.4278   5.800 6.62e-09 ***
doors5more          2.4816     0.4278   5.800 6.62e-09 ***
persons4           29.9652  1257.5256   0.024    0.981
personsmore        29.5843  1257.5255   0.024    0.981
lug_bootmed        -1.5172     0.3758  -4.037 5.40e-05 ***
lug_bootsmall      -4.4476     0.4750  -9.363  < 2e-16 ***
safetylow         -30.5045  1300.3428  -0.023    0.981
safetymed          -3.0044     0.3577  -8.400  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

We also saw the cryptic warning message "glm.fit: fitted probabilities numerically 0 or 1 occurred" which we will discuss later. Returning to our result we see a left column that is formed by concatenating variable names and variable values (values are called "levels" when they are strings). For example the label "buyinglow" is a combination of "buying" and "low" meaning a low purchase price. The next column (and the last one we will dig into) is the score associated with this combination of variable and level. The interpretation is that a care that has "buyinglow" is given a 5.0481 point score bonus. Whereas a car with "safetylow" is given a -30.5045 scoring penalty. In fact the complete prediction procedure for a new car is to look the levels specified for all 6 variables and add up the correct scores (plus the "(Intercept)" score of -28.4255 which is used as an initial score). Any value not found is assumed to be zero. This summed-up score is called the "link" and is essentially the model prediction. Positive link values are associated with acceptable cars and negative link values are associated with unacceptable cars. For example the first car in our data set is:

  buying maintenance doors persons lug_boot safety rating
   vhigh       vhigh     2       2    small    low  unacc

According to the columns: we see above our scoring procedure assigns a very bad score of -68 to this car- correctly predicting the "unacc" rating. We can examine the error rates of our model with the single line:

table(CarData$rating,predict(logisticModel,type='response')>=0.5)

While yields the result:

        FALSE TRUE
  acc      32  352
  good      0   69
  unacc  1166   44
  vgood     0   65

This diagram is called a "contingency table" and is a very powerful summary. The rows are labeled with the ratings assigned at training time (unacceptable, acceptable, good and very good). The columns FALSE and TRUE denote the model predicted the car was unacceptable or at least acceptable. From the row "unacc" we see that 1166 of the 1166+44 unacceptable cars were correctly predicted FALSE (or not at least acceptable). Also notice the only face negatives are the 32 FALSEs in the "acc" row- none of the good or very good cars were predicted to be unacceptable. We can also look at this graphically:

library(lattice)
densityplot(predict(logisticModel,type='link'),groups=CarData$rating!='unacc',auto.key=T)

Which yields the following graph:

This is an area density chart. Each car that was defined as being unacceptable adds a single blue circle to the bottom of the chart. Each car that was defined as being acceptable adds a single magenta circle to the bottom of the chart. The left-right position of each circle is the link score the model assigned to the circle. There are so many circles that they start to overlap into solid smudges. To help with this charting software adds the density curves above the circles. Density curves are a lot like histograms- the height of the curve indicates what fraction of the circles of the same color are under that region of the curve. So we can see most of the blue circles are in 3 clusters centered at -55, -30 and -5 while the magenta circles are largely clustered around +5. From a chart like this you can see that a decision procedure of saying a link score above zero is good and below zero is bad would be pretty accurate (most of the blue is to the left and most of the magenta is to the right). In fact this rule would be over 95% accurate (though accuracy is not a be-all measure, see: Accuracy Measures).

So far so good. We have built a model that accurately predicts conclusions based on inputs (so it could be used to generate new ratings) and furthermore our model has visible "Estimate" coefficients that report the relative strength of each feature (which we could use prescriptively in valuing trade-offs in designing a new car). Except we need to go back to the warning message: "glm.fit: fitted probabilities numerically 0 or 1 occurred." For a logistic regression the only way the model fitter can encounter "probabilities numerically 0 or 1" is if the link score was out of a reasonable range of zero (say +-20). Remember we saw link scores as low as -68. With a link score of -68 the probability of the car in question being acceptable is around 2*10^-16. This from a training set that only included 1728 items (so really can not be expected to see events much rarer than one in a thousand. We are deliberately confusing two different types of probabilities here- but it is a good way to think).

What is the cause of these extreme link scores? Extreme coefficient estimates. The +29.96 preference for "persons4" (cars that seat 4) is a huge vote that swamps out effects from purchase price and maintenance cost. The model has over fit and made some dangerous extreme assumptions. What causes this are variable and level combinations that have no falsification in the data set. For example: suppose only one car had the variable level "person4" and that car happened to be acceptable. The logistic modeling package could always raise the link score of that single car by putting a bit more weight on the "person4" estimate. Since this variable level shows up only in positive examples (and in this case only one example) there is absolutely no penalty for increasing the coefficient. Logistic regression models are built by an optimizer. And when an optimizer finds a situation with no penalty- it abuses the situation to no end. This is what the warning was reporting. All link numbers map to probabilities between zero and one; only ridiculously large link values map to probabilities near one (and only ridiculously small values map to probabilities near zero). The optimizer was caught trying to make some of the coefficients "run away" or "run to infinity." There are some additional diagnostic signs (such as the large coefficients, large standard errors and low significant levels), but there is no advice offered by the system in how to deal with this. The standard methods are to suppress problem variables and levels (or suppress data with the problem variables and levels present) from the model. But this is inefficient in that the only way we have of preventing a variable from appearing to be too useful is not to use it. These are exactly the variables we do not want to eliminate from the model, but they are unsafe to keep in the model (their presence can cause unreliable predictions on new data not seen during training).

What can we do to fix this? We need to ensure that running a coefficient to infinity is not without cost. One way to achieve this would be something like Laplace smoothing where we enter two made-up data items: one that has every level set on and is acceptable and one that has every level set on and is unacceptable. Unfortunately there is no easy way to do this from the user-layer in R. For example each datum can only have one value set for each categorical variable- so we can't define a datum that has all features on. Another way to fix this would be to directly penalize large coefficients (like Tychonoff regularization in linear algebra). Explicit regularization is a good idea and very much in the current style. Again, unfortunately, the R user layer does not expose a regularization control to the user. But one of the advantages of logistic regression is that it is relatively easy to implement (harder than Naive Bayes or standard regression in that it needs and optimizer, but easier than SVM in that the optimizer is fairly trivial).

The logistic regression optimizer works to find a model of probabilities p() that maximizes the sum:

Or in english: assign large probabilities to examples known to be positive and small probabilities to examples known to be negative. Now the logistic model assigns probabilities using a function of the form:

The beta is the model parameters and the x is the data associated with a given example. The dot product of beta and x is the link score we saw earlier. The rest of the function is called the sigmoid (also used by neural nets) and its inverse is called the "logit" which is where logistic regression gets its name. Now this particular function (and others have the so-called "canonical link") has the property that the gradient (the vector of derivative directions indicating the direction of most rapid score improvement) is particularly beautiful. The gradient vector is:

This quantity is a vector because it is a weighted sum over the data_i (which are all vectors of feature values and value/level indicators). As we expect the gradient to be zero at an optimal point we now have a set of equations we expect to be simultaneously satisfied at the optimal model parameters. In fact these equations are enough to essentially determine the model- find parameter values that satisfy all of this vector equation and you have found the model (this is usually done by the Newton–Raphson method or by Fisher Scoring). As an aside this is also the set of equations that the maximum entropy method must specify; which is why for probability problems maximum entropy and logistic regression models are essentially identical.

If we directly add a penalty for large coefficients to our original scoring function as below:

Beta is (again) our model weights (laid out in the same order as the per-datum variables) and epsilon (>=0) is the new user control for regularization. A small positive epsilon will cause regularization without great damage to model performance (for our example we used epsilon = 0.1). Now our optimal gradient equations (or set of conditions we must meet) become:

Or- instead of the model having to reproduce all know feature summaries (if a feature is on in 30% of the positive training data then it must be on for 30% of the modeled positive probabilities) we now have a slop term of 2 epsilon beta. To the extent a coefficient is large its matching summary is allowed slack (making it harder for the summary to drive the coefficient to infinity). This system of equations is as easy to solve as the original system (a slightly different update is used in the Newton-Raphson method) and we get a solution as below:

variable	kind		level	value
		Intercept		-2.071592578277024
buying		Categorical	high	-1.8456895489650629
buying		Categorical	low	2.024816396087424
buying		Categorical	med	1.257553912038549
buying		Categorical	vhigh	-3.508273337437926
doors		Categorical	2	-1.8414721595612646
doors		Categorical	3	-0.391932359146582
doors		Categorical	4	0.08090597021546056
doors		Categorical	5more	0.08090597021546052
lug_boot	Categorical	big	0.8368739895290729
lug_boot	Categorical	med	-0.2858686820997001
lug_boot	Categorical	small	-2.62259788570632
maintenance	Categorical	high	-1.274387434600252
maintenance	Categorical	low	1.3677275941857292
maintenance	Categorical	med	1.3677275941857292
maintenance	Categorical	vhigh	-3.5326603320481342
persons		Categorical	2	-8.360699258777752
persons		Categorical	4	3.2937880724973065
persons		Categorical	more	2.9953186080035197
safety		Categorical	high	4.160122946359636
safety		Categorical	low	-8.028169713081502
safety		Categorical	med	1.7964541884449603

This model has essentially identical performance and much smaller coefficients. From a performance point of view this is essentially the same model. What has changed is the model no longer is able to pick up a small bonus by "piling on" a coefficient. For example moving a link value to infinity moves the probabilities from 0.999 to 1.0 which in turn moves the data penalty (assuming the data point is positive) from log(0.999) to log(1.0) or from -0.001 to 0.0. The approximate 1/1000 score improvement is offset by a penalty proportional to the size of the coefficient- making the useless "adding of nines" no longer worth it. Or, as has become famous with large margin classifiers, it is important to improve what the model does on probability estimates near 0.5 not estimates already near 0 or 1.

As expected: the coefficients are significantly different than the standard logistic regression. For example the original model has 3 variables with extreme levels (the intercept, number of passengers and safety) while the new model sees only extreme values (but much smaller) for number of persons and safety (which are likely correlated). Also consider the difference between the buying levels low and very high in the original model (5 - -2 = 7) and in the new model (2 - -3.5 = 5.5) differ by 1.5 or around 3 of the reported standard deviations (indicating the significance summaries are not enough to certify the location of model parameters). It is not just that all of the coefficients have shifted, many of the differences are smaller (and others are not- changing the emphasis of the model). We don not want to overstate differences- we are not so much looking for something better than standard logistic regression as adding an automatic safety that saves us both the effort and loss of predictive power found in fixing models by suppressing unusually distributed (but useful) variables and levels.

An analyst is well served to have logistic regression (and the density plots plus contingency table summaries) as ready tools. These methods will take you quite far. And if you start hitting the limits of these tools you can, as we do, bring in custom tools that allow for explicit regularization yielding effective and reliable results.

Related posts:

1. The equivalence of logistic regression and maximum entropy models
2. The Simpler Derivation of Logistic Regression
3. Large Data Logistic Regression (with example Hadoop code)

The spirit of the list is to pick items such that: if you disagree with an item in this list then either you are wrong or you know something I would really like to hear about.

Encryption, disk images: TrueCrypt (open source: Linux, Windows, OSX)

TrueCrypt can create portable encrypted virtual disks (files that can be mounted as a disk on any operating system).

Encryption, files: GnuPG (open source: Linux, Windows, OSX)

GnuPG is the tool to use to encrypt files for email.

Presentation: Apple Keynote (commercial: OSX)

Keynote is not quite as friendly as Microsoft PowerPoint, but it quickly produces beautiful presentations.

Reference Library: Papers (commercial: OSX)

“iTunes for PDF.” Manage thousands of PDFs and references, annotate with meta-data, place papers into multiple project folders. An interesting runner-up is BibDesk (open source: OSX).

Spreadsheet: Microsoft Excel (commercial: Windows, OSX)

Open Office and Google Docs are getting better every day, but neither come close to Microsoft Excel in functionality and versatility of user interface. If you are on a platform that supports Excel, working regularly with spreadsheets and using something other than Excel: it really means that you do not value your time.

Statistics Software: R (open source: Linux, Windows, OSX)

R is rapidly becoming the platform of choice for statisticians and is (with the addition of lattice and ggplot2) the best way to produce graphs. R has fairly nasty programming language, but has so many statistical operations available that it can not be avoided.

Technical Documentation: LaTeX (open source: Linux, Windows, OSX)

It may seem antiquated but TeX/LaTex is still far more powerful than the “WSYWYG” pretenders. The separation of presentation from specification, automatic management of references, table of contents and being able
to include PDFs from external files (which get refreshed when you re-build the document) are all lifesavers.

Version Control: git (open source: Linux, Windows, OSX)

Just about the only version control system that: doesn’t damage the data you are trying to manage by adding dot-files into all of the directories, can routinely handle large files and can work productively without a network connection. Perforce is powerful central server commercial option (with the ability to have central policies, control and review).

I look forward to learning which of my choices are considered poor and what your must-haves are.

Related posts:

1. Exciting Technique #1: The “R” language.
2. “The Mythical Man Month” is still a good read
3. Microsoft Store Again

> l <- list(x=c(1,2,3))
> l$x
[1] 1 2 3

But suppose we wanted to automate this; that is pass in the name of the value we want in a variable. We are after all using a computer, so automating a step seems like a reasonable desire. R supplies a notation for this using the “[]” operator. But something slightly different comes out under the “[]” operator than under the “$” operator:

> varName <- 'x'
> l[varName]
$x
[1] 1 2 3

Notice that the printed outputs are slightly different (one echoes "$x" and one does not). Let's use the "class()" method to see what is actually being returned in each case.

> class(l$x)
[1] "numeric"
> class(l['x'])
[1] "list"

Completely different return types are returned (in one case a numeric vector in the other a general list, not interchangeable types).

At this point you may think it is time to turn in our "pro" label and call ourselves "newb" (Internet slang for "newbie" or "idiot"). But let's slow down for a bit. When two views of the same situation disagree (such as the difference in opinion between the authors of R and myself whether the "[]" and "$" operators should return the same type) you at most know that at least one of those views is wrong. You don't really know if one view is right or even if one view is right which one it is. I can, however, bring in some additional argument to try and show the design of R is in fact wrong. The additional argument is "The Principle of Least Astonishment." This principle roughly says that it is a mistake to introduce unnecessary differences in outcomes (which to the unprepared user are unpleasant surprises). There may be some deep (yet obscure) reasons the two operators prefer to return different results. But the fact you would have to find a way to document and explain these differences really should make one think that this situation is really a mis-design and the "explanation" is really an attempt at a work around. Or to put it more rudely: there may be an explanation, but there is no excuse.

For another example consider creating a 3 by 3 matrix:

> m <- matrix(c(1,2,3,1,1,1,0,0,1),nrow=3,ncol=3)
> m
     [,1] [,2] [,3]
[1,]    1    1    0
[2,]    2    1    0
[3,]    3    1    1

Now select the last two rows of the matrix.

> m[c(FALSE,TRUE,TRUE),]
     [,1] [,2] [,3]
[1,]    2    1    0
[2,]    3    1    1
>

Now (for the punchline) try to select just the middle row of the matrix.



> m[c(FALSE,TRUE,FALSE),]
[1] 2 1 0

Notice that once again (and without warning) the result is subtly different. I admit that it seems paranoid to worry about such small differences- but when you are debugging a system that should work these are exactly the killing mistakes you are looking for. In this case the problem is pretty bad. See what happens if you tried to ask for the dimension of each of these differing returns:

> dim(m[c(FALSE,TRUE,TRUE),])
[1] 2 3
> dim(m[c(FALSE,TRUE,FALSE),])
NULL

The first case works fine (reports 2 rows and 3 columns). The second case returns "NULL" (instead of 1 row and 3 columns). In R NULL is sometimes used as an error-value (instead of throwing an exception) and this value will poison any further conditions or calculations it is involved in. The main way to deal with the arbitrary introduction of such NULLs is the incredibly tedious uncertain defensive coding practices that we argue against in Postel’s Law: Not Sure Who To Be Angry With. Such code weakens both programs and programmers.

But what is going on in this example? Once again we use the "class()" method to inspect the subtly different results.

> class(m[c(FALSE,TRUE,TRUE),])
[1] "matrix"
> class(m[c(FALSE,TRUE,FALSE),])
[1] "numeric"

The result is disappointing. For a two-row select R returns a matrix (what we would expect). For a single-row select R does us the "favor" of converting the result into a vector. This is a disaster. A single row matrix is similar to a vector, but even R itself does not support the same set of operations and outcomes on vectors as it does on matrices (for example the failure of the "dim()" method). It is not safe to further calculate with these results (without by-hand converting the result back to a single row matrix which R can in fact represent). In my case this created crashing bugs deep in a long running analysis (and was hard to diagnose as the bug was in an "innocent operation" not in a "risky calculation").

All of this has to violate John Chambers' "Prime Directive" for data: "an obligation on all creators of software to program in such a way that the computations can be understood and trusted." Chambers' opinion being relevant as he is the author of the S language (of which R is an open source re-implementation). We continue to recommend R, but we also recommend being exceptionally careful when using it (which unfortunately adds time to projects).

Related posts:

1. R examine objects tutorial
2. CRU graph yet again (with R)
3. Survive R

If the re-constructed technique is close to what was originally done then so many bad moves were taken that you can’t learn much of anything from the original “result.” This points out some of the pratfalls of not performing hold-out tests, not examining the modeling diagnostics and not remembering that linear regression models fail to low-variance models (i.e. when they fail they do a good job predicting the mean and vastly under-estimate variance).

Our article not an article on global warming, but an article on analysis technique. Human driven global warming is either happening or not happening independent of any bad analysis. Finding the physical truth is a bigger harder job than eliminating some bad reports (the opposite of a bad report is not necessarily the truth). Bad analyses can have many different sources (mistakes, trying to jump ahead of your colleagues on something you believe is true, trying to fake something you believe is false or be figments of overly harsh critics) and we have not heard enough to make any accusations.

First: load the data (I re-formatted it at bit so R can read it: jonesmannrogfig2c.txt, data1400.dat_.txt ) , perform the principle components reduction and fit a first
model.

> library(lattice)
> d1400 <- read.table('data1400.dat.txt',sep='\t',header=FALSE)
> d1400r <- as.matrix(d1400[,2:23])
> pcomp <- prcomp(na.omit(d1400r))
> plot(pcomp)
> vars <- data.frame(cbind(Year=d1400[,1],d1400r %*% pcomp$rotation),row.names=d1400[,1])
> jones <- read.table('jonesmannrogfig2c.txt',sep='\t',header=TRUE)
> datUnion <- merge(vars,jones,all=TRUE)
> datUnion$avgTemp <- with(datUnion,(NH+CET+Central.Europe+Fennoscandia)/4.0)
> model <- lm(avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 ,dat=datUnion)
> summary(model)

Call:
lm(formula = avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5, data = datUnion)

Residuals:
       Min         1Q     Median         3Q        Max
-0.8811679 -0.2658117  0.0008174  0.2933058  1.0450044 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.0065252  0.5750696   0.011   0.9910
PC1         -0.0001683  0.0003912  -0.430   0.6679
PC2         -0.0003678  0.0010114  -0.364   0.7168
PC3          0.0003177  0.0014821   0.214   0.8307
PC4          0.0044084  0.0019351   2.278   0.0246 *
PC5          0.0188520  0.0205137   0.919   0.3601
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.4505 on 113 degrees of freedom
  (484 observations deleted due to missingness)
Multiple R-squared: 0.05223,	Adjusted R-squared: 0.01029
F-statistic: 1.245 on 5 and 113 DF,  p-value: 0.2927

We used only 5 principle components as modeling variables, because as is typical of principle component analysis- beyond the first few components the components become vanishingly small and unsuitable to use in modeling (see graph pcomp below).

However, this gave a model with far smaller R-squared than people are reporting, so lets add in a lot of components like everybody else does (bad!).

> model <- lm(avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 + PC6 + PC7 + PC8 + PC9 +PC10 +PC11 +PC12 + PC13 ,dat=datUnion)
> summary(model)

Call:
lm(formula = avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 + PC6 + PC7 +
    PC8 + PC9 + PC10 + PC11 + PC12 + PC13, data = datUnion)

Residuals:
     Min       1Q   Median       3Q      Max
-0.87249 -0.25951  0.03996  0.25055  0.99039 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  7.431e-01  1.424e+00   0.522   0.6028
PC1         -1.796e-04  3.665e-04  -0.490   0.6253
PC2         -4.179e-04  9.759e-04  -0.428   0.6694
PC3          3.306e-05  1.430e-03   0.023   0.9816
PC4          3.416e-03  1.803e-03   1.894   0.0609 .
PC5          4.032e-02  1.978e-02   2.039   0.0440 *
PC6         -3.260e-03  2.660e-02  -0.123   0.9027
PC7         -7.134e-02  3.620e-02  -1.971   0.0514 .
PC8         -1.339e-01  7.895e-02  -1.696   0.0928 .
PC9          7.577e-02  5.734e-02   1.321   0.1892
PC10         2.700e-01  5.878e-02   4.594 1.22e-05 ***
PC11         8.562e-02  6.741e-02   1.270   0.2068
PC12        -8.057e-02  1.053e-01  -0.765   0.4461
PC13        -4.099e-02  1.064e-01  -0.385   0.7008
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.4141 on 105 degrees of freedom
  (484 observations deleted due to missingness)
Multiple R-squared: 0.2558,	Adjusted R-squared: 0.1637
F-statistic: 2.777 on 13 and 105 DF,  p-value: 0.001961

This is a degenerate model that essentially didn’t fit (thought the significance on PC10 component fools the fitter, but PC10 can’t be usable- it is essentially noise). Graphically we can see the fit is not very useful (despite having a little bit of R-squared) by looking at the graph of the fit plotted in the region of fitting. Notice how the fit variance is much smaller than the true data variance even in the region of training data, this is typical of bad regression fits.

> dRange <- datUnion[datUnion$Year>=1856 & datUnion$Year<=1980,]
> xyplot(avgTemp + prediction ~Year,dat=dRange,type='l',auto.key=TRUE)

Now the statement they wanted to make is that the present looks nothing like the past. The past is only available through the fit model so what you would hope is that the model looks like the present and then the model itself separates the past and present. Instead as you see in the graphs above and below this fails two ways: the model looks nothing like the present and the model’s past looks a lot like the model’s present.

> datUnion$prediction <- predict(model,newdata=datUnion)
> xyplot(avgTemp + prediction ~Year,dat=datUnion,type=c('p','smooth'),auto.key=TRUE)

What we could do to falsely drive the conclusion (which itself may or may not be true, it just is not supported by this technique, model or data) is create the infamous graph where we switch from modeled data in the past to actual data in the present and then act surprised that the two did not line up (which they did at no step during the fitting). I don’t have the heart to unify the colors or remove the legend, but here is the graph below:

> datUnion$dinked <- datUnion$prediction
> datUnion$dinked[!is.na(datUnion$avg)] <- NA
> xyplot(avgTemp + dinked ~Year,dat=datUnion,type=c('p','smooth'),auto.key=TRUE)

The reason the blue points look different than the others is they came from the average temperature data instead of the model (where everything else came from). Switching the series is essentially assuming the conclusion that recent past looks very different than the far past.

Essentially this methodology was so poor it could not have illustrated or contradicted recent global warming. There are plenty of warning signs that the model fitting are problematic and the conclusion illustrated in the last graph can not actually be proved or disproved from this data (the proxy variables are too weak to be useful, that is not to say there are not other better proxy variables or modeling techniques). The problems of the presentation are, of course, not essential problems in detecting global warming (which likely is occurring and likely will be a drain on future quality of life) but problems found in a single bad analysis.

Related posts:

1. R annoyances
2. The cranky guide to trying R packages
3. Survive R

First let’s give R the commands to build a fake data set that has a variable y that turns out to be 3 times x (another variable) plus some noise:

> n <- 100
> x <- rnorm(n)
> y <- 3*x + 0.2*rnorm(n)
> d <- data.frame(x,y)

This data set (by design) has a nearly a linear relation between x and y. We can plot
the data as follows:

> library(ggplot2)
> ggplot(data=d) + geom_point(aes(x=x,y=y))

With data like this the most obvious statistical analysis is a linear regression. R can very quickly perform the linear regression and report the results.

> model <- lm(y~x,data=d)
> summary(model)

Call:
lm(formula = y ~ x, data = d)

Residuals:
     Min       1Q   Median       3Q      Max
-0.41071 -0.12762 -0.00651  0.10240  0.62772 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.02609    0.02102  -1.241    0.217
x            2.99150    0.02202 135.858   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.2102 on 98 degrees of freedom
Multiple R-squared: 0.9947,	Adjusted R-squared: 0.9947
F-statistic: 1.846e+04 on 1 and 98 DF,  p-value: < 2.2e-16

We can read the report and see that the estimated fit formula is: y = 2.99150*x – 0.02609 (which is very close to the true formula y = 3*x) . At this point the analysis is done (if the goal of the analysis is to just print the results). However, if we want to use the results in a calculation we need to get at the numbers shown in above printout. This printout contains a lot of information (such as the estimate fit coefficients, the standard errors, the t-values and the significances) that a statistician would want to see and want to use in further calculations. But it is unclear how to get at these numbers. For example: how do you get the “standard errors” (the numbers in the “Std. Error” column) from the returned model? Are we forced to cut and paste them from the printed report? What can you do?

The documentation nearly tells us what we need to know. help(lm) yields:

The functions summary and anova are used to obtain and print a summary and analysis of variance table of the results. The generic accessor functions coefficients, effects, fitted.values and residuals extract various useful features of the value returned by lm.

To a newer R user this may not be clear (as there are technical issues from both R and statistics quickly being run through). However, the experienced R user would immediately recognize from this help that what is returned form summary(model) is an object (not just a blob of text) and that looking at the class of the returned object (which turns out to be summary.lm) might tell them what they would need to know.

Typing:

>class(summary(model))
[1] "summary.lm"
> help(summary.lm)

Yields:

coefficients: a p x 4 matrix with columns for the estimated coefficient, its standard error, t-statistic and corresponding (two-sided) p-value. Aliased coefficients are omitted.

But if you are not very familiar with R you might miss that the summary function returns a useful object (instead of blob of text). Also you might only know to look at help(summary) which does not describe the location of the desired standard errors (but does have a reference to summary.lm, so if you are patient you might find it). We describe how to find the information you need by using R’s object inspection facilities. This is a “doing it the hard way” technique for when you do not understand the help system or you are using a package with less complete help documentation.

First (using the techniques described in the slides: Survive R) examine the model to see if the standard errors are there:

> names(model)
 [1] "coefficients"  "residuals"     "effects"       "rank"          "fitted.values" "assign"
 [7] "qr"            "df.residual"   "xlevels"       "call"          "terms"         "model"    

> model$coefficients
(Intercept)           x
-0.02609243  2.99150259

We found the coefficients, but did not find the standard errors. Now we know the standard errors are reported by summary(model), so they must be somewhere. Instead of performing a wild goose chase to find the standard errors let’s instead trace how the summary method works to find where it gets them. If we type print(summary) we don’t get any really useful information. This is because summary is a generic method and we need to know what type-qualified name the summary of a linear model is called.

> class(model)
[1] "lm"

So we see our model is of type lm so the summary(model) call would use a summary method called summary.lm (which as we saw is also the returned class of the summary(model) object). As we mentioned the solution is in help(summary.lm), but if the solution had not been there we could still make progress: we could dump the source of the summary.lm method:

> print(summary.lm)
function (object, correlation = FALSE, symbolic.cor = FALSE,
    ...)
{
    ....
    class(ans) <- "summary.lm"
    ans
}

We actually deleted the bulk of the print(summary.lm) result because the important thing to notice is that the method is huge and that it returns an object instead of a blob of text. The fact that the method summary.lm was huge means that it is likely calculating the things it reports (confirming that the standard errors are not part of the model object). The fact that an object is returned means that what we are looking for may sitting somewhere in the summary waiting for us. To find what we are looking for we convert the summary into a list (using the unclass() method) and look for something with the name or value we are looking for:

> unclass(summary(model))
$call
lm(formula = y ~ x, data = d)
...
$coefficients
               Estimate Std. Error    t value      Pr(>|t|)
(Intercept) -0.02609243 0.02102062  -1.241278  2.174662e-01
x            2.99150259 0.02201930 135.858209 2.095643e-113
...

And we have found it. The named slot summary(model)$coefficients is in fact a table that has what we are looking for in the second column. We can create a new list that will let us look up the standard errors by name (for the variable x and for the intercept):

> stdErrors <- as.list(summary(model)$coefficients[,2])

Now that we have the stdErrors in list form we can look up the numbers we wanted by name.

> stdErrors['x']
$x
[1] 0.0220193

> stdErrors['(Intercept)']
$`(Intercept)`
[1] 0.02102062

And we finally have the standard errors. But why did we want the standard errors? In this case I wanted the standard errors so I could plot the fit model and show the uncertainty of the model. As, is often the case, R already has a function that does all of this. Also (as is often the case) the R function that does this asks the right statistical question (instead of the obvious question) and can draw error bars that display the uncertainty of future predictions. The uncertainty in future prediction is in fact different than the uncertainty of the estimate (what was most obvious to calculate from the standard errors) and (after some reflection) is what I really wanted. Having these sort of distinctions already thought out is why we are using a statistics package like R instead of just coding everything up. These calculations are all trivial to implement- but remembering to perform the calculations that answer the right statistical questions can be difficult. The built in R solution of plotting the the fit model (black line) and the region of expected prediction uncertainty (blue lines) is as follows:

> pred <- predict.lm(model,interval='prediction')
> dfit <- data.frame(x,y,fit=pred[,1],lwr=pred[,2],upr=pred[,3])
> ggplot(data=dfit) + geom_point(aes(x=x,y=y)) +
    geom_line(aes(x,fit)) +
    geom_line(aes(x=x,y=lwr),color='blue') + geom_line(aes(x=x,y=upr),color='blue')

And we are done.

Related posts:

1. The cranky guide to trying R packages
2. Learn Logistic Regression (and beyond)
3. My Favorite Graphs