The spirit of the list is to pick items such that: if you disagree with an item in this list then either you are wrong or you know something I would really like to hear about.

Encryption, disk images: TrueCrypt (open source: Linux, Windows, OSX)

TrueCrypt can create portable encrypted virtual disks (files that can be mounted as a disk on any operating system).

Encryption, files: GnuPG (open source: Linux, Windows, OSX)

GnuPG is the tool to use to encrypt files for email.

Presentation: Apple Keynote (commercial: OSX)

Keynote is not quite as friendly as Microsoft PowerPoint, but it quickly produces beautiful presentations.

Reference Library: Papers (commercial: OSX)

“iTunes for PDF.” Manage thousands of PDFs and references, annotate with meta-data, place papers into multiple project folders. An interesting runner-up is BibDesk (open source: OSX).

Spreadsheet: Microsoft Excel (commercial: Windows, OSX)

Open Office and Google Docs are getting better every day, but neither come close to Microsoft Excel in functionality and versatility of user interface. If you are on a platform that supports Excel, working regularly with spreadsheets and using something other than Excel: it really means that you do not value your time.

Statistics Software: R (open source: Linux, Windows, OSX)

R is rapidly becoming the platform of choice for statisticians and is (with the addition of lattice and ggplot2) the best way to produce graphs. R has fairly nasty programming language, but has so many statistical operations available that it can not be avoided.

Technical Documentation: LaTeX (open source: Linux, Windows, OSX)

It may seem antiquated but TeX/LaTex is still far more powerful than the “WSYWYG” pretenders. The separation of presentation from specification, automatic management of references, table of contents and being able
to include PDFs from external files (which get refreshed when you re-build the document) are all lifesavers.

Version Control: git (open source: Linux, Windows, OSX)

Just about the only version control system that: doesn’t damage the data you are trying to manage by adding dot-files into all of the directories, can routinely handle large files and can work productively without a network connection. Perforce is powerful central server commercial option (with the ability to have central policies, control and review).

I look forward to learning which of my choices are considered poor and what your must-haves are.

Related posts:

1. Microsoft Store Again
2. Exciting Technique #1: The “R” language.
3. Public Service Article: JSTOR and other Useful Research Archives

> l <- list(x=c(1,2,3))
> l$x
[1] 1 2 3

But suppose we wanted to automate this; that is pass in the name of the value we want in a variable. We are after all using a computer, so automating a step seems like a reasonable desire. R supplies a notation for this using the “[]” operator. But something slightly different comes out under the “[]” operator than under the “$” operator:

> varName <- 'x'
> l[varName]
$x
[1] 1 2 3

Notice that the printed outputs are slightly different (one echoes "$x" and one does not). Let's use the "class()" method to see what is actually being returned in each case.

> class(l$x)
[1] "numeric"
> class(l['x'])
[1] "list"

Completely different return types are returned (in one case a numeric vector in the other a general list, not interchangeable types).

At this point you may think it is time to turn in our "pro" label and call ourselves "newb" (Internet slang for "newbie" or "idiot"). But let's slow down for a bit. When two views of the same situation disagree (such as the difference in opinion between the authors of R and myself whether the "[]" and "$" operators should return the same type) you at most know that at least one of those views is wrong. You don't really know if one view is right or even if one view is right which one it is. I can, however, bring in some additional argument to try and show the design of R is in fact wrong. The additional argument is "The Principle of Least Astonishment." This principle roughly says that it is a mistake to introduce unnecessary differences in outcomes (which to the unprepared user are unpleasant surprises). There may be some deep (yet obscure) reasons the two operators prefer to return different results. But the fact you would have to find a way to document and explain these differences really should make one think that this situation is really a mis-design and the "explanation" is really an attempt at a work around. Or to put it more rudely: there may be an explanation, but there is no excuse.

For another example consider creating a 3 by 3 matrix:

> m <- matrix(c(1,2,3,1,1,1,0,0,1),nrow=3,ncol=3)
> m
     [,1] [,2] [,3]
[1,]    1    1    0
[2,]    2    1    0
[3,]    3    1    1

Now select the last two rows of the matrix.

> m[c(FALSE,TRUE,TRUE),]
     [,1] [,2] [,3]
[1,]    2    1    0
[2,]    3    1    1
>

Now (for the punchline) try to select just the middle row of the matrix.



> m[c(FALSE,TRUE,FALSE),]
[1] 2 1 0

Notice that once again (and without warning) the result is subtly different. I admit that it seems paranoid to worry about such small differences- but when you are debugging a system that should work these are exactly the killing mistakes you are looking for. In this case the problem is pretty bad. See what happens if you tried to ask for the dimension of each of these differing returns:

> dim(m[c(FALSE,TRUE,TRUE),])
[1] 2 3
> dim(m[c(FALSE,TRUE,FALSE),])
NULL

The first case works fine (reports 2 rows and 3 columns). The second case returns "NULL" (instead of 1 row and 3 columns). In R NULL is sometimes used as an error-value (instead of throwing an exception) and this value will poison any further conditions or calculations it is involved in. The main way to deal with the arbitrary introduction of such NULLs is the incredibly tedious uncertain defensive coding practices that we argue against in Postel’s Law: Not Sure Who To Be Angry With. Such code weakens both programs and programmers.

But what is going on in this example? Once again we use the "class()" method to inspect the subtly different results.

> class(m[c(FALSE,TRUE,TRUE),])
[1] "matrix"
> class(m[c(FALSE,TRUE,FALSE),])
[1] "numeric"

The result is disappointing. For a two-row select R returns a matrix (what we would expect). For a single-row select R does us the "favor" of converting the result into a vector. This is a disaster. A single row matrix is similar to a vector, but even R itself does not support the same set of operations and outcomes on vectors as it does on matrices (for example the failure of the "dim()" method). It is not safe to further calculate with these results (without by-hand converting the result back to a single row matrix which R can in fact represent). In my case this created crashing bugs deep in a long running analysis (and was hard to diagnose as the bug was in an "innocent operation" not in a "risky calculation").

All of this has to violate John Chambers' "Prime Directive" for data: "an obligation on all creators of software to program in such a way that the computations can be understood and trusted." Chambers' opinion being relevant as he is the author of the S language (of which R is an open source re-implementation). We continue to recommend R, but we also recommend being exceptionally careful when using it (which unfortunately adds time to projects).

Related posts:

1. R examine objects tutorial
2. Relative returns: a banker versus trader paradox
3. Sorting Used in Anger

If the re-constructed technique is close to what was originally done then so many bad moves were taken that you can’t learn much of anything from the original “result.” This points out some of the pratfalls of not performing hold-out tests, not examining the modeling diagnostics and not remembering that linear regression models fail to low-variance models (i.e. when they fail they do a good job predicting the mean and vastly under-estimate variance).

Our article not an article on global warming, but an article on analysis technique. Human driven global warming is either happening or not happening independent of any bad analysis. Finding the physical truth is a bigger harder job than eliminating some bad reports (the opposite of a bad report is not necessarily the truth). Bad analyses can have many different sources (mistakes, trying to jump ahead of your colleagues on something you believe is true, trying to fake something you believe is false or be figments of overly harsh critics) and we have not heard enough to make any accusations.

First: load the data (I re-formatted it at bit so R can read it: jonesmannrogfig2c.txt, data1400.dat_.txt ) , perform the principle components reduction and fit a first
model.

> library(lattice)
> d1400 <- read.table('data1400.dat.txt',sep='\t',header=FALSE)
> d1400r <- as.matrix(d1400[,2:23])
> pcomp <- prcomp(na.omit(d1400r))
> plot(pcomp)
> vars <- data.frame(cbind(Year=d1400[,1],d1400r %*% pcomp$rotation),row.names=d1400[,1])
> jones <- read.table('jonesmannrogfig2c.txt',sep='\t',header=TRUE)
> datUnion <- merge(vars,jones,all=TRUE)
> datUnion$avgTemp <- with(datUnion,(NH+CET+Central.Europe+Fennoscandia)/4.0)
> model <- lm(avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 ,dat=datUnion)
> summary(model)

Call:
lm(formula = avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5, data = datUnion)

Residuals:
       Min         1Q     Median         3Q        Max
-0.8811679 -0.2658117  0.0008174  0.2933058  1.0450044 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.0065252  0.5750696   0.011   0.9910
PC1         -0.0001683  0.0003912  -0.430   0.6679
PC2         -0.0003678  0.0010114  -0.364   0.7168
PC3          0.0003177  0.0014821   0.214   0.8307
PC4          0.0044084  0.0019351   2.278   0.0246 *
PC5          0.0188520  0.0205137   0.919   0.3601
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.4505 on 113 degrees of freedom
  (484 observations deleted due to missingness)
Multiple R-squared: 0.05223,	Adjusted R-squared: 0.01029
F-statistic: 1.245 on 5 and 113 DF,  p-value: 0.2927

We used only 5 principle components as modeling variables, because as is typical of principle component analysis- beyond the first few components the components become vanishingly small and unsuitable to use in modeling (see graph pcomp below).

However, this gave a model with far smaller R-squared than people are reporting, so lets add in a lot of components like everybody else does (bad!).

> model <- lm(avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 + PC6 + PC7 + PC8 + PC9 +PC10 +PC11 +PC12 + PC13 ,dat=datUnion)
> summary(model)

Call:
lm(formula = avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 + PC6 + PC7 +
    PC8 + PC9 + PC10 + PC11 + PC12 + PC13, data = datUnion)

Residuals:
     Min       1Q   Median       3Q      Max
-0.87249 -0.25951  0.03996  0.25055  0.99039 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  7.431e-01  1.424e+00   0.522   0.6028
PC1         -1.796e-04  3.665e-04  -0.490   0.6253
PC2         -4.179e-04  9.759e-04  -0.428   0.6694
PC3          3.306e-05  1.430e-03   0.023   0.9816
PC4          3.416e-03  1.803e-03   1.894   0.0609 .
PC5          4.032e-02  1.978e-02   2.039   0.0440 *
PC6         -3.260e-03  2.660e-02  -0.123   0.9027
PC7         -7.134e-02  3.620e-02  -1.971   0.0514 .
PC8         -1.339e-01  7.895e-02  -1.696   0.0928 .
PC9          7.577e-02  5.734e-02   1.321   0.1892
PC10         2.700e-01  5.878e-02   4.594 1.22e-05 ***
PC11         8.562e-02  6.741e-02   1.270   0.2068
PC12        -8.057e-02  1.053e-01  -0.765   0.4461
PC13        -4.099e-02  1.064e-01  -0.385   0.7008
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.4141 on 105 degrees of freedom
  (484 observations deleted due to missingness)
Multiple R-squared: 0.2558,	Adjusted R-squared: 0.1637
F-statistic: 2.777 on 13 and 105 DF,  p-value: 0.001961

This is a degenerate model that essentially didn’t fit (thought the significance on PC10 component fools the fitter, but PC10 can’t be usable- it is essentially noise). Graphically we can see the fit is not very useful (despite having a little bit of R-squared) by looking at the graph of the fit plotted in the region of fitting. Notice how the fit variance is much smaller than the true data variance even in the region of training data, this is typical of bad regression fits.

> dRange <- datUnion[datUnion$Year>=1856 & datUnion$Year<=1980,]
> xyplot(avgTemp + prediction ~Year,dat=dRange,type='l',auto.key=TRUE)

Now the statement they wanted to make is that the present looks nothing like the past. The past is only available through the fit model so what you would hope is that the model looks like the present and then the model itself separates the past and present. Instead as you see in the graphs above and below this fails two ways: the model looks nothing like the present and the model’s past looks a lot like the model’s present.

> datUnion$prediction <- predict(model,newdata=datUnion)
> xyplot(avgTemp + prediction ~Year,dat=datUnion,type=c('p','smooth'),auto.key=TRUE)

What we could do to falsely drive the conclusion (which itself may or may not be true, it just is not supported by this technique, model or data) is create the infamous graph where we switch from modeled data in the past to actual data in the present and then act surprised that the two did not line up (which they did at no step during the fitting). I don’t have the heart to unify the colors or remove the legend, but here is the graph below:

> datUnion$dinked <- datUnion$prediction
> datUnion$dinked[!is.na(datUnion$avg)] <- NA
> xyplot(avgTemp + dinked ~Year,dat=datUnion,type=c('p','smooth'),auto.key=TRUE)

The reason the blue points look different than the others is they came from the average temperature data instead of the model (where everything else came from). Switching the series is essentially assuming the conclusion that recent past looks very different than the far past.

Essentially this methodology was so poor it could not have illustrated or contradicted recent global warming. There are plenty of warning signs that the model fitting are problematic and the conclusion illustrated in the last graph can not actually be proved or disproved from this data (the proxy variables are too weak to be useful, that is not to say there are not other better proxy variables or modeling techniques). The problems of the presentation are, of course, not essential problems in detecting global warming (which likely is occurring and likely will be a drain on future quality of life) but problems found in a single bad analysis.

Related posts:

1. R examine objects tutorial
2. Survive R
3. R annoyances

First let’s give R the commands to build a fake data set that has a variable y that turns out to be 3 times x (another variable) plus some noise:

> n <- 100
> x <- rnorm(n)
> y <- 3*x + 0.2*rnorm(n)
> d <- data.frame(x,y)

This data set (by design) has a nearly a linear relation between x and y. We can plot
the data as follows:

> library(ggplot2)
> ggplot(data=d) + geom_point(aes(x=x,y=y))

With data like this the most obvious statistical analysis is a linear regression. R can very quickly perform the linear regression and report the results.

> model <- lm(y~x,data=d)
> summary(model)

Call:
lm(formula = y ~ x, data = d)

Residuals:
     Min       1Q   Median       3Q      Max
-0.41071 -0.12762 -0.00651  0.10240  0.62772 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.02609    0.02102  -1.241    0.217
x            2.99150    0.02202 135.858   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.2102 on 98 degrees of freedom
Multiple R-squared: 0.9947,	Adjusted R-squared: 0.9947
F-statistic: 1.846e+04 on 1 and 98 DF,  p-value: < 2.2e-16

We can read the report and see that the estimated fit formula is: y = 2.99150*x – 0.02609 (which is very close to the true formula y = 3*x) . At this point the analysis is done (if the goal of the analysis is to just print the results). However, if we want to use the results in a calculation we need to get at the numbers shown in above printout. This printout contains a lot of information (such as the estimate fit coefficients, the standard errors, the t-values and the significances) that a statistician would want to see and want to use in further calculations. But it is unclear how to get at these numbers. For example: how do you get the “standard errors” (the numbers in the “Std. Error” column) from the returned model? Are we forced to cut and paste them from the printed report? What can you do?

The documentation nearly tells us what we need to know. help(lm) yields:

The functions summary and anova are used to obtain and print a summary and analysis of variance table of the results. The generic accessor functions coefficients, effects, fitted.values and residuals extract various useful features of the value returned by lm.

To a newer R user this may not be clear (as there are technical issues from both R and statistics quickly being run through). However, the experienced R user would immediately recognize from this help that what is returned form summary(model) is an object (not just a blob of text) and that looking at the class of the returned object (which turns out to be summary.lm) might tell them what they would need to know.

Typing:

>class(summary(model))
[1] "summary.lm"
> help(summary.lm)

Yields:

coefficients: a p x 4 matrix with columns for the estimated coefficient, its standard error, t-statistic and corresponding (two-sided) p-value. Aliased coefficients are omitted.

But if you are not very familiar with R you might miss that the summary function returns a useful object (instead of blob of text). Also you might only know to look at help(summary) which does not describe the location of the desired standard errors (but does have a reference to summary.lm, so if you are patient you might find it). We describe how to find the information you need by using R’s object inspection facilities. This is a “doing it the hard way” technique for when you do not understand the help system or you are using a package with less complete help documentation.

First (using the techniques described in the slides: Survive R) examine the model to see if the standard errors are there:

> names(model)
 [1] "coefficients"  "residuals"     "effects"       "rank"          "fitted.values" "assign"
 [7] "qr"            "df.residual"   "xlevels"       "call"          "terms"         "model"    

> model$coefficients
(Intercept)           x
-0.02609243  2.99150259

We found the coefficients, but did not find the standard errors. Now we know the standard errors are reported by summary(model), so they must be somewhere. Instead of performing a wild goose chase to find the standard errors let’s instead trace how the summary method works to find where it gets them. If we type print(summary) we don’t get any really useful information. This is because summary is a generic method and we need to know what type-qualified name the summary of a linear model is called.

> class(model)
[1] "lm"

So we see our model is of type lm so the summary(model) call would use a summary method called summary.lm (which as we saw is also the returned class of the summary(model) object). As we mentioned the solution is in help(summary.lm), but if the solution had not been there we could still make progress: we could dump the source of the summary.lm method:

> print(summary.lm)
function (object, correlation = FALSE, symbolic.cor = FALSE,
    ...)
{
    ....
    class(ans) <- "summary.lm"
    ans
}

We actually deleted the bulk of the print(summary.lm) result because the important thing to notice is that the method is huge and that it returns an object instead of a blob of text. The fact that the method summary.lm was huge means that it is likely calculating the things it reports (confirming that the standard errors are not part of the model object). The fact that an object is returned means that what we are looking for may sitting somewhere in the summary waiting for us. To find what we are looking for we convert the summary into a list (using the unclass() method) and look for something with the name or value we are looking for:

> unclass(summary(model))
$call
lm(formula = y ~ x, data = d)
...
$coefficients
               Estimate Std. Error    t value      Pr(>|t|)
(Intercept) -0.02609243 0.02102062  -1.241278  2.174662e-01
x            2.99150259 0.02201930 135.858209 2.095643e-113
...

And we have found it. The named slot summary(model)$coefficients is in fact a table that has what we are looking for in the second column. We can create a new list that will let us look up the standard errors by name (for the variable x and for the intercept):

> stdErrors <- as.list(summary(model)$coefficients[,2])

Now that we have the stdErrors in list form we can look up the numbers we wanted by name.

> stdErrors['x']
$x
[1] 0.0220193

> stdErrors['(Intercept)']
$`(Intercept)`
[1] 0.02102062

And we finally have the standard errors. But why did we want the standard errors? In this case I wanted the standard errors so I could plot the fit model and show the uncertainty of the model. As, is often the case, R already has a function that does all of this. Also (as is often the case) the R function that does this asks the right statistical question (instead of the obvious question) and can draw error bars that display the uncertainty of future predictions. The uncertainty in future prediction is in fact different than the uncertainty of the estimate (what was most obvious to calculate from the standard errors) and (after some reflection) is what I really wanted. Having these sort of distinctions already thought out is why we are using a statistics package like R instead of just coding everything up. These calculations are all trivial to implement- but remembering to perform the calculations that answer the right statistical questions can be difficult. The built in R solution of plotting the the fit model (black line) and the region of expected prediction uncertainty (blue lines) is as follows:

> pred <- predict.lm(model,interval='prediction')
> dfit <- data.frame(x,y,fit=pred[,1],lwr=pred[,2],upr=pred[,3])
> ggplot(data=dfit) + geom_point(aes(x=x,y=y)) +
    geom_line(aes(x,fit)) +
    geom_line(aes(x=x,y=lwr),color='blue') + geom_line(aes(x=x,y=upr),color='blue')

And we are done.

Related posts:

1. Survive R
2. CRU graph yet again (with R)
3. R annoyances

We at Win-Vector LLC appear to like R a bit more than some of our, perhaps wiser, colleagues ( see: Choose your weapon: Matlab, R or something else? and R and data ). While we do like R (see: Exciting Technique #1: The “R” language ) we also understand the need to defend oneself against the abuse regularly dished out by R. Here we will quickly share a few fighting techniques.

If you are not already using R the following will not mean much. If you are using R this may scratch a few itches.

• First: Write down everything- keep notes in a separate file.

  When you do figure out how to do something in R it will be concise, powerful and completely un-mnemonic and impossible to find again through the help system.

• Second: Find some way to search for R answers.

  http://stackoverflow.com/questions/102056/how-to-search-for-r-materials

• Third: Learn unclass().

  # Here is an example of fitting a linear model (from the help(glm) documentation)
  ## Dobson (1990) Page 93: Randomized Controlled Trial :
  > counts <- c(18,17,15,20,10,20,25,13,12)
  > outcome <- gl(3,1,9)
  > treatment <- gl(3,3)
  > glm.D93 <- glm(counts ~ outcome + treatment, family=poisson())
  
  

  Want to get the model coefficients and don't feel like suffering through the documentation/help system? You can't inspect the glm.D93 object because it has overridden the print() and summary() methods to hide details (in particular you can't find the member data). No problem, type this:

  > model <- unclass(glm.D93)
  

  The model is now a harmless list without a bunch of pesky methods hiding the information.

• Fourth: learn how to list class and methods.

  Often one of methods(), showMethods() or getS3Method() can show you what methods are on a class or object. Be prepared to try them all as they apply in different contexts.

  # lets make a tricky function
  > fe <- function(x) UseMethod("fe")
  > fe.formula <- function(x) { print('formula')}
  > fe.numeric <- function(x) { print('numeric')}
  

  How will anyone figure out what we have done?

  > class(fe)
  [1] "function"
  
  > methods(fe)
  # [1] fe.formula fe.numeric
  
  > getS3method('fe','numeric')
  # fe.numeric <- function(x) { print('numeric')}
  

• Fifth: Learn to stomp out attributes.

  Ever have this crud follow you around?

  > m <- summary(c(1,2))[4]
  > m
  Mean
   1.5
  

  Ah that’s cute: a little “Mean” tag is following the data around. But what if we try to use this value:

  > m*m
  Mean
  2.25
  

  Okay, now the “Mean” tag has outstayed its welcome. The fix:

  > attributes(m) <- c()
  > m*m
  [1] 2.25
  

  MUCH better.

• Sixth: Swallow your pride.

  My example: does R have map structures? I have no idea and I am too ashamed to ask. However I know I can fake it with environments (which may be “the R way to do this” or may be “a horrible abuse of the language”- I have no idea which).

  > map <- new.env(hash=TRUE)
  > assign('dog',7,map)
  > ls(map)
  [1] "dog"
  > get('dog',envir=map)
  [1] 7
  

  That (nearly) gives you maps with string keys. For maps with numeric keys we can fake something else up with findInterval(). For maps from generic comparable objects keys- I have no idea how you would trick R into helping. This is one reason we like to separate out all data-preparation into a pre-processing step implemented in Java or SQL.

  Note important correction from Eward Ratzer: use “map <- new.env(hash=TRUE,parent=emptyenv()), see comments.

• Seventh: Find and rely on “the one-liners.”

  Reading in an entire comma separated file in a single line ( read.table() ), re-aggregating data ( table() or doBy’s summaryBy() command ) or building an empirical density ( ecdf() ) in a single line of code is an experience not to be missed.

The overall all point is that while R has some (unnecessarily) sharp edges and pain-points it is a powerful tool worth using. I would much rather struggle through a minor R-language issue when trying to prepare my data than to do without the many special functions, distributions, fitters and plotters built into the R system.

Related posts:

1. R examine objects tutorial
2. Exciting Technique #1: The “R” language.
3. R annoyances

When a graph is made, quantitative and categorical information is encoded by a display method. Then the information is visually decoded. This visual perception is a vital link. No matter how clever the choice of the information, and no matter how technologically impressive the encoding, a visualization fails if the decoding fails. Some display methods lead to efficient, accurate decoding, and others lead to inefficient, inaccurate decoding. It is only through scientific study of visual perception that informed judgments can be made about display methods. The display methods of Elements rest on a foundation of scientific enquiry.

— from the preface of The Elements of Graphing Data

A revised edition of The Elements of Graphing Data was published in 1994, along with a companion volume, Visualizing Data, which is oriented towards the implementation and technical details of different graphing techniques. I highly recommend The Elements of Graphing Data as a guidebook for creating graphs, as well as for its excellent survey of several useful techniques. Cleveland, along with other colleagues at Bell Labs, developed the Trellis display system, a framework for the visualization of multivariable databases, using the ideas developed in his texts. Trellis, in turn, influenced Deepayan Sarkar’s Lattice graphics system for R. Lattice implements many of Cleveland’s ideas, and I also recommend Sarkar’s Lattice manual if you do data visualization in R.

It’s important to note here that Cleveland writes for researchers and decision-makers who use graphs to analyze data, or to convey scientific results to colleagues in an (ideally) objective manner. This distinguishes him from Darrell Huff, whose 1954 How to Lie with Statistics considered the use of graphs (and statistics in general) as rhetorical devices for convincing others of one’s point of view. Hence, some of Cleveland’s recommendations and guidelines actually contradict Huff’s. ¹

Edward Tufte also explored the idea that the choice of graphical display should be influenced by the viewer’s cognitive processes, in his 1990 book Envisioning Information. Tufte tends to be more broadly concerned with the gestalt of a graph, beyond its use as an analysis tool; he is also more concerned than Cleveland is with aesthetic considerations.

Cleveland’s philosophy might be summarized as: minimize the mental gymnastics that the viewer must go through to understand the graph. This leads to some obvious advice: avoid clutter and occlusion, make graphing symbols or color-coding unambiguous, use scale-lines on all four sides of the graph, and so on. It also leads to advice that perhaps should be as obvious, but isn’t: make the aspect of the data that you want to analyze as clear as possible. But what does this mean in practice?

Make important differences large enough to perceive

Weber’s Law is a well known observation from the psychophysics literature, which states that the “just noticeable” change in a stimulus is a constant ratio of the original stimulus. Put another way, people are only capable of detecting a change in a stimulus that is greater than a certain percentage k of the original stimulus. Here, “stimulus” can refer to any perceivable physical quantity: weight, intensity, length, orientation. The percentage k will vary with stimulus, and with observer.

Figure 1: From Cleveland, The Elements of Graphing Data

Figure 1 shows the application of Weber’s law to lengths. The bars A and B are of different lengths, but the difference is such a small fraction of the “base” length (say, A’s length, to be specific) that is difficult to tell whether or not they are different, or which is longer. On the right, the bars have been embedded in frames of identical length, and now it is easy to see that B is longer. Why? Because the difference in lengths of the white intervals is a much larger percentage of the white “base” length (say the white A interval). It is easy to see that the white B interval is shorter than the white A interval, and therefore, the black B interval is longer than the black A interval.

The moral is that you always want the viewer to be estimating changes or differences with respect to a short base length. You can do this with reference grids, as demonstrated below.

From Cleveland, The Elements of Graphing Data

Figure 2	Figure 3

Figure 2 shows eight curves. Which one dips to the lowest minimum? Are the high curves approaching the same value, and which one is rising the fastest? Are the low curves dipping to the same minimum? Are they going to the same steady state? Figure 3 shows the same curves, graphed with identical reference grids. The grids shorten the base lengths that are being compared, and it is now much easier to compare highs, lows, and steady state behavior.

But wouldn’t it be better to compare the graphs by superposing them? For two or three curves, perhaps. But in this case, eight curves can clutter the graph, and use up the symbol or color space, making it difficult to distinguish the different datasets — increasing the mental gymnastics.

Reference grids are useful even for a single curve, especially one with slowly varying segments, such as these graphs have. The reference grid makes it easier to answer questions like: does the process return to the initial state, or to a different steady state? Has the process reached steady state, or is it still growing?

Make important shape changes large enough to perceive: Banking to 45 degrees.

The aspect ratio of a graph is important when trying to understand shape. Rate of change information is encoded in the slope of the curve, which the viewer estimates by changes in the orientation of the local tangents at each point of the graph. Weber’s Law tells us that very small changes in this orientation will be difficult to detect. For a given (physical) curve, the local orientation changes will be dependent on the aspect ratio of its graphical presentation, as shown (to an exaggerated degree) in Figure 4. Here, the same curve (two line segments) is plotted at three different aspect ratios, one that centers the graph at 45 degrees, one that forces the curve to be nearly vertical, and another that forces it to be nearly horizontal. In the last two cases, the change in orientation of the two line segments is so small as to be nearly undetectable.

Figure 4: From Cleveland

For two line segments with positive, unequal slopes, a simple geometric argument shows that their absolute difference in orientation is maximized by the aspect ratio that sets their average orientation to 45 degrees (the first graph in Figure 4). Empirical studies by Cleveland and others have indeed verified that a viewer’s ability to judge the relative slopes of line segments on a graph is maximized when the absolute values of the orientations of the segments are centered on 45 degrees.

This result leads to a technique called Banking to 45, whereby the aspect ratio of the graph is chosen so that the average slope of the entire graph is 45 degrees. The details are discussed in Cleveland, and many of the plots in R’s Lattice package also have an option to bank the graph to 45 degrees.

This deliberate exaggeration of slope is something that Darrell Huff deplores. In How to Lie with Statistics, Huff refers to these graphs as “gee-whiz” graphs — and in the context of his discussion of statistics as rhetoric, they are:

Figure 5: From Huff, How to Lie With Statistics

To insist that a graph should always include a zero line and that units be in proportion may be good advice from a rhetorical perspective; but it is poor advice if the purpose of the graph is data analysis. As Figure 6 below demonstrates, we can lose resolution if we always insist on including the zero. Does the trend line in the left graph increase linearly, superlinearly, or sublinearly? The convexity of the curve is more apparent when it is banked to 45, as on the right. Assuming that the scientist reads the axis and is cognizant of the actual magnitude changes involved, the graph on the right conveys more information.

Figure 6: From Cleveland

Make sure all the data is equally well resolved.

It is quite common for positive data — word frequencies, populations, price distributions, just to name a few examples — to be skewed: most of the data is bunched towards low values, the rest of it is spread out on a very long tail. This long tail squashes the majority of the data into a tiny interval of a very narrow dynamic range, as in Figure 7, making it difficult to evaluate the data.

Figure 7: Long-tailed distribution of purchase sizes

Figure 8: Distribution of log(purchase size)

Imagine that Figure 7 represents the distribution of average purchase size across an online merchant’s customers: average purchase size is plotted on the x-axis, and the y-axis represents the fraction of the total customer population whose average purchase size is a given value (the area under the graph integrates to one). According to this graph, most customers make fairly small purchases on average, but there is a long tail of big spenders trailing out into the range of several thousand dollars. Obviously, one would like a little more resolution on the big spike of customers near zero. One could simply “zoom in” on this range, by chopping off some long chunk of the tail, but you may potentially lose sight of some global patterns in the data by doing so.

Graphing the distribution of log(purchase size) enables you to increase the resolution near zero, while preserving the global view. Figure 8 shows the distribution of log(purchase size), revealing two spending populations: a population of high spenders who tend to make purchases in the $3000 range (in log space), and another population whose purchases are centered (in log space) around $60. The existence of these two distinct populations is not apparent in the original graph.

Notice that Figure 8 has two x-axis scales: the top axis is marked in log units, while the bottom axis is marked in absolute dollars, spaced on a log scale. This accords with the principle of minimizing mental gymnastics, since the viewer of the graph will typically be concerned about prices in dollars, not log dollars. In fact, it would have been better yet to have plotted the distribution of log₂ or log₁₀ of the data; the former would allow us to see at a glance the doubling of price ranges, the latter to see price changes in factors of ten.

Figure 9: The 14 most abundant elements in meteorites. From Cleveland

Figure 9 shows another example: the fourteen most abundant elements in meteorites, specifically the average percent of each of the elements. If we graph the percentages directly, as on the left, we cannot easily distinguish the differences in the elements from aluminum on down. Graphing log₂ of the percentages, as on the right, improves the resolution. Again, we have two x-axes on the graph of the log data.

If you want to analyze the difference between two processes, then graph the difference, not the processes (or graph both).

Suppose that we are comparing the two processes f1 and f2 that are shown in Figure 10. As x increases, the two processes appear to be approaching each other — that is, the difference between the two seems to be decreasing. In reality, the difference between the two is constant: f2 = f1+1.

Figure 10: The illusion of convergence

Figure 11: British Imports and Exports. From Cleveland

It turns out that people are good at perceiving the perpendicular difference between two curves, but not the differences in height, which is what we are actually interested in here. When we try to infer the differences from the process graph, we may not only miss key information, we may actually draw incorrect conclusions.

A less toy example is given in Figure 11. Here the imports to and exports from England are graphed over the first 80 years of the 18th century. In the difference graph on the bottom, we can see a local peak in (imports-exports) just after 1760; this is not obvious from simply comparing the two processes (top graph).

If you are interested in rate of change, then graph rate of change.

In Figure 12, we see the population figures for a given community from 1990 to 2009. Obviously, the population is steadily increasing, but how quickly? Is the rate of population growth increasing over time, or is it decreasing? If we are interested in these questions, then simply graphing the population over time is not enough. We need to look at the rate of change directly.

Figure 12

Figure 13

The classic way to do this is by graphing the logarithm of the data. In Figure 13, we have graphed log₂ of the population over time, with the log scale printed on the right hand y-axis, and the actual population numbers printed at a log scale on the left hand axis. Now we can see that the population increased at a constant rate from 1990 to 2000, quadrupling approximately every four years, and then slowed down (to a lower constant rate) after 2000.

Graphs as a research tool

Throughout this discussion, we have considered graphs as a tool for data exploration and initial understanding. It is an iterative process — as questions arise, the data will be reprocessed and re-plotted to highlight the new issues to be examined. A good research graph must display this information directly, with a minimum of mental gymnastics, but — as with any research tool — there can be a learning curve. For example, densityplots (such as those shown in Figures 7 and 8) are in my opinion more useful than histograms for understanding how numerical data is distributed — and I am constantly surprised at the amount of explanation that they require when I show them to people who are unfamiliar with them. A number of very useful graphs that are discussed in Cleveland’s texts meet with the same reaction from people who encounter that style of graph for the first time. This is a disadvantage, relative to using a more fashionable graph, when attempting to communicate results. But the insight into the data that these graphs provide often make it worth spending the time to educate clients or peers on how to read the graph.

Even so, a good graph still may not be a quick read. As Cleveland writes:

While there is a place for rapidly-understood graphs, it is too limiting to make speed a requirement in science and technology, where the use of graphs ranges from detailed in-depth data analysis to quick presentation.
…

The important criterion for a graph is not simply how fast we can see a result; rather it is whether through the use of the graph we can see something that would have been harder to see otherwise or that could not have been seen at all.

- The Elements of Graphing Data, Chapter 2

Related posts:

1. A Demonstration of Data Mining
2. Living in A Lognormal World
3. The Data Enrichment Method

R is a language for statistical analysis available from http://cran.r-project.org/ . The things you can immediately do with it are incredible. You can import a spreadsheet and immediately spot relationships, trend and anomalies. R gives you instant access to top notch visualization methods and sophisticated statistical methods.

R is so hot (a strange thing to say about a statistics package) that it was the subject of a recent New York Times article: http://www.nytimes.com/2009/01/07/technology/business-computing/07program.html . If you read between the lines some of the interviewees come off as being slightly threatened by R (there is a slight hint of “R is very good for others”). In fact R is simply very good. A good statistician with R can do things that a great statistician without R can not. Like all tools R is dangerous, ask for the wrong analysis and you well draw wrong and misleading conclusions. Ask for the right analysis and R will correctly perform it while tracking critical implementation details that would take you hundreds of hours to master on you own.

Want to produce graphs using the theories of perception and analysis of W. S. Cleveland? Simple- use Deepayan Sarkar’s “Lattice” model, which even has a wonderful book.

Want to find subtle relationships in your data using logistic regression (one of the more complicated cousins of linear regression)? That is built into the base R system.

Need to re-run all of your analyses because the data has changed? R is script based and stores your command history. A single paste can re-run a 20 step analysis and re-build a 10 slide presentation.

Impressed by a particular type of analysis? Take, for example, Roger Koenker’s “Quantile Regression” (which is a brilliant idea backed by a masterpiece of a book). Guess what, the original author has supplied a free R-module that implements the ideas.

Want to give a client working software? Easy, R is open source and comes with very good automated installers for OSX, Linux and Windows.

Want to train somebody to use R? Easy, R has an extensive library of excellent books and there is even an exciting set of books with a series title “Use R!”

Want to learn the internals of R from John M. Chambers (one of the inventors of the “S” language that R is an implementation of)? You are in luck the latest book by Chambers is “Software for Data Analysis, Programming with R.” R is so popular that it has managed to pull one of the creators of S language and the proprietary S+ implementation into its world.

It is almost getting to the point where you need to justify not using R.

Related posts:

1. R annoyances
2. R examine objects tutorial
3. Survive R