> l <- list(x=c(1,2,3))
> l$x
[1] 1 2 3

But suppose we wanted to automate this; that is pass in the name of the value we want in a variable. We are after all using a computer, so automating a step seems like a reasonable desire. R supplies a notation for this using the “[]” operator. But something slightly different comes out under the “[]” operator than under the “$” operator:

> varName <- 'x'
> l[varName]
$x
[1] 1 2 3

Notice that the printed outputs are slightly different (one echoes "$x" and one does not). Let's use the "class()" method to see what is actually being returned in each case.

> class(l$x)
[1] "numeric"
> class(l['x'])
[1] "list"

Completely different return types are returned (in one case a numeric vector in the other a general list, not interchangeable types).

At this point you may think it is time to turn in our "pro" label and call ourselves "newb" (Internet slang for "newbie" or "idiot"). But let's slow down for a bit. When two views of the same situation disagree (such as the difference in opinion between the authors of R and myself whether the "[]" and "$" operators should return the same type) you at most know that at least one of those views is wrong. You don't really know if one view is right or even if one view is right which one it is. I can, however, bring in some additional argument to try and show the design of R is in fact wrong. The additional argument is "The Principle of Least Astonishment." This principle roughly says that it is a mistake to introduce unnecessary differences in outcomes (which to the unprepared user are unpleasant surprises). There may be some deep (yet obscure) reasons the two operators prefer to return different results. But the fact you would have to find a way to document and explain these differences really should make one think that this situation is really a mis-design and the "explanation" is really an attempt at a work around. Or to put it more rudely: there may be an explanation, but there is no excuse.

For another example consider creating a 3 by 3 matrix:

> m <- matrix(c(1,2,3,1,1,1,0,0,1),nrow=3,ncol=3)
> m
     [,1] [,2] [,3]
[1,]    1    1    0
[2,]    2    1    0
[3,]    3    1    1

Now select the last two rows of the matrix.

> m[c(FALSE,TRUE,TRUE),]
     [,1] [,2] [,3]
[1,]    2    1    0
[2,]    3    1    1
>

Now (for the punchline) try to select just the middle row of the matrix.



> m[c(FALSE,TRUE,FALSE),]
[1] 2 1 0

Notice that once again (and without warning) the result is subtly different. I admit that it seems paranoid to worry about such small differences- but when you are debugging a system that should work these are exactly the killing mistakes you are looking for. In this case the problem is pretty bad. See what happens if you tried to ask for the dimension of each of these differing returns:

> dim(m[c(FALSE,TRUE,TRUE),])
[1] 2 3
> dim(m[c(FALSE,TRUE,FALSE),])
NULL

The first case works fine (reports 2 rows and 3 columns). The second case returns "NULL" (instead of 1 row and 3 columns). In R NULL is sometimes used as an error-value (instead of throwing an exception) and this value will poison any further conditions or calculations it is involved in. The main way to deal with the arbitrary introduction of such NULLs is the incredibly tedious uncertain defensive coding practices that we argue against in Postel’s Law: Not Sure Who To Be Angry With. Such code weakens both programs and programmers.

But what is going on in this example? Once again we use the "class()" method to inspect the subtly different results.

> class(m[c(FALSE,TRUE,TRUE),])
[1] "matrix"
> class(m[c(FALSE,TRUE,FALSE),])
[1] "numeric"

The result is disappointing. For a two-row select R returns a matrix (what we would expect). For a single-row select R does us the "favor" of converting the result into a vector. This is a disaster. A single row matrix is similar to a vector, but even R itself does not support the same set of operations and outcomes on vectors as it does on matrices (for example the failure of the "dim()" method). It is not safe to further calculate with these results (without by-hand converting the result back to a single row matrix which R can in fact represent). In my case this created crashing bugs deep in a long running analysis (and was hard to diagnose as the bug was in an "innocent operation" not in a "risky calculation").

All of this has to violate John Chambers' "Prime Directive" for data: "an obligation on all creators of software to program in such a way that the computations can be understood and trusted." Chambers' opinion being relevant as he is the author of the S language (of which R is an open source re-implementation). We continue to recommend R, but we also recommend being exceptionally careful when using it (which unfortunately adds time to projects).

Related posts:

1. R examine objects tutorial
2. Relative returns: a banker versus trader paradox
3. Sorting Used in Anger

Postel’s Law (also called The Robustness Principle) comes from RFC 761 “Transmission Control Protocol” in 1980 and is: “Be conservative in what you do; be liberal in what you accept from others.” (Side note: RFC is the now ironic acronym used to describe Internet standards- the letters stood for “request for comments”).

This idea probably worked best where it started- in the TCP/IP world which is where a lot of hairy details of how computers network are handled. When your goal is the basic establishment of transient communications- success is measured by getting information through and not unnecessarily triggering a failure. It may be okay to tolerate mistakes here- because they don’t live long anyway.

Unfortunately, the law works less well other places where it is applied. The law is a downright hazard in dealing with archiving meaningful data (instead of managing transient signaling protocols). Sometimes the cost of obeying the law far outweighs the potential benefit.

A common arena for Postel’s law is now HTML, the markup language used to represent the content we view in web-browsers. In this arena Postel’s law has had two consequences: one good, one bad.

The good: almost anyone can create a working web-page or even a web-site because modern browsers have been designed to paper around almost every common HTML mistake. It has been pointed out that this ease of creation and “worse is better” (a deep principal due to Richard P. Gabriel see wikipedia: worse is better) has been one of the reasons that HTML out-competed and killed many other ideas. Philip Greenspun’s famous story of a 10-year-old building web site to get his mother medical attention happened in the sloppy world of HTML and could not have happened in the straight jacket of RDF (Resource Description Framework: the darling of the semantic web). I would not wish having to actually read or adhere to the incredibly long and irrelevant standards from w3.org (where the ratio of value to pedantry goes to zero) on an enemy. The web is only interesting due to its content and much of its content was only possible due to low barrier of entry.

The bad: to read HTML you almost have to re-create the entire history of web-browsers. This is a history of many hostile competitors (Microsoft, Netscape, Opera, WebKit, Mozzila, Google) and billions of dollars. Reproducing a significant fraction of this history is a significant (and useless) expense. For the most part I use a permissive parsing library like TagSoup or HTMLTidy but even these miss some things that browsers accept and are far more complicated than the task truly justifies.

Even worse is the cases of XML and RDF. These are often used for archival storage of semantic data. That is you may need to read and understand (not just display) data in XML for a long time. To be liberal in what you accept you have to again master a long set of useless complications (DTDs, namespaces incredibly inept character encoding and escapes) and still get burned by improperly encoded XML (that “used to work” because the bugs in the emitted XML matched the bugs in a library that is now out of date).

It is clear in the case of HTML and XML that Postel’s law’s cost is too high for what it delivers. Or at least half of the law is too expensive: no amount of being generous in what we accept makes up for the original data not have been impounded correctly (not being “conservative in what they do” and not having checked that at the time it was created). Some of this is that the producers of the data have no way of telling they are not being “conservative in what they do” because the “generous in what they accept” libraries they use to debug don’t tell them they are emitting bad data. And lets be honest- most systems are not designed for correctness, they are instead debugged until they seem to work. I would say that in fact HTML is not an example of the power of Postel’s law but of the pernicious influence of “worse is better.” Computer science has not risen to the level of “software engineering” we still are a horrible “fit to finish” industry.

Frankly for many things we need a simpler “fail early” discipline. Tools need to be better and standards need to be simpler so that if you write something that is wrong it is easy to see why it is wrong and easy to fix it. Postel’s law has helped hide the negative impacts of complicated standards, we need to push the cost of complications back on to standards committees. The need to be “generous in what you accept” overly favors large, rich entrenched players who have had the time and resources in incrementally invest in papering around every common mistake.

However, I am not sure if we can throw out half of Postel’s law or even if we want to. When Postel’s law fails it is not clear who to be mad at.

Sun, to kick somebody who is already down, was famous for making elaborate frameworks that correctly and brutally implement many details of RFCs. Sun’s Java includes huge frameworks for XML, UTF8 and email that scrupulously implement page after page of useless standard documentation but fail in the wild due to not being “generous in what they accept.” For example Sun’s GlassFish (which got listed named as one of four or five important assets during Sun’s various acquisition talks much like the fact the car has cup-holder somehow always gets mentioned in spec sheets) is an “open source production-quality enterprise software application server.” A supposedly major component of the GlassFish is its email component which is a huge unwieldy framework that implements many of the email related RFCs and protocols including IMAP. Unfortunately for all its hugeness it can not reliably read email folder names from one of the biggest IMAP servers: Google Mail. Google Mail includes “against standard” characters in the protocol and crashes the GlassFish software.

And here is where Postel’s law fails us: under Postel’s law both sides are at fault (Sun for failing to be generous in what they accepted, Google for failing to be conservative in what they did). We can’t assign only one villain. We have no proscription of who to ask for a fix. Postel’s law seems useful in that if either Google or Sun had followed it the two systems would work. But the law doesn’t pick one side to assign blame and help us to efficiently diagnose and fix the problem. It becomes difficult to find the critical bugs when they are masked by a see of “acceptable” bugs. Take a contrary example: the simpler law “implement the standard or fix the standard” would clearly assign blame to GMail.

Similar pain is encountered in Java’s handling of character encodings like UTF8. It is hard to move up the stack of artificial intelligence (from words, to concepts, to ideas, to reasoning to consciousness) when you can’t even reliably transcribe characters. When faced with bad character sequences (a common occurrence on the web) there is no practical way to get Java “mostly parse it,” Java libraries and frameworks authors seem to extract a perverse joy in throwing a program-killing exception (it does not matter if you catch it the library has already stopped doing what you wanted) because they are concerned that a diacritical mark was not properly encoded (web browsers, on the other hand, lose the mark or show some sort of damage near the mistake and blunder on). And here is were the frustration sets in, how can you make applications that are generous in what they accept when the libraries and frameworks are overly proud and picky? This, at first, seems like an argument for Postel’s law- if everybody else (especially the library authors) were generous in what they accepted your life could be easy. That is certainly one possibility- but I argue it often becomes a matter of semantics to assign blame where there is no pre-existing specification or performance agreement. In the end you will waste more time dealing with errors that should never have made it to you than the time you save emitting the odd error of your own.

The unit testing people have a somewhat better idea: fail early, fail at the factory where it is cheap to fix. Don’t litter all of your code with indecisive statements like:

  Set matches = computeMatches();
  if( matches!=null ) {
     for(String match: matches) {
         ...
     }
  }

Instead: write a unit test to document you expectation that the empty set is expressed in single consistent way:

   Set matches = computeMatches();
   assertNotNull(matches);

And from then on write more confident code:

  for(String match: computeMatches()) {
         ...
  }

This may seem overly optimistic and overly strict- but I have a point. One of the few good principles in computer science (and perhaps one of computer science’s contributions to knowledge, computers are a huge contribution to society- but they were made by engineers) is composition. A plan for getting from A to B followed by (or composed with) a plan for getting from B to C is a plan for getting from A to C. Well a correct plan for getting from A to B when composed with a correct plan for getting from B to C, if each of the plans “is mostly right if the piece after is so nice to fix up a few mistakes” you really don’t know what you have. You may have nothing.

That is my complaint- you can’t put an a priori bound on how expensive attempting to allow both sides of Postel’s law will be. You would like others to paper over your mistakes, but it is becoming too expensive to paper over the mistakes of others. In the end Postel’s law is of little help when cleaning up the inevitable mess.

Related posts:

1. Map Reduce: A Good Idea
2. R annoyances
3. Something I don’t get about business and bailouts

Q: What is the difference between a banker and a trader?
A: A banker will try and tell you a 10% loss followed by a 10% gain is breaking even.

This is a bit less arcane than some of the issues we usually discuss on the Win-Vector Blog, but it is a fun one. And it does take some effort to disabuse yourself of the banker’s fallacy.

It turns out that a lot of our instincts about something as simple as ratios is not quite right. Likely this is because the innate skills of counting leads to a deep understanding of addition and not of multiplication. Take for example the opening joke: a 10% loss followed by a 10% gain sounds like it should be exactly breaking even. But in fact it is exactly a 1% loss.

To compute the loss and gain on $100 we would say after the 10% loss we have $100*(1-0.1) = $90. A 10% gain on this remaining portion would be written as $90*(1 + 0.1) = $99 which, as predicted, is missing a dollar. An incorrect explanation would be something along the lines “well the loss was first, so it applied to a larger number than the gain.” But relative losses and gains work by multiplying and therefore is insensitive to order. It is a fact that a 10% loss followed by a 10% gain is exactly the same as a 10% gain followed by a 10% loss (which eliminates the attempted explanation). The correct explanation is the flaw was far earlier than you would think: you should not believe that the opposite of 10% loss is a 10% gain. To undo the effect of a 10% loss you need just over an 11% gain (a 11.1111111% gain).

For a more dramatic example consider the Dow Jones Industrial Average. It was at $12827 on January 7th 2008, by March 5th of 2009 it had fallen to $6594 or a 48% loss. By January 4th 2010 it had experienced a 60% gain relative to March 5th 2009- but that only got us to $10583, still a 17% loss relative to January 7th 2008. The opposite of 48% loss is in fact 192% gain (which obviously has not happened).

Bankers typically quote interest rates as if they were additive. Things like points and fees are all added. This is almost correct for small interest rates. This nearly right (but actually wrong) language is why we have a bestiary of confusing terms describing interest: simple interest, compound interest and yield. The bankers need some way to signal which numbers will actually be used for computing your mortgage payments versus which numbers will be used for advertising (and in the US they tended not to tell you many of the more important numbers until they were required to by law).

Traders, on the other hand, are very comfortable with multiplying relative losses and relative gains. The main trick of achieving such mastery is to convert multiplication into addition. The way to do this is the log() function (or the logarithm).

The log() function is simple function that has the property that log(a*b) = log(a) + log(b). For connivence lets pick our notation so that log(10) = 1. From this we can deduce that it must be the case that:

log() can not be used on zero or negative numbers (at least not if you expect a real number as a result). For other values we use our calculator.

A trader uses logarithms to think additively about relative changes (also called “returns”). Breaking even is represented as 0 (our friend log(1)), relative increases are represented as positive numbers and relative decreases are represented as negative numbers. For example a 10% loss is represented additively using logarithms as log(1- 0.1) = -0.0458. Now in this logarithm notation the additive opposite of a -0.0458 is in fact (as you would hope) +0.0458. You can even double check: log(1 + 0.1111111) = 0.0458. In this notation the mathematics and the language work together- the opposite of a loss is a gain with the same magnitude (and positive sign).

Returning to our initial example: a 10% loss is represented as -0.0458 and a 10% gain is represented as log(1 + 0.1) = 0.0414, so if we add them (how we combing operations in the logarithmic notation) we get -0.0044. Notice this is not zero, and is in fact equal to log(0 – 0.01) or a net-loss of 1%.

The point is that even trivial math becomes difficult if you are forced, by language or convention, to work from false premises.

Related posts:

1. R annoyances
2. Thievery considered harmful
3. Programs reduced to statistics

If the re-constructed technique is close to what was originally done then so many bad moves were taken that you can’t learn much of anything from the original “result.” This points out some of the pratfalls of not performing hold-out tests, not examining the modeling diagnostics and not remembering that linear regression models fail to low-variance models (i.e. when they fail they do a good job predicting the mean and vastly under-estimate variance).

Our article not an article on global warming, but an article on analysis technique. Human driven global warming is either happening or not happening independent of any bad analysis. Finding the physical truth is a bigger harder job than eliminating some bad reports (the opposite of a bad report is not necessarily the truth). Bad analyses can have many different sources (mistakes, trying to jump ahead of your colleagues on something you believe is true, trying to fake something you believe is false or be figments of overly harsh critics) and we have not heard enough to make any accusations.

First: load the data (I re-formatted it at bit so R can read it: jonesmannrogfig2c.txt, data1400.dat_.txt ) , perform the principle components reduction and fit a first
model.

> library(lattice)
> d1400 <- read.table('data1400.dat.txt',sep='\t',header=FALSE)
> d1400r <- as.matrix(d1400[,2:23])
> pcomp <- prcomp(na.omit(d1400r))
> plot(pcomp)
> vars <- data.frame(cbind(Year=d1400[,1],d1400r %*% pcomp$rotation),row.names=d1400[,1])
> jones <- read.table('jonesmannrogfig2c.txt',sep='\t',header=TRUE)
> datUnion <- merge(vars,jones,all=TRUE)
> datUnion$avgTemp <- with(datUnion,(NH+CET+Central.Europe+Fennoscandia)/4.0)
> model <- lm(avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 ,dat=datUnion)
> summary(model)

Call:
lm(formula = avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5, data = datUnion)

Residuals:
       Min         1Q     Median         3Q        Max
-0.8811679 -0.2658117  0.0008174  0.2933058  1.0450044 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.0065252  0.5750696   0.011   0.9910
PC1         -0.0001683  0.0003912  -0.430   0.6679
PC2         -0.0003678  0.0010114  -0.364   0.7168
PC3          0.0003177  0.0014821   0.214   0.8307
PC4          0.0044084  0.0019351   2.278   0.0246 *
PC5          0.0188520  0.0205137   0.919   0.3601
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.4505 on 113 degrees of freedom
  (484 observations deleted due to missingness)
Multiple R-squared: 0.05223,	Adjusted R-squared: 0.01029
F-statistic: 1.245 on 5 and 113 DF,  p-value: 0.2927

We used only 5 principle components as modeling variables, because as is typical of principle component analysis- beyond the first few components the components become vanishingly small and unsuitable to use in modeling (see graph pcomp below).

However, this gave a model with far smaller R-squared than people are reporting, so lets add in a lot of components like everybody else does (bad!).

> model <- lm(avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 + PC6 + PC7 + PC8 + PC9 +PC10 +PC11 +PC12 + PC13 ,dat=datUnion)
> summary(model)

Call:
lm(formula = avgTemp ~ PC1 + PC2 + PC3 + PC4 + PC5 + PC6 + PC7 +
    PC8 + PC9 + PC10 + PC11 + PC12 + PC13, data = datUnion)

Residuals:
     Min       1Q   Median       3Q      Max
-0.87249 -0.25951  0.03996  0.25055  0.99039 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept)  7.431e-01  1.424e+00   0.522   0.6028
PC1         -1.796e-04  3.665e-04  -0.490   0.6253
PC2         -4.179e-04  9.759e-04  -0.428   0.6694
PC3          3.306e-05  1.430e-03   0.023   0.9816
PC4          3.416e-03  1.803e-03   1.894   0.0609 .
PC5          4.032e-02  1.978e-02   2.039   0.0440 *
PC6         -3.260e-03  2.660e-02  -0.123   0.9027
PC7         -7.134e-02  3.620e-02  -1.971   0.0514 .
PC8         -1.339e-01  7.895e-02  -1.696   0.0928 .
PC9          7.577e-02  5.734e-02   1.321   0.1892
PC10         2.700e-01  5.878e-02   4.594 1.22e-05 ***
PC11         8.562e-02  6.741e-02   1.270   0.2068
PC12        -8.057e-02  1.053e-01  -0.765   0.4461
PC13        -4.099e-02  1.064e-01  -0.385   0.7008
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.4141 on 105 degrees of freedom
  (484 observations deleted due to missingness)
Multiple R-squared: 0.2558,	Adjusted R-squared: 0.1637
F-statistic: 2.777 on 13 and 105 DF,  p-value: 0.001961

This is a degenerate model that essentially didn’t fit (thought the significance on PC10 component fools the fitter, but PC10 can’t be usable- it is essentially noise). Graphically we can see the fit is not very useful (despite having a little bit of R-squared) by looking at the graph of the fit plotted in the region of fitting. Notice how the fit variance is much smaller than the true data variance even in the region of training data, this is typical of bad regression fits.

> dRange <- datUnion[datUnion$Year>=1856 & datUnion$Year<=1980,]
> xyplot(avgTemp + prediction ~Year,dat=dRange,type='l',auto.key=TRUE)

Now the statement they wanted to make is that the present looks nothing like the past. The past is only available through the fit model so what you would hope is that the model looks like the present and then the model itself separates the past and present. Instead as you see in the graphs above and below this fails two ways: the model looks nothing like the present and the model’s past looks a lot like the model’s present.

> datUnion$prediction <- predict(model,newdata=datUnion)
> xyplot(avgTemp + prediction ~Year,dat=datUnion,type=c('p','smooth'),auto.key=TRUE)

What we could do to falsely drive the conclusion (which itself may or may not be true, it just is not supported by this technique, model or data) is create the infamous graph where we switch from modeled data in the past to actual data in the present and then act surprised that the two did not line up (which they did at no step during the fitting). I don’t have the heart to unify the colors or remove the legend, but here is the graph below:

> datUnion$dinked <- datUnion$prediction
> datUnion$dinked[!is.na(datUnion$avg)] <- NA
> xyplot(avgTemp + dinked ~Year,dat=datUnion,type=c('p','smooth'),auto.key=TRUE)

The reason the blue points look different than the others is they came from the average temperature data instead of the model (where everything else came from). Switching the series is essentially assuming the conclusion that recent past looks very different than the far past.

Essentially this methodology was so poor it could not have illustrated or contradicted recent global warming. There are plenty of warning signs that the model fitting are problematic and the conclusion illustrated in the last graph can not actually be proved or disproved from this data (the proxy variables are too weak to be useful, that is not to say there are not other better proxy variables or modeling techniques). The problems of the presentation are, of course, not essential problems in detecting global warming (which likely is occurring and likely will be a drain on future quality of life) but problems found in a single bad analysis.

Related posts:

1. R examine objects tutorial
2. Survive R
3. R annoyances

I have two areas of concern.

My first concern is: what is going on with the ads on Google search? Are they safe to click on? Check out this ad (from a Google search page):

Where do you think the link would lead to if you clicked on it? First guess: “www.dmb.com/duns” which is further labeled “Official Site.” This sounds like we would directly link to the venerable company “Dun and Bradstreet.” This is not the case. The text and the green site are apparently part of the ad. Hovering the mouse or capturing the link yields the following huge and hard to decipher URL:

http://www.google.com/aclk?sa=L&ai=CwmvMvAVpSsyeDpvusAO4y5D8D5aTi3qAv83oDPXK5f0CEAEguVQoA1Dw0pOr-_____8BYMn2-IbIo6AZyAEBqgQdT9AhbH2fVXmJHnIy-TNNj_HkY7JsaGV106RyaVw&num=1&sig=AGiWqtwX5zxePZHhDkpqJBojcybMzKkFSw&

q=http://track.did-it.com/n%3Flid%3D13619863%26tid%3D3fe34bfe02723%26eng_creative%3D3345771492%26eng_keyword%3Dd-u-n-s%2520number%26eng_placement
%3D%26url%3Dhttp://smallbusiness.dnb.com/webapp/wcs/stores/servlet/SmbHome%3FstoreId%3D10001%26cm_mmc%3DGoogle-_-Adword-_-online-_-d-u-n-s%2520number%26LID%3D13619863

Can you anticipate where this goes? You can see the following three URL fragments inside the URL:

• http://www.google.com/
• http://track.did-it.com/
• http://smallbusiness.dnb.com/

Reading from left to right you can be pretty sure the URL goes to a Google server that uses the rest of the URL as an argument. We can then assume that this Google server performs some bookkeeping and redirects to http://track.did-it.com/ . We can then further assume that track.did-it.com performs some more bookkeeping and redirects to http://smallbusiness.dnb.com/. Notice the first non-Google URL does not match the advertised URL. We don’t know what is encoded into each of these pieces and we are only assuming the track.did-it.com server unpacks the URL from the argument; maybe that is a red herring and it redirects us to somewhere else entirely.

Frankly I have no idea if this really happens. I do know from experience that Google did re-direct to track.did-it.com because my anti-spyware traps caught this, blocked the page load and issued a warning. This sort of thing could be a very irritating late night call from Mom- she gets a web-safety warning (anti-phishing, anti-virus, cookie request or something else) from a site she is not on, she is not loading inclusions from (images, scripts, iframes and so on) and she is not knowingly clicking on.

Is there a finite list of partners that Google allows to re-direct? Can anybody do this? Can I produce and place ads with arbitrary redirections?

My second concern is: do you even need to click to leave the Google search page?

Check out what happens with a fresh copy of Mozilla Firefox (no plugins and nothing added to the browser) when you search for “Bing” on Google:

Let’s look at that pop-up a little closer:

Why would www.bing.com get a chance to set a cookie? Doesn’t that only happen in response to a request from my copy of Firefox? Did some script on the Google search page or some obscure “web acceleration” option in Firefox pre-fetch the content of the first Google link or first Google Ad (triggering the cookie request)? Let’s not worry too much over cause (Google is the major funding source for Firefox), but look for possible effects. Does the destination site (Bing) see traffic from the Google page even if the user never clicked on the link? Something like that would inflate the already huge stature of Google as a traffic source. How much of my site’s “web traffic” comes from phantom clicks (from abandoned searches)? Finally, how safe is it? How much more than the cookie is being loaded? What if the site linked to has malware- is this a no-click infection route?

Some of these subtle features are necessary to support an ad network. But the implementation does not seem minimal. Allowing hosts to differ from ad content and performing a pre-fetch with every search both expose searchers to additional risk.

EDIT:

Found the prefetch. A Firefox tag that Google knows to set (see What is firefox prefetching?)- and sure enough if I perform Google search using Firefox we see:

<link rel="prefetch" href="http://www.bing.com/">

(and I don’t see this tag when using Safari).
So that is the mechanism- still wondering how much risk this is to users. Also wondering how much this skews traffic statistics into Firefox’s and Google’s favor (I haven’t seen the prefetch tag on Yahoo search or Bing search).

Related posts:

1. YAYGDA (Yet Another Yahoo Google Deal Article)
2. Is Search Advertising a Market for Lemons?
3. Public Service Article: JSTOR and other Useful Research Archives

Microsoft has tried retail outlets before. So we have some knowledge of what to expect.

In particular I am thinking of the Microsoft store that was part of the new San Francisco Metreon Mall (opened in June of 1999). At the time Microsoft had no good reason to produce such a store. They were not ready for a major consumer product push and didn’t release the original Xbox until 2001 or the Zune until 2006. This era was also a gap in their consumer operating system release cycle (Windows 98 was already out, Windows ME didn’t come out until Q2 2000 and Windows XP Q3 2001). In 1999 there still were Gateway computer retail stores and Apple Stores were still two years in the future. This Microsoft Store satisfied no discernible need, promoted no major strategy and was deservedly forgotten. From their own press release the purpose of the store was to sell “more than 90 items of apparel, desk accessories, office supplies and other items with the microsoftSF logo” ( see presspass). A fairly low marketing ambition- which predictably failed.

Fast forward to 2009. A Microsoft store could work, if Microsoft were willing to make a good one. I doubt they are willing. Likely they feel hey can easily cut themselves a lucrative slice of retail pie (now that they have major consumer products like Xbox, Zune and Windows Media Center). This will fail- these are all significant products, but together they don’t make for an interesting store or cohesive shopping experience or lifestyle pitch.

Instead, imaging the following store.

• A Microsoft store where a competent sales staff sells major brand PCs (like a Sony, HP, Leveno and so on). But for a small mark-up they wipe out the manufacturer’s defective Windows install and spyware. They install a clean full version of Windows 7 (or Windows XP). A store where they consult with you and which version of Office really is best for you and then sell and install this.

Even I would go there (and I use Apple OSX and Linux almost exclusively). Microsoft is too big an ecosystem to ignore. We need to be able to buy mainstream (and that means Microsoft) PCs, operating systems and software packages at retail (and not just at box stores).

Let me illustrate my point using the last time I purchased a PC. After an hour of trying to deal with Leveno’s atrocious web-site (they really do not want to sell computers retail) I got up and drove to the Metreon Sony Store and purchased a Sony Vaio.

Here is what happened:

• The Sony Vaio could not handle the Vista it came installed with.
• The Sony Vaio came without restore media- making a clean re-install of the operating system impossible.
• The Sony Vaio came laden with all kinds of crapware (trial anti-virus software, Internet poker offers).
• I (later) spent around $400 on Office Professional just to find out I could not uninstall enough of the Sony demo version of Office to get rid of some sort of killing dependence on “Contact Manager” (some sort of enterprise wide service that a home PC would not have).
• I purchased two old copies of XP. I used one to reformat the Vaio and donated that to my father in law for web-surfing. We then used the other copy of XP to host Office Professional (which we really wanted and needed) inside a virtual machine on a hand-me-down Macbook. End of Microsoft Windows (outside of virtual machines) in our household. If Windows can do something good we now have no way of seeing it (since we now only run Windows inside a virtual machine, which is unfair to Windows).

A careful reader would say “this is unfair, it was Sony that was bad not Microsoft.” That is correct. But there really (right now) is no way for an individual consumer to directly deal with Microsoft. Eliminate some of the wall of stupid between me and Microsoft and I probably would use more of their products.

Here is what I (and probably most other consumers would pay big for).

• A Microsoft store that re-sells major brand PCs (Sony, HP so on) that have been opened, re-imaged with a clean OS from Microsoft (not the evil OEMs) and given out with install/restore media.
• Ability to buy a copy of Office Professional that will work on my PC. Here is my proposition. I drop off my PC and $400, next day I come back and take back only the PC and I have a copy of Office Professional that works and isn’t whining about a bunch of unresolved dependencies to servers and services I do not own. I also need to know which version of Office to get (and I know it may cost more than the cheapest one) without reading some impenetrable “enterprise white paper.”
• A store with a staff that lets me safely say “I don’t know Dad, maybe you should take it back in to the Microsoft Store.” I’ll pay. I’d buy training, support, warrantees, read marketing materials and try product demos

Again, a careful reader would point out that all of these activities are labor intensive and going to be very low margin. This too can be fixed. Once Microsoft installers are wasting Microsoft’s time and money (instead of wasting my time and money) the installers will be fixed. Installing Microsoft Office could be come as easy as selecting a new application in Ubuntu (my current gold standard in easy installs).

I would welcome and support a good Microsoft retail effort, I just don’t believe one to be very likely.

Related posts:

1. Must Have Software
2. Living in A Lognormal World
3. Should your mom use Google search?

• Special Agent Michael G. McSwain’s charges
• Mathew Goldstein’s Reuters article
• Zero Hedge blog entry

For me this triggers some strong (and sad) personal memories.

No matter what inappropriate “Robin Hood” intellectual property fantasies you have this (if true) is just wrong. I have never been a huge fan of the ACM Code of Ethics (which does cover this situation, but fails to seriously address much beyond having responsibilities to your employer) but this sort of incident reminds me why computer science needs some approximation of a shared set of ethics that we can try to refer to.

A particularly sad part of the story that attracted my attention was the reliance on “bash history” to try and establish what happened. Attempting to “prove” something using a “bash history” is something I have painful experience with. The “bash history” system is incredibly inadequate even for what it was designed for (caching recent commands). Simply having two shells open can cause non-deterministic overwriting, deletion, clobbering and time disorderings in the history file. Furthermore bash history has no dates, times, directories or any other contextual hints written into it. Finally bash history has no hashes, signatures, nonces, sequence numbers or any other device that helps establish authenticity.

Now for my story. We (by chance) caught somebody walking off with our group’s entire source tree. In the end all we had to go on was the bash history. To hostile eyes bash history is nowhere near what you would call “forensic grade evidence.” Unfortunately for us the theft was intramural, the thief was merely taking the code to another group in the same company to later mine and represent as their own work. At this point even language worked against us- every time we accidentally said something like “our code” (as in the code we produced, not the code we own) we were perceived as being anti-company. Evil prevailed (the thief was promoted) and I looked stupid for working so hard to try to interpret such low-quality evidence. But we live in an objective world- just because you can’t prove something doesn’t mean there isn’t some buried ugly truth.

So what was stolen? Not the code, that moved from one pocket of the corporation that owned it to another pocket of the same corporation. What was stolen was reputation. The thief presumably appeared to out-produce both his old colleagues and his new ones (who don’t have a few absconded person-years of development to draw from). So an apology to anyone who was asked why they could not code as fast as our escaped “genius,” it was certainly not our intent to so equip him. And a larger apology to the rest of the team, sorry we could not prove the misappropriation of your work.

Of course Shakespeare said it much better (from Othello):

Good name in man and woman, dear my lord,
Is the immediate jewel of their souls:
Who steals my purse steals trash; ’t is something, nothing;
’T was mine, ’t is his, and has been slave to thousands;
But he that filches from me my good name
Robs me of that which not enriches him
And makes me poor indeed.

Related posts:

1. Sorting Used in Anger
2. Relative returns: a banker versus trader paradox
3. A Quick Appreciation of the Sharpe Ratio

However, the reading experience is marred by some flaw in their RSS system that keeps marking the article “MapReduce: A major step backwards” as a new article. This causes the article to appear in my RSS reader every few weeks as “new.” This wouldn’t bother me too much except that the article runs so counter to experience that it is itself offensive.

I have no desire to defend Google (the home of MapReduce)- they don’t need it and are clearly laughing all the way to the bank. However the points used to kick at MapReduce are so broad and so devalue practitioner experience that they are insulting. I find the individual arguments offensive and wish to stand against them. I am not that concerned about the conclusion, use MapReduce or don’t. For some things MapReduce is a good tool and for some things it is not.

Let’s limit ourselves to the 5 primary complaints from the article. The article (verbatim) says MapReduce is:

1. A giant step backward in the programming paradigm for large-scale data intensive applications.

2. A sub-optimal implementation, in that it uses brute force instead of indexing.

3. Not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago.

4. Missing most of the features that are routinely included in current DBMS.

5. Incompatible with all of the tools DBMS users have come to depend on.

Now let us comment:

1. “A giant step backward in the programming paradigm for large-scale data intensive applications.”

Actually, no.

MapReduce represents a continuity in a stream of ideas that made UNIX great: composable transient tools. Not everything is a database or data warehouse. A lot of the grungy UNIX tools (like sort, sed, awk, join) have often been combined to do large scale (at the time) research because they all worked “out of core” fairly well. This makes for a horrible bailing-wire set-up. However, it often handles problems of a size much larger than would have been possible on the hardware at the time.

In addition the author trots out the “it’s Codasyl all over again” argument. This argument refers to the ongoing pain and expense derived from binding algorithmic details too close to the data representation. In earlier writing it was a fantastic point that warned that the up and coming object oriented databases were going to be the same nasty pointer chasing nightmares that hierarchical databases had been. I can see why an author might feel that just saying “it’s Codasyl” could win any argument.

2. “A sub-optimal implementation, in that it uses brute force instead of indexing.”

MapReduce does not use brute force.

MapReduce uses the idea (one that goes back to merge sort) that parallel traversals (that is: running through two lists in the same order synchronously) are a very powerful technique that can, among other things, produce indices. MapReduce is so efficient that it has been shown to be competitive with the best large scale sorting algorithms on their home-turf: sorting.

MapReduce looks brutish because it drops a lot of popular design features. One such feature is trying to speed things up through local caching and combining. However, on the data that MapReduce is commonly used (free form written text) it is a provable property of the data that local caching is an ineffective complication (due to the heavy-tailedness of the data). So many of the graceful features missing from MapReduce are actually no help on the types of data it is used on. There is a certain grace in leaving only only the features that are actually helping.

3. “Not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago.”

A nasty attack.

MapReduce is a good explanation of some merging techniques that have been common knowledge for quite a while. This is a legitimate expository goal: explaining something everybody already “knows” better. In fact this is very hard to do and considered a legitimate accomplishment in many fields (for example Rota stated it was a legitimate goal in mathematics). I myself looked at some of my own older code for dealing with very large data sets after reading the MapReduce paper. I saw that the paper was describing what I was already doing (splitting the data into streams for later re-joining) and explaining it so well that it was now a method and no longer a hack. When a paper successfully teaches about you something you already “know” it is a good work.

The attack is is also inaccurate- the ideas are not 25 years old it is closer to 120 years old.
We could easily trace the lineage of MapReduce back to Hollerith style sorting machines that pre-date general purpose computers (i.e. going back to before 1889) . MapReduce refers back to a time when all computation was performed by what we now call external sorting and tabulation. These 19th century technologies may seem archaic but they were developed in a word similar to ours: worlds where the amount of data is in excess of your conveniently reconfigurable computational resources.

4. “Missing most of the features that are routinely included in current DBMS.”

Unfortunate.

I miss a lot of those features.

However, because MapReduce is such a lean technique I have seen engineers implement their own MapReduce systems in a day (to solve a problem they are working on). That is they are successfully sorting, joining, indexing and summarizing hundreds of gigabytes of data on a consumer PC within a couple of days of being asked to. This is from scratch after reading the MapReduce paper.

The “make versus buy” decision should not always come out “make.” But it is not wise to artificially bloat up requirements so that the decision can only be “buy.”

5. “Incompatible with all of the tools DBMS users have come to depend on.”

Good.

Frankly for a lot of analytic practitioners many DMBS systems and tools have become expensive obstacles in the way getting results. Yes, we enjoy humiliating an interview candidate that does not know all of the Codd normal forms (or can’t remember which of the alphabet soups of OLTP or OLAP is the “good one” ) as much as the next person. But to many of us a lot of these tools and procedures are more obstacles than a solutions.

This may sound nasty, but if were not the case why would companies like Vertica be producing radical new database tools? The fact is existing DBMS tools were designed for a different type and scale of data than we regularly see on the web (and column oriented database designers seem to share this view). The situation is so bad that “roach motel” is a common analyst’s slang for “data warehouse” (derived from: “data checks in but it never checks out”).

This isn’t meant to be a hagiography of MapReduce, but given that MapReduce has paid the bills I feel it deserves a small show of respect along the lines of “dance with the one who brung you.”

MapReduce is not a panacea. One of the tasks I have hated most in my career was maintaining a seven step MapReduce based system. I would love to have avoided all the detail fiddling that set-up required. However, the system paid our bills by performing a calculation that was beyond the scale of simpler methods and it would have been unaffordable to buy a solution.

Related posts:

1. Sorting Used in Anger
2. Must Have Software
3. Exciting Technique #1: The “R” language.

If the reason to bail-out Wall Street is to “protect retirement savings” why not just dump the $700 Billion into Social Security and make Social Security a needs based program?

Here is one story we are being told. Without the bailout the credit markets freeze. If the credit markets freeze then “main street” is hurt because businesses can not borrow money. This causes actual economic damage and everybody is really poorer.

I understand the credit markets increase the money supply and generally promote growth. But it might be okay if they were temporarily unavailable. My experience in trading is that when you buy or sell in a finance market: most of the time the counter-party in a trade is another hedge fund or speculator- not somebody actually doing anything productive with the money. So most of transactions lost to a freeze really helped nobody you would care about.

Also, why should businesses always need credit? Many respected businesses (Apple, Google, Microsoft) have tens of billions in cash reserves and likely do not need credit. Others (Oracle) have huge cash positions and debt positions as the same time. I have a feeling that a credit crunch will punish over-extended business that have weird balance sheets more than businesses that run on a sound financial basis.

If you think all businesses have a “rocket science” accounting ability (they produce reports that nobody else can understand and we just balance our check books), consider this. I have regularly seen offers of early payment (net-10 days instead of net-20 days) if a you accept a 10% cut in your bill. Now if you think of this as a favor you might accept it (a supplier should always be willing to offer 10% off to make a client happier). If you think of this as a payday loan (which is what it is) the supplier would be paying an interest rate that compounds to 569% annually just to get their money 20 days earlier. If this is what is considered “financial engineering” we do not need it in this world.

Related posts:

1. It is not all the quants’ fault.
2. A Quick Appreciation of the Sharpe Ratio
3. I know, I am the one being a jerk

Here is a question from the interview:

Andrew: One of the few projects of yours that hasn’t been embraced by a widespread community is literate programming. What are your thoughts about why literate programming didn’t catch on? And is there anything you’d have done differently in retrospect regarding literate programming?

Professor Knuth had a good and interesting answer, which I will not go into here. Also, it was a good question- Literate Programming is a good idea, yet we have only seen weak imitations of it like Doxygen and JavaDoc (which automatically document the syntactic structure of code instead of really helping the programer become an author and explain their meaning and intent).

Mr. Binstock even includes a link to a site promoting the concept. Lets as the kids these days say “click through” and see what awaits us.

This is great. Literate programing is definitely Web 2.0 (rounded corners, use of light pastels and cool gradients). This is now, this is modern, sign me on. The site even has links to original articles by the masters:

Literate Programming - CACM Series
	Programming Pearls: Literate Programming, CACM (May 1986)
	Programming Pearls: A Literate Program, CACM (June 1986)
	Programming Pearls: Abstract Data Types, CACM (April 1987)
	Announcing Literate Programming, CACM (July 1987)
	LP: Processing Transactions, CACM (December 1987)
	LP: Expanding Generalized Regular Expressions, CACM (December 1988)
	LP: A File Difference Program, CACM (June 1989)
	LP: Weaving a Language-Independent WEB, CACM (September 1989)
	LP: An Assessment, CACM (March 1990)
	The Literate-Programming Paradigm
	Donald Knuth. "Literate Programming (1984)" in Literate Programming. CSLI, 1992, pg. 99.

Lets click through and see how the Association for Computing Machinery helps disseminate, guide and educate:

Oh, maybe this is part of why Literate Programming hasn’t been embraced: the whole purpose of Literate Programming is lost when you keep it a secret.

I am sure I have been a paid ACM member from time to time, but I don’t remember the online credentials and they have probably lapsed by now. I tried applying for the free temporary credential (the online form ended up not sending me anything- ACM not so good with the computers). I can afford pay (yet again) to re-join ACM but why would I want to give my money to support an organization so far from my (and common) academic values?

So in conclusion:

• Sorry Professor Knuth, you remain one of my heroes, but I’ll have to get to Literate Programing a bit later. I would say that the marketing campaign behind Literate Programming has excessive “breakage.”
• ACM: that was a funny joke, great head-fake, impeccable comic timing, good fun and I certainly learned something. Oh, and I will see you in hell.

Related posts:

1. Programs reduced to statistics
2. Something I don’t get about business and bailouts
3. Map Reduce: A Good Idea