Our Example Problem

Our small scale problem is this: we have a number of target points on a map and we want to pick a central point to directly connect to all of these points with wire. Our goal is to minimize the total amount of wire used. This problem is called the “Geometric Median”. So we are trying to find a point that minimizes the sum of distances from our chosen center to every target point. If we were trying to minimize the sum of squared distances from our chosen center to every target point the answer would be obvious: the average or mean (which by Hooke’s law is also the point where a set of identical springs would relax to). The mean is in fact a fairly good guess, but you can do better (which could important if the “wire” is expensive, such as cutting irrigation or drainage ditches). For example given the three target points (20,0), (-1,-1) and (-1,1) the optimal point is (-0.42,0) not the mean (6,0) and the choice of optimal point represents an over 19% savings in total wiring distance (see figure).

This is a substantial saving in cost.

The problem changes as we consider variations. If indirect connections (such as routing one point through another, which may or may not be possible for reasons of capacity or safety) and multiple new centers are allowed we then have an instance of the Steiner Tree Problem which is harder to solve (since it is known to be NP complete). If no new centers are allowed (all routing must be between pre-existing target points) then we have a Spanning Tree Problem- which admits very quick solutions.

We bring up the geometric median as a mere example. We don’t intend for our code to solve only the geometric median problem and we don’t intend to touch on the literature of specialized methods for solving the geometric median problem. Instead we are trying to demonstrate the speed you can develop prototype solutions if you have a few good tools (like various optimizers) available in your toolkit. Numeric optimizers may sound exotic, but they often are the kind of thing you want to experiment with and link directly into your code.

Optimization as General Tool

Now that we have the example problem we can describe a solution strategy. In this case the solution uses code “we wished we had lying around” before we started on the problem. We will pretend we have the tools we want ready to solve our problem and then we will pay our debt and build the required tools. The issue is that there is not an obvious closed form for the solution of the geometric median problem. So we are forced to work a bit harder. In this case harder means we need to solve an optimization problem. Consider the contour plot of the total wiring cost as function of where we choose to place our center. Our optimal point (-0.42,0) had wiring cost of 22.73 and the contour plot given here shows concentric regions of solution positions with higher cost.

In general it is unwise to throw an optimizer at an arbitrary problem and hope to find the globally best solution. But in this case (and in many similar situations) we can prove that a simple local optimizer will in fact find the unique best solution. This is a property of the problem not of the optimizer. The concentric regions shown in the contour plot have a very nice shape: they are convex. That is: they have no intrusions- for any two points drawn from one of these shapes the straight line segment between these points stays inside the given shape. We don’t have to depend on observation- we can actually prove this is always the case for this problem. The wiring cost from a proposed center to any single target point is a convex function of where we choose to place our center (a convex function is a function whose graph never reaches above the secant line drawn between any two points on its graph). The total wiring cost is just the sum of the wiring costs to each target point. And to finish: the sum of a collection of convex functions is itself a convex function. Since the contour plot of a convex function has only convex shapes and we have proven the statement.

But how does this help us? There is a standard technique to find “local minima” of a function by inspecting a function for places where the gradient is zero (points where there is no obvious down hill direction on the contour plot). This technique usually can only be guaranteed to find local minima (places where no small change improves your situation). But there is no guarantee that the local minimum you find is in fact the global minimum (the best possible solution). Except when you are dealing with a convex function. When a function is convex then all of the local minima are always grouped together into a single convex connected shape (if not a line drawn between two remote minima would violate the convexity definition). And if the function is never flat then this set is a single unique point: the unique best solution. Our inspection technique will be a gradient driven optimizer- that is an optimizer that when the gradient is non-zero improves its objective by running down hill and halts when the gradient is zero.

The stated function to minimize is to sum the distance from our proposed center to each target point. We can write this as the sum of the distances:

( which is the traditional Euclidean or L2 distance). This function actually has one one subtle flaw that we will deal with in the appendix (see: Fixing Smoothness).

Using Scala to Apply the Optimization Solution

To find our optimal center placement using Scala we first write our cost or objective as a Scala function:

    val dat:Array[Array[Double]] = Array(
      Array( 20, 0.0),
      Array( -1.0, 1.0),
      Array( -1.0, -1.0)
    )

    def fx(p:Array[Double]):Double = {
      val dim = p.length
      val npoint = dat.length
      var total = 0.0
      for(k <- 0 to (npoint-1)) {
        var term = 0.0
        for(i <- 0 to (dim-1)) {
          val diff = p(i) - dat(k)(i)
          term = term + diff*diff
        }
        total = total + scala.math.sqrt(term)
      }
      total
    }

Scala is succinct and it is a great connivence to have a function definition capture data from its environment. What we would like to do is generate an initial guess as the solution (we use the mean as our initial guess) and then call an optimizer (in this case a conjugate gradient optimizer) to do all the work:

 val p0:Array[Double] = mean(dat)
 val (pF,fpF) = CG.minimize(fx,p0)

At this point we would be done, except the conjugate gradient method (which is superior to gradient descent and many the non-gradient methods) requires a gradient.
We could provide a numeric estimate of the gradient by the following divided difference method:

  def gradientD(f:Array[Double]=>Double,p:Array[Double]):Array[Double] = {
    val xdim = p.length
    val p2 = copy(p)
    val base = f(p2)
    val ret = new Array[Double](xdim)
    val delta = 1.0e-6
    for(i <- 0 to (xdim-1)) {
      p2(i) = p(i) + delta
      val fplus = f(p2)
      p2(i) = p(i)
      val diff = (fplus-base)/delta
      ret(i) = diff
    }
    ret
  }

This numeric divided difference method often outperforms non-derivative optimization methods (like Powell’s Method and the Nelder-Mead Amoeba method). But the technique can run into numeric difficulties. We can remedy this if we are willing to write our function in a slightly more general way. If we re-encode our function in a generic manner we can use automatic differentiation (not to be confused with numeric differentiation or with symbolic differentiation) to produce a reliable gradient for optimization. What we need to do is re-write our function to work over an abstract field of numbers instead of only the machine supplied doubles. In fact what we need to do is specify a generic function that will work over any field, with the field to be determined later. The code to do this in Scala is very similar to the non-generic code:

   val genericFx = new VectorFN {
      def apply[Y <: NumberBase[Y]](p:Array[Y]):Y = {
        val field = p(0).field
        val dim = p.length
        val npoint = dat.length
        var total = field.zero
        for(k <- 0 to (npoint-1)) {
          var term = field.zero
          for(i <- 0 to (dim-1)) {
            val diff = p(i) - field.inject(dat(k)(i))
            term = term + diff*diff
          }
          total = total + smoothSQRT(term)
        }
        total
      }
    }

Notice that code is very similar to the “def fx()” code. The key differences are that we had to define genericFx as extending a trait (a type of Scala interface) called VectorFN and inside this trait extension we defined a parameterized function name apply(). apply() is a generic function that is willing to work over any type Y where Y is at least of type NumberBase[Y] (we will get more into what that means in a moment). The difference in notation is that while the Scala function syntax can not specify a generic function with free type parameters (the incompletely specified Y) the Scala semantics are strong enough to implement this. In fact standard function definitions (such as “def fx()”) are just syntactic sugar for extending the Scala built-in Function1 trait. With a generic objective function in hand all we need is conjugate gradient code that is expecting a VectorFN (and willing to call apply() instead of just using naked function parenthesis) and some type NumberBase[Y] that can compute gradients for us. The Scala compiler can specialize our genericFx() into one version for quick calculation and another for gradients. How this is done is what we will discuss next. From our point of view our problem is solved with the following one line of code:

val (pF,fpF) = CG.minimize(genericFx,p0)

This should always be your goal- build sufficient preparation so your last step is a “obvious one liner.”

What Tools we Wish we Had Lying Around

We supply in our example some workable conjugate gradient code, but that is standard so we will not discuss it. What is of interest (and facilitated by Scala’s parametrized type system) is the implementation of dual numbers as a framework to supply automatic differentiation. An implementation of dual numbers as a NumerBase[DualNumber] type is the core of our demonstration.

Dual numbers are an algebraic structure written as pairs of real numbers “(a,b)”. The arithmetic table for dual numbers is given below:

In a dual number (a,b) “a” is the “large” or “standard” part of the number. You can check from the arithmetic table that the pair of dual numbers (a,0) and (c,0) behave just as we would expect the real numbers a and c to behave. In the dual number (a,b) “b” is the “small” or “ideal” portion of the number. From the multiplication rule above we can observe two rules: (0,b) * (c,0) = (0,b*c) (something small times anything else is small) and (0,b)*(0,d) = (0,0) (two small things become zero when multiplied). Essentially the dual numbers are carrying around the first two terms of a Taylor series: we get as a result both the function value and the function derivative. For a function f() over the real numbers we extend f() to work over the dual number by defining: f((a,b)) = (f(a),b f’(a)) (which is consistent with the previously defined arithmetic). We can check that the dual numbers numbers obey the usual laws of arithmetic (associative, commutative, distributive, identities and inverses). The punchline is that over the dual numbers the divided difference estimate of f’(x) (the derivative of f() evaluated at x) is in fact exact in the sense that f((x,1)) = (f(x),f’(x)) (or f((x,0)+(0,1)) – f((x,0)) = (0, f’(x))). Implementing the DualNumber class is little more than transcribing the above arithmetic table into Scala.

We have already seen how to write code that uses NumberBase[Y] types (genericFx() itself is an example). A more complicated example is the CG.minimize() code which not only accepts a generic function (in the form of VectorFN) but then specializes it to NumberBase[DualNumber] to compute gradients and also specializes to NumberBase[MDouble] for quick calculation during line searches (MDouble is just an adapter for machine Doubles, used for speed). The ability to re-specialize a function is one of the advantages of a parameterized type system. The DualNumbers are an example of forward automatic differentiation. We could also use the same object framework to capture a representation of the computation path and apply more sophisticated methods such as reverse automatic differentiation.

We give a link to a jar containing complete Scala source code including this example, the DualNumber implementation, a conjugate gradient implementation and some JUnit tests (all under a GPLv3 license) and will go on to describe some of the design decisions. The code is the bulky part of this work, so we will move on to discuss something more compact: types.

Types

If code is ever beautiful it is only when it is succinct. Among the most succinct forms of code are individual type signatures and interfaces (though the indiscriminate repetition of type signatures is rightly considered ugly bloat, which Scala works to avoid). Since we are distributing complete source we will describe only types and method signatures. The entry points to the code are the JUnit tests (organized in the ScalaDiff/test source directory and depending on JUnit which was not included) and the demo program in ScalaDiff/src/demo/Demo.scala).

To be a usable arithmetic type (like DualNumber or MDouble) you must extend the following parameterized abstract class:

abstract class NumberBase[NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  // basic arithmetic
  def + (that: NUMBERTYPE):NUMBERTYPE
  def - (that: NUMBERTYPE):NUMBERTYPE
  def unary_-():NUMBERTYPE
  def * (that: NUMBERTYPE):NUMBERTYPE
  def / (that: NUMBERTYPE):NUMBERTYPE  // that not equal to zero
  // more complicated
  def pow(that:Double):NUMBERTYPE
  def exp:NUMBERTYPE
  def log:NUMBERTYPE // this is positive
  // comparison functions
  def > (that: NUMBERTYPE):Boolean
  def >= (that: NUMBERTYPE):Boolean
  def == (that: NUMBERTYPE):Boolean
  def != (that: NUMBERTYPE):Boolean
  def < (that: NUMBERTYPE):Boolean
  def <= (that: NUMBERTYPE):Boolean
  // utility
  def field:Field[NUMBERTYPE]
}

In particular DualNumber extends NumberBase[DualNumber]. This deliberate circular reference has a big purpose: it allows publicly visible contravariant return types (returning nearly the exact type we really are instead of a base type). This allows us to have strict type arguments so that trying to add a MDouble to DualNumber is a type error (even though they both extend the same base class). The automatic differentiation technique encapsulated in the DualNumber class only works if all of the calculation is in the DualNumber types and this strict type enforcement allows the compiler to help prevent results sneaking in and out through other types. All of the methods on NumberBase are obviously related to arithmetic except the field() method. This method gives us access to a Field object which is responsible for carrying around the runtime type information (this is a common problem in Java and Scala, that some type information known at compile type such choice of template types is not easily accessed at runtime). The Field class is as follows:

abstract class Field [NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  def zero:NUMBERTYPE            // return canonical zero in field
  def one:NUMBERTYPE             // return canonical one in field
  def inject(v:Double):NUMBERTYPE  // return canonical representation of number in field
  def project(v:NUMBERTYPE):Double // return standard-number represented in field
  def array(n:Int):Array[NUMBERTYPE] // return an array of this type

The Field class is where we have factories for numbers (zero, one, arrays, injection from standard Doubles), casting (projection back to standard Doubles).

With these types defined we can actually read intent off some of the method signatures.

For example our conjugate gradient optimizer is accessed through the following method signature:

 def minimize(fn:VectorFN,x0:Array[Double]):(Array[Double],Double) // return x,f(x)

The above can be read as: CG.minimize() requires a VectorFN (our trait representing single argument functions with a free type parameter) and an initial point (in standard Doubles). The code will the return a pair of the optimum point and the function evaluated at the optimum point. From the type signature we can see that CG.minimize() expects to re-specialize the function “fn” to types of its own choosing (else it could have accepted a parameterized argument instead of our custom trait) and will handle all up-conversion and down-conversion between machine Doubles and NumberBase[Y]‘s itself. This sort of type information is hard to express (let alone enforce) in a dynamically typed language.

A slightly more complicated example is the lineMinD() method:

def lineMinD[Y<:NumberBase[Y]](field:Field[Y],
 f:Array[Y]=>Y,
 xm:Array[Double],
 di:Array[Double]):(Array[Double],Double)

Notice it is willing to work with any type parameterized function (which means it is willing to let the caller pick the actual type of NumberBase[Y] and work with that). Most callers will call with Y=MDouble (the wrapper for machine Doubles) and lineMin() will then work with that (without ever really knowing the actual underlying type).

A lot of fans of dynamic languages consider type systems to be mere hairshirt penance. But that is not so. Broken type systems (like Java’s collections before erasure parameters were introduced in Java 1.5) are indeed more trouble than they are worth. Working type systems (like C++ Templates/STL, Java 1.5+ and Scala) allow you to solve problems (and enforce decisions) during the design phase (which is much much cheaper than during the deployment phase). You can’t set your types in stone (you are likely going to have them subtly wrong for the first few iteration). You must be willing to think like a “language lawyer” to find out what parts of your work can be specified and enforced in the language type system. To use an analogy: static types are your blueprint or your underpainting.

Tests

One argument against static types is that you can get much of their benefit from unit tests. My opinion is you never have enough unit tests, so putting more pressure on your test suite is not wise. Static types plus tests are strictly more powerful than static types alone or tests alone.

Even for this example toy-scale project we have include a JUnit test set to pursue a number of goals:

• Confirm our number implementations (DualNumber and MDouble) correctly model machine Doubles (perform parallel calculations and compare).
• Confirm DualNumber obeys expected laws of algebra composition and cancellation including the portions that can not be modeled in machine Doubles.
• Confirm DualNumbers compute gradients.
• Confirm operations of optimizers and optimizer components.

Many of these tests are related, but they don’t all imply each other and give different perspective on the errors they catch. For example no amount of parallel computation between DualNumbers and machine Doubles is going to confirm the infinitesimal portion of the DualNumber is propagating correctly (since this is not a property of machine Doubles). So we add extra tests that expect DualNumber to obey algebraic relations like: a*(b+c) = a*b + a*c hold. It is then another step to confirm that whatever the DualNumbers calculate is not only self-consistent, but also models a truncated Taylor Series or differentiation.

Conclusion

We hope we have demonstrated how the complexity of a mathematical programming problem can be managed by breaking the problem into an objective function that is separate from the optimizer (allowing the optimizer to be both good and hidden) and a static type system (such as Scala) to help enforce required properties of a calculation (such as all numbers being routed though a required representation). With these sort of tools available many formerly hard problems (that are often, unfortunately solved by over-specifying direct inefficient iterative improvement techniques) become “if I can write a reasonable objective function this may already by solved by an optimizer in my library.” The more of these tools you have (either in your code or in your reference library) the more of these problems become easy (this is the topic of my earlier paper: The Local to Global Principle).

Appendix: Fixing Smoothness

Our chosen example objective function is very nice (i.e. convex) but it has a small (but correctable) problem. The derivative or gradient or gradient has some jump discontinuities that could cause an optimizer to exit prematurely (not at the global optimum). Consider the simple form of this for wiring a center to a single point at the origin (even in 1 dimension). The wiring cost function is sqrt(x*x) has a cost graph as shown here.

This is convex- but derivative is not smooth as we see in the included graph of the derivative of sqrt(x*x).

So: in this case if the optimizer stops at one of the target points we can’t be sure that it stopped at the global optimum (it may have stopped due to the discontinuity in the gradient). For some simple problems the optimum is necessarily at a target point. For example on the number line take the target points 0,1 and x. As long as x≥0 and x≤1 the optimum placement will be x itself.

One way to defend against this is to use some sort of smoothed version of sqrt() that essentially decreases a little faster near the origin. Our cost function becomes:

Related posts:

1. Gradients via Reverse Accumulation
2. R examine objects tutorial
3. Survive R

Postel’s Law (also called The Robustness Principle) comes from RFC 761 “Transmission Control Protocol” in 1980 and is: “Be conservative in what you do; be liberal in what you accept from others.” (Side note: RFC is the now ironic acronym used to describe Internet standards- the letters stood for “request for comments”).

This idea probably worked best where it started- in the TCP/IP world which is where a lot of hairy details of how computers network are handled. When your goal is the basic establishment of transient communications- success is measured by getting information through and not unnecessarily triggering a failure. It may be okay to tolerate mistakes here- because they don’t live long anyway.

Unfortunately, the law works less well other places where it is applied. The law is a downright hazard in dealing with archiving meaningful data (instead of managing transient signaling protocols). Sometimes the cost of obeying the law far outweighs the potential benefit.

A common arena for Postel’s law is now HTML, the markup language used to represent the content we view in web-browsers. In this arena Postel’s law has had two consequences: one good, one bad.

The good: almost anyone can create a working web-page or even a web-site because modern browsers have been designed to paper around almost every common HTML mistake. It has been pointed out that this ease of creation and “worse is better” (a deep principal due to Richard P. Gabriel see wikipedia: worse is better) has been one of the reasons that HTML out-competed and killed many other ideas. Philip Greenspun’s famous story of a 10-year-old building web site to get his mother medical attention happened in the sloppy world of HTML and could not have happened in the straight jacket of RDF (Resource Description Framework: the darling of the semantic web). I would not wish having to actually read or adhere to the incredibly long and irrelevant standards from w3.org (where the ratio of value to pedantry goes to zero) on an enemy. The web is only interesting due to its content and much of its content was only possible due to low barrier of entry.

The bad: to read HTML you almost have to re-create the entire history of web-browsers. This is a history of many hostile competitors (Microsoft, Netscape, Opera, WebKit, Mozzila, Google) and billions of dollars. Reproducing a significant fraction of this history is a significant (and useless) expense. For the most part I use a permissive parsing library like TagSoup or HTMLTidy but even these miss some things that browsers accept and are far more complicated than the task truly justifies.

Even worse is the cases of XML and RDF. These are often used for archival storage of semantic data. That is you may need to read and understand (not just display) data in XML for a long time. To be liberal in what you accept you have to again master a long set of useless complications (DTDs, namespaces incredibly inept character encoding and escapes) and still get burned by improperly encoded XML (that “used to work” because the bugs in the emitted XML matched the bugs in a library that is now out of date).

It is clear in the case of HTML and XML that Postel’s law’s cost is too high for what it delivers. Or at least half of the law is too expensive: no amount of being generous in what we accept makes up for the original data not have been impounded correctly (not being “conservative in what they do” and not having checked that at the time it was created). Some of this is that the producers of the data have no way of telling they are not being “conservative in what they do” because the “generous in what they accept” libraries they use to debug don’t tell them they are emitting bad data. And lets be honest- most systems are not designed for correctness, they are instead debugged until they seem to work. I would say that in fact HTML is not an example of the power of Postel’s law but of the pernicious influence of “worse is better.” Computer science has not risen to the level of “software engineering” we still are a horrible “fit to finish” industry.

Frankly for many things we need a simpler “fail early” discipline. Tools need to be better and standards need to be simpler so that if you write something that is wrong it is easy to see why it is wrong and easy to fix it. Postel’s law has helped hide the negative impacts of complicated standards, we need to push the cost of complications back on to standards committees. The need to be “generous in what you accept” overly favors large, rich entrenched players who have had the time and resources in incrementally invest in papering around every common mistake.

However, I am not sure if we can throw out half of Postel’s law or even if we want to. When Postel’s law fails it is not clear who to be mad at.

Sun, to kick somebody who is already down, was famous for making elaborate frameworks that correctly and brutally implement many details of RFCs. Sun’s Java includes huge frameworks for XML, UTF8 and email that scrupulously implement page after page of useless standard documentation but fail in the wild due to not being “generous in what they accept.” For example Sun’s GlassFish (which got listed named as one of four or five important assets during Sun’s various acquisition talks much like the fact the car has cup-holder somehow always gets mentioned in spec sheets) is an “open source production-quality enterprise software application server.” A supposedly major component of the GlassFish is its email component which is a huge unwieldy framework that implements many of the email related RFCs and protocols including IMAP. Unfortunately for all its hugeness it can not reliably read email folder names from one of the biggest IMAP servers: Google Mail. Google Mail includes “against standard” characters in the protocol and crashes the GlassFish software.

And here is where Postel’s law fails us: under Postel’s law both sides are at fault (Sun for failing to be generous in what they accepted, Google for failing to be conservative in what they did). We can’t assign only one villain. We have no proscription of who to ask for a fix. Postel’s law seems useful in that if either Google or Sun had followed it the two systems would work. But the law doesn’t pick one side to assign blame and help us to efficiently diagnose and fix the problem. It becomes difficult to find the critical bugs when they are masked by a see of “acceptable” bugs. Take a contrary example: the simpler law “implement the standard or fix the standard” would clearly assign blame to GMail.

Similar pain is encountered in Java’s handling of character encodings like UTF8. It is hard to move up the stack of artificial intelligence (from words, to concepts, to ideas, to reasoning to consciousness) when you can’t even reliably transcribe characters. When faced with bad character sequences (a common occurrence on the web) there is no practical way to get Java “mostly parse it,” Java libraries and frameworks authors seem to extract a perverse joy in throwing a program-killing exception (it does not matter if you catch it the library has already stopped doing what you wanted) because they are concerned that a diacritical mark was not properly encoded (web browsers, on the other hand, lose the mark or show some sort of damage near the mistake and blunder on). And here is were the frustration sets in, how can you make applications that are generous in what they accept when the libraries and frameworks are overly proud and picky? This, at first, seems like an argument for Postel’s law- if everybody else (especially the library authors) were generous in what they accepted your life could be easy. That is certainly one possibility- but I argue it often becomes a matter of semantics to assign blame where there is no pre-existing specification or performance agreement. In the end you will waste more time dealing with errors that should never have made it to you than the time you save emitting the odd error of your own.

The unit testing people have a somewhat better idea: fail early, fail at the factory where it is cheap to fix. Don’t litter all of your code with indecisive statements like:

  Set matches = computeMatches();
  if( matches!=null ) {
     for(String match: matches) {
         ...
     }
  }

Instead: write a unit test to document you expectation that the empty set is expressed in single consistent way:

   Set matches = computeMatches();
   assertNotNull(matches);

And from then on write more confident code:

  for(String match: computeMatches()) {
         ...
  }

This may seem overly optimistic and overly strict- but I have a point. One of the few good principles in computer science (and perhaps one of computer science’s contributions to knowledge, computers are a huge contribution to society- but they were made by engineers) is composition. A plan for getting from A to B followed by (or composed with) a plan for getting from B to C is a plan for getting from A to C. Well a correct plan for getting from A to B when composed with a correct plan for getting from B to C, if each of the plans “is mostly right if the piece after is so nice to fix up a few mistakes” you really don’t know what you have. You may have nothing.

That is my complaint- you can’t put an a priori bound on how expensive attempting to allow both sides of Postel’s law will be. You would like others to paper over your mistakes, but it is becoming too expensive to paper over the mistakes of others. In the end Postel’s law is of little help when cleaning up the inevitable mess.

Related posts:

1. Map Reduce: A Good Idea
2. R annoyances
3. Something I don’t get about business and bailouts

The Local to Global Principle

John Mount¹

Date: November 11, 2009

Abstract:

Contents

• Introduction
• The Examples
  • Web Page Link Analysis
  • Natural Language Processing
  • Machine Learning

• Some Methods
  • Local Methods
  • Globalization Methods

• Conclusion
• Bibliography
• Acknowledgement

Introduction

A common vain hope of computer scientists and algorithm designers is that a domain expert has already “boiled down” a problem to a precise, but unsolved, algorithmic core. On this point the mathematician Gian-Carlo Rota wrote:

One of the rarest mathematical talents is the talent for applied mathematics, for picking out of a maze of experimental data the two or three parameters that are relevant, and to discard all other data. This talent is rare. It is taught only at the shop level.[Rot97, ``A Mathematician's Gossip'']

We describe a useful tool for designing algorithmic applications and solutions which we call “the local to global principle.” The local to global principle is the method of deriving applications and solutions by specifying “local” (and deliberately myopic) heuristics, critiques and methods followed by using a powerful general method to “globalize” this specification into a complete solution.

There are many important problem solving prescriptions and methods of thought already systematically described and taught:

• Bacon’s “New Organon” and Mill’s principles of inductive logic.[Mil02]
• Feynman’s genius method.[Rot97, ``Ten Lessons I Wish I Had Been Taught'']
• Reductionism (top down and bottom up).
• Divide and conquer.[CLRS09]
• Forward deduction, backwards induction.
• Root Cause Analysis.
• Polya’s heuristic and conjecture and prove patterns [Pol71,Pol54a,Pol54b]
• Doron Zeilberger’s “Method of Undetermined Generalization and Specialization.” [Zei95]
• Zbigniew Michalewicz and David B. Fogel’s presentation of evolutionary algorithms.[MF00]

The local to global principle is more of an organizational pattern than “computer aided technique” as no one specific species of software or family of notation is required.

The local to global principle can be identified in a number of previous important applications, but it is not currently an identified principle.² The principle is very general, so any succinct description of it is going to be painfully vague. Instead, we explain the principle by discussing some example applications and methods. For each of our example applications we deliberately use a different globalization technique. The effective algorithmist or practitioner must in fact come to each problem already familiar with a reasonably large set of already known local and global techniques, so we conclude with some appropriate fields of study and preparation.

The local to global principle is divided into two parts: local encoding of the problem followed by a globalization step that uses the encoding. The guiding feature of local encodings is that they are usually easy to compute from the data at hand. Any extension that looks like enumeration, search or optimization is best left to the global step. The local step is essentially the translation of your problem into an abstract language that is ready for the globalization step. In contrast globalization methods are often “off the shelf” in that once you abstract and encode the particulars of your problem you can look for pre-existing useful methods or software to finish your solution. The idea of globalization is to find a best overall or global compromise between competing local criteria. The local step does not so much have to avoid conflicts but instead “price them.” There is also an important trade-off that sophisticated local techniques allow the use of simpler globalization methods and more powerful globalization methods allow the use of simpler local techniques.

The Examples

To demonstrate the breadth of the local to global principle we choose a diverse collection of example applications: web page link analysis, natural language processing and machine learning. For each example application we will set up the problem, introduce a reasonable set of local criteria and pick an appropriate globalization technique. We will favor finishing each example without describing the globalization technique in detail, as this would distract from our point and is best left to the given references. These examples are previously solved problems, our contribution is demonstrating the shared underlying principle.

Web Page Link Analysis

For our first example application we demonstrate web page link analysis in the form of the famous PageRank score.[PBMW98]

One of the many good ideas leading up to the early Google search engine was the design of a non-text based measure of importance or interestingness of web pages. A search engine that could fold “interestingness” or popularity into its notion of relevance could better sort important pages into the search user’s view. When the web got so large that there were many pages that were exact matches to any common user query popularity became a critical consideration. A link based notion of popularity exploits what is important about the web (the link structure, for example see [Kle97]) and avoids having to depend on a lot of natural language understanding technology. This technique also uses authority outside of the given page, so has some hope at being resistant (though not immune) to web-spam.

Taken all at once, the task of designing a score of page importance is a daunting task. However, by working in stages (as the local to global principle prescribes) we can quickly derive interesting scores including the famous PageRank score. We start with the idea that popularity (or the amount of web traffic a page receives) is (loosely) correlated with importance. So for our first approximation step we decide to try to estimate popularity (or web traffic) and use this estimate as our importance score. Accurately estimating web traffic is itself a hard problem and a big industry (just a few of the major companies involved in this are: Google/Urchin, Quantcast, Nielsen, comScore, Alexa, Hitwise and LookSmart). For our second approximation step we are going to try and estimate popularity from the link structure⁴ of the web (using no other measurements or historic data) and use this as our score. This link based estimate is unlikely to completely reproduce real web surfing patterns, but it is very interesting in its own right and has been proven in the market to be a useful score.

Now the problem is to try to estimate the popularity of a web page from the link structure of the web. We claim: we can generate a useful (but not necessarily accurate) estimate of web traffic from the web’s link structure alone. Consider Figure 1 where we have a universe of three web pages A,B and C that link to each other in the pattern illustrated by what is called a graph⁵

Figure 1: A set of Mutually Linked Web Pages

In Figure 1 we can consider each link to another page as evidence the other page is interesting or popular. One idea is to simulate a very simple web surfer who clicks on the links on a page uniformly at random. This is called “the random surfer model” and even a model this simple allows us to read some useful information from the link structure of the web. For instance, we could ask what fraction of their time the random surfer spends on each web page, with an eye to the idea that the pages the random surfer visits more often are the more important ones. Let denote the proportion of time the random web surfer spends on page A (and define and similarly). While we do not know any of
or we can derive some relationships between them by inspecting the link graph:

The first equation is just reading from the graph that: all visits on page-A must come from pages B and C, half of the visitors on page-B continue on to A and all of the visitors on page-C continue on to A. The second and third equations are the appropriate summaries of how traffic is routed to pages B and C. We can insist that
as we want these numbers to represent the fraction of time the random web surfer spends on each page. A more sophisticated model would add more features⁶ to get a more useful result.

It turns out we have already encoded enough local rules to completely determine
and . In this example application an algorithmist already familiar with linear algebra [Str76] would recognize these local conditions as “a system of linear equations.” Solving even web-scale systems of linear systems is considered easy with modern techniques and modern computers. For our small example example the solution is:
,
, and
. The role of the local steps was to reduce a new problem (estimating the importance or popularity of web page from the link structure) to something with its already known known techniques (like solving a linear system as illustrated in Figure 2).

Figure 2: Linear Algebra Solution: As Taught in School

So page-A is the most important page by the PageRank measure.

In this example application the local step was setting up the system of linear equalities (which are easy to derive from the web link graph) and the global step was solving the entire system for the final scores (which were not obvious). You spend most of your time encoding the problem and then use a known technique (in this case solving a linear system) to finish the solution.

Natural Language Processing

Our next example application is natural language processing [Cha96,Cha97]. Speech recognition (the alignment or transcription of recognized intelligible segments of sound to written text) is an important problem in natural language processing. An example problem is the need to find the most likely text matching a sequence of sounds such as is shown in Figure 3.

Figure 3: A Sequence of Sounds

Consider Figure 4 (which shows a bad transcription) and Figure 5 (which shows a good transcription).

Figure 4: A Bad Transcription

Figure 5: A Good Transcription

Our claim: we can (given access to training data, and this is the age of data [HNP09]) solve this problem with a local step that is a set of simple criticisms of proposed transcriptions. A good starting point is a database of previous sounds to text transcriptions. This database allows the construction of a series of tables that give the historic frequency (or probability) of all of the following:

• Prior probability of each sound
• Probability of each sound given the immediately previous sound
• Prior probability of each word
• Probability of each word given the immediately previous word
• Which combinations of word fragments are legitimate words
• Probability of each sound being assigned to each word fragment (syllables, phonemes and so on).

These tables encode a “speech model” (the rules involving sounds only), a language model (the rules involving text or words only) and the linkage between the two models. These models are deliberately simple in that they capture only local interactions (like probability of a word given the word before it) but no long range interactions (like subject predicate agreement).

Each box, nested box and arrow on our diagram represents one possible local critique. For each item in our diagram (again, the boxes and arrows) we can use our tables to assign a goodness or plausibility score. For instance bad word to word transitions (like “won”
“won”) will be rare in our historic tables so, just looking up probabilities from the tables (or, better, using the logarithms of probabilities) gives as a “plausibility score” that prefers known patterns of language. Then a score for the overall transcription can be derived by multiplying all of the local scores together. These local scores (though simple) already have encoded enough evidence to prefer the good transcription to the bad transcription without requiring any deep knowledge of speech, text or the meaning of the text. This is because the bad transcription has a series of obvious flaws such as: unlikely sound to word fragment assignments and unlikely word to word transitions.

Figure 6: Naively Extending a Partial Transcription

For example consider Figure 6 where a naive solver is in the process of considering selecting the word “one” as the third word to fill in. The only local critiques they need to consider are:

• how likely the word “one” is in general (call this )
• how likely the word “one” is to follow the word “nine” (call this
  )
• how likely the letter sequence “o” is given the sound “w” (call this
  )
• how likely the letter sequence “ne” is given the sound “n” (call this
     n ).

So the local plausibility of the fill-in word “one” is:
. We will call this the critique of “one” in position 3 and write as
where is the word known to be in position 2. Similarly we can generate all of the possible critiques ,
,
,
and the overall criticize of a sequence
:
from our pre-computed tables of probabilities. Notice for all of these critiques only the immediately previous word and the nearby sounds were used to determine the plausibility of the word we are attempting to fit in. Instead of using these critiques to directly fill in a possible solution (or using search) we will package up these critiques (in the form of the ) and pass them on to a powerful separate globalization step called Dynamic Programming [Bel57].

The globalization or finding of a best overall transcription is not trivial even though our score is simple. This is because the overall best sequence could depend on clever non-local fill-ins (like deliberately picking a less likely first word to allow a later favored transition to a fantastically good third word). Dynamic Programing does not fill in the transcription from left to right, but instead uses a table of scores derived from the left to right arrows and the . In our example Dynamic Programming consists of building a table of information as shown in Figure 7. Let represent the word position we are working looking at (so ranges from 1 to 4) and let be a variable that ranges over every word in the dictionary. Our table is indexed by and and when filled in stores what the highest “plausibility score” of a partial sequence of words where words 1 through have been filled in and the -th word is .

Figure 7: Dynamic Programming: Back Chaining in for a Solution

If we already had this magic table we could find a best possible sequence by “back chaining.” We start by finding a fourth word ( ) such that is maximal (in this case “one”). We then find a best third word ( ) by enumerating all words and picking such that
. We continue back until we had found words and to get a complete best sequence. Notice that we work from right to left (backwards) and except for the starting step we pick each word to match the calculation we are trying to un-roll, not to be the maximal entry in the column. For instance we pick
even though it does not have a the highest score, but because
is the maximal complete chain.

Of course, we don’t start with the table already filled in- so we need a procedure to build it. This procedure is the heart of the Dynamic Programming method (for more examples see: “Introduction to Algorithms” [CLRS09]). Notice that can be filled in for all just by plugging in words and computing the critiques (i.e.
). Once all the are filled in we can fill in the the with the general (and slightly trickier) formula:

as we illustrate for in Figure 8.

Figure 8: Dynamic Programming: Building the Table

The magic of the Dynamic Programing technique is: by being careful to not store too much in the table we avoid an explosion in record keeping that would render the method inefficient. Dynamic Programming exploits the small dependence structure encoded in (each box in our diagram depending on only a few arrows) and as we have shown can find “clever” solutions (such as taking a sub-optimal first word to get better transitions into preferred later words). For those who want more detail on solving this problem we recommend [Cha96] (as our goal is not to fully explain Dynamic Programming, but to demonstrate how it could be applied to the transcription problem as a pre-packaged globalizer).

In this example the local step was the graph link based critiques and the globalization step was Dynamic Programming. The separation of concerns from the local scoring to the globalizing step is a strength of the local to global principle.

Machine Learning

Our final example application is machine learning. Machine learning is loosely defined as computer programs that adapt or learn from data. Thomas Mitchell helps distinguish this activity as a specialty of artificial intelligence that concentrates on “well-posed learning problems.” [Mit97] Trevor Hastie, Robert Tibshirani, Jerome Friedman emphasize the relation to statistics (versus more traditional symbolic AI) [TH09]. A simple demonstration can be found in [Mou09b].

Machine learning is perhaps the strongest example of the local to global principle and is inspired by the work of Kristin P. Bennett and Emilio Parrado-Hernandez [BPH06]. In hindsight many machine learning algorithms (each of which has had a turn at being “the most exciting breakthrough ever” for a while) can be seen as the pairing of a performance criterion (which we call a local criterion as it applies to one specific set of parameter values at a time) and an optimization method (what we have been calling the globalization step). The work of Bennett and Parrado-Hernandez calls this distinction out and shows how it is not productive to present machine learning systems as unique named monolithic units, but instead to consider how to break them into an objective function and an optimizer. This allows both choice of better optimizers (such as replacing the inferior method of gradient descent method wherever it occurs) and for explicit control of important concepts such as hypothesis regularization and control of over-fitting (which some algorithms claim to achieve by deliberately using early exit from a an inferior optimizer).

At a “30,000 feet level” we can build a table of common machine learning techniques and name what is commonly used to implement their local and global steps. When a machine learning algorithm is defined by what conditions are meant to be true at the optimum we are no longer bound by details of the original implementation and can examine fix and improve the components.⁷ Table 1 is a crude summary of a wide selection for machine learning algorithms that may be more likely to offend everybody than just offend somebody. But this is also the point: it is the algorithmist’s job to think fluidly (beyond given names and provenances) and to invent scaffolding to convert partial analogies into practical correspondences.

This table is a necessarily crude summary. For example: notice that several known techniques can not even be distinguished from each other by the local and global columns of the table.

There are a few points we would like to make. Back propagation was considered unique to Neural Nets for quite a while because it was so entwined with the technique it was not recognized as the simple application of Automatic Differentiation [RC96] that it is. Support Vector Machines (SVM) are remarkable for their uniform very good choice of component methods (maximum L1 margin objective regularization, Kernel Methods [STC04] and sophisticated optimization methods [Joa06]). Many of the machine learning methods that SVM outperforms become again competitive when they adopt some of SVM’s technologies (especially using kernel methods to produce synthetic features).

Beyond these points we invoke a “globalizers are pre-packaged” principle and leave the discussion of machine learning and optimization to our reference: [BPH06]. In this example the local step is a per-example score or penalty and the globalization step is optimization.

Some Methods

The application of the local to global principle is similar to the Feynman “genius method.” Feynman’s method is to always have in mind a list of problems and a list of solution methods. The genius step is: anytime you see a new problem or a new solution method to immediately try it against every item from the complementary list. [Rot97, ``Ten Lessons I Wish I Had Been Taught''] This deliberate retention and activity greatly increases your problem solving ability. The power of the local to global principle is itself proportional to the number of local methods times the number of globalization strategies. Of course, to even start: the practitioner must already have available a number of candidate local and globalization methods. We list some methods and some guidance on variation and invention.

Local Methods

Good sources of ideas and analogies for local methods include:

• Introduce a Graph Structure

  A graph structure is a network of nodes connected by edges. Use of graphs was demonstrated both in the natural language processing and web page link analysis examples. We can dress up how we solved these problems and say we used a “Hidden Markov Model”, but the real power was we encoded our problem in a simple graph. Some problems (especially those from logic or those involving time) are essentially solved once they are translated out of their original form and into graph notation (for an example see: [Mou00]).

• Appeal to Physical Conservation Laws

  A good example physical law is Kirchhoff’s law or conservation of flow. All of the web page link analysis’s equations were derived by saying that the attention of at node is essentially the sum of attentions from other nodes (more sophisticated versions of the analysis actually do create and destroy flow, but they do it in a principled way).

• Encode the Problem into an Objective Function

  This method is essentially your declaration that you intend to use an optimizer for the globalization step. In operations research this specific technique has long been the practice (with no disrespect: a very productive part of operations research has been translating different problems into linear programs so the simplex method can be applied, for an example see [Mou09a]).

• Gradient Like Computations

  Includes Gradients, Secants, Lagrangians and other ideas from calculus. Gradients can drive optimizer based globalizers and techniques like Lagrangians are often powerful enough use mere inspection as the globalization step.

• Violation Driven Updates

  This method is particularly effective when your problem is not amenable to continuous optimization. A good example is the Lin-Kernighan heuristic for solving the traveling salesman problem.[LK73] This heuristic looks at subsets of the problem and suggests improving “surgeries” (until no more such improvements are possible).

• Introduction of Symbols

  Often, as with the web page link analysis example, you can not specify specific values for the unknowns, but you can specify relationships. You often can then solve for the symbols or introduce additional conditions and use an optimizer to complete the solution (see for example the maximum entropy method as described in [Ski88]).

• Over Specification

  If we anticipate using a global step like search, enumeration, summation or integration then over specification is a good local idea.

  For example: consider computing the probability that a fair count flipped 10 times comes up with heads exactly 3 times. The easiest way to perform this calculation is to specify exactly which 3 coins come up heads (the local over-specification) and then sum over all choices of 3 out of 10 coins (the global step). In mathematical notation this is:

  exactly 3 heads out of 10 flips

  or just under 12%.

• Under Specification

  One of the core principles of Dynamic Programming is to forget as much as possible about partial solutions, keeping only partial solution cost and just enough information to extend partial solutions. If you anticipate using something like Dynamic Programming as your globalization step then your goal should be to under specify.

• Tables

  A key step of the natural language processing example was the use of tables of past experience to determine which sounds likely corresponded to which words, which words likely followed each other (and so on). Encoding domain knowledge or expertise as probability tables is a very effective problem solving strategy (especially if the globalization strategy is going to be search or Dynamic Programming). In natural language processing examples tables and statistics are much easier to manage than comprehensive rules or grammars.

• Set up as Ranking or Machine Learning Problem

  This tactic is especially appropriate if your solution success metric is counts, frequencies or probabilities (instead of having to always be correct or always be optimal).

Globalization Methods

The universe of possible globalization methods is very diverse (in particular globalization is not always optimization).

• Search / Enumeration

  Search can be slow, but it is always an option to consider. If your problem translates naturally into a graph structure or your solutions are naturally seen as being composed of small pieces search should be considered. One of the big advantages using the local phase to formally encode your problem’s structure and putting search off to the global phase is: you can use advanced search techniques. Once you are freed from your specific problem details it becomes much easier to consider search techniques like branch and bound, A*, game theoretic search and general speed-up techniques like hashing and caching.

• Dynamic Programming

  If your problem has a bit more structure (in that partial solutions summarize and compose easily) then you can likely replace search with Dynamic Programming. The advantage is that Dynamic Programming typically offers an incredible speed up when compared to search.

• Optimization

  If your problem is continuous (involves numbers instead of discrete or categorical decisions), can be encoded as a reasonable objective function (linear, positive definite quadratic) and has reasonable constraints (linear or convex) then you can immediately apply an optimizer as your globalization step. Typical optimization methods include: conjugate gradient, Newton methods, quasi Newton methods, linear programming and quadratic programming.

• Combinatorial Optimization

  If your problem includes a “discrete variables” (that is variables that take on one of fixed set of values instead of values from a numeric range) then you may not be able to apply standard optimization techniques. At this point you may want to use more expensive combinatorial optimization techniques like integer linear programing or constraint satisfaction.

• Fixed Point Methods / Iteration

  Fixed point methods are based on the idea: “incrementally improve until there is no incremental improvement possible.” If the problem is continuous this is similar to steepest descent. If the problem is discrete then this is similar the Lin-Kernighan heuristic.

• Linear Algebra

  The web page link analysis and optimization examples were essentially solved once we reduced them to linear algebra. If you can write your problem as a linear relationship between unknowns or as the fixed-point of a linear operator (i.e. an such that ) then you can immediately use linear algebra to solve the problem at very large scale (e.g. web scale).

• Sampling / Problem Kernels

  A very successful line of attack on large problems is to reduce to a smaller problem containing most of the essential difficulty. David Karger has produced a number of effective algorithms for graph cuts and flows using a theory of sampling [Kar98]. Rod Downey and M. Fellows have demonstrated an effective theory of “problem kernels” that finds solution by focusing on smaller sub-problems (on which we can afford to use more expensive procedures).[DF98]

• Amortized Analysis / Economic Mechanism Methods

  Daniel Sleator and Robert Tarjan’s ideas of amortized analysis [ST85] allow approximation schemes similar to problem kernels. The method is to approximately optimize by pairing a bunch of unavoidable large penalties (conditions we can’t meet) with some accounting credits (say bonuses from other conditions we are meeting very well). We then isolate these paired items and optimize the rest of the problem exactly. The technique often works by showing the approximation can not be too bad because, due to the pairing of large penalties to good credits, there can not be too many large penalties. An informal example is: if it is impossible to pick someplace where all of an office will eat for lunch, perhaps you can solve the problem by paying one person to accept a restaurant they do not like (if the removal of their objection opens up a venue that is acceptable to everybody else).

• Relaxation / Homotopic methods

  These methods involve changing hard constraints to soft penalties (so allowing the constraints to be violated, but at a slowly increasing cost). After such a relaxation the homotopic (or continuous deformation) method is to increase the cost of violation and re-solve to try and get a trajectory of better and better nearly acceptable solutions that point to a possible overall solution.

Conclusion

The purpose of this article has been to make more visible an idea we call the local to global principle. This principle is an organizing tool useful both in designing and analyzing a wide variety of applications. Essentially the whole point of this writeup is to set up enough framework to quickly write down a table of advice such as Table 2 (and for such a table to mean something).

The principle is not universal; not everything can be fit into such a table. For example the local to global decoupling is not a feature of the famous EM algorithm [DLR77], which depends on mixing predictions and corrections.

To conclude: the recipe is as follows. If you come to a problem with a large shopping bag of possible ways to build local criteria and powerful globalization procedures then you stand a very good chance of solving the problem quickly. Also, if you keep the local to global principle in mind you are more likely to identify and retain potential local tricks and globalizers when you see them and thus have a larger more nimble set of tools available to solve problems when the time comes.

Bibliography

AC06

Nir Ailon and Bernard Chazelle, Approximate nearest neighbors and the fast johnson-lindenstrauss transform, STOC (2006).

AI06

Alexandr Andoni and Piotr Indyk, Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions.

BD02

Avrim Blum and John Dunagan, Smoothed analysis of the perceptron algorithm for linear programming, SODA (2002), 11.

Bel57

Richard Bellman, Dynamic programming, Princeton University Press, 1957.

BF97

Leo Breiman and Jerome H Friedman, Predicting multivariate responses in multiple linear regression, Journal of the Royal Statistical Society, Series B (Methodological) 59 (1997), no. 1, 3-54.

BFSO84

Leo Breiman, Jerome Friedman, Charles J. Stone, and R. A. Olshen, Classification and regression trees, Chapman & Hall/CRC, January 1984.

BNJ03

David M Blei, Andrew Y Ng, and Michael I Jordan, Latent dirichlet allocation, Journal of Machine Learning Research 3 (2003), 993-1022.

BPH06

Kristin P. Bennett and Emilio Parrado-Hernandez, The interplay of optimization and machine learning research, Journal of Machine Learning Research 7 (2006), 1265-1281.

Bre00

Leo Breiman, Special invited paper. additive logistic regression: A statistical view of boosting: Discussion, Ann. Statist. 28 (2000), no. 2, 374-377.

BRS91

R Beigel, N Reingold, and D Spielman, The perceptron strikes back, Structure in Complexity Theory Conference 6 (1991), 286-291.

Cha96

Eugene Charniak, Statistical language learning, MIT Press, 1996.

Cha97

to3em, Statistial techniques for natural language parsing, AI Magazine 18 (1997), no. 4, 33-44.

CLRS09

Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein, Introduction to algorithms, MIT Press, 2009.

CSS02

Michael Collins, Robert E Schapire, and Yoram Singer, Logistic regression, adaboost and bregman distances, Machine Learning 48 (2002), no. 1/2/3, 30.

CV05

Rudi Cilibrasi and Paul M.B Vitanyi, Clustering by compression, IEEE Transactions on Information Theory 51 (2005), no. 4, 1523-1545.

DF98

Rod G. Downey and M. R. Fellows, Parameterized complexity, Monographs in Computer Science, Springer, November 1998.

DLR77

A P Dempster, N M Laird, and D B Rubin, Maximum likelihood from incomplete data via the em algorithm, Journal of the Royal Statistical Society, Series B (Methodological) 39 (1977), no. 1, 1-38.

Fis36

Ronald A Fisher, The use of multiple measurements in taxonomic problems, Annals of Eugenics 7 (1936), 179-188.

FS99

Yoav Freund and Robert E Schapire, A short introduction to boosting, Journal of Japanese Society for Artificial Intelligence 14 (1999), no. 5, 771-780.

GD04

Peter D Grunwald and A Philip Dawid, Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory, Ann. Statist. 32 (2004), no. 4, 1367-1433.

Gru00

PD Grunwald, Maximum entropy and the glasses you are looking through, Conference on Uncertainty in Artificial Intelligence (2000), 238-246.

HNP09

Alon Halevy, Peter Norvig, and Fernando Pereira, The unreasonable effectiveness of data, IEEE Intellegent Systems (2009).

Hus99

Dirk Husmeier, Neural networks for conditional probability estimation, Springer, 1999.

IM99

Piotr Indyk and Rajeev Motwani, Approximate nearest neighbors: Towards removing the curse of dimensionality.

Joa98

Thorsten Joachims, Making large-scale svm learning practical, Advances in Kernel Methods – Support Vector Learning (1998).

Joa06

to3em, Training linear svms in linear time, KDD (2006).

Kar98

David R Karger, Randomization in graph optimization problems: A survey, Optima: Mathematical Programming Society Newsletter 58 (1998).

KCVM04

Trausti Kristjansson, Aron Culotta, Paul Viola, and Andrew Kachites McCallum, Interactive information extraction with constrained conditional random fields, AAAI (2004).

Kle97

Jon M Kleinberg, Authoritative souces in a hyperlinked environment, ACM SIAM Symposium on Discrete Algorithms (1997).

Kom08

Paul Komarek, Logistic regression for data mining and high-dimensional classification, CMU CS Thesis (2008), 138.

KWA95

J Kivinen, Manfred K Warmuth, and P Auer, The perceptron algorithm v.s. winnowo: Linear v.s. logarithmic mistake bounds when few input variables are relevant, COLT (1995), 289-296.

Lew98

David D Lewis, Naive (bayes) at forty: The independence assumption in information retrieval, find journal (1998).

LK73

S Lin and BW Kernighan, An effective heuristic algorithm for the traveling-salesman problem, Operations Research (1973), 498-516.

Mar61

M E Maron, Automatic indexing: An experimental inquiry, RAND Technical Report (1961), 404-417.

MF00

Zbigniew Michalewicz and David B. Fogel, How to solve it: Modern heuristics, Springer, 2000.

Mil02

John Stuart Mill, A system of logic, University Press of the Pacific, 2002.

Mit97

Thomas Mitchell, Machine learning, McGraw-Hill, 1997.

MK00

M E Maron and J L Kuhns, On relevance, probabilistic indexing and information retrieval, 1960 (2000), 1-29.

Mou00

John A Mount, Automatic detection of potential deadlock, Dr. Dobbs Journal (2000).

Mou09a

John Mount, Automatic generation and testing of un-rolls for profitable technical trades, http://www.win-vector.com/blog/2007/10/paper-on-stock-trading/, 2009.

Mou09b

to3em, A demonstration of data mining, http://www.win-vector.com/blog/2009/08/a-demonstration-of-data-mining/, 2009.

PBMW98

Lawrence Page, Sergey Brin, Rajeev Motwani, and Tery Winograd, The pagerank citation ranking: Bringing order to the web, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.31.1768 (1998).

Pol54a

G. Polya, Induction and analogy in mathematics, Princeton University Press, 1954.

Pol54b

to3em, Patterns of plausible inference, Princeton University Press, 1954.

Pol71

to3em, How to solve it, Princeton University Press, November 1971.

RC96

Louis B Rall and George F Corliss, An introduction to automatic differentiation, SIAM: Computational Differentiation: Techniques, Applications and Tools (1996), 1-18.

Rot97

Gian-Carlo Rota, Indiscrete thoughts, Birkhauser, 1997.

Ski88

John Skilling, The axioms of maximum entropy, Maximum Entropy and Bayesian Methods in Science and Engineering 1 (1988), no. 173-187.

ST85

Daniel Dominic Sleator and Robert Endre Tarjan, Amortized efficiency of list update and paging rules, Communications of the ACM 28 (1985), no. 2.

STC00

Jown Shawe-Taylor and Nello Cristianini, Support vector machines, Cambridge University Press, 2000.

STC04

to3em, Kernel methods for pattern analysis, Cambridge University Press, 2004.

Str76

Gilbert Strang, Linear algebra and its applications, Academic Press, Inc., 1976.

TH09

Jerome Friedman Trevor Hastie, Robert Tibshirani, The elements of statistical learning: Data mining, inference and prediction, Springer, 2009.

TTV08

Luca Trevisan, Madhur Tulsiani, and Salil Vadhan, Regularity, boosting, and efficiently simulating every high-entropy distribution, Electronic Colloquium on Computational Complexity (2008), 18.

Zei95

Doron Zeilberger, The method of undetermined generalization and specialization illustrated with fred galvin’s amazing proof of the dinitz conjecture, http://arxiv.org/abs/math/9506215, 1995.

Acknowledgement

A thank you to readers who supplied help and comments on earlier drafts.

Footnotes

… Mount¹

email: mailto:jmount@win-vector.com web: http://www.win-vector.com/

… principle.²

The pre-existing practice that comes cloesest to the local o global principle is found in operations research where encoding a problem to be solved by an optimizer is a central technique. We claim the natural statement of the local to global principle is more general than always encoding constraints for a particular optimizer (in particular globalization is not always optimization).

… structure⁴

By “link structure” we mean which web pages link to which other web pages.

… graph⁵

Remember, a graph is diagram consisting of nodes and edges (here depicted as arrows).

… features⁶

For example the model could account for:

• surfers entering and leaving the model
• link odds that vary where they are on a page
• surfers staying on a page proportional to how much text is on the page
• matching known traffic and click behavior where we have such data.

For simplicity we will just stick with the example given example.

… components.⁷

When a system is named and defined as an exact set of procedures the system can, by definition, not be improved. This is because with any change in procedure we have a new system that no longer matches the original definition and therefore requires a new name.

Related posts:

1. Automatic Differentiation with Scala
2. A Demonstration of Data Mining
3. Should your mom use Google search?

The frenzy of anticipation and opinion about “The Cloud” is so intense and so pointless it becomes “parody proof.”
It is as Jerry Holkins and Mike Krahulik wrote (regarding a different situation):

It’s like trying to make fun of a clown. What, are you going to make fun of his tiny car? His floppy shoes? It just doesn’t work.

I would like to point out that (by computer science standards) the cloud is not new and has for some time been considered inevitable.

But what is “The Cloud?” What the cloud is depends a bit on what conversation you are being drawn into. If the conversation is about computing then the cloud is remote computers, software and services like Wikipedia, GMail, SalesForce.com, Google Docs, Amazon EC2/S3 and Google App Engine. If the conversation is about human interaction then the cloud is ecosystems like Facebook, Twitter and RSS. Each of these are facets of important longer term trends, but for individual companies and technologies the pendulum is about as fast on the down-swing as it was on the up-swing. At this time we can safely declare a number of recent important players dead: Friendster, AltaVista, WSDL, Usenet, IRC and Web2.0.

It is true that the network itself is more useful than the computer, but this idea is not new to our third millennium. The current people getting rich promoting this idea did not invent this idea, they grew up in its shadow. The early big thinkers on computers had big plans. Plans much larger than Tetris, payroll processing, COBOL and punched cards.

Take the article “As We May Think” (by Vannevar Bush, The Atlantic Monthly (1945)). In it Vannevar Bush writes:

Consider a future device for individual use, which is a sort of mechanized private file and library. It needs a name, and, to coin one at random, “memex” will do. A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.

At first this sounds like nothing more than “Danny Dunn and the Homework Machine” (by Jay Williams (1964), Scholastic Press)

.

But in his essay Vannevar Bush uses the phrase “it can presumably be operated from a distance” and ends his essay with a long section of how many professions would benefit from a Memex (we show here only one):

Wholly new forms of encyclopedias will appear, ready made with a mesh of associative trails running through them, ready to be dropped into the memex and there amplified. The lawyer has at his touch the associated opinions and decisions of his whole experience, and of the experience of friends and authorities.

Obviously we are reading this with a modern eye, but here we have the antecedents of hypertext and the Wikipedia.

We can trace this thread further forward to “Augmenting Human Intellect: A Conceptual Framework” (by Douglas C Engelbart (1962)) and the famous 1968 demo.

And we can further trace the ideas passing through: “Literary Machines: The report on, and of, Project Xanadu concerning word processing, electronic publishing, hypertext, thinkertoys, tomorrow’s intellectual revolution, and certain other topics including knowledge, education and freedom” (by Ted Nelson (1981), Mindful Press, Sausalito, California.)

These works were all about knowledge engineering, information storage, networking and communication. There was an extreme urgency in these works. Both Engelbart and Nelson felt we had a limited window to gain the ability to organize the world’s information before some catastrophic error or misunderstanding eliminated us all. This feeling of urgency and doom came from another exciting application of real time networked computers: SAGE. SAGE was the “Semi Automatic Ground Environment” first made operational in 1959. It involved networked computers, light pen based operator terminals and was the system that the United States had ready to fight World War III.

This was the era of near infinite budgets, block sized computer complexes, massive mainframes and IT priesthoods that ran the whole show.

The inevitable march was on. Some large fraction of the GDP would be forever dedicated to building and maintaining monument sized networked computing facilities. Your degree of relevance and power in society would be directly determined by how close you could get to these facilities. Then something happened and distracted everyone. The distraction was so immediate and so complete that by the time the inevitable march restarted (block sized Google data centers and a proposed Yahoo data center to be built attached to Niagara falls) everyone thought it was a new thing.

What happened was the 1958 demonstrations of successful integrated circuits. This and the transistor started an era of micro-miniaturization that took the world by storm. By 1971 Intel had released a single chip CPU (the 4004) as a commercial product. This chip implemented the core of a computer in a fingertip size package that contained 2300 transistors.

From here on everything was desktop calculators, pocket calculators and digital watches. And then the personal computer and the personal computer revolution hit.

IBM kicked the PC revolution into high gear when they pushed into the market in 1981. The personal computer was a supreme distraction that pulled attention away from the monolithic computers for fifteen years. And for a long while networking and shared information were both nearly forgotten. Computers were for spreadsheets, desktop publishing and other non-networked tasks.

However, out of public view the monolithic network continued to develop. The Internet was started as ARPAnet and grew connecting universities and defense contractors from 1969 through now. The messaging formats (it is inappropriate to use the more common term “technology” to describe HTTP and HTML) we call “The World Wide Web” were invented (without much fanfare) in 1989. Netscape was founded in 1994 and made the World Wide Web and Internet available to the PC. And then the Internet hit like a Tsunami. Electronic commerce and speculation funded the the initial burst. Then on-line advertising took over and we are back to building new encyclopedias, tracking everyone and once again building city block sized computers (now called data centers).

Once again we are being told our data is too important to be locked in our desk (or PC) and everything is migrating back to the mainframe (now called “the cloud”).

Will the cycle reverse? If applications are moving into the cloud now will they ever move back out?

Moore’s law has a way of shrinking things (a current smart phone outperforms many early mainframes, super computers and data centers). Will individual PCs once again be more important than the network? Some of the more useful parts of the Internet (like the Wikipedia) are small enough to put on current PCs. The data centers and networks will not go away any time soon, but excitement and attention could move on to something else. Devices that you could carry everywhere and that have intermittent or expensive connections to the Internet might have an advantage in being able to cache some of the Internet. And excitement follows what is new, so a stable pervasive cloud would likely be taken for granted (like roads, power, telephone and other utilities).

Another thing that could migrate applications back out of the cloud (assuming they migrate in) is if access to the user becomes too important to delegate to the cloud. eCommerce applications take user access when they can get it, but many other applications may depend more on immediate access to the user than on grabbing fresh data from the network. For example a pacemaker is likely to run most of its application from an embedded computer- this computer might talk to the cloud when it can, but the application will be designed to stand alone as long as possible.

In the end evangelizing the coming triumph of factory scale computing and networking is pointless. It is already here and has no great need for cheerleaders.

Related posts:

1. Postel’s Law: Not Sure Who To Be Angry With
2. Sorting Used in Anger
3. I know, I am the one being a jerk

The longer answer is that a number of times different forms of algorithmic art have been invented. Algorithmic art is art generated by mathematical procedures. Such art is similar to earlier mechanical and kinetic art forms. One branch of mathematics often used to generate such art is called “fractals.” We looked somewhere else for our inspiration (our art is not strictly fractal in nature). What we worked on we called “genetic art” to emphasize the role of encoding and re-combination in the works.

In the early 90′s Karl Sims presented a number of art installations based on at least three interesting ideas:

• Transforming images
• Evolving combinations of transforms
• Direct participation

(see: Karl Sims. Artificial Evolution for Computer Graphics. Proceedings of SIGGRAPH 1991 and Karl Sims’ homepage).

The part that caught a number of people’s imaginations was the evolution aspect. Karl Sims defined a method of combining transformations of original source images. He then allowed people to manipulate his art installations and “vote” on art they liked best. The more popular pieces were combined (or bred) to create newer works that then put up against criticism. After many breedings (or generations) the combinations of transforms were quite complicated and a number of unexpected images were created.

At CMU Shumeet Baluja, Dean Pomerleau and Todd Jochem were interested both in the evolutionary aspects of the art and also seeing if a machine could learn to model user tastes (see Shumeet Baluja, Dean Pomerleau and Todd Jochem. Simulating User’s Preferences: Towards Automated Artificial Evolution for Computer Generated Images. Technical Report CMU-CS-93-198. Carnegie Mellon University. Pittsburgh, PA. October 1993. ). They built a much simpler art system that combined primitive elements (elements closer to brush strokes than to original pictures) and tried to learn user preferences for complex pictures.

Figures of this era looked much like this (well better than this, this comes from a scan of a black and white printing of the paper):

.

Scott Neal Reilly built a new, more simplified system; and with Michael Witbrock put the whole thing on the Web. This was an unimaginably primitive time on the Web. Cutting edge interaction was sites like “Blue Dog Can Count.” The Mac had no forms capable browser and Amazon.com was still a year away from launching. An interactive art exhibition running directly on the Web (and manipulated by anybody) was a significant step forward.

Michael Witbrock was influenced by the stories of Heikegani Crabs and Alan Turing’s 1952 paper “The Chemical Basis of Morphogenesis” (which theorized how simple systems could develop textures).

At this point I (John Mount) got interested in the project and felt that much more could be done with how such systems handled color. The art had been simplified to primitive elements that one could think of as brushes (really more like gradients) but the art was essentially grey-scale with a false-color map applied at the last step. Karl Sims had made transformations on images his primitive operations, I wanted my primitive operation to be transformations on color.

Being a math-nerd I chose to encode color inside a mathematical system called “Quaternions” (see Ebbinghaus et al. Numbers Springer-Verlag, Second Edition, 1988). Colors are often represented as three brightness terms- for example intensity of red, intensity of green and intensity of blue. The Quaternions were discovered by Sir William Rowan Hamilton in 1843 (see Wikipeida: Quaterion). Sir Hamilton was trying to solve the problem of encoding positions in space in a nice structure and was so excited by his discovery he carved his fundamental formula for them in the Brougham Bridge the night he had his breakthrough. Quaternions are represented as four standard numbers (so they have enough “slots” to encode a position in Sir Hamilton’s case or in our case a color) and they their selves behave a lot like individual numbers. There are rules for adding, subtracting, multiplying and even dividing Quaterions. This means you can write formulas over them and these formulas are now directly manipulating colors (instead of manipulating geometry or intensities as in the earlier systems).

So, as with Neal Reilly’s system, we represented all of our transformations as formulas and represented “breeding” as ripping a bit of one formula out and combining it with another. For instance these two rather uninteresting color gradients were represented by the formulas:

( x – i y ) :
( x – i y – j x – k y ) :	.

One of our arithmetic operations was named “mod” and we could use it to combine the two items into a more complicated formula and somewhat more interesting picture:

( mod ( x – i y) ( x – i y – j x – k y ) ) :

.

After enough generations of selection and breeding the formulas get long and complicated (luckily nobody but the machines have to look at them) and the pictures get interesting:

.

Michael Witbrock and Scott Neal Reilly supplied an updated web interface. And at this point we got our 15 minutes of fame:

From Wired 3.01 January 1995 p. 147 Kristin Spence’s “Net Surf” column:

.

We also made large prints (using a parallel computation system named “WAX” by Peter Stout) and set up an exhibition in a Pittsburgh coffee house. The development work was largely done on a machine with a black and white monitor (later we got access to a grey scale monitor) so it really was a treat that color was able to fend for itself.

On an unrelated track in 1991 another CMU student, Scott Draves, was pursuing a serious project and building art based on iterated function systems (more related to fractals than our work) and in 1999 added some genetic ideas and released the Electronic Sheep client/server oriented screen saver (inspired by SETI@home).

Recently we have gotten back to some more of Sims’ ideas and allowed more geometric transformations and real source images, such as incorporating some of Dover’s royalty free Japanese textile designs:

.

The original system was group interactive and an incredible hit (and had its own Zephyr discussion instance- the then equivalent of Twitter). The current demo is stand alone and server-free, so it is a single player game.

All the user has to do is point a Java 1.4 (or better) capable browser at: Genetic Art Program. Click (and hold) on the “Action Menu” of any of the sub-windows and select “take over left selection” on one picture that interests you and “take over right selection” on another picture that interests you. You will notice doing this copies the pictures to the left and right of the “breed pictures” button. Now press “breed pictures” as many times as you want. Each time you press it you will get a new picture build by combining elements of the two small pictures. At any time you can have some other picture you like take over a breeding position (again by using its action menu). And you can also scroll the strip of five pictures around and click on them to introduce previous favorites from the earlier server version of the program into your space of opportunities.

We were originally a bit uncomfortable calling the work “Art” but a number of the images have made significant impressions on us and others- so perhaps it qualifies.

Related posts:

1. Algorithmic Movie (with texture)
2. Automatic Differentiation with Scala
3. Relative returns: a banker versus trader paradox

I have had a soft spot for this kind of study ever since I read: Donald E. Knuth: An Empirical Study of FORTRAN Programs. Softw., Pract. Exper. 1(2): 105-133 (1971). In that article Knuth admits to breaking into people’s accounts to collect statistics on what evil people were feeding into the FORTRAN complier.

Let’s look at the gestalt of a few popular programing languages following button-sized excerpts from Marceau’s article:

.

To build these graphs 19 challenge problems were implemented in 72 programming languages. Each square is programming language, the x-axis is runtime size and the y-axis is code size (large is bad on both of these). Each line segment connects the code size and run-time of one example program run to the centroid of all such runs for the language. We all know code size is not a very good stand-in for programming difficulty (compare C a merely primitive language to C++ an outright programmer hostile language), but the pictures actually tell a credible story.

• GCC (or C) is very very fast but takes a lot of code (its graph is a vertical bar running up and down the left).
• Java mostly works like C, but every once and a lets you down on performance (this is leaving out that Java is far safer than C and far more wasteful of memory).
• Javascript and Ruby have such bad implementations that their centers are off the graph (this brings up a point the original authors well understand- you can not benchmark a language only a specific run of a specific program using a specific language implementation).
• Perl and Erlang have similar run time performance (though are completely opposite poles of elegance, elegance not plotted on graph).
• Ruby’s implementation makes Python look fast.
• OCaml lives up to its reputation of being simultaneously very expressive and efficient (but expressive power is not a direct measure of ease of use, think of APL).

The benchmarking depends on people donating example programs and the problem types are heavily biased towards the puzzle are (where C, Java and OCaml excel) and not to the “its a one-liner because it is already done in a frame work” (Perl, Python, Ruby).

For all the problems inherent in such a study I think it is actually interesting what a little quantitative data lets us think about.

Related posts:

1. Sorting Used in Anger
2. I know, I am the one being a jerk
3. Survive R

However, the reading experience is marred by some flaw in their RSS system that keeps marking the article “MapReduce: A major step backwards” as a new article. This causes the article to appear in my RSS reader every few weeks as “new.” This wouldn’t bother me too much except that the article runs so counter to experience that it is itself offensive.

I have no desire to defend Google (the home of MapReduce)- they don’t need it and are clearly laughing all the way to the bank. However the points used to kick at MapReduce are so broad and so devalue practitioner experience that they are insulting. I find the individual arguments offensive and wish to stand against them. I am not that concerned about the conclusion, use MapReduce or don’t. For some things MapReduce is a good tool and for some things it is not.

Let’s limit ourselves to the 5 primary complaints from the article. The article (verbatim) says MapReduce is:

1. A giant step backward in the programming paradigm for large-scale data intensive applications.

2. A sub-optimal implementation, in that it uses brute force instead of indexing.

3. Not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago.

4. Missing most of the features that are routinely included in current DBMS.

5. Incompatible with all of the tools DBMS users have come to depend on.

Now let us comment:

1. “A giant step backward in the programming paradigm for large-scale data intensive applications.”

Actually, no.

MapReduce represents a continuity in a stream of ideas that made UNIX great: composable transient tools. Not everything is a database or data warehouse. A lot of the grungy UNIX tools (like sort, sed, awk, join) have often been combined to do large scale (at the time) research because they all worked “out of core” fairly well. This makes for a horrible bailing-wire set-up. However, it often handles problems of a size much larger than would have been possible on the hardware at the time.

In addition the author trots out the “it’s Codasyl all over again” argument. This argument refers to the ongoing pain and expense derived from binding algorithmic details too close to the data representation. In earlier writing it was a fantastic point that warned that the up and coming object oriented databases were going to be the same nasty pointer chasing nightmares that hierarchical databases had been. I can see why an author might feel that just saying “it’s Codasyl” could win any argument.

2. “A sub-optimal implementation, in that it uses brute force instead of indexing.”

MapReduce does not use brute force.

MapReduce uses the idea (one that goes back to merge sort) that parallel traversals (that is: running through two lists in the same order synchronously) are a very powerful technique that can, among other things, produce indices. MapReduce is so efficient that it has been shown to be competitive with the best large scale sorting algorithms on their home-turf: sorting.

MapReduce looks brutish because it drops a lot of popular design features. One such feature is trying to speed things up through local caching and combining. However, on the data that MapReduce is commonly used (free form written text) it is a provable property of the data that local caching is an ineffective complication (due to the heavy-tailedness of the data). So many of the graceful features missing from MapReduce are actually no help on the types of data it is used on. There is a certain grace in leaving only only the features that are actually helping.

3. “Not novel at all — it represents a specific implementation of well known techniques developed nearly 25 years ago.”

A nasty attack.

MapReduce is a good explanation of some merging techniques that have been common knowledge for quite a while. This is a legitimate expository goal: explaining something everybody already “knows” better. In fact this is very hard to do and considered a legitimate accomplishment in many fields (for example Rota stated it was a legitimate goal in mathematics). I myself looked at some of my own older code for dealing with very large data sets after reading the MapReduce paper. I saw that the paper was describing what I was already doing (splitting the data into streams for later re-joining) and explaining it so well that it was now a method and no longer a hack. When a paper successfully teaches about you something you already “know” it is a good work.

The attack is is also inaccurate- the ideas are not 25 years old it is closer to 120 years old.
We could easily trace the lineage of MapReduce back to Hollerith style sorting machines that pre-date general purpose computers (i.e. going back to before 1889) . MapReduce refers back to a time when all computation was performed by what we now call external sorting and tabulation. These 19th century technologies may seem archaic but they were developed in a word similar to ours: worlds where the amount of data is in excess of your conveniently reconfigurable computational resources.

4. “Missing most of the features that are routinely included in current DBMS.”

Unfortunate.

I miss a lot of those features.

However, because MapReduce is such a lean technique I have seen engineers implement their own MapReduce systems in a day (to solve a problem they are working on). That is they are successfully sorting, joining, indexing and summarizing hundreds of gigabytes of data on a consumer PC within a couple of days of being asked to. This is from scratch after reading the MapReduce paper.

The “make versus buy” decision should not always come out “make.” But it is not wise to artificially bloat up requirements so that the decision can only be “buy.”

5. “Incompatible with all of the tools DBMS users have come to depend on.”

Good.

Frankly for a lot of analytic practitioners many DMBS systems and tools have become expensive obstacles in the way getting results. Yes, we enjoy humiliating an interview candidate that does not know all of the Codd normal forms (or can’t remember which of the alphabet soups of OLTP or OLAP is the “good one” ) as much as the next person. But to many of us a lot of these tools and procedures are more obstacles than a solutions.

This may sound nasty, but if were not the case why would companies like Vertica be producing radical new database tools? The fact is existing DBMS tools were designed for a different type and scale of data than we regularly see on the web (and column oriented database designers seem to share this view). The situation is so bad that “roach motel” is a common analyst’s slang for “data warehouse” (derived from: “data checks in but it never checks out”).

This isn’t meant to be a hagiography of MapReduce, but given that MapReduce has paid the bills I feel it deserves a small show of respect along the lines of “dance with the one who brung you.”

MapReduce is not a panacea. One of the tasks I have hated most in my career was maintaining a seven step MapReduce based system. I would love to have avoided all the detail fiddling that set-up required. However, the system paid our bills by performing a calculation that was beyond the scale of simpler methods and it would have been unaffordable to buy a solution.

Related posts:

1. Sorting Used in Anger
2. Must Have Software
3. Exciting Technique #1: The “R” language.

Here is a question from the interview:

Andrew: One of the few projects of yours that hasn’t been embraced by a widespread community is literate programming. What are your thoughts about why literate programming didn’t catch on? And is there anything you’d have done differently in retrospect regarding literate programming?

Professor Knuth had a good and interesting answer, which I will not go into here. Also, it was a good question- Literate Programming is a good idea, yet we have only seen weak imitations of it like Doxygen and JavaDoc (which automatically document the syntactic structure of code instead of really helping the programer become an author and explain their meaning and intent).

Mr. Binstock even includes a link to a site promoting the concept. Lets as the kids these days say “click through” and see what awaits us.

This is great. Literate programing is definitely Web 2.0 (rounded corners, use of light pastels and cool gradients). This is now, this is modern, sign me on. The site even has links to original articles by the masters:

Literate Programming - CACM Series
	Programming Pearls: Literate Programming, CACM (May 1986)
	Programming Pearls: A Literate Program, CACM (June 1986)
	Programming Pearls: Abstract Data Types, CACM (April 1987)
	Announcing Literate Programming, CACM (July 1987)
	LP: Processing Transactions, CACM (December 1987)
	LP: Expanding Generalized Regular Expressions, CACM (December 1988)
	LP: A File Difference Program, CACM (June 1989)
	LP: Weaving a Language-Independent WEB, CACM (September 1989)
	LP: An Assessment, CACM (March 1990)
	The Literate-Programming Paradigm
	Donald Knuth. "Literate Programming (1984)" in Literate Programming. CSLI, 1992, pg. 99.

Lets click through and see how the Association for Computing Machinery helps disseminate, guide and educate:

Oh, maybe this is part of why Literate Programming hasn’t been embraced: the whole purpose of Literate Programming is lost when you keep it a secret.

I am sure I have been a paid ACM member from time to time, but I don’t remember the online credentials and they have probably lapsed by now. I tried applying for the free temporary credential (the online form ended up not sending me anything- ACM not so good with the computers). I can afford pay (yet again) to re-join ACM but why would I want to give my money to support an organization so far from my (and common) academic values?

So in conclusion:

• Sorry Professor Knuth, you remain one of my heroes, but I’ll have to get to Literate Programing a bit later. I would say that the marketing campaign behind Literate Programming has excessive “breakage.”
• ACM: that was a funny joke, great head-fake, impeccable comic timing, good fun and I certainly learned something. Oh, and I will see you in hell.

Related posts:

1. Programs reduced to statistics
2. Something I don’t get about business and bailouts
3. Map Reduce: A Good Idea

The other day I had a bit of time to kill before an appointment. Luck was with me: there was a nearby bookstore and I was able to pass some of the time skimming through a book called “Beautiful Code.” Everything started out fun and nostalgic. The book title reminded me of “The Art of Computer Programming” (a book that probably did as much through the grace of its title as it did through its incredible contents to attract minds into theoretical computer science). One of the chapters of “Beautiful Code” was by Jon Bentley (a hero of sharp reasoning and clever coding) and as I flipped to the chapter my day was ruined. There it was: Quicksort an algorithm that I have a long love and hate relationship with.

Bentley’s code (combining his skill with a really good idea Bentley credits to Nico Lomuto) is indeed a masterpiece:

void quicksort(int l, int u)
{   int i, m;
    if (l >= u) return;
    swap(l, randint(l, u));
    m = l;
    for (i = l+1; i < = u; i++)
        if (x[i] < x[l])
            swap(++m, i);
    swap(l, m);
    quicksort(l, m-1);
    quicksort(m+1, u);
}

How can I have such strong feelings for something so abstract and small as this algorithm? Part of the answer is that I had just the other day written some Quicksort code that, until I flipped open the book, I had been quite proud of. The rest of the answer is that Quicksort and I have a history.

Coding is all about compromises. Some of the greats in our field work to teach us simplicity and purity but are often accused of being idealists. Not so: their point was that there will be more than enough compromise and lessening of your vision as your project progresses. So to have anything left you must start with a lot (which we usually do not). C. A R. Hoare himself (the inventor of Quicksort) worked hard to build frameworks that were able to prove small programs like Quicksort functioned correctly. This effort is often criticized as being impractical for large software systems. But you cannot build large systems without a library of well understood reliable components. Why tolerate any unnecessary flaws when you will certainly face plenty of necessary flaws?

How I have managed and negotiated compromises has been a major source of pride in my work. I have pushed hard to move “nimble activities” such as research and prototyping away from looser languages (like Perl) into safer and stricter (though somewhat more tedious to work with) languages like Java.

This isn’t a pure position, it represents compromises and has consequences. Java itself has a number of unnecessary flaws. One flaw is that Java is very inefficient in storing huge collections of small objects. That is: Java is bad at a task that underlies a lot of the basic record keeping needed for a lot of the research I do (machine learning and data mining). I had, until opening “Beautiful Code,” been feeling very good about a compromise I had recently made. To avoid giving up on Java (and moving to an even more tedious language like C++) I figured out how to store all of my many many small bits of information without creating a great number of objects. By introducing complicated code I could stay in Java and keep many of the advantages of the language. But, the compromise was my data lost access to a number of essential Java supplied services- such as sorting and lookup.

The solution was to re-implement sorting. Re-imlementing sorting in this day and age (especially in a language such as Java that supplies some great sorted and ordered data structures) seems stupid. But none of the Java built-in functions are willing to sort a large collection of data without create a great many objects (which is prohibitively wasteful). If I had code that was able to sort my “poor man’s records” directly I would have a complete solution. The most expedient sort procedure to code-up is Quicksort (which tends to run very well and does not need a lot of extra space for record keeping). So I coded up what I that was a pretty clean and clever Quicksort for my problem. Had I more time I would have preferred to use Mergesort which has better guarantees that Quicksort (Quicksort is usually very fast, Mergesort is always fast).

Seeing Bentley’s code I realized my code was “bloated and flabby.” What I did in four separate ways he did in one. My Quicksort was quick and correct, but not perfect. I began to think that perhaps more of my reasoning was flabby and expedient. Should I have moved the whole project out of Java? Should I have implemented Mergesort? Why did my code seem so filled with fret and worry?

Then I saw it. Bentley’s example was wrong. The code was so unworried that it failed to anticipate an important possibility. When you are really thinking hard about an algorithm you can often simulated it in your head. Bentley’s clear writing made thinking easier. Suppose you gave Bentley’s Quicksort code a set of items to sort that were all identical. Then the “if”-statement would never be true and the variable “m” would never be increased. At the end of each run you would always have “m” pointing to the left-most portion of the array (despite the clever choice of pivot made earlier in the code). The Quicksort procedure depends on calling itself to re-sort shorter arrays, but in this case the re-sort would be on arrays only a single item shorter than the original. This means Quicksort would be doing about the same amount of work over and over again- instead of working on smaller and smaller pieces. Quicksort would be very very slow.

Why do I notice this? Part of it is my training: theoretical computer scientists are trained to work examples and trained to worry. The other part is that I had been burned by this in the past. Years ago I had been with a group doing very interesting biotech research on what at the time were moderately large data sets. One of the clever steps was that the group had reduced a lot of the research steps to standard computer science techniques like sorting and joining (much like Google’s famous MapReduce reduces complicated computations to primitives). Then the standard “commercial package” failed us. The built-in sorting function was often very very slow. The often happened if there were a lot of repeated values in the data we were trying to sort.

Years ago we fixed the problem by implementing our own Mergesort. Mergesort is a bit harder to implement than Quicksort but there are no “special situations” that make Mergesort slow down. Quicksort itself can, at the loss of some elegance, be coded in such a way to avoid any problems with duplicate entries (as I had recently done). However, there is always a unlucky data set that Quicksort will be slow on. The unlucky data set is rare, but it can happen (unlike with Mergesort).

Most implementations of Quicksort have this flaw (nearly guaranteed slowness when there are many duplicate keys in the data to sort). One thing to look for is if when Quicksort calls itself is it using a single mid-value variable to represent the partition (like the “m-1” and “m+1” or does it use two variables to try and suppress all values similar to the pivot. For instance the code found at http://www.csse.monash.edu.au/~lloyd/tildeAlgDS/Sort/Quick/ (by Lloyd Allison who attributes it to Niklaus Wirth’s 1986 book “Algorithms and Data Structures”) performs some careful accounting to try and avoid allowing duplicate elements into the split in addition to the code to do this and the comments documenting what is being attempted the key they to look for is the use of two separate variables (“right” and “left” in addition to “lo” and “hi”) in the recursive call:

quicksort(a, lo, right);// divide and conquer
quicksort(a, left,  hi);

One think that always fundamentally bugged me about Quicksort: why settle for a procedure that “almost always works quickly” when there are other procedures that “always works quickly?” Why would someone like C. A. R. Hoare promote a needless compromise?

And that is just it- it isn’t needless. We live in the real world with bounded resources, compromise is inevitable. If you want to sort a single list and guarantee it will be done quickly you really should be using a provably good sorting procedure (like Mergesort). If, however, you are asked to solve many sorting problems then a procedure like Quicksort is so often faster than others that even when you add in the time lost on the bad rare-situations (which you will eventually run into) you will still be (on-average) ahead for using Quicksort. It is a compromise having to pick a “most appropriate method” instead of a “best method” (because “best” varies by situation) but it is not an unnecessary compromise. Seeing and managing there trade-offs is the essence of design and programming.

Related posts:

1. Map Reduce: A Good Idea
2. Programs reduced to statistics
3. Thievery considered harmful

February 19, 2008

Computer scientists have usually dodged questions of intent, purpose or meaning. While there are theories that assign deep mathematical meaning to computer programs[13] we computer scientists usually avoid discussion of meaning and talk more about utility and benefit. Discussions of the rhetorical meaning of programs is even less common. However, there is a famous computer program that has a clean an important rhetorical point. This program is called “hello world” and its entire action is to write out the phrase “hello world.” The action is simple but the “hello world” program actually has a fairly significant purpose and meaning.

I would like to briefly trace the known history of “hello world” and show how the rhetorical message it presents differs from the rhetoric embodied in earlier programs. In this sense we can trace a change in the message computer scientists felt they needed to communicate (most likely due to changes in the outside world).

Since the late 1970’s it has been a tradition in computer science that when facing a new system to start by writing the traditional “first program”, which is called “hello world.” The history of this tradition is not well documented, but WikiBooks claims the origin is in Brain Kerninghan’s 1974 tutorial for the computer language “B.”[14, 8]

The program itself is as follows:

main( ) {
  extrn a, b, c;
  putchar(a); putchar(b); putchar(c); putchar('!*n');
}
a 'hell';
b 'o, w';
c 'orld';

From the original context in [8] we can say this is not a pure “hello world” program. The purpose of this program is to illustrate a few functions of the language (“extrn” variables, use of multiple lines and so on) and not to test if the system is running.
The most famous (and a pure) example of “hello world” is found in Brian Kerninghan and Dennis Ritchie’s famous 1978 C book[9]. The code looks like the following:

main() {
        printf("hello, world\n");
}

Equally famous are the early versions in “BASIC”:

10 print "Hello, World!"

In both of these examples the purpose of “hello world” is clear: it is a trivial computer program that does nothing but print a single line of text. If this can not be made to work then nothing can be made to work. “hello world” is in fact a somewhat confrontational program. The author is saying “it isn’t obvious your computer system will work, so I am not going to invest a lot of time in it until I see it can at least print one line of text.”

Writing “hello world” as the first program on a new system was certainly a well known tradition by the late 1970’s. We can ask: did this tradition showed up late or early in the history of computer science?

The first modern computer is commonly credited to John von Neumann in 1945. This attribution of “first” was made before a lot of previously secret information on Bletchley Park’s and Konrad Zuse’s work in Germany were well known, but von Neumann’s work was definitely the known foundation that later work built on. Donald Knuth credits von Neumann with the first modern computer program[10]. In any case we can securely place the invention of the modern computer to the early to mid 1940’s. It seems like “hello world” was not written at that time because it made entirely the wrong rhetorical point.

In fact von Neumann’s first program (in 1945) was a sorting algorithm (proving the computers could at least replace tabulators)- it was important to prove that not only could the computer turn on but that it could do something.

The 1957 description of FORTRAN starts with a program that solves for roots of an equation.[4] John Backus himself says the prior to 1954 “almost all programming was done in machine language or assembly language”[3] (which would make “hello world” an unlikely first program as numeric operations are typically much more succinct than string manipulations in these languages).

The 1958 UNIVAC Math-Matic Programming Manual first examples are (predictably) equations involving trigonometric functions.[5]

In 1959 the Algo 58 standard [2] had no concrete example programs- the purpose of this document was to show that program syntax could be specified and to discuss the techniques of specification (so a runnable example did not serve their argument).
John McCarthy’s paper on recursive functions (LISP) starts with examples partial functions and conditionals (to get to terminating recursion as fast as possible).[11] The Lisp1 manual [6] concentrates on the translation of information into data-structures (an important point) and the Lisp1.5 manual [12] is written for the famous “bottom of page 13” moment where “eval” (essentially the semantic core of LISP) is defined in a few lines LISP code.

The computer language “BASIC” seems almost designed to support “hello world.” Instead the 1964 BASIC manual[7] starts with a lively discussion of what a computer program is (as a process) and the first example program is a highly useful program that solves simultaneous equations. The point again being that useful work can be done.

Even as late as 1972 “hello world” does not seem to be the obvious message. For example the popular BASIC tutorial called “My Computer Likes Me, When I Speak in BASIC”[1] does indeed start with a simple program that only writes a line of text. But the line it writes is “MY HUMAN UNDERSTANDS ME.” This is making a very different point than the point made by “hello world.”

While I have skipped a number of languages of the period (Autocode, COBOL, APL, SNOBOL, PL/1, Logo, BCPL, Forth, Smalltalk …) we can see the general trend: fist programs had to make a point. By the dates it seems that once invented the “hello world” tradition took off and spread very quickly. Writing such a program serves a useful purpose as a test so the practice probably spread quickly once the rhetoric of the earlier programs was no longer needed. That is “hello world” became popular once computer scientists no longer felt that society needed to be persuaded of the ultimate utility of computer systems.

With the increasing complexity of modern systems “hello world” is a more important test than ever. “hello world” is sometimes called the hardest application to deploy. The idea is that once you learn your lessons from deploying it then deploying a second, more sophisticated, application seems relatively easy. I often find that I can lean the true nature of a system by deploying “hello world.” If writing and deploying “hello world” is a sufficiently unpleasant task in a system then it is likely that every other task in that system will be similarly unpleasant.

In the end “hello world” serves the same purpose that it always has (testing if a system in fact works) and it stands as a rhetorical marker signifying that we are now living long after the skirmishes of the computer revolution.

References

[1] Albrecht, B. My Computer Likes Me, When I Speak in BASIC. Dymax, 1972.

[2] Backus, J. W. The syntax and semantics of the proposed international algebraic language of the zurich acm-gamm conference.

[3] Backus, J. W. The history of fortran i, ii and iii. ACM SIGPLAN Notices 13, 8 (Aug 1978).

[4] Backus, J. W., Beeber, R. J., Best, S., Goldberg, R., Haibt, L. M., Herrick, H. L., Nelson, R. A., Sayre, D., Sheridan, P. B., Stern, H., Ziller, I., Hughes, R. A., and Nutt, R. The fortran automatic coding system. Proceedings of the Western Joint Computer Conference (Feb 1957).

[5] Corp., R. R. U. UNIVAC math-matic programming system. Sperry Rand, 1958.

[6] Fox, P. Lisp 1 programmers manual. 165.

[7] Kemeny, J., and Kurtz, T. Basic.

[8] Kerninghan, B. W. A tutorial introduction to the language b.

[9] Kerninghan, B. W., and Ritchie, D. M. The C Programming Language. Prentice Hall, February 1978.

[10] Knuth, D. E. von neumann’s first computer program. Comp. Surveys 2, 4 (1970), 247–260.

[11] McCarthy, J. Recursive functions of symbolic expressions and their computation by machine, part i.

[12] McCarthy, J., Abrahams, P. W., Edwards, D. J., Hart, T. P., and Levin, M. I. Lisp 1.5 programmers manual. 116.

[13] Tennent, R. D. The denotational semantics of programming languages. Communications of the ACM 18, 8 (Aug 1976), 437–453.

[14] Wikibooks. http://en.wikibooks.org/wiki/Computer_programming/Hello_world#B, 2008.

Related posts:

1. Programs reduced to statistics
2. I know, I am the one being a jerk
3. Sorting Used in Anger