The ergodic theorem (http://en.wikipedia.org/wiki/Ergodic theory (link)) is an important principle of recurrence and averaging in dynamical systems. However, there are some inconsistent uses of the term, much of the machinery is intended to work with deterministic dynamical systems (not probabilistic systems, as is often implied) and often the conclusion of the theory is mis-described as its premises.

By “interested computer scientists” we mean people who know math and work with probabilistic systems1, but know not to accept mathematical definitions without some justification (actually a good attitude for mathematicians also).Please click through to read Ergodic Theory for Interested Computer Scientists.

Related posts:

1. Hello World: An Instance Of Rhetoric in Computer Science
2. Six Fundamental Methods to Generate a Random Variable
3. What Did Theorists Do Before The Age Of Big Data?

To implement many numeric simulations you need a sophisticated source of instances of random variables. The question is: how do you generate them?

The literature is full of algorithms requiring random samples as inputs or drivers (conditional random fields, Bayesian network models, particle filters and so on). The literature is also full of competing methods (pseudorandom generators, entropy sources, Gibbs samplers, Metropolis–Hastings algorithm, Markov chain Monte Carlo methods, bootstrap methods and so on). Our thesis is: this diversity is supported by only a few fundamental methods. And you are much better off thinking in terms of a few deliberately simple composable mechanisms than you would be in relying on some hugely complicated black box “brand name” technique.

We will discuss the half dozen basic methods that all of these techniques are derived from.To our mind all of the famous random variate generation/sampling techniques are derived from combinations of the following six fundamental methods:

1. Physical sources.
2. Empirical resampling.
3. Pseudo random generators.
4. Simulation/Game-play.
5. Rejection Sampling.
6. Transform methods.

The technical fights (such as: “is Gibbs sampling superior to, or even distinguishable from, Markov chain Monte Carlo?”) are all in the details, history and citation conventions. Each field and particular method accretes its own traditions. We will quickly discuss the fundamental methods we listed. As we will see: complexity goes up as we move through the list (so at some point things are no longer fundamental but instead derived, allowing us to end the list).

The Methods

Physical sources

This is the most basic way (though not as practical in the computer age) to generate random variables. Observe the flip of a real coin, shuffle actual cards, mix numbered balls or count the number of ticks from an actual radioactive source. In all of these the randomness comes from physical principles (such chaotic dynamics for coin flips or quantum mechanics for radioactive decay).

These sources are “outside of computer science” so we will say the least about them.

Empirical resampling

This is what used to be called “tables” (which were themselves often generated from physical processes). The observation is: that sometimes
to run a simulation you need access to instances of random variables that are distributed in a very precise way- but you don’t have a usable description of the desired distribution. You would think that in this case you could do nothing. But the principle of empirical resampling is that you can approximately generate new samples by taking samples (with repetition or replacement) from an old sample. This is the cornerstone of Bootstrap methods.

As an example: suppose we were given the sample of numbers 5, 5, 10, 5, 5 which has mean equal to 6. Further suppose we have no
description of how these number were generated but we wanted to know if a mean of at least 8 is likely or unlikely for five more numbers drawn the same way. We can approximate this by drawing many samples of size five from this original sample (allow the same number to be in our new
sample multiple times) and get the bootstrap estimate of the probability of seeing mean of at least 8 as having a probability around 0.6%.

This may seem trivial- but it is very important.

Pseudo random generators

In the computer age, to avoid need for external tables or expensive and slow peripherals we tend to use pseudo random generators. That is the output of deterministic iterative procedures as equivalent to true random sources. The science of pseudo randomness has evolved from cobbled together procedures passing ad-hoc tests (such as in Knuth Volume 2) to more formal pseudo randomness based on important properties (like provably being k-wise independent) or complexity (being computationally indistinguishable from a truly random on a time or space bounded machine). Behind the canned routines of all of the basic “random generators” commonly available is a pseudo random source.

Good references for the modern theory include:

• “Pseudorandomness and Cryptographic Applications” Michael Luby 1996.
• “Modern Cryptography, Probabilistic Proofs and Pseudorandomness” Oded Goldreich, 1999.

The most basic form of a sequential pseudo random generator is a sequence of states s(1), s(2), s(3) … . Where s(i+1) = g(s(i)) where g() is our deterministic function that maps state to state. The observed random variables are then h(s(i)) where h() is some deterministic function maps state to observables. For example for the linear congruential generator found in glibc we have g(x) = (1103515245*x + 12345) modulo 2^32 and h(x) = x modulo 2^30 (x an integer from 0 to 2^32 – 1). An example application: this generator when divided by (2^30 – 1) might return numbers passably uniformly distributed in the interval [0,1]. Two such variates might be uses as a uniform sample from the unit square.

That a simple iterated deterministic system (like the modulo arithmetic or even a physical system like coin flipping) would even superficially appear random (let alone be safe to use as pseudo random source) turns out to be the main consequence of Ergodic theory (which we will touch on in a later article). The point is: it should not be obvious (without bringing in some more theory) why you should trust pseudo-random sources.

Simulation/Game-play

Another fundamental method is direct simulation or game play. If we wanted a random variable that was 1 with probability equal to the odds of being dealt a full house from a standard shuffled deck of 52 cards (and zero otherwise). We can generate such a variable by simulating shuffling a deck, drawing a hand and returning 1 if the hand draw is a full house (and returning 0 otherwise). Notice in this case we are combining many random variables to get a single result.

One of the most important simulation techniques is Markov chain Monte Carlo methods (related to Gibbs sampling, simulated annealing and many other variations). These method implement a complex procedure over a stream of random inputs to generate a more difficult to achieve sequence of random outputs.

For example: Let T be the set of pairs of non-negative integers x, y such that x + y ≤ 1000. We could implement a Markov chain on this set from a source of coin flips. Given a point (x,y) in T we take three coin flips and move to new point (x’,y’) (also in T) using the following procedure:

1. Let m = 1 if the first flip is heads and m=0 if the first flip is tails.
2. Let v = (1,0) if the second flip is heads and v=(0,1) if the second flip is tails.
3. Let d = +1 if the third flip is heads and d = -1 if the third flip is tails.
4. If (x,y) + m*d*v is in T let (x’,y’) = (x,y) + m*d*v, otherwise let (x’,y’) = (x,y) (stay put).

Repeating this procedure a large number of times produces a sequence of points (x,y) such that (x,y) is distributed uniformly on S (again this follows from ergodic principles). The correctness of this simulation of or game of following a Markov chain is a very fundamental method in generating more complicated random variates and something we will write more about in an article dealing with the ergodic principle (the relation of connectedness to showing averages over time equal averages over space).

For simple shapes (rectangle, triangles) there are more efficient ways to generate points uniformly at random. For squares we exploit independence and just generate the coordinates independently. For triangles we could rejection sample from a bounding rectangle. Or we could use a tranform method: write down a counting function that indexes all the points in the triangle and generate points by index (for example it is easy to work out there are 501501 points in our example S so if we generate a random integer uniformly from 1 to 501501 can just pick the point with given index as our sample).

For general convex shapes (in high dimensions) these methods become intractible and Markov chain methods are one of the few options remaining.

Rejection Sampling

Rejection sampling is another way to convert one sequence of random variables into another. If we assume we can generate a random variable according to the distribution p(x) we can “rejection sample” to a new distribution using an “acceptance function” q(x) which returns a number in the interval [0,1]. Our procedure is to
repeat the following: generate x with probability p(x), generate a random variable y with uniformly in the interval [0,1] if y ≤ q(x) accept x as
our answer and quit (otherwise draw a new x and repeat).

When the distribution that rejection sampling draws with is such that if x and y had a ratio of being drawn of p(x)/p(y) then under the rejection procedure they have relative odds of (p(x)q(x))/(p(y)q(y)). An important special case is when q() is always 0 or 1, in this case we are drawing with relative odds proportional to p(x) from the subset of x with q(x)=1.

As an example: consider the problem of trying to draw a point (x,y) such that x^2 + y^x < 1 (the open unit disk) uniformly at random. The rejection sampling solution is: repeat the following until you have a success: generate x and y independently uniformly in the interval [-1,1], if x^2 + y^2 < then 1 accept them as our sample (otherwise repeat). This procedure is very fast as the unit disk that represents our acceptance region has area pi and the square we are generating trials from has area 4: so we over a 78% chance of success on each trial or expect to only have to run fewer that 1.28 trials (on average) to get a sample.

Transform methods

A transform method is used when we have the ability to generate instances of a random variable according to one distribution and we would like instances according to another distribution.

One method is used when we have access to the inverse of the cumulative distribution function of the distribution we are trying to generate. In this case we can use this function to convert uniform variants from the interval [0,1] into our target distribution. The commutative distribution function is the function cdf() where cdf(x) is the probability a random variate generated according to our distribution is less than or equal to x. The inverse function function icdf() where icdf(y) is such that cdf(icdf(y)) = y. For example the exponential distribution has an inverse cumulative distribution function icdf(y) = -ln(1-y)/lamda . So if y is
generated uniformly in the interval [0,1] then icdf(y) is a random variable generated according to the exponential distribution with parameter lambda.

A great example of transform methods is generating Gaussian random variables. We could directly use the inverse cumulative distribution function method described above- but to do this we would require a special function library to perform the required calculation of the inverse cummulative distribution (or inverse of erf()). Another way is the polar method: generate x,y uniformly from the open unit disk (by, for example rejection sampling as described earlier), set s = x^2 + y^2 and return x*sqrt(-2 ln(s)/s), y*sqrt(-2 ln(s)/s) as two independent Gaussian random variables. The trick being: the distribution function of r = sqrt(s) is of the form r*e^(-r*r/2) which leads to an elementary cumulative distribution function (unlike the original Gaussian density of the form e^(-r*r/2)) that is easy to invert.

Conclusion

Our thesis is: all major methods to generate random variables use aspects of the six methods we have listed here as fundamental. Or you should at least have a fluid understanding of at least these methods. You should be able to break down big “brand name” methods (like Gibbs sampling) roughly into their constituent parts (so you can reason about them). One example: notice how ratios of probabilities enter into Markov chain Monte Carlo methods (they cause step rejections); from this you can reason if your problem has bounded ratios it is a good candidate for direct application of the technique (and if it does not you need to add some more ideas, as was demonstrated in: “Sampling from Log-Concave Distributions,” Alan Frieze , Ravi Kannan , Nick Polson, Ann. Appl. Prob, 1994 ).

The first two methods we discuss (physical sources and empirical re-sampling) are of the class of solutions “already have the right answer.” Pseudo random generators are the primary way to negate the need for physical sources and resampling techniques. Simulation, rejection sampling and transform methods are the main tools for building new distributions out of old.

It is a matter of taste if a given trick fits into this ad-hoc taxonomy or not. You can invent new and better generation methods- but these methods are easily derived using ideas from the fundamental methods we mentioned here.

Related posts:

1. What is a large enough random sample?
2. Kernel Methods and Support Vector Machines de-Mystified
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)

Early computers were most limited by their paltry memory sizes. von Neumann himself commented that even a room full of genius mathematicians would not be capable of much if all they could communicate, think upon or remember were the characters on a single type written page (much more memory than the few hundred words available to the Eniac). The most visible portions of early computers are their external memories or secondary stores: card readers, paper tape readers and tape drives.

SDC 920 computer, Computer History Museum, Mountain View CA

Historically computer scientists have concentrated on streaming or online algorithms (that is algorithms that work with the data in the order it is available and use limited memory). For many problems we have found this an insufficient model and it is much better to assume you can re-order and replicate data (such as scattering data to many processors and re-collecting it to sort). The scatter/gather paradigm is ubiquitous and is the underpinning of large scale sorting, databases and Map Reduce. So in one sense databases and Map Reduce different APIs on top of very related technologies (journaling, splitting and merging). Replicating data (or even delaying duplicate elimination) that is already “too large to handle” may seem counterintuitive; but it is exploiting the primary property of secondary storage: that secondary storage tends to be much larger than primary storage (typically by 2 orders of magnitude, compare a 2 terabyte drive to an 8 gigabyte memory stick).In our web age, the typical big data problems are inverting indices (for fast search lookup) and computing term frequencies (for TF/IDF scoring or for things like Naive Bayes classifiers). Since these are over-worked examples we will use a mathematical problem from “Additive Combinatorics”, Terence Tao, Van Vu, (ISBN-13: 9780521853866; ISBN-10: 0521853869)

We take one problem from the field of additive combinatorics: sum sets. For two sets of integers A = {a_1, … a_s} and B {b_1, …, b_t} the sum set is defined as the set (without repetition) A + B = { a_i + b_j | i = 1,…s, j=1…t }. For sets of integers the size of A+B (denoted as |A+B|) can vary from |A| + |B| – 1 to |A| * |B| depending on the relations between the numbers in A and B (or the structure of A and B). If instead of working with integers we work with integers modulo p where p is a prime number (or equivalently we treat all numbers as remainders of division by p) then by the Cauchy-Davenport inequality we have |A + B| ≥ min(|A|+|B|-1,p) (so essentially the same result, except when we run out of possible integers modulo p).

For example we would say (working modulo 19) that [0, 1, 4, 5] + [10, 11, 14, 15] = [0, 1, 10, 11, 12, 14, 15, 16, 18]. In fact there are 19 pairs of sets that add up to [0, 1, 10, 11, 12, 14, 15, 16, 18] ( for instance [5, 6, 9, 10] + [5, 6, 9, 10] is another such pair). Just to move forward assume we were interested in determining how many ways a set can be written as the sum of a pair of sets (each of size 4). For a given sum result we might try search or integer programming to find all possible summands. However, if we want the statistics on all sums simultaneously, we can work much quicker and without need for big gun mathematics.

The straightforward solution is this case is a bit of code like:

for set A from all possible sets of 4 integers from 0 to 18
    for set B from all possible sets of 4 integers from 0 to 18
        let set C = A + B modulo 19
        use set C as a key and add the pair (A,B) to the list associated with C
for all key sets C tracked above
     compute the size of the list of summand pairs found for C
print how many result sets C have a given number of summand pairs

The relations C which have a summand of form A can be collected by any bit of Java code implementing the interface below (just call insertReln(C,(A,B)) to store the relations and then entries() to get them back). A small interface that declares the needed methods is given below:

public interface RelnCollector<A,B> {
	void insertReln(A a, B b) throws IOException;
	Iterable<Map.Entry<C,Iterable<B>>> entries() throws IOException, InterruptedException;
	void close() throws IOException;
}

An in-memory relation collector is trivially implemented by a nested map adjusted to declare the above interface, as we see in the next code snippet:

public final class InMemoryRelnCollector<A,B>
	implements RelnCollector<A,B> {
	private final DataAdapter<A> adapterA;
	private final DataAdapter<B> adapterB;
	private Map<A,Iterable<B>> atoBs;

	public InMemoryRelnCollector(final DataAdapter<A> adapterA,
		final DataAdapter<B> adapterB) {
		this.adapterA = adapterA;
		this.adapterB = adapterB;
		atoBs = new TreeMap<A,Iterable<B>>(this.adapterA);
	}

	@Override
	public void insertReln(final A a, final B b) {
		Set<B> set = (Set<B>) atoBs.get(a);
		if(null==set) {
			set = new TreeSet<B>(adapterB);
			atoBs.put(a,set);
		}
		if(!set.contains(b)) {
			set.add(b);
		}
	}

	@Override
	public Iterable<Map.Entry<A,Iterable<B>>> entries() {
		return atoBs.entrySet();
	}

	@Override
	public void close() {
		atoBs = null;
	}
}

The great savings in time is that we work from summands to results sums (but keep many sets of results indexed by result sets). Thus we don’t have to figure out how to invert the sum operation (as we do our bookkeeping forward). However, this very bookkeeping may overwhelm us. As we can see below, a Java implementation of the above procedure runs out of memory when trying to characterize which sets of integers modulo 19 can be split into two sets of size four (and how many ways each such set can be split). However, this was with the deliberately small default allocation of memory available to Java processes (so for this particular instance we could avoid trouble by allocating more memory, we ran out of allocation not system memory). What happens when we don’t manage memory is illustrated below:

Start	com.winvector.consolidate.impl.InMemoryRelnCollector
	Tue Dec 06 10:04:38 PST 2011
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.TreeMap.put(TreeMap.java:554)
	at java.util.TreeSet.add(TreeSet.java:238)
	at com.winvector.consolidate.example.AdditiveSets.sum(AdditiveSets.java:25)
	at com.winvector.consolidate.example.AdditiveSets.main(AdditiveSets.java:55)

An out of core solution can solve the entire problem without needing any additional system memory (just some disk space which is still of a much greater size than primary memory). The complete calculated result is given below:

Examining sums of 4 integers chosen from 0 through 18 modulo 19.
Start	com.winvector.consolidate.impl.FileRelnCollector
	Tue Dec 06 09:54:20 PST 2011
	Inserted 15023376 relations.
 [0, 1, 4, 5] + [10, 11, 14, 15] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [0, 1, 15, 16] + [0, 14, 15, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [0, 3, 4, 18] + [11, 12, 15, 16] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [0, 14, 15, 18] + [0, 1, 15, 16] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [1, 2, 5, 6] + [9, 10, 13, 14] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [1, 2, 16, 17] + [13, 14, 17, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [2, 3, 6, 7] + [8, 9, 12, 13] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [2, 3, 17, 18] + [12, 13, 16, 17] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [3, 4, 7, 8] + [7, 8, 11, 12] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [4, 5, 8, 9] + [6, 7, 10, 11] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [5, 6, 9, 10] + [5, 6, 9, 10] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [6, 7, 10, 11] + [4, 5, 8, 9] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [7, 8, 11, 12] + [3, 4, 7, 8] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [8, 9, 12, 13] + [2, 3, 6, 7] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [9, 10, 13, 14] + [1, 2, 5, 6] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [10, 11, 14, 15] + [0, 1, 4, 5] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [11, 12, 15, 16] + [0, 3, 4, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [12, 13, 16, 17] + [2, 3, 17, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [13, 14, 17, 18] + [1, 2, 16, 17] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
	Examined 128820 sums and 15023376 summands.
	found 3705 sums with 19 distinct summands
	found 39900 sums with 38 distinct summands
	found 26847 sums with 76 distinct summands
	found 22230 sums with 114 distinct summands
	found 10602 sums with 152 distinct summands
	found 8892 sums with 190 distinct summands
	found 2736 sums with 228 distinct summands
	found 5016 sums with 266 distinct summands
	found 2736 sums with 304 distinct summands
	found 1710 sums with 342 distinct summands
	found 171 sums with 361 distinct summands
	found 1710 sums with 380 distinct summands
	found 855 sums with 418 distinct summands
	found 342 sums with 456 distinct summands
	found 342 sums with 532 distinct summands
	found 342 sums with 570 distinct summands
	found 171 sums with 722 distinct summands
	found 171 sums with 760 distinct summands
	found 171 sums with 912 distinct summands
	found 171 sums with 1026 distinct summands
Done:	com.winvector.consolidate.impl.FileRelnCollector
   elapsed time: 618473MS
   Tue Dec 06 10:04:38 PST 2011

We performed the calculation be using a different implementation of RelnCollector called FileRelnCollector. What this implementation does is write relations to a file as they are made available. That is FileRelnCollector implementation of insertReln is literally a println(). Something not more more complicated than the following:

	@Override
	public void insertReln(final A a, final B b) {
		System.out.println("" + a + "\t" + b);
	}

The heavy lifting is done when entries() is called. When the entries are wanted the FileRelnCollector calls GNU sort on the saved file to get all the results ordered by result sum (instead of by summand). GNU sort can sort files larger than memory by a split and merge strategy involving temporary files. We provide such a file plus GNU sort based implementation of RelnCollector.

Note that this runtime can be deceptively low. If running on a machine with a modern operating system and enough memory the file being used as "external storage" actually gets cached into memory (and gets near memory speed performance). To get a reliable timing you need to test a problem of the size you are interested in on the size machine you are going to deploy on (not on a larger machine).

For better or worse this method should seem familiar as a lot of science has been done using the Unix text tools (sort, join and a few more). This is also the basis of Map Reduce and we demonstrate a Hadoop implementation of RelnCollector as well. Or we can link up with the other technology designed for beyond memory size data manipulation and get a database based implementation of RelnCollector.

In all cases the implementations we call depend on journaling (in the sense of keeping a sequential log of operations to be done instead of immediately performing the operations), scattering (splitting into multiple temp files and structures) and merging (combining data form multiple ordered files). We could write our own code to perform all of these operations (obliviating any need for GNU sort, Hadoop or a database), but it is much less code to do as we have here and write an adapter to use existing implementations.

The sum-set example is deliberately artificial. More common examples are, as we mentioned, index inversion and term frequency calculation. All of our example code is available here: https://github.com/WinVector/OutOfCore including JUnit tests and an example program. The code depends on libraries for JUnit 4.10, h2 database, Hadoop 0.21.0 for the various implementations.

The main trick is basing your code on a very thin storage abstraction (like the RelnCollector interface, instead of explicitly known data structures) and then using this abstraction to hide all of the details away from the rest of your code (keeping complexity at a manageable level). The two things to avoid are either infecting your code with too much knowledge of your storage plans (i.e. pushing implementation details into your important code to "speed things up") or being forced to re-design your entire project to fit within some framework (like re-writing all of your code as a database stored procedure or an explicit Hadoop map/reduce pair as this over-commits you to one technology).

Related posts:

1. Automatic Differentiation with Scala
2. Large Data Logistic Regression (with example Hadoop code)
3. SQL Screwdriver

Related posts:

1. A Quick Appreciation of the Sharpe Ratio
2. The Local to Global Principle
3. Automatic Differentiation with Scala

• System architects once were the people who said “no” to features to maintain design consistency and coherency. Now architects are the people who buy and bring in external frameworks and technologies (killing any chance of consistency or coherency). Kind of like the Fahrenheit 451 quote “I remember firemen used to fight fires.”
• By far the thing that aged the worst was the reverence for the WIMP (windows, icons, menus, pointing) paradigm. At this point I think we can argue that WIMP codified a lot of provably bad decisions: desktops, icons, menus and mouse out of visual field. Maybe some of the ideas prior to WIMP (like SAGE’s light-pens) or after WIMP (application launcher noun-verb theories like Quicksilver, search, touch pads, full screen apps, versioning and not forcing the user to adapt to the file storage abstraction) are actually much more fundamental. I think we all were seduced by the 1968 Engelbart demo but forget that the Semi Automated Ground Environment was a production deployed direct (light pen) multi user information sharing point and click system since 1959.
  
  

  SAGE station, Computer History Museum- Mountain View, CA

• Most everything else ages very well. The discussions of pain of having to work “out of core” remain relevant as this is what we now call “big data” (though in Brooks’ time this pain extends to documentation, source code and binaries all of which are too big to hold in memory or even in machine accessible format in the time of the IBM System/360).

  Though in the old days- “out of core” meant punched cards, punched tape, magnetic tape or very slow hard disks (which were a new luxury for the period Brooks writes about).
  

  
  

  SDS 920 with built in tape-drive, Computer History Museum- Mountain View, CA

• Linkers were among the biggest problems in the 1960s and remain the so now (though we now call it late binding, jars, shared libraries and APIs). At one point Brooks throws up his hands and says that it would be faster to just re-compile everything than to deal with some relocating linkers.
• Brooks definitely advocates and anticipates things like developer wikis (though he had to use microfiche as the computers of his day didn’t have enough storage to manage their own documentation).
• “Literate Programming” is clearly anticipated.
• Version control procedures are definitely written about, but Brooks seems not to anticipate version control software.

Overall: very well written and still interesting and relevant.

Related posts:

1. Map Reduce: A Good Idea
2. Good Graphs: Graphical Perception and Data Visualization
3. Must Have Software

public class SomeBigProject1 {
	public static double logStirlingApproximation(final int n) {
		return n*(Math.log(n)-1) + 0.5*Math.log(2*Math.PI*n);
	}

	public static double logFactorial(final int n) {
		double r = 0.0;
		for(int i=n;i>1;--i) {
			r += Math.log(i);
		}
		return r;
	}

	public static void main(final String[] args) {
		int nbad = 0;
		for(int n=1000;n<10000;++n) {
			if(Math.abs(logFactorial(n)-logStirlingApproximation(n))>=1.0e-5) {
				++nbad;
			}
		}
		System.out.println("nbad: " + nbad);
	}
}

Imagine that this is some humongous project to use Stirling’s Approximation as a replacement for factorial. All the code up until main is great. But the unfortunate developer has hard-coded an acceptance test into main(). If they run their big project all they get out is:

nbad: 7334

The developer needs to re-code and re-build to diagnose the failure, tweak their acceptance criteria or add more measurements.

I strongly recommend a different work pattern. Instead of bringing criteria into the code, bring the data out:

public class SomeBigProject2 {
	public static void main(final String[] args) {
		System.out.println("n"
				+ "\t" + "logFactorial"
				+ "\t" + "logStirlingApproximation");
		for(int n=1000;n<10000;++n) {
			System.out.println(String.valueOf(n)
					+ "\t" + SomeBigProject1.logFactorial(n)
					+ "\t" + SomeBigProject1.logStirlingApproximation(n));
		}
	}
}

Capture this output in a file named “data.tsv” and both Microsoft Excel and R can open it. Naturally I prefer to use R (so that is what I will demonstrate). To read the results into R you start up an R and type in a command like the following:

 > d <- read.table('data.tsv',
        header=T,sep='\t',quote='',as.is=T,
        stringsAsFactors=F,comment.char='',allowEscapes=F)

Most of the arguments controlling the style of file R is to expected (what the field separator is, weather to expect escapes and quotes and so on). The settings I suggest here are the “ultra hardened” settings. If you make sure none of your fields have a tab or line-break in them when you print then it is guaranteed R can read the data (not matter what whacky symbols are in it). On the java side that usually means making sure any varying text fields are run through .replaceAll("\\s+"," ") “just in case.” At this point you can already look at your data with the summary() command:

 > summary(d)

       n         logFactorial   logStirlingApproximation
 Min.   :1000   Min.   : 5912   Min.   : 5912
 1st Qu.:3250   1st Qu.:23034   1st Qu.:23034
 Median :5500   Median :41870   Median :41870
 Mean   :5500   Mean   :42536   Mean   :42536
 3rd Qu.:7749   3rd Qu.:61653   3rd Qu.:61653
 Max.   :9999   Max.   :82100   Max.   :82100

This immediately hints that you should have been thinking in terms of relative error instead of absolute error (since insisting on high absolute accuracy on large results does not always make sense).

You also have access to standard statistical measures of agreement like correlation:

 > with(d,cor(logFactorial,logStirlingApproximation))

result: 1

You can see where your failures were:

 > library(ggplot2)
 > d$bad <- with(d,abs(logFactorial-logStirlingApproximation)>=1.0e-5)
 > ggplot(d) + geom_point(aes(x=n,y=bad))

Yields the graph:

You can see all your failures are in the initial interval. You can then drill in:

 > ggplot(d) + geom_point(aes(x=n,y=logFactorial-logStirlingApproximation))
                + scale_y_log10()

And here we see some things (that are in general true for Stirling’s approximation):

1. It is very accurate.
2. It is always an under estimate.
3. It gets better as n gets larger.

Essentially by poking around with graphs in R you can figure out the nature of your errors (telling you what to fix) and generate findings that tell you how to fix your criteria (perhaps your code is working- but your test wasn’t sensible). The “dump everything and then use R” technique is also particularly good for generating reports on code timings using either geom_histogram or geom_density.

For example, if we had data with a field runTimeMS then it is a simple one-liner to get plot like the following:

 > ggplot(t) + geom_density(aes(x=runTimeMS))

From this graph we can immediately see:

1. Most of our run-times are very fast.
2. We have a heavy right-tail (evidence of “contagion” or one slow-down causing others, like CPU or IO contention).
3. Data is truncated at 100MS (could be something “censoring” the measurement, an exception being thrown or an abort).
4. There is a spike at 30MS (something is true and slow for some subset of the data that isn’t present in the majority).

This is a lot more that would be seen in a mean-only or mean and standard deviation summary. We may even being seeings signs of two different bugs (the truncation and the spike).

In all cases the key is to dump a lot of data in machine readable form and then come back to to analyze. This is far more flexible than hoping to code in the right summaries and then further hoping the summaries don’t miss something important (or that you at least get a chance to notice if they do miss something). Being able to do exploratory statistics on dumps from your code (both results and timing) gives you incredible measurement, tuning and debugging powers. The scriptability of R means any later analysis is as easy as cut and paste.

Related posts:

1. Automatic Differentiation with Scala
2. R examine objects tutorial
3. Survive R

Related posts:

1. Automatic Differentiation with Scala
2. Programs reduced to statistics
3. I know, I am the one being a jerk

Back to the tar file issue. We had been meaning to get our code up on github or some other public source control system. Instead we have HTMLified it (with some cross reference links, it still isn’t pretty).

And Antonio Piccolboni, thanks for the great article.

Related posts:

1. What to do when you run out of memory
2. SQL Screwdriver
3. Programmers Should Know R

Hal 9000

There have been vast improvements in artificial intelligence. Much of the improvement has been driven by the engineering effects of Moore’s Law (resulting in my mobile phone’s processor having 12 times the clock speed and over 32 times the memory of an $8 million Cray 1 super computer) and significant machine learning research results. These machine size changes happened during the productive careers of many researchers, so ideas are often evaluated at a series of radically different machine capabilities and data scales.

Cray 1

von Neuman himself commented that scale was a major limiting factor in early computers. He asked the question how you could be expected to achieve anything significant even from a roomful of geniuses if (as with his early computers) all notes, communication and memory were limited to less than a single typed page. von Neuman’s comment stands in contrast to science fiction scientists and early boosters of artificial intelligence who always seem to be in awe of their own creations. Computers are certainly much larger- but we need to be humble and put off deciding if we are yet in the era of large computers (compared to human or animal brains). Everything we are doing now may still just be artificial intelligence’s pre-history and prologue. Feynman in his lectures on computation mentions that RNA transcription can be estimated to take around 100 kT of energy to transcribe a bit while a transistor may easily use 100,000,000 kT energy units to switch states. This means for the amount of heat the human head dissipates (energy supply and heat dissipation are rapidly becoming the most relevant measures of computational power) you could do a million times more work using RNA techniques (if you knew how) than with transistors. So computers may not yet be what we should call large (though they are likely getting there). What we currently call “datacenters” are in fact block sized computers (consuming an enormous amount of energy and dissipating a huge amount of heat).

A datacenter (or a block sized computer)

Not all improvements in machine intelligence have come from (or are to come from) improvements in hardware. Many of the improvements came from machine learning research results and these are what I will outline below.

Early machine learning algorithms were driven by analogy. This led us to perceptrons (1957, fairly early in the history of computer science) and neural nets. These methods have their successes but were largely over used and developed before researchers developed a good list of desirable properties of a machine learning method.

Neural Net diagram

These methods live on but are, in my opinion, not currently competitive. Some of their important ideas and contributions have been revived from time to time, such as the online update rules becoming what we now call stochastic gradients.

A list of (often incompatible) desirable properties of a machine learning algorithm is the following:

• Able to represent complicated functions
• Good generalization performance (quality predictions on data not seen during training)
• Unique optimal model for a given set of data and feature definitions
• Efficient and well characterized solution method
• Consistent summary statistics
• Preference for simple models

We divert from this list for a bit of background and context.

The neural net was largely celebrated for its ability to represent complex functions and the perceived efficiency of its newer back-propagation based training method (related to the efficient calculation of gradients). The downsides were you never knew if your neural net was the right one (even assuming you had the right features, layout and training data) and could not be sure you were biasing towards simple models that might perform well on novel queries. Great effort was expended in extending neural nets based on the supposition they should work as they were an analogy to how we imagined biological neurons might function. An almost mystic hope was derived from the non-linear nature and special properties of the sigmoid curve (which was in fact a curve already known to statisticians).

Other methods than neural nets also had early success. The field of information retrieval (which was not “sexy” prior to the Web) had huge success since the 1960s with Naive Bayes, Rocchio Classification, and TF/IDF methods. The early success of these methods may have in fact delayed research on current hot research areas such as segmentation and author topic models.

Theoretical computer science initially sought to characterize machine learning methods in non-statistical language. In the 1980s a great amount of ink was spilled on “learning boolean functions.” Papers proving nothing was learnable (by picking a function related to cryptography) alternated with papers proving everything was learnable (for example via amplification techniques like boosting). Generalization of models to new data remained a theoretical problem that was dealt with by appeals to model complexity and MDL (minimum description length). A major breakthrough in characterizing generalization performance was the PAC model (probably approximately correct) framework which finally allowed direct treatment of generalization performance.

We now have enough context to discuss some of the current best of breed machine learning techniques (that address many of the desired properties mentioned above):

• Support Vector Machines
• Kernel Methods
• Logistic Regression
• Maximum Entropy Methods
• Regularization
• Graphical Models
• Conditional Random Fields




Typical SVM maximum margin diagram

Not all of these methods are new (Logistic Regression for example dates from 1925 and is itself based on regression which goes back to Gauss). But the concerns these methods address are all much more statistical than artificial intelligence in nature. For example we don’t suppose that there is some cryptographically obscured combination of features that we need to find to make the best prediction. We instead worry about detecting which features are useful and note that it is a significant (though solvable) problem to correctly use combinations of useful features (phrased as statistical concerns: feature to feature dependencies and higher order interactions). Machine learning has always run where statisticians fear to tread. But more and more often we are seeing that the methods and concerns of statisticians are what are needed to achieve many of the listed desired properties of machine learning models.

The methods I have singled out for praise are very effective and achieve a number of our listed desired properties. For example: both logistic regression and maximum entropy have a unique solution that is easy to find. They are also both consistent with all summaries known during training. That is: if 30% of the positive training data has a feature present then 30% of the data also has the feature present when weighted by the model’s score (so the model score shares a lot of properties with training truth). Support Vector Machines also have well understood solutions and a theory (called maximum margin) that directly addresses generalization (good predictions on new data). Kernel Methods (both as used in SVMs and elsewhere) allow controlled introduction of very complex functions. Graphical Models and Conditional Random Fields also allow the controlled introduction of modeled dependencies in the data.

It is now common to call what was previously thought of as artificial intelligence or machine learning: “statistical machine learning.” This reflects that the kind of prediction and characterization we expect from machine learning algorithms are in fact statistical concerns that we can deal with if we have enough data and enough computational resources.

The current important issues for statistical machine learning include:

• Dealing with very large datasets (driving the return of simpler methods like Naive Bayes)
• Dealing with lack of training data (driving interest in clustering and manifold regularization methods)
• Dealing with unstructured data and text mining (driving interest in information extraction and segmentation via generative models)

Just as Wigner famously wrote about “The Unreasonable Effectiveness of Mathematics” in the 1960s Halevy,Norvig and Pereira write about the “Unreasonable Effectiveness of Data.” They argue that we are in the age of big data (or the age of analysts). Or, as Varian observed: “it is a good time to supply a good complementary to data” (i.e. it is a good time to be an analyst). I would temper this with we are likely in the age of unmarked data and unstructured data. Less often are we asked to automate a known prediction and more often we are asked to cluster, characterize and segment wild data. In my opinion the hard problem in machine learning has moved from prediction to characterization. With enough marked training data (that is data for which we know both the observables and desired outcome) it is now quite possible to use standard techniques and libraries to build a very good predictive model. However, it is still hard to characterize, segment or extract useful information from the wealth of unstructured and unmarked data that is upon us. And this is where a lot of the current research in statistical machine learning is directed.

Or course characterization and clustering have their own infamous history. Rota wrote: “… Or a subject is important, but nobody understands what is going on; such is the case with quantum field theory, the distribution of primes, pattern recognition and cluster analysis.” Artificial intelligence may be moving from areas where computer scientists have over-promised to areas where statisticians have over-promised. But this is not a disaster: the most valuable research tends to be done in hectic times in messy fields, not in calm times in neat fields. And the already large scale adoption of statistical machine learning techniques means there is immediate great client value in even seemingly small improvements in understanding, explanation, documentation, training, tools, libraries and techniques.

Classic attempt to add structure to text

(images from Wikipedia)

Related posts:

1. A Demonstration of Data Mining
2. Kernel Methods and Support Vector Machines de-Mystified
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)

The problem that got me thinking is this:

Given a sequence of n integers x1 through xn and an integer k (1 ≤ k ≤ n), find the mean value of all of the medians of the k-sized selections from x1 through xn. Or as a formula:

where x_s is defined as the sequence of integers whose indices are in the set s (not necessarily a contiguous sequence). The median is the “value in the middle” (a value such that half of the selected data are above it and half are below) and “(n choose k)” is the number of ways to choose k items from a collection of n items (which is just the number: n!/((n-k)! k!)). So our sum is adding up a number of terms and then dividing by the number of such terms to get the average or mean of the terms. We will call this sum a “mean of medians”.

Some obvious special cases are: for k=1 the
expression simplifies to the sum of the x_i divided by n (or just the mean of all the x_i) and for k=n it simplifies to the median of all of the x_i. For intermediate values of k it is not immediately obvious how to efficiently compute the value of the sum. Directly adding all (n choose k) terms (as the sum is written) would be very slow for large n with even moderate sized k. Instead we look to find some method that has the same value of the total sum, without directly computing all of the terms.

This gets us to the ad-hoc side of theoretical computer science. We need a clever idea. In this case the idea is simple. To keep things simple: suppose all of the n integers in our sequence have different values and that k is odd (neither of these conditions are important they just let us avoid some non-essential technicalities). What values can median(x_s) take on? median(x_s) is always x_i form some i in the subset s. In fact our sum is equivalent to:

This new sum has a reasonable number of terms- so we can actually calculate it directly if we knew the values of the terms. Without loss of generality assume the x_i are sorted in increasing order. Then the number of times x_i is the median of some x_s is exactly:

The complete solution calculating the mean of medians for distinct sorted x_i is:

A statistician would recognize this expression as a kind of centrally weighted Winsorized mean. The shape of the graph of weights (in this case the n=10, k=5) is suggestive of
a bounded normal window (though i is a rank, not a free-ranging value):

Likely we have re-invented a data treatment known to statisticians. But the above steps were really just combinatorics. What a theorist does is abstract something down to this sort of problem and think of variations and solutions. The formal side of theoretical computer science is attempting to organize all of the problems in the world into two classes: problems presumed easy and problems presumed hard.

For example- what if we had wanted to know the median of many means instead of the mean of many medians?
It turns out a small variation of the median of means problem is already known to be difficult. The hard version of the reversed problem is called “Kth largest subset” (this is a different K than we have been using up until now). The Kth largest subset problem is: given a sequence of integers and constants K and B do K or more distinct subsets have a sum of no more than B? The Kth largest subset problem is known to be “NP hard” which means that a whole host of other thought to be hard problems could be solved if we had the ability to solve this problem (see “Computers and Intractability: A Guide to the Theory of NP-Completeness” Michael R. Garey and David S. Johnson, 1979). The median of many means is not quite as expressive as the Kth largest subset problem (so we have not proven the median of many means is itself NP hard) but the relation is strong: to even verify that we had the right median we would need to solve a Kth largest subset problem with K=(n choose k)/2 and B= k*the_claimed_median (assuming we modified the subset problem to restrict itself to size k sub-sequences). If fact even checking if a given number is one of the possible means (let alone if it is the median of the means) is equivalent to the NP Complete knapsack problem. This correspondence makes us suspect that something as simple as the trick we devised for the mean of medians problem will not solve the median of means problem. One should not take this argument too seriously though, as the same argument would seem to apply to the pair of problems “min of means” and “mean of mins” both of which are in fact easy. We have not proven the median of means problem to be hard, we have just found relevant algorithms and related problems.

What theorists do is find these analogs and then do the heavy lifting (continue the work even when the solution is not simple) to find a way to either efficiently solve the problem at hand or prove it is in fact as hard as other important problems. This kind of thing is hard work for small gains, but the value accumulates because each of these gains is permanent. Finally additional variations of the problem are tried and characterized, to help check we hare not “leaving money on the table” (missing nearby improvements). Some of this may seem like pointless work on toy problems- but every once in a while one of these solutions or characterizations is exactly what is needed to take a large system (like a search engine) to the next level of performance.

Related posts:

1. Good Graphs: Graphical Perception and Data Visualization
2. The Data Enrichment Method
3. A Demonstration of Data Mining