Early computers were most limited by their paltry memory sizes. von Neumann himself commented that even a room full of genius mathematicians would not be capable of much if all they could communicate, think upon or remember were the characters on a single type written page (much more memory than the few hundred words available to the Eniac). The most visible portions of early computers are their external memories or secondary stores: card readers, paper tape readers and tape drives.

SDC 920 computer, Computer History Museum, Mountain View CA

Historically computer scientists have concentrated on streaming or online algorithms (that is algorithms that work with the data in the order it is available and use limited memory). For many problems we have found this an insufficient model and it is much better to assume you can re-order and replicate data (such as scattering data to many processors and re-collecting it to sort). The scatter/gather paradigm is ubiquitous and is the underpinning of large scale sorting, databases and Map Reduce. So in one sense databases and Map Reduce different APIs on top of very related technologies (journaling, splitting and merging). Replicating data (or even delaying duplicate elimination) that is already “too large to handle” may seem counterintuitive; but it is exploiting the primary property of secondary storage: that secondary storage tends to be much larger than primary storage (typically by 2 orders of magnitude, compare a 2 terabyte drive to an 8 gigabyte memory stick).In our web age, the typical big data problems are inverting indices (for fast search lookup) and computing term frequencies (for TF/IDF scoring or for things like Naive Bayes classifiers). Since these are over-worked examples we will use a mathematical problem from “Additive Combinatorics”, Terence Tao, Van Vu, (ISBN-13: 9780521853866; ISBN-10: 0521853869)

We take one problem from the field of additive combinatorics: sum sets. For two sets of integers A = {a_1, … a_s} and B {b_1, …, b_t} the sum set is defined as the set (without repetition) A + B = { a_i + b_j | i = 1,…s, j=1…t }. For sets of integers the size of A+B (denoted as |A+B|) can vary from |A| + |B| – 1 to |A| * |B| depending on the relations between the numbers in A and B (or the structure of A and B). If instead of working with integers we work with integers modulo p where p is a prime number (or equivalently we treat all numbers as remainders of division by p) then by the Cauchy-Davenport inequality we have |A + B| ≥ min(|A|+|B|-1,p) (so essentially the same result, except when we run out of possible integers modulo p).

For example we would say (working modulo 19) that [0, 1, 4, 5] + [10, 11, 14, 15] = [0, 1, 10, 11, 12, 14, 15, 16, 18]. In fact there are 19 pairs of sets that add up to [0, 1, 10, 11, 12, 14, 15, 16, 18] ( for instance [5, 6, 9, 10] + [5, 6, 9, 10] is another such pair). Just to move forward assume we were interested in determining how many ways a set can be written as the sum of a pair of sets (each of size 4). For a given sum result we might try search or integer programming to find all possible summands. However, if we want the statistics on all sums simultaneously, we can work much quicker and without need for big gun mathematics.

The straightforward solution is this case is a bit of code like:

for set A from all possible sets of 4 integers from 0 to 18
    for set B from all possible sets of 4 integers from 0 to 18
        let set C = A + B modulo 19
        use set C as a key and add the pair (A,B) to the list associated with C
for all key sets C tracked above
     compute the size of the list of summand pairs found for C
print how many result sets C have a given number of summand pairs

The relations C which have a summand of form A can be collected by any bit of Java code implementing the interface below (just call insertReln(C,(A,B)) to store the relations and then entries() to get them back). A small interface that declares the needed methods is given below:

public interface RelnCollector<A,B> {
	void insertReln(A a, B b) throws IOException;
	Iterable<Map.Entry<C,Iterable<B>>> entries() throws IOException, InterruptedException;
	void close() throws IOException;
}

An in-memory relation collector is trivially implemented by a nested map adjusted to declare the above interface, as we see in the next code snippet:

public final class InMemoryRelnCollector<A,B>
	implements RelnCollector<A,B> {
	private final DataAdapter<A> adapterA;
	private final DataAdapter<B> adapterB;
	private Map<A,Iterable<B>> atoBs;

	public InMemoryRelnCollector(final DataAdapter<A> adapterA,
		final DataAdapter<B> adapterB) {
		this.adapterA = adapterA;
		this.adapterB = adapterB;
		atoBs = new TreeMap<A,Iterable<B>>(this.adapterA);
	}

	@Override
	public void insertReln(final A a, final B b) {
		Set<B> set = (Set<B>) atoBs.get(a);
		if(null==set) {
			set = new TreeSet<B>(adapterB);
			atoBs.put(a,set);
		}
		if(!set.contains(b)) {
			set.add(b);
		}
	}

	@Override
	public Iterable<Map.Entry<A,Iterable<B>>> entries() {
		return atoBs.entrySet();
	}

	@Override
	public void close() {
		atoBs = null;
	}
}

The great savings in time is that we work from summands to results sums (but keep many sets of results indexed by result sets). Thus we don’t have to figure out how to invert the sum operation (as we do our bookkeeping forward). However, this very bookkeeping may overwhelm us. As we can see below, a Java implementation of the above procedure runs out of memory when trying to characterize which sets of integers modulo 19 can be split into two sets of size four (and how many ways each such set can be split). However, this was with the deliberately small default allocation of memory available to Java processes (so for this particular instance we could avoid trouble by allocating more memory, we ran out of allocation not system memory). What happens when we don’t manage memory is illustrated below:

Start	com.winvector.consolidate.impl.InMemoryRelnCollector
	Tue Dec 06 10:04:38 PST 2011
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.util.TreeMap.put(TreeMap.java:554)
	at java.util.TreeSet.add(TreeSet.java:238)
	at com.winvector.consolidate.example.AdditiveSets.sum(AdditiveSets.java:25)
	at com.winvector.consolidate.example.AdditiveSets.main(AdditiveSets.java:55)

An out of core solution can solve the entire problem without needing any additional system memory (just some disk space which is still of a much greater size than primary memory). The complete calculated result is given below:

Examining sums of 4 integers chosen from 0 through 18 modulo 19.
Start	com.winvector.consolidate.impl.FileRelnCollector
	Tue Dec 06 09:54:20 PST 2011
	Inserted 15023376 relations.
 [0, 1, 4, 5] + [10, 11, 14, 15] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [0, 1, 15, 16] + [0, 14, 15, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [0, 3, 4, 18] + [11, 12, 15, 16] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [0, 14, 15, 18] + [0, 1, 15, 16] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [1, 2, 5, 6] + [9, 10, 13, 14] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [1, 2, 16, 17] + [13, 14, 17, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [2, 3, 6, 7] + [8, 9, 12, 13] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [2, 3, 17, 18] + [12, 13, 16, 17] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [3, 4, 7, 8] + [7, 8, 11, 12] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [4, 5, 8, 9] + [6, 7, 10, 11] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [5, 6, 9, 10] + [5, 6, 9, 10] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [6, 7, 10, 11] + [4, 5, 8, 9] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [7, 8, 11, 12] + [3, 4, 7, 8] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [8, 9, 12, 13] + [2, 3, 6, 7] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [9, 10, 13, 14] + [1, 2, 5, 6] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [10, 11, 14, 15] + [0, 1, 4, 5] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [11, 12, 15, 16] + [0, 3, 4, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [12, 13, 16, 17] + [2, 3, 17, 18] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
 [13, 14, 17, 18] + [1, 2, 16, 17] = [0, 1, 10, 11, 12, 14, 15, 16, 18]
	Examined 128820 sums and 15023376 summands.
	found 3705 sums with 19 distinct summands
	found 39900 sums with 38 distinct summands
	found 26847 sums with 76 distinct summands
	found 22230 sums with 114 distinct summands
	found 10602 sums with 152 distinct summands
	found 8892 sums with 190 distinct summands
	found 2736 sums with 228 distinct summands
	found 5016 sums with 266 distinct summands
	found 2736 sums with 304 distinct summands
	found 1710 sums with 342 distinct summands
	found 171 sums with 361 distinct summands
	found 1710 sums with 380 distinct summands
	found 855 sums with 418 distinct summands
	found 342 sums with 456 distinct summands
	found 342 sums with 532 distinct summands
	found 342 sums with 570 distinct summands
	found 171 sums with 722 distinct summands
	found 171 sums with 760 distinct summands
	found 171 sums with 912 distinct summands
	found 171 sums with 1026 distinct summands
Done:	com.winvector.consolidate.impl.FileRelnCollector
   elapsed time: 618473MS
   Tue Dec 06 10:04:38 PST 2011

We performed the calculation be using a different implementation of RelnCollector called FileRelnCollector. What this implementation does is write relations to a file as they are made available. That is FileRelnCollector implementation of insertReln is literally a println(). Something not more more complicated than the following:

	@Override
	public void insertReln(final A a, final B b) {
		System.out.println("" + a + "\t" + b);
	}

The heavy lifting is done when entries() is called. When the entries are wanted the FileRelnCollector calls GNU sort on the saved file to get all the results ordered by result sum (instead of by summand). GNU sort can sort files larger than memory by a split and merge strategy involving temporary files. We provide such a file plus GNU sort based implementation of RelnCollector.

Note that this runtime can be deceptively low. If running on a machine with a modern operating system and enough memory the file being used as "external storage" actually gets cached into memory (and gets near memory speed performance). To get a reliable timing you need to test a problem of the size you are interested in on the size machine you are going to deploy on (not on a larger machine).

For better or worse this method should seem familiar as a lot of science has been done using the Unix text tools (sort, join and a few more). This is also the basis of Map Reduce and we demonstrate a Hadoop implementation of RelnCollector as well. Or we can link up with the other technology designed for beyond memory size data manipulation and get a database based implementation of RelnCollector.

In all cases the implementations we call depend on journaling (in the sense of keeping a sequential log of operations to be done instead of immediately performing the operations), scattering (splitting into multiple temp files and structures) and merging (combining data form multiple ordered files). We could write our own code to perform all of these operations (obliviating any need for GNU sort, Hadoop or a database), but it is much less code to do as we have here and write an adapter to use existing implementations.

The sum-set example is deliberately artificial. More common examples are, as we mentioned, index inversion and term frequency calculation. All of our example code is available here: https://github.com/WinVector/OutOfCore including JUnit tests and an example program. The code depends on libraries for JUnit 4.10, h2 database, Hadoop 0.21.0 for the various implementations.

The main trick is basing your code on a very thin storage abstraction (like the RelnCollector interface, instead of explicitly known data structures) and then using this abstraction to hide all of the details away from the rest of your code (keeping complexity at a manageable level). The two things to avoid are either infecting your code with too much knowledge of your storage plans (i.e. pushing implementation details into your important code to "speed things up") or being forced to re-design your entire project to fit within some framework (like re-writing all of your code as a database stored procedure or an explicit Hadoop map/reduce pair as this over-commits you to one technology).

Related posts:

1. Automatic Differentiation with Scala
2. Large Data Logistic Regression (with example Hadoop code)
3. SQL Screwdriver

• System architects once were the people who said “no” to features to maintain design consistency and coherency. Now architects are the people who buy and bring in external frameworks and technologies (killing any chance of consistency or coherency). Kind of like the Fahrenheit 451 quote “I remember firemen used to fight fires.”
• By far the thing that aged the worst was the reverence for the WIMP (windows, icons, menus, pointing) paradigm. At this point I think we can argue that WIMP codified a lot of provably bad decisions: desktops, icons, menus and mouse out of visual field. Maybe some of the ideas prior to WIMP (like SAGE’s light-pens) or after WIMP (application launcher noun-verb theories like Quicksilver, search, touch pads, full screen apps, versioning and not forcing the user to adapt to the file storage abstraction) are actually much more fundamental. I think we all were seduced by the 1968 Engelbart demo but forget that the Semi Automated Ground Environment was a production deployed direct (light pen) multi user information sharing point and click system since 1959.
  
  

  SAGE station, Computer History Museum- Mountain View, CA

• Most everything else ages very well. The discussions of pain of having to work “out of core” remain relevant as this is what we now call “big data” (though in Brooks’ time this pain extends to documentation, source code and binaries all of which are too big to hold in memory or even in machine accessible format in the time of the IBM System/360).

  Though in the old days- “out of core” meant punched cards, punched tape, magnetic tape or very slow hard disks (which were a new luxury for the period Brooks writes about).
  

  
  

  SDS 920 with built in tape-drive, Computer History Museum- Mountain View, CA

• Linkers were among the biggest problems in the 1960s and remain the so now (though we now call it late binding, jars, shared libraries and APIs). At one point Brooks throws up his hands and says that it would be faster to just re-compile everything than to deal with some relocating linkers.
• Brooks definitely advocates and anticipates things like developer wikis (though he had to use microfiche as the computers of his day didn’t have enough storage to manage their own documentation).
• “Literate Programming” is clearly anticipated.
• Version control procedures are definitely written about, but Brooks seems not to anticipate version control software.

Overall: very well written and still interesting and relevant.

Related posts:

1. Map Reduce: A Good Idea
2. Good Graphs: Graphical Perception and Data Visualization
3. Must Have Software

public class SomeBigProject1 {
	public static double logStirlingApproximation(final int n) {
		return n*(Math.log(n)-1) + 0.5*Math.log(2*Math.PI*n);
	}

	public static double logFactorial(final int n) {
		double r = 0.0;
		for(int i=n;i>1;--i) {
			r += Math.log(i);
		}
		return r;
	}

	public static void main(final String[] args) {
		int nbad = 0;
		for(int n=1000;n<10000;++n) {
			if(Math.abs(logFactorial(n)-logStirlingApproximation(n))>=1.0e-5) {
				++nbad;
			}
		}
		System.out.println("nbad: " + nbad);
	}
}

Imagine that this is some humongous project to use Stirling’s Approximation as a replacement for factorial. All the code up until main is great. But the unfortunate developer has hard-coded an acceptance test into main(). If they run their big project all they get out is:

nbad: 7334

The developer needs to re-code and re-build to diagnose the failure, tweak their acceptance criteria or add more measurements.

I strongly recommend a different work pattern. Instead of bringing criteria into the code, bring the data out:

public class SomeBigProject2 {
	public static void main(final String[] args) {
		System.out.println("n"
				+ "\t" + "logFactorial"
				+ "\t" + "logStirlingApproximation");
		for(int n=1000;n<10000;++n) {
			System.out.println(String.valueOf(n)
					+ "\t" + SomeBigProject1.logFactorial(n)
					+ "\t" + SomeBigProject1.logStirlingApproximation(n));
		}
	}
}

Capture this output in a file named “data.tsv” and both Microsoft Excel and R can open it. Naturally I prefer to use R (so that is what I will demonstrate). To read the results into R you start up an R and type in a command like the following:

 > d <- read.table('data.tsv',
        header=T,sep='\t',quote='',as.is=T,
        stringsAsFactors=F,comment.char='',allowEscapes=F)

Most of the arguments controlling the style of file R is to expected (what the field separator is, weather to expect escapes and quotes and so on). The settings I suggest here are the “ultra hardened” settings. If you make sure none of your fields have a tab or line-break in them when you print then it is guaranteed R can read the data (not matter what whacky symbols are in it). On the java side that usually means making sure any varying text fields are run through .replaceAll("\\s+"," ") “just in case.” At this point you can already look at your data with the summary() command:

 > summary(d)

       n         logFactorial   logStirlingApproximation
 Min.   :1000   Min.   : 5912   Min.   : 5912
 1st Qu.:3250   1st Qu.:23034   1st Qu.:23034
 Median :5500   Median :41870   Median :41870
 Mean   :5500   Mean   :42536   Mean   :42536
 3rd Qu.:7749   3rd Qu.:61653   3rd Qu.:61653
 Max.   :9999   Max.   :82100   Max.   :82100

This immediately hints that you should have been thinking in terms of relative error instead of absolute error (since insisting on high absolute accuracy on large results does not always make sense).

You also have access to standard statistical measures of agreement like correlation:

 > with(d,cor(logFactorial,logStirlingApproximation))

result: 1

You can see where your failures were:

 > library(ggplot2)
 > d$bad <- with(d,abs(logFactorial-logStirlingApproximation)>=1.0e-5)
 > ggplot(d) + geom_point(aes(x=n,y=bad))

Yields the graph:

You can see all your failures are in the initial interval. You can then drill in:

 > ggplot(d) + geom_point(aes(x=n,y=logFactorial-logStirlingApproximation))
                + scale_y_log10()

And here we see some things (that are in general true for Stirling’s approximation):

1. It is very accurate.
2. It is always an under estimate.
3. It gets better as n gets larger.

Essentially by poking around with graphs in R you can figure out the nature of your errors (telling you what to fix) and generate findings that tell you how to fix your criteria (perhaps your code is working- but your test wasn’t sensible). The “dump everything and then use R” technique is also particularly good for generating reports on code timings using either geom_histogram or geom_density.

For example, if we had data with a field runTimeMS then it is a simple one-liner to get plot like the following:

 > ggplot(t) + geom_density(aes(x=runTimeMS))

From this graph we can immediately see:

1. Most of our run-times are very fast.
2. We have a heavy right-tail (evidence of “contagion” or one slow-down causing others, like CPU or IO contention).
3. Data is truncated at 100MS (could be something “censoring” the measurement, an exception being thrown or an abort).
4. There is a spike at 30MS (something is true and slow for some subset of the data that isn’t present in the majority).

This is a lot more that would be seen in a mean-only or mean and standard deviation summary. We may even being seeings signs of two different bugs (the truncation and the spike).

In all cases the key is to dump a lot of data in machine readable form and then come back to to analyze. This is far more flexible than hoping to code in the right summaries and then further hoping the summaries don’t miss something important (or that you at least get a chance to notice if they do miss something). Being able to do exploratory statistics on dumps from your code (both results and timing) gives you incredible measurement, tuning and debugging powers. The scriptability of R means any later analysis is as easy as cut and paste.

Related posts:

1. Automatic Differentiation with Scala
2. R examine objects tutorial
3. Survive R

Related posts:

1. Automatic Differentiation with Scala
2. Programs reduced to statistics
3. I know, I am the one being a jerk

Back to the tar file issue. We had been meaning to get our code up on github or some other public source control system. Instead we have HTMLified it (with some cross reference links, it still isn’t pretty).

And Antonio Piccolboni, thanks for the great article.

Related posts:

1. What to do when you run out of memory
2. SQL Screwdriver
3. Programmers Should Know R

Previously we discussed some of the issues of large scale data analytics. A lot of the work done at the MapReduce scale is necessarily limited to mere aggregation and report generation. But what of medium scale? That is data too large to perform all steps in your favorite tool (R, Excel or something else) but small enough that you are expected to produce sophisticated models, decisions and analysis. At this scale, if properly prepared, you don’t need large scale tools and their limitations. With extra preparation you can continue to use your preferred tools. We call this the realm of medium scale data and discuss a preparation tool style we call “screwdriver” (as opposed to larger hammers).

We stand the “no SQL” movement on its head and discuss the beneficial use of SQL without a server (as opposed to their vision of a key-value store without SQL). Database servers can be a nuisance- but that is not enough reason to give up the power of relational query languages.

One of the tenants of the MAD analytics movement is that you have to constantly move new data towards your decision problem. This wisdom is compatible with the machine learning rule that 90% of your effort is spent on feature design and another 90% is needed for data-tubing. As an example: suppose you are attempting to predict or model the probability that a candidate will make a given purchase or take out a given loan. You might want to merge what you know about candidates (both past candidates which form your training data and future candidates that you are trying to characterize) with other data sources before you start your machine learning process. An example “other data source” is the Census ZCTA arranged data which is aggregated census data (age, income, education and so on) keyed by ZCTA (ZIP Code Tabulation Areas). If you had a table of per-person data called “people” then what you want is to merge the ZCTA data into this table as additional columns. By far the most reasonable way to express this known as a “join”. The SQL expression for such a join looks like the following:

SELECT people.*, zctaSummaries.*
      FROM people LEFT JOIN ( zctaSummaries )
      ON ( people.ZIPCODE = zctaSummaries.ZCTA )

This is the SQL way of saying “make a new table where each row is a row from my people data with the ZCTA data appended as additional columns.” Notice, and
this is the one grace of SQL, that we do not have to specify how this assembly is to be done (no loops, variables or explicit sorting). This is what we want. What we don’t want is the pain of setting up a persistent database server just so we can run some SQL. A server involves processes listening on ports, passwords, provisioning, maintenance and so on. The idea is we should only specify how we want our data prepared for analysis (or de-normalized in database terms), we shouldn’t have to specify how it is done or manage a server just to get it done.

Luckily there are a number of no-server databases that implement SQL. In particular we call out H2. H2 is a pure Java SQL engine, so any place we can run Java we can run H2. And one of the better graphical database clients (SQuirreL SQL) is both free and compatible with H2. So not only can you run your SQL- you can explore your data interactively!

To go further with this example you need a few minor database tools we are releasing under the GPL3 Affero License. The latest copy of WinVectorLogistic.Hadoop0.20.2.jar contains both compiled Java classes and Java 6 compatible source code. This jar includes the tools that talk to H2 (or any other JDBC compatible database) that embody our SQL Screwdriver idea. To complete the tutorial you will need to download the WinVector Logistic jar, a H2 distribution (contains h2-1.2.147.jar) and the pre-processed ZCTA data (zctaSummaries.tsv).

Next we build a Java XML style properties file describing our database. In our case we leave user and password blank, specify use of the H2 embedded driver and name the file that will be the backing store for our small database (in this case H2TestDB).

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
  <comment>testdb</comment>
  <entry key="user"></entry>
  <entry key="password"></entry>
  <entry key="driver">org.h2.Driver</entry>
  <entry key="url">jdbc:h2:H2TestDB/H2DB
         ;LOG=0;CACHE_SIZE=65536;LOCK_MODE=0;UNDO_LOG=0</entry>
</properties>

Once we have done this we can transfer the tab-separated data into our database by running the following java command:

  java -cp WinVectorLogistic.Hadoop0.20.2.jar:h2-1.2.147.jar
     com.winvector.db.LoadTable
     file:h2Test.xml t file:zctaSummaries.tsv zctaSummaries

The four final arguments are the location of the XML file we just created, “t” to denote tab separated data (“|” for pipe separated), the location of the ZCTA tab separated data and what name to give the new table in the database. Once this has been run the contents of the ZCTA table are in the database. To see that we start up Squirrel SQL, configure Squirrel SQL’s H2 driver to point to our h2-1.2.147.jar and configure a database alias identical to our XML file. One of the great merits of Squirrel SQL is it uses JDBC configuration just like any other Java program- so if you can get it to run in some other program you can get Squirrel SQL to work (and vise versa).

Once Squirrel SQL is up we can examine our table by running a simple select:

   select * from zctaSummaries

And we see our data:

We can now use another small tool to dump our freshly joined data into a tab separated format ready for use by an analysis tool (like R):

java -cp WinVectorLogistic.Hadoop0.20.2.jar:h2-1.2.147.jar
   com.winvector.db.DBDump
   file:h2Test.xml
   "SELECT people.*, zctaSummaries.*
       FROM people LEFT JOIN ( zctaSummaries )
       ON ( people.ZIPCODE = zctaSummaries.ZCTA )"
   mergedData.tsv

The last three arguments being: the database definition XML again, the exact query we want and the name of a file to write tab separated results into. At this point we are done. The file “mergedData.tsv” can be moved into R for modeling and analysis. We can, for neatness or security, now dispose of the database file if we wish. We have deliberately avoided the built-in bulk table import and export tools as they tend to be finicky and database implementation dependent (our screwdriver tools can be used to move in and out of persistent databases like MySQL just by specifying the correct JDBC driver, URL and driver jars). But the tools are not as important as the attitude of using powerful relational tools (e.g. SQL) in a batch manner on transient data stores (very different than OLTP and OLAP which have high maintenance costs and tend to trap data).

What we have done (with minimal preparation) is: brought the full relational power of SQL to perform the joins required to bring new data into an analysis. We can execute arbitrary SQL (much more powerful than Unix command line “join” and typical R table manipulation tools) and quickly get our data organized for machine learning analysis. We can work on datasets larger than machine memory if needed and have not incurred the cost of configuring or getting access to a server.

Instead of “no SQL” we say: “no server” (which is appropriate in the medium sized data regime so common in predictive analytics).

Related posts:

1. What to do when you run out of memory
2. Programmers Should Know R
3. Brevity is a Virtue

Stochastic gradient descent is in fact an appropriate method for big data. For example: if our model complexity is held constant and our data set size is allowed to grow; then stochastic gradient descent will achieve its convergence condition before it even completes a single random order traversal of the data. However, stochastic gradient descent has a control called the learning rate and one can easily imagine a series of problems that require the learning rate to be set arbitrarily slow. For example a data set formed as the union of very many “typical” examples where a given variable is independent of the outcome and small minority of “special” examples where the same variable helps influence the outcome presents a problem. Training on the “typical” examples causes the stochastic gradient descent method to perform a random walk on the given variable coefficient. So the learning rate must be slow enough that the expected drift does not swamp out the rare contributions from the “special” examples (meaning the learning rate must slow roughly proportionally to the square root of the ratio of the typical to special examples).

Not too much must be made of artificial problems designed to slow stochastic gradient descent. The traditional Fisher scoring (or the Newton/Raphston method) can simply be killed by specifying a problem with a great number of levels for categorical variables. In this case traditional methods have to solve a linear system that can in fact be much larger than the entire data set (causing representation, work and numeric stability problems). So it takes little imagination to design problems that kill the traditional methods. Other intermediate complexity methods (like conjugate gradient) avoid the storage size problem; but can require a many more passes through the training data.

There is a common situation where Fisher scoring makes good sense: you are trying to fit a relatively simple model to an enormous amount of data (often to predict a rare event). One could sub-sample the training data to shrink the scale of the problem- but this is a case of the analyst being forced to accede to poor tools. What one would naturally want is a training method that can fit reasonable sized models (that is models with a reasonable number of variables and levels) onto enormous data sets. The software package R can work with fairly large data sets (in the gigabytes range) and has some parallel flavors, but R is mostly an in-memory system. It is appropriate to want a direct method that both “works out of core” (i.e. in the terabytes and petabytes ranges), parallelizes to hundreds of machines (using current typical infrastructure- like a Hadoop cluster) and is exact (without additional parameters like learning rate).

We demonstrate here an example implementation in Java for both single machine “out of core” training (allowing filesystem sized datasets) and MapReduce style parallelism (allowing even larger scale). The method also includes the problem regularization steps discussed in our recent logistic regression article. The code (packaged in: WinVectorLogistic.Hadoop0.20.2.jar ) is being distributed under the GNU Affero General Public License version 3. This is an open source license that (roughly) requires (among other things) redistribution of source code of systems linked against the licensed project to anyone receiving a compiled version or using the system as a network service. The license also promises no warranty or implied fitness. The distribution is a standalone runnable Jar (source code and license inside the jar) and is the minimal object required to run on Hadoop (which is itself a Java project). More advanced versions of the library (with better linear algebra libraries, better problem slice control, unit tests, JDBC bindings and with different license arrangements) can be arranged from the code owners: Win-Vector LLC. This jar was built for Apache Hadoop version 0.20.2 (the latest version Amazon Elastic Map Reduce runs at this time) and we use as many of the newer interfaces as possible (so the code will run against the current Hadoop 0.21.0 if re-built against Hadoop 0.21.0, the jar can not switch versions without being re-built due to how Hadoop calls methods).

For our example we will work on a small data set. The code is designed to pass through data directly from disk, storing only the Fisher structures- which require storage proportional to the square of the number of variables and levels but is independent of the number of data rows. The data format is what we call “naive TSV” or “naive tab separated values.” This is a file where each line has exactly the same number of values (separated by tabs) and the first line of the file is the header line naming each column. This is compatible with Microsoft Excel and R with the proviso that this file format does not allow any sort of escapes, quoting or multiple line fields. Our data set is taken from the UCI machine learning database ( data, description ) and converted into the naive TSV format (split into training and testing subsets: uciCarTrain.tsv, uciCarTest.tsv).

The first few lines of the training file are given here:

buying	maintenance	doors	persons	lug_boot	safety	rating
vhigh	vhigh	2	2	small	med	FALSE
vhigh	vhigh	2	2	med	low	FALSE
vhigh	vhigh	2	2	med	med	FALSE

The first experiment is to use the Java program standalone (without Hadoop) to train a model. The method used is Fisher scoring by multiple passes over the data file. Only the Fisher structures are stored in memory- so in principle the data set could be arbitrarily large. To run the logistic training program download the files WinVectorLogistic.Hadoop0.20.2.jar and uciCarTrain.tsv . You will also need some libraries ( commons-logging-*.jar and commons-logging-api-*.jar , and sometimes hadoop-*-core.jar and log4j-*.jar ) from the appropriate Hadoop distribution. Before running the code you can examine the source (and re-build the project using an IDE like Eclipse) by extracting the code in an empty directory using the Java jar command:

jar xvf WinVectorLogistic.Hadoop0.20.2.jar

To run the code type at the command line (all in a single line, we have inserted line breaks for clarity, we are also assuming you are using a Unix style shell on Linux, OSX or Cygwin on Windows):

java -cp WinVectorLogistic.Hadoop0.20.2.jar:commons-logging-1.0.4.jar:commons-logging-api-1.0.4.jar
   com.winvector.logistic.demo.LogisticTrain
   file:uciCarTrain.tsv "rating ~ buying + maintenance + doors + persons + lug_boot + safety" model.ser

The portion of interest is the last three arguments:

• file:uciCarTrain.tsv : The URI pointing to the file containing the training data.
• “rating ~ buying + maintenance + doors + persons + lug_boot + safety” : The formula specifying that rating will be predicted as a function of buying, maintenance, doors, persons, lug_boot and safety.
• model.ser : Where to write the Java Serialized model result.

After that we can run the scoring procedure on the held-out test data:

java -cp WinVectorLogistic.Hadoop0.20.2.jar:commons-logging-1.0.4.jar:commons-logging-api-1.0.4.jar
   com.winvector.logistic.demo.LogisticScore
   model.ser file:uciCarTest.tsv scored.tsv

In this case the last three arguments are:

• model.ser : Where to read the Java Serialized model from.
• file:uciCarTest.tsv : The URI pointing to the file to make predictions for.
• scored.tsv : Where to write the predictions to.

The first few lines of the result file are:

predict.rating.FALSE	predict.rating.TRUE	buying	maintenance	doors	persons	lug_boot	safety	rating
0.9999999999999392	6.091299561082107E-14	vhigh	vhigh	2	2	small	low	FALSE
0.9999999824028766	1.759712345446162E-8	vhigh	vhigh	2	2	small	high	FALSE

These lines are just lines from the file uciCarTest.tsv (same format is uciCarTrain.tsv) copied over with the addition of the first two columns that show the modeled probabilities of rating acceptable being FALSE or TRUE. The accuracy of the prediction is computed and written into the runlog if the data had the rating outcomes in it (else we just get a file of predictions- which is the usual application of machine learning).

The details of running the Hadoop versions of the same process depend on the configuration of your Hadoop environment. Just unpacking the 0.20.2 version of Hadoop will let you try the single-machine version of the MapReduce Logistic Regression process (which will be much slower than the standalone Java version). To run the training step the Hadoop command line is as follows (notice this time we do not have to specify the logging jars as they are part of the Hadoop environment):

hadoop-0.20.2/bin/hadoop jar WinVectorLogistic.Hadoop0.20.2.jar
   logistictrain
   uciCarTrain.tsv "rating ~ buying + maintinance + doors + persons + lug_boot + safety" model.ser

And the scoring procedure is below:

hadoop-0.20.2/bin/hadoop jar WinVectorLogistic.Hadoop0.20.2.jar
   logisticscore
   model.ser uciCarTest.tsv scoredDir

The only operational differences are that the results are written into the file scoredDir/part-r-00000 (as is Hadoop convention) instead of scored.tsv (and an extra “offset” column is also included) and data is handled in Files (to allow Hadoop Paths to be formed) instead of URIs. The Hadoop training and test steps are able to run in this manner because we have constructed WinVectorLogistic.Hadoop0.20.2.jar as an executable jar file with the class com.winvector.logistic.demo.DemoDriver as the class to execute. This class uses that standard org.apache.hadoop.util.ProgramDriver pattern to run our jobs under the org.apache.hadoop.util.Tool interface. This means that the standard Hadoop generic flags for specifying cluster configuration will be respected.

The big benefit of all of this packaging is: if this command is run on a large Hadoop cluster (instead of on a single machine) then the input file could be split up and processed in parallel on many machines. The easiest way to do this is to use Amazon.com’s Amazon Elastic MapReduce. This service (used in conjunction with S3 storage and EC2 virtual machines) allows the immediate remote provisioning and execution on a version 0.20.* Hadoop cluster. To demonstrate this service we created a new S3 Bucket named wvlogistic. Into wvlogistic we copied our jar of our code compiled against Hadoop 0.20.2 APIs ( WinVectorLogistic.Hadoop0.20.2.jar ) and a moderate sized synthetic training data set ( bigProb.tsv, created by running: java -cp WinVectorLogistic.Hadoop0.20.2.jar com.winvector.logistic.demo.BigExample bigProb.tsv ). Once this has been set up (and you have signed up for the Amazon Elastic MapReduce credentials) you can run the training procedure from the Amazon web UI. In five steps (following the direcitons found in Tutorial: How to Create and Debug an Amazon Elastic MapReduce Job Flow ) the job can be configured and launched.

First: press “Crate New Job Flow” and choose a job name, check “Run your own application” and select “Cusom Jar”.

Step 1/5

Second: specify the location of the jar in your Bucket and give the command line arguments (prepending S3 paths with “s3n://”).

Step 2/5

Third: select the type and number of machine instances you want, run without and EC2 key pair, enable logging and send the log back to your S3 bucket.

Step 3/5

Fourth: add the default bootstrap action of configuring the Hadoop cluster.

Step 4/5

Fifth: confirm and launch the job.

Step 5/5

When the job completes transfer the result ( bigModel.ser ) back to your local system and you have your new map reduced produced logistic model. We can confirm and use the model locally with a Java command similar to our earlier examples:

java -cp WinVectorLogistic.Hadoop0.20.2.jar:commons-logging-1.0.4.jar:commons-logging-api-1.0.4.jar:hadoop-0.20.2-core.jar:log4j-1.2.15.jar
   com.winvector.logistic.demo.LogisticScore
   bigModel.ser bigProb.tsv bigScored.tsv

Be aware that at this tens of megabytes scale there is no advantage in running on a Hadoop cluster (versus using the stand-alone program). At moderate scale parallelism may not even be attempted (due to block size) and the costs of data motion can overcome the benefit of parallel scans. The biggest gain is being able to train many models from many gigabytes of data on a single machine without sub-sampling. While we have the ability to build a logistic model at “web scale” (terabytes or petabytes of data) you would not want to use the MapReduce calling pattern until you had a web-scale amount of training data.

The point of this exercise was to take a solid implementation of regularized logistic regression article and use the decomposition into the ” Statistical Query Model” (as suggested in the NIPS paper “Map-Reduce for Machine Learning on Multicore”) to quickly get an intermediate sophistication machine learning method (more sophisticated than Naive Bayes, less sophisticated than Kernelized Support Vector Machines) working at large (beyond RAM) scale. Briefly: most of the technique is in an interface that considers the mis-fit, gradient if mis-fit and hessian of mis-fit as a linear (summable) function over the data. Or in the “book’s worth of preparation so we can write the result in one line” paradigm: all of the machinery we have been discussing is support so the following summable interface (part of the source code we are distributing) can be used to do all of the work:

Summable Interface

Of course once you have the framework up that makes one non-trivial task easy you have likely made many other non-trivial tasks easy.

We hope this demonstration and examining the source code in our WinVectorLogistic.Hadoop0.20.2.jar will help you find ways to tackle your large data machine learning problems.

Code License:

Packages com.winvector.*, extra.winvector.*
Code for performing logistic regression on Hadoop.
Copyright (C) Win Vector LLC 2010 (contact: John Mount jmount@win-vector.com).
Distributed under GNU Affero General Public License version 3 (2007, see http://www.gnu.org/licenses/agpl.html ).
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, only version 3 of the
License.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see .
(Source code in jar, see also http://www.win-vector.com/blog/2010/12/large-data-logistic-regression-with-example-hadoop-code/ )

usage: com.winvector.logistic.LogisticTrain
 -formula <arg>      formula to fit
 -inmemory           if set data is held in memory during training
 -resultSer <arg>    (optional) file to write seriazlized results to
 -resultTSV <arg>    (optional) file to write TSV results to
 -trainClass <arg>   (optional) alternate class to use for training
 -trainHDL <arg>     XML file to get JDBC connection to training data
                     table
 -trainTBL <arg>     table to use from database for training data
 -trainURI <arg>     URI to get training TSV data from

usage: com.winvector.logistic.LogisticScore
 -dataHDL <arg>      XML file to get JDBC connection to scoring data table
 -dataTBL <arg>      table to use from database for scoring data
 -dataURI <arg>      URI to get scoring data from
 -modelFile <arg>    file to read serialized model from
 -resultFile <arg>   file to write results to

Related posts:

1. The equivalence of logistic regression and maximum entropy models
2. The Simpler Derivation of Logistic Regression
3. Learn Logistic Regression (and beyond)

The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

then the following is a picture of reverse accumulation:

Related posts:

1. Automatic Differentiation with Scala
2. Automatic Detection of Potential Deadlock
3. Must Have Software

Our Example Problem

Our small scale problem is this: we have a number of target points on a map and we want to pick a central point to directly connect to all of these points with wire. Our goal is to minimize the total amount of wire used. This problem is called the “Geometric Median”. So we are trying to find a point that minimizes the sum of distances from our chosen center to every target point. If we were trying to minimize the sum of squared distances from our chosen center to every target point the answer would be obvious: the average or mean (which by Hooke’s law is also the point where a set of identical springs would relax to). The mean is in fact a fairly good guess, but you can do better (which could important if the “wire” is expensive, such as cutting irrigation or drainage ditches). For example given the three target points (20,0), (-1,-1) and (-1,1) the optimal point is (-0.42,0) not the mean (6,0) and the choice of optimal point represents an over 19% savings in total wiring distance (see figure).

This is a substantial saving in cost.

The problem changes as we consider variations. If indirect connections (such as routing one point through another, which may or may not be possible for reasons of capacity or safety) and multiple new centers are allowed we then have an instance of the Steiner Tree Problem which is harder to solve (since it is known to be NP complete). If no new centers are allowed (all routing must be between pre-existing target points) then we have a Spanning Tree Problem- which admits very quick solutions.

We bring up the geometric median as a mere example. We don’t intend for our code to solve only the geometric median problem and we don’t intend to touch on the literature of specialized methods for solving the geometric median problem. Instead we are trying to demonstrate the speed you can develop prototype solutions if you have a few good tools (like various optimizers) available in your toolkit. Numeric optimizers may sound exotic, but they often are the kind of thing you want to experiment with and link directly into your code.

Optimization as General Tool

Now that we have the example problem we can describe a solution strategy. In this case the solution uses code “we wished we had lying around” before we started on the problem. We will pretend we have the tools we want ready to solve our problem and then we will pay our debt and build the required tools. The issue is that there is not an obvious closed form for the solution of the geometric median problem. So we are forced to work a bit harder. In this case harder means we need to solve an optimization problem. Consider the contour plot of the total wiring cost as function of where we choose to place our center. Our optimal point (-0.42,0) had wiring cost of 22.73 and the contour plot given here shows concentric regions of solution positions with higher cost.

In general it is unwise to throw an optimizer at an arbitrary problem and hope to find the globally best solution. But in this case (and in many similar situations) we can prove that a simple local optimizer will in fact find the unique best solution. This is a property of the problem not of the optimizer. The concentric regions shown in the contour plot have a very nice shape: they are convex. That is: they have no intrusions- for any two points drawn from one of these shapes the straight line segment between these points stays inside the given shape. We don’t have to depend on observation- we can actually prove this is always the case for this problem. The wiring cost from a proposed center to any single target point is a convex function of where we choose to place our center (a convex function is a function whose graph never reaches above the secant line drawn between any two points on its graph). The total wiring cost is just the sum of the wiring costs to each target point. And to finish: the sum of a collection of convex functions is itself a convex function. Since the contour plot of a convex function has only convex shapes and we have proven the statement.

But how does this help us? There is a standard technique to find “local minima” of a function by inspecting a function for places where the gradient is zero (points where there is no obvious down hill direction on the contour plot). This technique usually can only be guaranteed to find local minima (places where no small change improves your situation). But there is no guarantee that the local minimum you find is in fact the global minimum (the best possible solution). Except when you are dealing with a convex function. When a function is convex then all of the local minima are always grouped together into a single convex connected shape (if not a line drawn between two remote minima would violate the convexity definition). And if the function is never flat then this set is a single unique point: the unique best solution. Our inspection technique will be a gradient driven optimizer- that is an optimizer that when the gradient is non-zero improves its objective by running down hill and halts when the gradient is zero.

The stated function to minimize is to sum the distance from our proposed center to each target point. We can write this as the sum of the distances:

( which is the traditional Euclidean or L2 distance). This function actually has one one subtle flaw that we will deal with in the appendix (see: Fixing Smoothness).

Using Scala to Apply the Optimization Solution

To find our optimal center placement using Scala we first write our cost or objective as a Scala function:

    val dat:Array[Array[Double]] = Array(
      Array( 20, 0.0),
      Array( -1.0, 1.0),
      Array( -1.0, -1.0)
    )

    def fx(p:Array[Double]):Double = {
      val dim = p.length
      val npoint = dat.length
      var total = 0.0
      for(k <- 0 to (npoint-1)) {
        var term = 0.0
        for(i <- 0 to (dim-1)) {
          val diff = p(i) - dat(k)(i)
          term = term + diff*diff
        }
        total = total + scala.math.sqrt(term)
      }
      total
    }

Scala is succinct and it is a great connivence to have a function definition capture data from its environment. What we would like to do is generate an initial guess as the solution (we use the mean as our initial guess) and then call an optimizer (in this case a conjugate gradient optimizer) to do all the work:

 val p0:Array[Double] = mean(dat)
 val (pF,fpF) = CG.minimize(fx,p0)

At this point we would be done, except the conjugate gradient method (which is superior to gradient descent and many the non-gradient methods) requires a gradient.
We could provide a numeric estimate of the gradient by the following divided difference method:

  def gradientD(f:Array[Double]=>Double,p:Array[Double]):Array[Double] = {
    val xdim = p.length
    val p2 = copy(p)
    val base = f(p2)
    val ret = new Array[Double](xdim)
    val delta = 1.0e-6
    for(i <- 0 to (xdim-1)) {
      p2(i) = p(i) + delta
      val fplus = f(p2)
      p2(i) = p(i)
      val diff = (fplus-base)/delta
      ret(i) = diff
    }
    ret
  }

This numeric divided difference method often outperforms non-derivative optimization methods (like Powell’s Method and the Nelder-Mead Amoeba method). But the technique can run into numeric difficulties. We can remedy this if we are willing to write our function in a slightly more general way. If we re-encode our function in a generic manner we can use automatic differentiation (not to be confused with numeric differentiation or with symbolic differentiation) to produce a reliable gradient for optimization. What we need to do is re-write our function to work over an abstract field of numbers instead of only the machine supplied doubles. In fact what we need to do is specify a generic function that will work over any field, with the field to be determined later. The code to do this in Scala is very similar to the non-generic code:

   val genericFx = new VectorFN {
      def apply[Y <: NumberBase[Y]](p:Array[Y]):Y = {
        val field = p(0).field
        val dim = p.length
        val npoint = dat.length
        var total = field.zero
        for(k <- 0 to (npoint-1)) {
          var term = field.zero
          for(i <- 0 to (dim-1)) {
            val diff = p(i) - field.inject(dat(k)(i))
            term = term + diff*diff
          }
          total = total + smoothSQRT(term)
        }
        total
      }
    }

Notice that code is very similar to the “def fx()” code. The key differences are that we had to define genericFx as extending a trait (a type of Scala interface) called VectorFN and inside this trait extension we defined a parameterized function name apply(). apply() is a generic function that is willing to work over any type Y where Y is at least of type NumberBase[Y] (we will get more into what that means in a moment). The difference in notation is that while the Scala function syntax can not specify a generic function with free type parameters (the incompletely specified Y) the Scala semantics are strong enough to implement this. In fact standard function definitions (such as “def fx()”) are just syntactic sugar for extending the Scala built-in Function1 trait. With a generic objective function in hand all we need is conjugate gradient code that is expecting a VectorFN (and willing to call apply() instead of just using naked function parenthesis) and some type NumberBase[Y] that can compute gradients for us. The Scala compiler can specialize our genericFx() into one version for quick calculation and another for gradients. How this is done is what we will discuss next. From our point of view our problem is solved with the following one line of code:

val (pF,fpF) = CG.minimize(genericFx,p0)

This should always be your goal- build sufficient preparation so your last step is a “obvious one liner.”

What Tools we Wish we Had Lying Around

We supply in our example some workable conjugate gradient code, but that is standard so we will not discuss it. What is of interest (and facilitated by Scala’s parametrized type system) is the implementation of dual numbers as a framework to supply automatic differentiation. An implementation of dual numbers as a NumerBase[DualNumber] type is the core of our demonstration.

Dual numbers are an algebraic structure written as pairs of real numbers “(a,b)”. The arithmetic table for dual numbers is given below:

In a dual number (a,b) “a” is the “large” or “standard” part of the number. You can check from the arithmetic table that the pair of dual numbers (a,0) and (c,0) behave just as we would expect the real numbers a and c to behave. In the dual number (a,b) “b” is the “small” or “ideal” portion of the number. From the multiplication rule above we can observe two rules: (0,b) * (c,0) = (0,b*c) (something small times anything else is small) and (0,b)*(0,d) = (0,0) (two small things become zero when multiplied). Essentially the dual numbers are carrying around the first two terms of a Taylor series: we get as a result both the function value and the function derivative. For a function f() over the real numbers we extend f() to work over the dual number by defining: f((a,b)) = (f(a),b f’(a)) (which is consistent with the previously defined arithmetic). We can check that the dual numbers numbers obey the usual laws of arithmetic (associative, commutative, distributive, identities and inverses). The punchline is that over the dual numbers the divided difference estimate of f’(x) (the derivative of f() evaluated at x) is in fact exact in the sense that f((x,1)) = (f(x),f’(x)) (or f((x,0)+(0,1)) – f((x,0)) = (0, f’(x))). Implementing the DualNumber class is little more than transcribing the above arithmetic table into Scala.

We have already seen how to write code that uses NumberBase[Y] types (genericFx() itself is an example). A more complicated example is the CG.minimize() code which not only accepts a generic function (in the form of VectorFN) but then specializes it to NumberBase[DualNumber] to compute gradients and also specializes to NumberBase[MDouble] for quick calculation during line searches (MDouble is just an adapter for machine Doubles, used for speed). The ability to re-specialize a function is one of the advantages of a parameterized type system. The DualNumbers are an example of forward automatic differentiation. We could also use the same object framework to capture a representation of the computation path and apply more sophisticated methods such as reverse automatic differentiation.

We give a link to a jar containing complete Scala source code including this example, the DualNumber implementation, a conjugate gradient implementation and some JUnit tests (all under a GPLv3 license) and will go on to describe some of the design decisions. The code is the bulky part of this work, so we will move on to discuss something more compact: types.

Types

If code is ever beautiful it is only when it is succinct. Among the most succinct forms of code are individual type signatures and interfaces (though the indiscriminate repetition of type signatures is rightly considered ugly bloat, which Scala works to avoid). Since we are distributing complete source we will describe only types and method signatures. The entry points to the code are the JUnit tests (organized in the ScalaDiff/test source directory and depending on JUnit which was not included) and the demo program in ScalaDiff/src/demo/Demo.scala).

To be a usable arithmetic type (like DualNumber or MDouble) you must extend the following parameterized abstract class:

abstract class NumberBase[NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  // basic arithmetic
  def + (that: NUMBERTYPE):NUMBERTYPE
  def - (that: NUMBERTYPE):NUMBERTYPE
  def unary_-():NUMBERTYPE
  def * (that: NUMBERTYPE):NUMBERTYPE
  def / (that: NUMBERTYPE):NUMBERTYPE  // that not equal to zero
  // more complicated
  def pow(that:Double):NUMBERTYPE
  def exp:NUMBERTYPE
  def log:NUMBERTYPE // this is positive
  // comparison functions
  def > (that: NUMBERTYPE):Boolean
  def >= (that: NUMBERTYPE):Boolean
  def == (that: NUMBERTYPE):Boolean
  def != (that: NUMBERTYPE):Boolean
  def < (that: NUMBERTYPE):Boolean
  def <= (that: NUMBERTYPE):Boolean
  // utility
  def field:Field[NUMBERTYPE]
}

In particular DualNumber extends NumberBase[DualNumber]. This deliberate circular reference has a big purpose: it allows publicly visible covariant return types (returning nearly the exact type we really are instead of a base type). This allows us to have strict type arguments so that trying to add a MDouble to DualNumber is a type error (even though they both extend the same base class). The automatic differentiation technique encapsulated in the DualNumber class only works if all of the calculation is in the DualNumber types and this strict type enforcement allows the compiler to help prevent results sneaking in and out through other types. All of the methods on NumberBase are obviously related to arithmetic except the field() method. This method gives us access to a Field object which is responsible for carrying around the runtime type information (this is a common problem in Java and Scala, that some type information known at compile type such choice of template types is not easily accessed at runtime). The Field class is as follows:

abstract class Field [NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  def zero:NUMBERTYPE            // return canonical zero in field
  def one:NUMBERTYPE             // return canonical one in field
  def inject(v:Double):NUMBERTYPE  // return canonical representation of number in field
  def project(v:NUMBERTYPE):Double // return standard-number represented in field
  def array(n:Int):Array[NUMBERTYPE] // return an array of this type

The Field class is where we have factories for numbers (zero, one, arrays, injection from standard Doubles), casting (projection back to standard Doubles).

With these types defined we can actually read intent off some of the method signatures.

For example our conjugate gradient optimizer is accessed through the following method signature:

 def minimize(fn:VectorFN,x0:Array[Double]):(Array[Double],Double) // return x,f(x)

The above can be read as: CG.minimize() requires a VectorFN (our trait representing single argument functions with a free type parameter) and an initial point (in standard Doubles). The code will the return a pair of the optimum point and the function evaluated at the optimum point. From the type signature we can see that CG.minimize() expects to re-specialize the function “fn” to types of its own choosing (else it could have accepted a parameterized argument instead of our custom trait) and will handle all up-conversion and down-conversion between machine Doubles and NumberBase[Y]‘s itself. This sort of type information is hard to express (let alone enforce) in a dynamically typed language.

A slightly more complicated example is the lineMinD() method:

def lineMinD[Y<:NumberBase[Y]](field:Field[Y],
 f:Array[Y]=>Y,
 xm:Array[Double],
 di:Array[Double]):(Array[Double],Double)

Notice it is willing to work with any type parameterized function (which means it is willing to let the caller pick the actual type of NumberBase[Y] and work with that). Most callers will call with Y=MDouble (the wrapper for machine Doubles) and lineMin() will then work with that (without ever really knowing the actual underlying type).

A lot of fans of dynamic languages consider type systems to be mere hairshirt penance. But that is not so. Broken type systems (like Java’s collections before erasure parameters were introduced in Java 1.5) are indeed more trouble than they are worth. Working type systems (like C++ Templates/STL, Java 1.5+ and Scala) allow you to solve problems (and enforce decisions) during the design phase (which is much much cheaper than during the deployment phase). You can’t set your types in stone (you are likely going to have them subtly wrong for the first few iteration). You must be willing to think like a “language lawyer” to find out what parts of your work can be specified and enforced in the language type system. To use an analogy: static types are your blueprint or your underpainting.

Tests

One argument against static types is that you can get much of their benefit from unit tests. My opinion is you never have enough unit tests, so putting more pressure on your test suite is not wise. Static types plus tests are strictly more powerful than static types alone or tests alone.

Even for this example toy-scale project we have include a JUnit test set to pursue a number of goals:

• Confirm our number implementations (DualNumber and MDouble) correctly model machine Doubles (perform parallel calculations and compare).
• Confirm DualNumber obeys expected laws of algebra composition and cancellation including the portions that can not be modeled in machine Doubles.
• Confirm DualNumbers compute gradients.
• Confirm operations of optimizers and optimizer components.

Many of these tests are related, but they don’t all imply each other and give different perspective on the errors they catch. For example no amount of parallel computation between DualNumbers and machine Doubles is going to confirm the infinitesimal portion of the DualNumber is propagating correctly (since this is not a property of machine Doubles). So we add extra tests that expect DualNumber to obey algebraic relations like: a*(b+c) = a*b + a*c hold. It is then another step to confirm that whatever the DualNumbers calculate is not only self-consistent, but also models a truncated Taylor Series or differentiation.

Conclusion

We hope we have demonstrated how the complexity of a mathematical programming problem can be managed by breaking the problem into an objective function that is separate from the optimizer (allowing the optimizer to be both good and hidden) and a static type system (such as Scala) to help enforce required properties of a calculation (such as all numbers being routed though a required representation). With these sort of tools available many formerly hard problems (that are often, unfortunately solved by over-specifying direct inefficient iterative improvement techniques) become “if I can write a reasonable objective function this may already by solved by an optimizer in my library.” The more of these tools you have (either in your code or in your reference library) the more of these problems become easy (this is the topic of my earlier paper: The Local to Global Principle).

Appendix: Fixing Smoothness

Our chosen example objective function is very nice (i.e. convex) but it has a small (but correctable) problem. The derivative or gradient or gradient has some jump discontinuities that could cause an optimizer to exit prematurely (not at the global optimum). Consider the simple form of this for wiring a center to a single point at the origin (even in 1 dimension). The wiring cost function is sqrt(x*x) has a cost graph as shown here.

This is convex- but derivative is not smooth as we see in the included graph of the derivative of sqrt(x*x).

So: in this case if the optimizer stops at one of the target points we can’t be sure that it stopped at the global optimum (it may have stopped due to the discontinuity in the gradient). For some simple problems the optimum is necessarily at a target point. For example on the number line take the target points 0,1 and x. As long as x≥0 and x≤1 the optimum placement will be x itself.

One way to defend against this is to use some sort of smoothed version of sqrt() that essentially decreases a little faster near the origin. Our cost function becomes:

Related posts:

1. Gradients via Reverse Accumulation
2. Programmers Should Know R
3. Automatic Detection of Potential Deadlock

The spirit of the list is to pick items such that: if you disagree with an item in this list then either you are wrong or you know something I would really like to hear about.

Encryption, disk images: TrueCrypt (open source: Linux, Windows, OSX)

TrueCrypt can create portable encrypted virtual disks (files that can be mounted as a disk on any operating system).

Encryption, files: GnuPG (open source: Linux, Windows, OSX)

GnuPG is the tool to use to encrypt files for email.

Presentation: Apple Keynote (commercial: OSX)

Keynote is not quite as friendly as Microsoft PowerPoint, but it quickly produces beautiful presentations.

Reference Library: Papers (commercial: OSX)

“iTunes for PDF.” Manage thousands of PDFs and references, annotate with meta-data, place papers into multiple project folders. An interesting runner-up is BibDesk (open source: OSX).

Spreadsheet: Microsoft Excel (commercial: Windows, OSX)

Open Office and Google Docs are getting better every day, but neither come close to Microsoft Excel in functionality and versatility of user interface. If you are on a platform that supports Excel, working regularly with spreadsheets and using something other than Excel: it really means that you do not value your time.

Statistics Software: R (open source: Linux, Windows, OSX)

R is rapidly becoming the platform of choice for statisticians and is (with the addition of lattice and ggplot2) the best way to produce graphs. R has fairly nasty programming language, but has so many statistical operations available that it can not be avoided.

Technical Documentation: LaTeX (open source: Linux, Windows, OSX)

It may seem antiquated but TeX/LaTex is still far more powerful than the “WSYWYG” pretenders. The separation of presentation from specification, automatic management of references, table of contents and being able
to include PDFs from external files (which get refreshed when you re-build the document) are all lifesavers.

Version Control: git (open source: Linux, Windows, OSX)

Just about the only version control system that: doesn’t damage the data you are trying to manage by adding dot-files into all of the directories, can routinely handle large files and can work productively without a network connection. Perforce is powerful central server commercial option (with the ability to have central policies, control and review).

I look forward to learning which of my choices are considered poor and what your must-haves are.

Related posts:

1. Exciting Technique #1: The “R” language.
2. “The Mythical Man Month” is still a good read
3. Microsoft Store Again