Hal 9000

There have been vast improvements in artificial intelligence. Much of the improvement has been driven by the engineering effects of Moore’s Law (resulting in my mobile phone’s processor having 12 times the clock speed and over 32 times the memory of an $8 million Cray 1 super computer) and significant machine learning research results. These machine size changes happened during the productive careers of many researchers, so ideas are often evaluated at a series of radically different machine capabilities and data scales.

Cray 1

von Neuman himself commented that scale was a major limiting factor in early computers. He asked the question how you could be expected to achieve anything significant even from a roomful of geniuses if (as with his early computers) all notes, communication and memory were limited to less than a single typed page. von Neuman’s comment stands in contrast to science fiction scientists and early boosters of artificial intelligence who always seem to be in awe of their own creations. Computers are certainly much larger- but we need to be humble and put off deciding if we are yet in the era of large computers (compared to human or animal brains). Everything we are doing now may still just be artificial intelligence’s pre-history and prologue. Feynman in his lectures on computation mentions that RNA transcription can be estimated to take around 100 kT of energy to transcribe a bit while a transistor may easily use 100,000,000 kT energy units to switch states. This means for the amount of heat the human head dissipates (energy supply and heat dissipation are rapidly becoming the most relevant measures of computational power) you could do a million times more work using RNA techniques (if you knew how) than with transistors. So computers may not yet be what we should call large (though they are likely getting there). What we currently call “datacenters” are in fact block sized computers (consuming an enormous amount of energy and dissipating a huge amount of heat).

A datacenter (or a block sized computer)

Not all improvements in machine intelligence have come from (or are to come from) improvements in hardware. Many of the improvements came from machine learning research results and these are what I will outline below.

Early machine learning algorithms were driven by analogy. This led us to perceptrons (1957, fairly early in the history of computer science) and neural nets. These methods have their successes but were largely over used and developed before researchers developed a good list of desirable properties of a machine learning method.

Neural Net diagram

These methods live on but are, in my opinion, not currently competitive. Some of their important ideas and contributions have been revived from time to time, such as the online update rules becoming what we now call stochastic gradients.

A list of (often incompatible) desirable properties of a machine learning algorithm is the following:

• Able to represent complicated functions
• Good generalization performance (quality predictions on data not seen during training)
• Unique optimal model for a given set of data and feature definitions
• Efficient and well characterized solution method
• Consistent summary statistics
• Preference for simple models

We divert from this list for a bit of background and context.

The neural net was largely celebrated for its ability to represent complex functions and the perceived efficiency of its newer back-propagation based training method (related to the efficient calculation of gradients). The downsides were you never knew if your neural net was the right one (even assuming you had the right features, layout and training data) and could not be sure you were biasing towards simple models that might perform well on novel queries. Great effort was expended in extending neural nets based on the supposition they should work as they were an analogy to how we imagined biological neurons might function. An almost mystic hope was derived from the non-linear nature and special properties of the sigmoid curve (which was in fact a curve already known to statisticians).

Other methods than neural nets also had early success. The field of information retrieval (which was not “sexy” prior to the Web) had huge success since the 1960s with Naive Bayes, Rocchio Classification, and TF/IDF methods. The early success of these methods may have in fact delayed research on current hot research areas such as segmentation and author topic models.

Theoretical computer science initially sought to characterize machine learning methods in non-statistical language. In the 1980s a great amount of ink was spilled on “learning boolean functions.” Papers proving nothing was learnable (by picking a function related to cryptography) alternated with papers proving everything was learnable (for example via amplification techniques like boosting). Generalization of models to new data remained a theoretical problem that was dealt with by appeals to model complexity and MDL (minimum description length). A major breakthrough in characterizing generalization performance was the PAC model (probably approximately correct) framework which finally allowed direct treatment of generalization performance.

We now have enough context to discuss some of the current best of breed machine learning techniques (that address many of the desired properties mentioned above):

• Support Vector Machines
• Kernel Methods
• Logistic Regression
• Maximum Entropy Methods
• Regularization
• Graphical Models
• Conditional Random Fields




Typical SVM maximum margin diagram

Not all of these methods are new (Logistic Regression for example dates from 1925 and is itself based on regression which goes back to Gauss). But the concerns these methods address are all much more statistical than artificial intelligence in nature. For example we don’t suppose that there is some cryptographically obscured combination of features that we need to find to make the best prediction. We instead worry about detecting which features are useful and note that it is a significant (though solvable) problem to correctly use combinations of useful features (phrased as statistical concerns: feature to feature dependencies and higher order interactions). Machine learning has always run where statisticians fear to tread. But more and more often we are seeing that the methods and concerns of statisticians are what are needed to achieve many of the listed desired properties of machine learning models.

The methods I have singled out for praise are very effective and achieve a number of our listed desired properties. For example: both logistic regression and maximum entropy have a unique solution that is easy to find. They are also both consistent with all summaries known during training. That is: if 30% of the positive training data has a feature present then 30% of the data also has the feature present when weighted by the model’s score (so the model score shares a lot of properties with training truth). Support Vector Machines also have well understood solutions and a theory (called maximum margin) that directly addresses generalization (good predictions on new data). Kernel Methods (both as used in SVMs and elsewhere) allow controlled introduction of very complex functions. Graphical Models and Conditional Random Fields also allow the controlled introduction of modeled dependencies in the data.

It is now common to call what was previously thought of as artificial intelligence or machine learning: “statistical machine learning.” This reflects that the kind of prediction and characterization we expect from machine learning algorithms are in fact statistical concerns that we can deal with if we have enough data and enough computational resources.

The current important issues for statistical machine learning include:

• Dealing with very large datasets (driving the return of simpler methods like Naive Bayes)
• Dealing with lack of training data (driving interest in clustering and manifold regularization methods)
• Dealing with unstructured data and text mining (driving interest in information extraction and segmentation via generative models)

Just as Wigner famously wrote about “The Unreasonable Effectiveness of Mathematics” in the 1960s Halevy,Norvig and Pereira write about the “Unreasonable Effectiveness of Data.” They argue that we are in the age of big data (or the age of analysts). Or, as Varian observed: “it is a good time to supply a good complementary to data” (i.e. it is a good time to be an analyst). I would temper this with we are likely in the age of unmarked data and unstructured data. Less often are we asked to automate a known prediction and more often we are asked to cluster, characterize and segment wild data. In my opinion the hard problem in machine learning has moved from prediction to characterization. With enough marked training data (that is data for which we know both the observables and desired outcome) it is now quite possible to use standard techniques and libraries to build a very good predictive model. However, it is still hard to characterize, segment or extract useful information from the wealth of unstructured and unmarked data that is upon us. And this is where a lot of the current research in statistical machine learning is directed.

Or course characterization and clustering have their own infamous history. Rota wrote: “… Or a subject is important, but nobody understands what is going on; such is the case with quantum field theory, the distribution of primes, pattern recognition and cluster analysis.” Artificial intelligence may be moving from areas where computer scientists have over-promised to areas where statisticians have over-promised. But this is not a disaster: the most valuable research tends to be done in hectic times in messy fields, not in calm times in neat fields. And the already large scale adoption of statistical machine learning techniques means there is immediate great client value in even seemingly small improvements in understanding, explanation, documentation, training, tools, libraries and techniques.

Classic attempt to add structure to text

(images from Wikipedia)

Related posts:

1. A Demonstration of Data Mining
2. Kernel Methods and Support Vector Machines de-Mystified
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)

We start with W. Edwards Deming. Deming was a statistician who designed many of the production methods of post-war occupied Japan. Deming’s work on quality quantification, measurement and continuous improvement formed the fundamental basis of Japan’s later rise as a respected manufacturing super power. Many of the further improved techniques were later imported into the United States as “eastern wisdom.” However, some of the lesser ideas were perverted by eager followers into destructive cargo-cult rituals like “six sigma” (we must remember that it was the depth and power of Deming’s ideas that attracted the imitators).

One of Deming’s most fundemental ideas was the “PDCA loop.”

The PDCA loop is cycle of conceptual and analytic effort that sequences repeatedly through the stages Plan, Do, Check and Act. The cycle starts with a plan and the next cycle’s plan is influenced by results of the previous cycle. The explicit Check and Act steps show the presumption that the Do step will always need measurement and correction. This cycle is designed to help mitigate Clausewitz’s observation that “no campaign plan survives first contact with the enemy.” Deming’s idea is essentially the systematic application of the scientific method (“propose/test”- or Francis Bacon’s Novum Organum of 1620) to adaption and implementation of plans.

While Deming was teaching planning and “statistical process control” to boost US wartime production a number of other statisticians were having great success in developing reactive strategies. One of the best stories is that of Abraham Wald. Wald became interested in allied aircraft mortality during World War II. He prepared a number of studies and charts of surviving aircraft, tabulating where bullet and shrapnel damage was most extensive. He could, for example, combine inspections of many returning bombers to determine where the returning bombers had the most damage (say the bulk area of fuselage and the leading edges of the wings):

Wald then had the genius idea of proposing additional armor on the parts of the aircraft that never showed any hits on surviving aircraft (reasoning that aircraft routinely took damage everywhere so the undamaged areas in surviving aircraft must be the areas more often damaged in the unobserved, non-returning lost aircraft). From the above diagram we might propose to add more armor near the pilots, engines and trailing control surfaces. Wald later published sophisticated statistical techniques for imputing the distribution of hits (and therefore the distribution of vulnerabilities) on the unobserved aircraft: “A Method of Estimating Plane vulnerability Based on Damage of Survivors,” Abraham Wald, Center for Naval Analyses (1943).

This art of reactive observation was later systematized by Colonel John Boyd. Boyd invented what he called the “OODA loop.”

This loop cycles similarly to Demings’s through a pattern of Observe, Orient, Decide and Act. The OODA loop differs from the PDCA loop in that it assumes a world that looks back and adapts against your actions. Boyd added ideas of tempo and pace such as “short cutting the loop” (skipping from act to orient or even act to decide) to adapt faster than nature or than your enemy.

Boyd is also famous for applying his and Wald’s ideas in the design of the A-10 Warthog. The A-10 is a unique non-stealth, sub-sonic close air support plane. It is considered one of the ugliest things to every fly. The A-10 was not state of the art when it was introduced but it was scientifically designed for survival in the style of Wald. The engine intakes are partially protected by the wings, there is extra titanium armor around the pilot and a primitive direct lever control system in addition to the traditional hydraulics. The A-10 is known for its “lingering ability” or ability to stay near troops under fire to deliver support. It has also allowed pilots like Major Kim Reed-Campbell to fly for an hour and return to base after losing pieces of wing and all hydraulics. Here is a picture Reed-Campbell inspecting her damaged A-10 in 2003 after safely landing:

Deming, Wald and Boyd were able to move statistics and analytics beyond description and use mathematics for prescription. The techniques they developed for planning, measurement and reasoning remain relevant to this day.

Related posts:

1. Statsmanship: Failure Through Analytics Sabotage

The longer answer is that a number of times different forms of algorithmic art have been invented. Algorithmic art is art generated by mathematical procedures. Such art is similar to earlier mechanical and kinetic art forms. One branch of mathematics often used to generate such art is called “fractals.” We looked somewhere else for our inspiration (our art is not strictly fractal in nature). What we worked on we called “genetic art” to emphasize the role of encoding and re-combination in the works.

In the early 90′s Karl Sims presented a number of art installations based on at least three interesting ideas:

• Transforming images
• Evolving combinations of transforms
• Direct participation

(see: Karl Sims. Artificial Evolution for Computer Graphics. Proceedings of SIGGRAPH 1991 and Karl Sims’ homepage).

The part that caught a number of people’s imaginations was the evolution aspect. Karl Sims defined a method of combining transformations of original source images. He then allowed people to manipulate his art installations and “vote” on art they liked best. The more popular pieces were combined (or bred) to create newer works that then put up against criticism. After many breedings (or generations) the combinations of transforms were quite complicated and a number of unexpected images were created.

At CMU Shumeet Baluja, Dean Pomerleau and Todd Jochem were interested both in the evolutionary aspects of the art and also seeing if a machine could learn to model user tastes (see Shumeet Baluja, Dean Pomerleau and Todd Jochem. Simulating User’s Preferences: Towards Automated Artificial Evolution for Computer Generated Images. Technical Report CMU-CS-93-198. Carnegie Mellon University. Pittsburgh, PA. October 1993. ). They built a much simpler art system that combined primitive elements (elements closer to brush strokes than to original pictures) and tried to learn user preferences for complex pictures.

Figures of this era looked much like this (well better than this, this comes from a scan of a black and white printing of the paper):

.

Scott Neal Reilly built a new, more simplified system; and with Michael Witbrock put the whole thing on the Web. This was an unimaginably primitive time on the Web. Cutting edge interaction was sites like “Blue Dog Can Count.” The Mac had no forms capable browser and Amazon.com was still a year away from launching. An interactive art exhibition running directly on the Web (and manipulated by anybody) was a significant step forward.

Michael Witbrock was influenced by the stories of Heikegani Crabs and Alan Turing’s 1952 paper “The Chemical Basis of Morphogenesis” (which theorized how simple systems could develop textures).

At this point I (John Mount) got interested in the project and felt that much more could be done with how such systems handled color. The art had been simplified to primitive elements that one could think of as brushes (really more like gradients) but the art was essentially grey-scale with a false-color map applied at the last step. Karl Sims had made transformations on images his primitive operations, I wanted my primitive operation to be transformations on color.

Being a math-nerd I chose to encode color inside a mathematical system called “Quaternions” (see Ebbinghaus et al. Numbers Springer-Verlag, Second Edition, 1988). Colors are often represented as three brightness terms- for example intensity of red, intensity of green and intensity of blue. The Quaternions were discovered by Sir William Rowan Hamilton in 1843 (see Wikipeida: Quaterion). Sir Hamilton was trying to solve the problem of encoding positions in space in a nice structure and was so excited by his discovery he carved his fundamental formula for them in the Brougham Bridge the night he had his breakthrough. Quaternions are represented as four standard numbers (so they have enough “slots” to encode a position in Sir Hamilton’s case or in our case a color) and they their selves behave a lot like individual numbers. There are rules for adding, subtracting, multiplying and even dividing Quaterions. This means you can write formulas over them and these formulas are now directly manipulating colors (instead of manipulating geometry or intensities as in the earlier systems).

So, as with Neal Reilly’s system, we represented all of our transformations as formulas and represented “breeding” as ripping a bit of one formula out and combining it with another. For instance these two rather uninteresting color gradients were represented by the formulas:

( x – i y ) :
( x – i y – j x – k y ) :	.

One of our arithmetic operations was named “mod” and we could use it to combine the two items into a more complicated formula and somewhat more interesting picture:

( mod ( x – i y) ( x – i y – j x – k y ) ) :

.

After enough generations of selection and breeding the formulas get long and complicated (luckily nobody but the machines have to look at them) and the pictures get interesting:

.

Michael Witbrock and Scott Neal Reilly supplied an updated web interface. And at this point we got our 15 minutes of fame:

From Wired 3.01 January 1995 p. 147 Kristin Spence’s “Net Surf” column:

.

We also made large prints (using a parallel computation system named “WAX” by Peter Stout) and set up an exhibition in a Pittsburgh coffee house. The development work was largely done on a machine with a black and white monitor (later we got access to a grey scale monitor) so it really was a treat that color was able to fend for itself.

On an unrelated track in 1991 another CMU student, Scott Draves, was pursuing a serious project and building art based on iterated function systems (more related to fractals than our work) and in 1999 added some genetic ideas and released the Electronic Sheep client/server oriented screen saver (inspired by SETI@home).

Recently we have gotten back to some more of Sims’ ideas and allowed more geometric transformations and real source images, such as incorporating some of Dover’s royalty free Japanese textile designs:

.

The original system was group interactive and an incredible hit (and had its own Zephyr discussion instance- the then equivalent of Twitter). The current demo is stand alone and server-free, so it is a single player game.

All the user has to do is point a Java 1.4 (or better) capable browser at: Genetic Art Program. Click (and hold) on the “Action Menu” of any of the sub-windows and select “take over left selection” on one picture that interests you and “take over right selection” on another picture that interests you. You will notice doing this copies the pictures to the left and right of the “breed pictures” button. Now press “breed pictures” as many times as you want. Each time you press it you will get a new picture build by combining elements of the two small pictures. At any time you can have some other picture you like take over a breeding position (again by using its action menu). And you can also scroll the strip of five pictures around and click on them to introduce previous favorites from the earlier server version of the program into your space of opportunities.

We were originally a bit uncomfortable calling the work “Art” but a number of the images have made significant impressions on us and others- so perhaps it qualifies.

Related posts:

1. Algorithmic Movie (with texture)

February 19, 2008

Computer scientists have usually dodged questions of intent, purpose or meaning. While there are theories that assign deep mathematical meaning to computer programs[13] we computer scientists usually avoid discussion of meaning and talk more about utility and benefit. Discussions of the rhetorical meaning of programs is even less common. However, there is a famous computer program that has a clean an important rhetorical point. This program is called “hello world” and its entire action is to write out the phrase “hello world.” The action is simple but the “hello world” program actually has a fairly significant purpose and meaning.

I would like to briefly trace the known history of “hello world” and show how the rhetorical message it presents differs from the rhetoric embodied in earlier programs. In this sense we can trace a change in the message computer scientists felt they needed to communicate (most likely due to changes in the outside world).

Since the late 1970’s it has been a tradition in computer science that when facing a new system to start by writing the traditional “first program”, which is called “hello world.” The history of this tradition is not well documented, but WikiBooks claims the origin is in Brain Kerninghan’s 1974 tutorial for the computer language “B.”[14, 8]

The program itself is as follows:

main( ) {
  extrn a, b, c;
  putchar(a); putchar(b); putchar(c); putchar('!*n');
}
a 'hell';
b 'o, w';
c 'orld';

From the original context in [8] we can say this is not a pure “hello world” program. The purpose of this program is to illustrate a few functions of the language (“extrn” variables, use of multiple lines and so on) and not to test if the system is running.
The most famous (and a pure) example of “hello world” is found in Brian Kerninghan and Dennis Ritchie’s famous 1978 C book[9]. The code looks like the following:

main() {
        printf("hello, world\n");
}

Equally famous are the early versions in “BASIC”:

10 print "Hello, World!"

In both of these examples the purpose of “hello world” is clear: it is a trivial computer program that does nothing but print a single line of text. If this can not be made to work then nothing can be made to work. “hello world” is in fact a somewhat confrontational program. The author is saying “it isn’t obvious your computer system will work, so I am not going to invest a lot of time in it until I see it can at least print one line of text.”

Writing “hello world” as the first program on a new system was certainly a well known tradition by the late 1970’s. We can ask: did this tradition showed up late or early in the history of computer science?

The first modern computer is commonly credited to John von Neumann in 1945. This attribution of “first” was made before a lot of previously secret information on Bletchley Park’s and Konrad Zuse’s work in Germany were well known, but von Neumann’s work was definitely the known foundation that later work built on. Donald Knuth credits von Neumann with the first modern computer program[10]. In any case we can securely place the invention of the modern computer to the early to mid 1940’s. It seems like “hello world” was not written at that time because it made entirely the wrong rhetorical point.

In fact von Neumann’s first program (in 1945) was a sorting algorithm (proving the computers could at least replace tabulators)- it was important to prove that not only could the computer turn on but that it could do something.

The 1957 description of FORTRAN starts with a program that solves for roots of an equation.[4] John Backus himself says the prior to 1954 “almost all programming was done in machine language or assembly language”[3] (which would make “hello world” an unlikely first program as numeric operations are typically much more succinct than string manipulations in these languages).

The 1958 UNIVAC Math-Matic Programming Manual first examples are (predictably) equations involving trigonometric functions.[5]

In 1959 the Algo 58 standard [2] had no concrete example programs- the purpose of this document was to show that program syntax could be specified and to discuss the techniques of specification (so a runnable example did not serve their argument).
John McCarthy’s paper on recursive functions (LISP) starts with examples partial functions and conditionals (to get to terminating recursion as fast as possible).[11] The Lisp1 manual [6] concentrates on the translation of information into data-structures (an important point) and the Lisp1.5 manual [12] is written for the famous “bottom of page 13” moment where “eval” (essentially the semantic core of LISP) is defined in a few lines LISP code.

The computer language “BASIC” seems almost designed to support “hello world.” Instead the 1964 BASIC manual[7] starts with a lively discussion of what a computer program is (as a process) and the first example program is a highly useful program that solves simultaneous equations. The point again being that useful work can be done.

Even as late as 1972 “hello world” does not seem to be the obvious message. For example the popular BASIC tutorial called “My Computer Likes Me, When I Speak in BASIC”[1] does indeed start with a simple program that only writes a line of text. But the line it writes is “MY HUMAN UNDERSTANDS ME.” This is making a very different point than the point made by “hello world.”

While I have skipped a number of languages of the period (Autocode, COBOL, APL, SNOBOL, PL/1, Logo, BCPL, Forth, Smalltalk …) we can see the general trend: fist programs had to make a point. By the dates it seems that once invented the “hello world” tradition took off and spread very quickly. Writing such a program serves a useful purpose as a test so the practice probably spread quickly once the rhetoric of the earlier programs was no longer needed. That is “hello world” became popular once computer scientists no longer felt that society needed to be persuaded of the ultimate utility of computer systems.

With the increasing complexity of modern systems “hello world” is a more important test than ever. “hello world” is sometimes called the hardest application to deploy. The idea is that once you learn your lessons from deploying it then deploying a second, more sophisticated, application seems relatively easy. I often find that I can lean the true nature of a system by deploying “hello world.” If writing and deploying “hello world” is a sufficiently unpleasant task in a system then it is likely that every other task in that system will be similarly unpleasant.

In the end “hello world” serves the same purpose that it always has (testing if a system in fact works) and it stands as a rhetorical marker signifying that we are now living long after the skirmishes of the computer revolution.

References

[1] Albrecht, B. My Computer Likes Me, When I Speak in BASIC. Dymax, 1972.

[2] Backus, J. W. The syntax and semantics of the proposed international algebraic language of the zurich acm-gamm conference.

[3] Backus, J. W. The history of fortran i, ii and iii. ACM SIGPLAN Notices 13, 8 (Aug 1978).

[4] Backus, J. W., Beeber, R. J., Best, S., Goldberg, R., Haibt, L. M., Herrick, H. L., Nelson, R. A., Sayre, D., Sheridan, P. B., Stern, H., Ziller, I., Hughes, R. A., and Nutt, R. The fortran automatic coding system. Proceedings of the Western Joint Computer Conference (Feb 1957).

[5] Corp., R. R. U. UNIVAC math-matic programming system. Sperry Rand, 1958.

[6] Fox, P. Lisp 1 programmers manual. 165.

[7] Kemeny, J., and Kurtz, T. Basic.

[8] Kerninghan, B. W. A tutorial introduction to the language b.

[9] Kerninghan, B. W., and Ritchie, D. M. The C Programming Language. Prentice Hall, February 1978.

[10] Knuth, D. E. von neumann’s first computer program. Comp. Surveys 2, 4 (1970), 247–260.

[11] McCarthy, J. Recursive functions of symbolic expressions and their computation by machine, part i.

[12] McCarthy, J., Abrahams, P. W., Edwards, D. J., Hart, T. P., and Levin, M. I. Lisp 1.5 programmers manual. 116.

[13] Tennent, R. D. The denotational semantics of programming languages. Communications of the ACM 18, 8 (Aug 1976), 437–453.

[14] Wikibooks. http://en.wikibooks.org/wiki/Computer_programming/Hello_world#B, 2008.

Related posts:

1. Living in A Lognormal World
2. What is “Genetic Art?”