The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

then the following is a picture of reverse accumulation:

Related posts:

1. Automatic Differentiation with Scala
2. “Easy” Portfolio Allocation
3. A Quick Appreciation of the Sharpe Ratio

Our Example Problem

Our small scale problem is this: we have a number of target points on a map and we want to pick a central point to directly connect to all of these points with wire. Our goal is to minimize the total amount of wire used. This problem is called the “Geometric Median”. So we are trying to find a point that minimizes the sum of distances from our chosen center to every target point. If we were trying to minimize the sum of squared distances from our chosen center to every target point the answer would be obvious: the average or mean (which by Hooke’s law is also the point where a set of identical springs would relax to). The mean is in fact a fairly good guess, but you can do better (which could important if the “wire” is expensive, such as cutting irrigation or drainage ditches). For example given the three target points (20,0), (-1,-1) and (-1,1) the optimal point is (-0.42,0) not the mean (6,0) and the choice of optimal point represents an over 19% savings in total wiring distance (see figure).

This is a substantial saving in cost.

The problem changes as we consider variations. If indirect connections (such as routing one point through another, which may or may not be possible for reasons of capacity or safety) and multiple new centers are allowed we then have an instance of the Steiner Tree Problem which is harder to solve (since it is known to be NP complete). If no new centers are allowed (all routing must be between pre-existing target points) then we have a Spanning Tree Problem- which admits very quick solutions.

We bring up the geometric median as a mere example. We don’t intend for our code to solve only the geometric median problem and we don’t intend to touch on the literature of specialized methods for solving the geometric median problem. Instead we are trying to demonstrate the speed you can develop prototype solutions if you have a few good tools (like various optimizers) available in your toolkit. Numeric optimizers may sound exotic, but they often are the kind of thing you want to experiment with and link directly into your code.

Optimization as General Tool

Now that we have the example problem we can describe a solution strategy. In this case the solution uses code “we wished we had lying around” before we started on the problem. We will pretend we have the tools we want ready to solve our problem and then we will pay our debt and build the required tools. The issue is that there is not an obvious closed form for the solution of the geometric median problem. So we are forced to work a bit harder. In this case harder means we need to solve an optimization problem. Consider the contour plot of the total wiring cost as function of where we choose to place our center. Our optimal point (-0.42,0) had wiring cost of 22.73 and the contour plot given here shows concentric regions of solution positions with higher cost.

In general it is unwise to throw an optimizer at an arbitrary problem and hope to find the globally best solution. But in this case (and in many similar situations) we can prove that a simple local optimizer will in fact find the unique best solution. This is a property of the problem not of the optimizer. The concentric regions shown in the contour plot have a very nice shape: they are convex. That is: they have no intrusions- for any two points drawn from one of these shapes the straight line segment between these points stays inside the given shape. We don’t have to depend on observation- we can actually prove this is always the case for this problem. The wiring cost from a proposed center to any single target point is a convex function of where we choose to place our center (a convex function is a function whose graph never reaches above the secant line drawn between any two points on its graph). The total wiring cost is just the sum of the wiring costs to each target point. And to finish: the sum of a collection of convex functions is itself a convex function. Since the contour plot of a convex function has only convex shapes and we have proven the statement.

But how does this help us? There is a standard technique to find “local minima” of a function by inspecting a function for places where the gradient is zero (points where there is no obvious down hill direction on the contour plot). This technique usually can only be guaranteed to find local minima (places where no small change improves your situation). But there is no guarantee that the local minimum you find is in fact the global minimum (the best possible solution). Except when you are dealing with a convex function. When a function is convex then all of the local minima are always grouped together into a single convex connected shape (if not a line drawn between two remote minima would violate the convexity definition). And if the function is never flat then this set is a single unique point: the unique best solution. Our inspection technique will be a gradient driven optimizer- that is an optimizer that when the gradient is non-zero improves its objective by running down hill and halts when the gradient is zero.

The stated function to minimize is to sum the distance from our proposed center to each target point. We can write this as the sum of the distances:

( which is the traditional Euclidean or L2 distance). This function actually has one one subtle flaw that we will deal with in the appendix (see: Fixing Smoothness).

Using Scala to Apply the Optimization Solution

To find our optimal center placement using Scala we first write our cost or objective as a Scala function:

    val dat:Array[Array[Double]] = Array(
      Array( 20, 0.0),
      Array( -1.0, 1.0),
      Array( -1.0, -1.0)
    )

    def fx(p:Array[Double]):Double = {
      val dim = p.length
      val npoint = dat.length
      var total = 0.0
      for(k <- 0 to (npoint-1)) {
        var term = 0.0
        for(i <- 0 to (dim-1)) {
          val diff = p(i) - dat(k)(i)
          term = term + diff*diff
        }
        total = total + scala.math.sqrt(term)
      }
      total
    }

Scala is succinct and it is a great connivence to have a function definition capture data from its environment. What we would like to do is generate an initial guess as the solution (we use the mean as our initial guess) and then call an optimizer (in this case a conjugate gradient optimizer) to do all the work:

 val p0:Array[Double] = mean(dat)
 val (pF,fpF) = CG.minimize(fx,p0)

At this point we would be done, except the conjugate gradient method (which is superior to gradient descent and many the non-gradient methods) requires a gradient.
We could provide a numeric estimate of the gradient by the following divided difference method:

  def gradientD(f:Array[Double]=>Double,p:Array[Double]):Array[Double] = {
    val xdim = p.length
    val p2 = copy(p)
    val base = f(p2)
    val ret = new Array[Double](xdim)
    val delta = 1.0e-6
    for(i <- 0 to (xdim-1)) {
      p2(i) = p(i) + delta
      val fplus = f(p2)
      p2(i) = p(i)
      val diff = (fplus-base)/delta
      ret(i) = diff
    }
    ret
  }

This numeric divided difference method often outperforms non-derivative optimization methods (like Powell’s Method and the Nelder-Mead Amoeba method). But the technique can run into numeric difficulties. We can remedy this if we are willing to write our function in a slightly more general way. If we re-encode our function in a generic manner we can use automatic differentiation (not to be confused with numeric differentiation or with symbolic differentiation) to produce a reliable gradient for optimization. What we need to do is re-write our function to work over an abstract field of numbers instead of only the machine supplied doubles. In fact what we need to do is specify a generic function that will work over any field, with the field to be determined later. The code to do this in Scala is very similar to the non-generic code:

   val genericFx = new VectorFN {
      def apply[Y <: NumberBase[Y]](p:Array[Y]):Y = {
        val field = p(0).field
        val dim = p.length
        val npoint = dat.length
        var total = field.zero
        for(k <- 0 to (npoint-1)) {
          var term = field.zero
          for(i <- 0 to (dim-1)) {
            val diff = p(i) - field.inject(dat(k)(i))
            term = term + diff*diff
          }
          total = total + smoothSQRT(term)
        }
        total
      }
    }

Notice that code is very similar to the “def fx()” code. The key differences are that we had to define genericFx as extending a trait (a type of Scala interface) called VectorFN and inside this trait extension we defined a parameterized function name apply(). apply() is a generic function that is willing to work over any type Y where Y is at least of type NumberBase[Y] (we will get more into what that means in a moment). The difference in notation is that while the Scala function syntax can not specify a generic function with free type parameters (the incompletely specified Y) the Scala semantics are strong enough to implement this. In fact standard function definitions (such as “def fx()”) are just syntactic sugar for extending the Scala built-in Function1 trait. With a generic objective function in hand all we need is conjugate gradient code that is expecting a VectorFN (and willing to call apply() instead of just using naked function parenthesis) and some type NumberBase[Y] that can compute gradients for us. The Scala compiler can specialize our genericFx() into one version for quick calculation and another for gradients. How this is done is what we will discuss next. From our point of view our problem is solved with the following one line of code:

val (pF,fpF) = CG.minimize(genericFx,p0)

This should always be your goal- build sufficient preparation so your last step is a “obvious one liner.”

What Tools we Wish we Had Lying Around

We supply in our example some workable conjugate gradient code, but that is standard so we will not discuss it. What is of interest (and facilitated by Scala’s parametrized type system) is the implementation of dual numbers as a framework to supply automatic differentiation. An implementation of dual numbers as a NumerBase[DualNumber] type is the core of our demonstration.

Dual numbers are an algebraic structure written as pairs of real numbers “(a,b)”. The arithmetic table for dual numbers is given below:

In a dual number (a,b) “a” is the “large” or “standard” part of the number. You can check from the arithmetic table that the pair of dual numbers (a,0) and (c,0) behave just as we would expect the real numbers a and c to behave. In the dual number (a,b) “b” is the “small” or “ideal” portion of the number. From the multiplication rule above we can observe two rules: (0,b) * (c,0) = (0,b*c) (something small times anything else is small) and (0,b)*(0,d) = (0,0) (two small things become zero when multiplied). Essentially the dual numbers are carrying around the first two terms of a Taylor series: we get as a result both the function value and the function derivative. For a function f() over the real numbers we extend f() to work over the dual number by defining: f((a,b)) = (f(a),b f’(a)) (which is consistent with the previously defined arithmetic). We can check that the dual numbers numbers obey the usual laws of arithmetic (associative, commutative, distributive, identities and inverses). The punchline is that over the dual numbers the divided difference estimate of f’(x) (the derivative of f() evaluated at x) is in fact exact in the sense that f((x,1)) = (f(x),f’(x)) (or f((x,0)+(0,1)) – f((x,0)) = (0, f’(x))). Implementing the DualNumber class is little more than transcribing the above arithmetic table into Scala.

We have already seen how to write code that uses NumberBase[Y] types (genericFx() itself is an example). A more complicated example is the CG.minimize() code which not only accepts a generic function (in the form of VectorFN) but then specializes it to NumberBase[DualNumber] to compute gradients and also specializes to NumberBase[MDouble] for quick calculation during line searches (MDouble is just an adapter for machine Doubles, used for speed). The ability to re-specialize a function is one of the advantages of a parameterized type system. The DualNumbers are an example of forward automatic differentiation. We could also use the same object framework to capture a representation of the computation path and apply more sophisticated methods such as reverse automatic differentiation.

We give a link to a jar containing complete Scala source code including this example, the DualNumber implementation, a conjugate gradient implementation and some JUnit tests (all under a GPLv3 license) and will go on to describe some of the design decisions. The code is the bulky part of this work, so we will move on to discuss something more compact: types.

Types

If code is ever beautiful it is only when it is succinct. Among the most succinct forms of code are individual type signatures and interfaces (though the indiscriminate repetition of type signatures is rightly considered ugly bloat, which Scala works to avoid). Since we are distributing complete source we will describe only types and method signatures. The entry points to the code are the JUnit tests (organized in the ScalaDiff/test source directory and depending on JUnit which was not included) and the demo program in ScalaDiff/src/demo/Demo.scala).

To be a usable arithmetic type (like DualNumber or MDouble) you must extend the following parameterized abstract class:

abstract class NumberBase[NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  // basic arithmetic
  def + (that: NUMBERTYPE):NUMBERTYPE
  def - (that: NUMBERTYPE):NUMBERTYPE
  def unary_-():NUMBERTYPE
  def * (that: NUMBERTYPE):NUMBERTYPE
  def / (that: NUMBERTYPE):NUMBERTYPE  // that not equal to zero
  // more complicated
  def pow(that:Double):NUMBERTYPE
  def exp:NUMBERTYPE
  def log:NUMBERTYPE // this is positive
  // comparison functions
  def > (that: NUMBERTYPE):Boolean
  def >= (that: NUMBERTYPE):Boolean
  def == (that: NUMBERTYPE):Boolean
  def != (that: NUMBERTYPE):Boolean
  def < (that: NUMBERTYPE):Boolean
  def <= (that: NUMBERTYPE):Boolean
  // utility
  def field:Field[NUMBERTYPE]
}

In particular DualNumber extends NumberBase[DualNumber]. This deliberate circular reference has a big purpose: it allows publicly visible contravariant return types (returning nearly the exact type we really are instead of a base type). This allows us to have strict type arguments so that trying to add a MDouble to DualNumber is a type error (even though they both extend the same base class). The automatic differentiation technique encapsulated in the DualNumber class only works if all of the calculation is in the DualNumber types and this strict type enforcement allows the compiler to help prevent results sneaking in and out through other types. All of the methods on NumberBase are obviously related to arithmetic except the field() method. This method gives us access to a Field object which is responsible for carrying around the runtime type information (this is a common problem in Java and Scala, that some type information known at compile type such choice of template types is not easily accessed at runtime). The Field class is as follows:

abstract class Field [NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  def zero:NUMBERTYPE            // return canonical zero in field
  def one:NUMBERTYPE             // return canonical one in field
  def inject(v:Double):NUMBERTYPE  // return canonical representation of number in field
  def project(v:NUMBERTYPE):Double // return standard-number represented in field
  def array(n:Int):Array[NUMBERTYPE] // return an array of this type

The Field class is where we have factories for numbers (zero, one, arrays, injection from standard Doubles), casting (projection back to standard Doubles).

With these types defined we can actually read intent off some of the method signatures.

For example our conjugate gradient optimizer is accessed through the following method signature:

 def minimize(fn:VectorFN,x0:Array[Double]):(Array[Double],Double) // return x,f(x)

The above can be read as: CG.minimize() requires a VectorFN (our trait representing single argument functions with a free type parameter) and an initial point (in standard Doubles). The code will the return a pair of the optimum point and the function evaluated at the optimum point. From the type signature we can see that CG.minimize() expects to re-specialize the function “fn” to types of its own choosing (else it could have accepted a parameterized argument instead of our custom trait) and will handle all up-conversion and down-conversion between machine Doubles and NumberBase[Y]‘s itself. This sort of type information is hard to express (let alone enforce) in a dynamically typed language.

A slightly more complicated example is the lineMinD() method:

def lineMinD[Y<:NumberBase[Y]](field:Field[Y],
 f:Array[Y]=>Y,
 xm:Array[Double],
 di:Array[Double]):(Array[Double],Double)

Notice it is willing to work with any type parameterized function (which means it is willing to let the caller pick the actual type of NumberBase[Y] and work with that). Most callers will call with Y=MDouble (the wrapper for machine Doubles) and lineMin() will then work with that (without ever really knowing the actual underlying type).

A lot of fans of dynamic languages consider type systems to be mere hairshirt penance. But that is not so. Broken type systems (like Java’s collections before erasure parameters were introduced in Java 1.5) are indeed more trouble than they are worth. Working type systems (like C++ Templates/STL, Java 1.5+ and Scala) allow you to solve problems (and enforce decisions) during the design phase (which is much much cheaper than during the deployment phase). You can’t set your types in stone (you are likely going to have them subtly wrong for the first few iteration). You must be willing to think like a “language lawyer” to find out what parts of your work can be specified and enforced in the language type system. To use an analogy: static types are your blueprint or your underpainting.

Tests

One argument against static types is that you can get much of their benefit from unit tests. My opinion is you never have enough unit tests, so putting more pressure on your test suite is not wise. Static types plus tests are strictly more powerful than static types alone or tests alone.

Even for this example toy-scale project we have include a JUnit test set to pursue a number of goals:

• Confirm our number implementations (DualNumber and MDouble) correctly model machine Doubles (perform parallel calculations and compare).
• Confirm DualNumber obeys expected laws of algebra composition and cancellation including the portions that can not be modeled in machine Doubles.
• Confirm DualNumbers compute gradients.
• Confirm operations of optimizers and optimizer components.

Many of these tests are related, but they don’t all imply each other and give different perspective on the errors they catch. For example no amount of parallel computation between DualNumbers and machine Doubles is going to confirm the infinitesimal portion of the DualNumber is propagating correctly (since this is not a property of machine Doubles). So we add extra tests that expect DualNumber to obey algebraic relations like: a*(b+c) = a*b + a*c hold. It is then another step to confirm that whatever the DualNumbers calculate is not only self-consistent, but also models a truncated Taylor Series or differentiation.

Conclusion

We hope we have demonstrated how the complexity of a mathematical programming problem can be managed by breaking the problem into an objective function that is separate from the optimizer (allowing the optimizer to be both good and hidden) and a static type system (such as Scala) to help enforce required properties of a calculation (such as all numbers being routed though a required representation). With these sort of tools available many formerly hard problems (that are often, unfortunately solved by over-specifying direct inefficient iterative improvement techniques) become “if I can write a reasonable objective function this may already by solved by an optimizer in my library.” The more of these tools you have (either in your code or in your reference library) the more of these problems become easy (this is the topic of my earlier paper: The Local to Global Principle).

Appendix: Fixing Smoothness

Our chosen example objective function is very nice (i.e. convex) but it has a small (but correctable) problem. The derivative or gradient or gradient has some jump discontinuities that could cause an optimizer to exit prematurely (not at the global optimum). Consider the simple form of this for wiring a center to a single point at the origin (even in 1 dimension). The wiring cost function is sqrt(x*x) has a cost graph as shown here.

This is convex- but derivative is not smooth as we see in the included graph of the derivative of sqrt(x*x).

So: in this case if the optimizer stops at one of the target points we can’t be sure that it stopped at the global optimum (it may have stopped due to the discontinuity in the gradient). For some simple problems the optimum is necessarily at a target point. For example on the number line take the target points 0,1 and x. As long as x≥0 and x≤1 the optimum placement will be x itself.

One way to defend against this is to use some sort of smoothed version of sqrt() that essentially decreases a little faster near the origin. Our cost function becomes:

Related posts:

1. Gradients via Reverse Accumulation
2. R examine objects tutorial
3. Survive R

The spirit of the list is to pick items such that: if you disagree with an item in this list then either you are wrong or you know something I would really like to hear about.

Encryption, disk images: TrueCrypt (open source: Linux, Windows, OSX)

TrueCrypt can create portable encrypted virtual disks (files that can be mounted as a disk on any operating system).

Encryption, files: GnuPG (open source: Linux, Windows, OSX)

GnuPG is the tool to use to encrypt files for email.

Presentation: Apple Keynote (commercial: OSX)

Keynote is not quite as friendly as Microsoft PowerPoint, but it quickly produces beautiful presentations.

Reference Library: Papers (commercial: OSX)

“iTunes for PDF.” Manage thousands of PDFs and references, annotate with meta-data, place papers into multiple project folders. An interesting runner-up is BibDesk (open source: OSX).

Spreadsheet: Microsoft Excel (commercial: Windows, OSX)

Open Office and Google Docs are getting better every day, but neither come close to Microsoft Excel in functionality and versatility of user interface. If you are on a platform that supports Excel, working regularly with spreadsheets and using something other than Excel: it really means that you do not value your time.

Statistics Software: R (open source: Linux, Windows, OSX)

R is rapidly becoming the platform of choice for statisticians and is (with the addition of lattice and ggplot2) the best way to produce graphs. R has fairly nasty programming language, but has so many statistical operations available that it can not be avoided.

Technical Documentation: LaTeX (open source: Linux, Windows, OSX)

It may seem antiquated but TeX/LaTex is still far more powerful than the “WSYWYG” pretenders. The separation of presentation from specification, automatic management of references, table of contents and being able
to include PDFs from external files (which get refreshed when you re-build the document) are all lifesavers.

Version Control: git (open source: Linux, Windows, OSX)

Just about the only version control system that: doesn’t damage the data you are trying to manage by adding dot-files into all of the directories, can routinely handle large files and can work productively without a network connection. Perforce is powerful central server commercial option (with the ability to have central policies, control and review).

I look forward to learning which of my choices are considered poor and what your must-haves are.

Related posts:

1. Microsoft Store Again
2. Exciting Technique #1: The “R” language.
3. Public Service Article: JSTOR and other Useful Research Archives

Since the mid 90′s we have been dabbling off and on with a combination of algorithmic and genetic art (see: What is “Genetic Art?” or try running the Java code directly in your browser). Every once in a while we return to the project and generate something we would like to share.

For this project we have used formulas over the variables “x” and “y” to describe how color varies as a function of position on our canvas.

This has allowed formulas like:

( + ( mod ( iexp k ) ( isin ( / j ( / ( x + i y + j x + k y ) k ) ) ) ) ( mod ( iexp k ) ( isin ( / j ( / ( x + i y + j x + k y ) k ) ) ) ) )

To generate pictures like this:

We then add a source-texture from C. Estrade’s “Full-Color Japanese Textile Designs CD-ROM and Book” (Dover, unrestricted use):

Which (with a slightly modified formula) yields a picture like this:

( + ( subst ( mod ( iexp k ) ( isin ( / j ( / ( x + i y + j x + k y ) k ) ) ) ) Img23 ) ( mod ( iexp k ) ( isin ( / j ( / ( x + i y + j x + k y ) k ) ) ) ) )

We can further modify the formula to depend on time (represented by the new variable “z”):

( + ( subst ( mod ( iexp k ) ( isin ( / j ( / ( x + i y + j (x +z) + k (y + z) ) k ) ) ) ) Img23 ) ( mod ( iexp k ) ( isin ( / j ( / ( x + i y + j (x +z) + k (y + z) ) k ) ) ) ) )

And get a movie like this:

What we have previously called “genetic art” was the system of automatically combining and re-combining fragments of formulas using user votes and preferences (so nobody would have to see or understand these ugly formulas to produce art). What we now present is a larger “algebra” of “simple picture plus pattern = complicated pictures” and “picture plus time transformations = movie.”

Related posts:

1. What is “Genetic Art?”
2. Gradients via Reverse Accumulation

Introduction

Combinatorics, as it matures, becomes harder to succinctly describe. The field has progressed from the basic study of finite sets and counting techniques to being the discipline where questions involving counting, graphs, connectivity, mappings and partial orders all naturally reside. But the objects that combinatorics studies turn out not to be the correct foundation to support modern combinatorial methods. Many combinatorial methods were dismissed as mere technique until combinatorics expanded to include the natural domains of these methods: lattices, formal power series, valuation rings, matroids and many diverse algebras. One person who pushed hard for this coherence and unity was Gian-Carlo Rota.

An example of a high-school level combinatorial trick is proving the equation

by applying the binomial theorem to . This trick is transformed into a method when you recognize that you really should be working in the ring of formal power series and invent the Umbral Calculus. With the Umbral Calculus you can use the equivalence of the following two equations:

(i.e. is equivalent to ) to prove that for any two arbitrary infinite sequences the following two statements are also equivalent:

		for all	(1)
		for all	(2)

For example: we could pick and substitute it into Equation 1. With some work we see this implies .¹Then by the Umbral result we know Equation 2 must also be true so we get a new identity: . This algebraic production of a new identity is very different than the classical method of “counting two ways” (or being lucky enough to come up with a clever bijection to prove the identity).

Summary

The book “Combinatorics the Rota Way” is itself hard to succinctly describe. The first and third authors tell of writing this book using notes from the Massachusetts Institute of Technology’s course 18.315 collected over a span of more than 30 years. Gian-Carlo Rota himself was added as a posthumous author. The book itself contains more than a single course-year’s worth of material and is packed very densely.

The book’s emphasis is abstract and algebraic. The exercises are not to teach, but are instead to identify applications of combinatorics in other mathematical disciplines. The book is the product of a strong push to demonstrate many combinatorial methods in their most powerful, but not most obvious, forms. This work is clearly a labor of love and contains some remarkable material. However, due to the large breadth of the work not much time is spent on motivation or on concrete examples.

Chapter 1: Sets, Functions and Relations

The first chapter covers the definitional foundations of combinatorics: sets, lattices, partial orders, functions and relations. These are the discrete objects that the book will reason about by later building more complicated algebraic objects. This section is very dense and reads like a compressed Bourbaki treatment of discrete mathematics.

One portion of this chapter that is problematic is the section on entropy that seems to serve no purpose other than to prepare the reader for exercise 1.4.10 which demonstrates an abstraction of entropy. Also, exercises 1.2.5(j,k) are needlessly cruel in asking the reader to recreate the Robertson-Seymour graph minor theorem. There have been books where the reader is successfully guided through a major result by exercises, such as the Weak Perfect Graph Theorem in Lovász’s “Combinatorial Problems and Exercises”, but this book is not structured in that manner.

Chapter 2: Matching Theory

The second chapter is a welcome change in tone and opens with a quote from Harper and Rota describing matching theory and a clever 1979 Putnam exam problem is worked into the exercises and solutions. Central to the chapter is “marriage theorem”, which determines when matchings are possible. Also discussed is Birkhoff’s Theorem, which states that every doubly stochastic matrix is a convex combination of permutations matrices, which relates matchings to matrices. The text is lively and includes a number of well-researched asides, such as the origin of the name “The Hungarian Method.” However, there are some problems with forward reference: for example the reader is asked to work a couple of exercise (2.4.5 and 2.4.6) using the Binet-Cauchy formula, which isn’t discussed at length until chapter 6.

Chapter 3: Partially Ordered Sets and Lattices

This chapter begins with a very exciting presentation of the Möbius Function (the convolutional inverse of what is essentially the indicator function of a partial order). It is a real pleasure to see this material well presented in a general lattice setting, instead of the more common and specialized number theoretic setting. The chapter moves on to chains (ordered sequences in lattices) and anti-chains (sets of incomparable elements) in partial orders. The authors present Dilworth’s theorem which states that every partial can be covered by a number of chains no larger than the size of the largest anti-chain.² The chapter continues with Sperner Theory, which relates counting anti-chains to binomial coefficients. Chapter 3 concludes with valuation rings and Möbius Algebras: a transition to the more algebraic style found in Chapter 4.

Chapter 4: Generating Functions and the Umbral Calculus

This is a key chapter. The book introduces the Umbral Calculus, a transform space automating the manipulation of generating functions. The algebra of delta operators is introduced, which provides an abstraction of differentiation. Finally co-algebras are explored, which abstract the processes of factoring.

A rare (and unfortunate) typo on page-190 mis-defines a basic sequence for the delta operator as obeying instead of the correct equation: . A careful reader can spot the mistake as it is inconsistent with the the subsequent demonstrations and uses.

Chapter 5: Symmetric Functions and Baxter Algebras

This chapter treats a number of important algebraic topics. Symmetric functions are studied and identified as being the obvious class of functions that contains all of the well know generating functions already studied. Pólya’s Enumeration Theory, which is the method of counting the number of equivalence classes of distinct arrangements, is given a very interesting exposition. But the book skips the classic examples and exercises, such as counting the number of ways to construct distinct necklaces from colored beads, that would be needed for the topic to be fully approachable. Baxter Algebras, which abstract both summation and integration by parts, are introduced and via a study the sequence shift operator. By this point the book has abstract versions of both differentiation and integration, providing a combinatorial groundwork to prove theorems on “the calculus” that are more general than is possible in any one theory of differentiation or integration.

Chapter 6: Determinants, Matrices and Polynomials

This chapter is most similar to classical polynomial invariant theory, the study of symmetric functions of the roots of polynomials such as the discriminant. A major theme of this chapter is the study of the relations between properties of polynomial coefficients and the locations of roots of the polynomials. The study of matrices brings us to the remarkable Binet-Cauchy Formula for the determinant of a product of matrices. The results are deep, but it is a shame that more time isn’t spent on simple concrete applications such as using the Binet-Cauchy formula to count the number of spanning trees in a graph. This chapter reveals the parts of combinatorics that come from analysis and the study of locations of roots of polynomials (via group theory), in contrast to the parts that come from enumerating finite sets, linear algebra and abstract algebra. This is also the chapter where the exterior algebra, a favorite tool of Rota’s, is most discussed.

A typo on page 275 (a potentially confusing comma in the definition of the operation) can be recovered from because the authors have the nice habit of explicitly calling out the domain and range of functions.

Opinion

Some important questions about this book are: is Gian-Carlo Rota a coauthor, what is the purpose of the book and who is the best audience?

Gian-Carlo Rota seems appropriately labeled as a co-author, as clearly a lot of his work went into the book. The book is not suitable to be used as an introductory text book or as a reference. It is a book meant to be read. The ideal audience is capable of graduate level mathematics, is comfortable with a high degree of abstraction and algebra and is already familiar with many of the structures and techniques of combinatorics: sets, graphs, matrices, alternating sequences and generating functions. A mathematician or computer scientist wanting to learn more about the science of combinatorics will find a good read here.

The book works best as a second read of the topics covered. If you already know of a combinatorial method, like Pólya’s Enumeration Theory, this book is a good place to find the starting point for an alternate and powerful treatment of the topic. The book admits to not being self contained, and has a few forward-reference problems. However, this is forgivable when you realize the goal of this book is not to teach some easy discrete mathematics before you move on to analysis, but to extract the important combinatorial methods and themes from all of mathematics.

The content is well written, very accurate and well edited. The index is good, but not quite up to the job. The bibliography is very good and divided into three useful sections: papers by Gian-Carlo Rota and coworkers, books for further reading and a section of references.

We close with a extract from the book at hand. Many mathematicians have used the phrase “merely combinatorial proof” as a phrase of dismissal. However, when properly founded, combinatorial proofs are in fact more general than proofs that depend on additional specific details from the original problem domain. The authors take some justifiable pleasure in including points like: “Hilbert’s basis theorem is equivalent to the ‘trivial combinatorial fact’ given in Gordan’s lemma.” This is certainly a taste of combinatorics the Rota way.

Footnotes

….¹

For this use the binomial theorem to expand , differentiate with respect to and then substitute in .

… anti-chain.²

From this they derive just about the only Ramsey-theoretic style result in the book: any large partial order must have a large chain or large anti-chain.

Related posts:

1. What is Mathematics, Really?
2. The Joy of Calculation
3. Sorting Used in Anger

We start with W. Edwards Deming. Deming was a statistician who designed many of the production methods of post-war occupied Japan. Deming’s work on quality quantification, measurement and continuous improvement formed the fundamental basis of Japan’s later rise as a respected manufacturing super power. Many of the further improved techniques were later imported into the United States as “eastern wisdom.” However, some of the lesser ideas were perverted by eager followers into destructive cargo-cult rituals like “six sigma” (we must remember that it was the depth and power of Deming’s ideas that attracted the imitators).

One of Deming’s most fundemental ideas was the “PDCA loop.”

The PDCA loop is cycle of conceptual and analytic effort that sequences repeatedly through the stages Plan, Do, Check and Act. The cycle starts with a plan and the next cycle’s plan is influenced by results of the previous cycle. The explicit Check and Act steps show the presumption that the Do step will always need measurement and correction. This cycle is designed to help mitigate Clausewitz’s observation that “no campaign plan survives first contact with the enemy.” Deming’s idea is essentially the systematic application of the scientific method (“propose/test”- or Francis Bacon’s Novum Organum of 1620) to adaption and implementation of plans.

While Deming was teaching planning and “statistical process control” to boost US wartime production a number of other statisticians were having great success in developing reactive strategies. One of the best stories is that of Abraham Wald. Wald became interested in allied aircraft mortality during World War II. He prepared a number of studies and charts of surviving aircraft, tabulating where bullet and shrapnel damage was most extensive. He could, for example, combine inspections of many returning bombers to determine where the returning bombers had the most damage (say the bulk area of fuselage and the leading edges of the wings):

Wald then had the genius idea of proposing additional armor on the parts of the aircraft that never showed any hits on surviving aircraft (reasoning that aircraft routinely took damage everywhere so the undamaged areas in surviving aircraft must be the areas more often damaged in the unobserved, non-returning lost aircraft). From the above diagram we might propose to add more armor near the pilots, engines and trailing control surfaces. Wald later published sophisticated statistical techniques for imputing the distribution of hits (and therefore the distribution of vulnerabilities) on the unobserved aircraft: “A Method of Estimating Plane vulnerability Based on Damage of Survivors,” Abraham Wald, Center for Naval Analyses (1943).

This art of reactive observation was later systematized by Colonel John Boyd. Boyd invented what he called the “OODA loop.”

This loop cycles similarly to Demings’s through a pattern of Observe, Orient, Decide and Act. The OODA loop differs from the PDCA loop in that it assumes a world that looks back and adapts against your actions. Boyd added ideas of tempo and pace such as “short cutting the loop” (skipping from act to orient or even act to decide) to adapt faster than nature or than your enemy.

Boyd is also famous for applying his and Wald’s ideas in the design of the A-10 Warthog. The A-10 is a unique non-stealth, sub-sonic close air support plane. It is considered one of the ugliest things to every fly. The A-10 was not state of the art when it was introduced but it was scientifically designed for survival in the style of Wald. The engine intakes are partially protected by the wings, there is extra titanium armor around the pilot and a primitive direct lever control system in addition to the traditional hydraulics. The A-10 is known for its “lingering ability” or ability to stay near troops under fire to deliver support. It has also allowed pilots like Major Kim Reed-Campbell to fly for an hour and return to base after losing pieces of wing and all hydraulics. Here is a picture Reed-Campbell inspecting her damaged A-10 in 2003 after safely landing:

Deming, Wald and Boyd were able to move statistics and analytics beyond description and use mathematics for prescription. The techniques they developed for planning, measurement and reasoning remain relevant to this day.

No related posts.

> l <- list(x=c(1,2,3))
> l$x
[1] 1 2 3

But suppose we wanted to automate this; that is pass in the name of the value we want in a variable. We are after all using a computer, so automating a step seems like a reasonable desire. R supplies a notation for this using the “[]” operator. But something slightly different comes out under the “[]” operator than under the “$” operator:

> varName <- 'x'
> l[varName]
$x
[1] 1 2 3

Notice that the printed outputs are slightly different (one echoes "$x" and one does not). Let's use the "class()" method to see what is actually being returned in each case.

> class(l$x)
[1] "numeric"
> class(l['x'])
[1] "list"

Completely different return types are returned (in one case a numeric vector in the other a general list, not interchangeable types).

At this point you may think it is time to turn in our "pro" label and call ourselves "newb" (Internet slang for "newbie" or "idiot"). But let's slow down for a bit. When two views of the same situation disagree (such as the difference in opinion between the authors of R and myself whether the "[]" and "$" operators should return the same type) you at most know that at least one of those views is wrong. You don't really know if one view is right or even if one view is right which one it is. I can, however, bring in some additional argument to try and show the design of R is in fact wrong. The additional argument is "The Principle of Least Astonishment." This principle roughly says that it is a mistake to introduce unnecessary differences in outcomes (which to the unprepared user are unpleasant surprises). There may be some deep (yet obscure) reasons the two operators prefer to return different results. But the fact you would have to find a way to document and explain these differences really should make one think that this situation is really a mis-design and the "explanation" is really an attempt at a work around. Or to put it more rudely: there may be an explanation, but there is no excuse.

For another example consider creating a 3 by 3 matrix:

> m <- matrix(c(1,2,3,1,1,1,0,0,1),nrow=3,ncol=3)
> m
     [,1] [,2] [,3]
[1,]    1    1    0
[2,]    2    1    0
[3,]    3    1    1

Now select the last two rows of the matrix.

> m[c(FALSE,TRUE,TRUE),]
     [,1] [,2] [,3]
[1,]    2    1    0
[2,]    3    1    1
>

Now (for the punchline) try to select just the middle row of the matrix.



> m[c(FALSE,TRUE,FALSE),]
[1] 2 1 0

Notice that once again (and without warning) the result is subtly different. I admit that it seems paranoid to worry about such small differences- but when you are debugging a system that should work these are exactly the killing mistakes you are looking for. In this case the problem is pretty bad. See what happens if you tried to ask for the dimension of each of these differing returns:

> dim(m[c(FALSE,TRUE,TRUE),])
[1] 2 3
> dim(m[c(FALSE,TRUE,FALSE),])
NULL

The first case works fine (reports 2 rows and 3 columns). The second case returns "NULL" (instead of 1 row and 3 columns). In R NULL is sometimes used as an error-value (instead of throwing an exception) and this value will poison any further conditions or calculations it is involved in. The main way to deal with the arbitrary introduction of such NULLs is the incredibly tedious uncertain defensive coding practices that we argue against in Postel’s Law: Not Sure Who To Be Angry With. Such code weakens both programs and programmers.

But what is going on in this example? Once again we use the "class()" method to inspect the subtly different results.

> class(m[c(FALSE,TRUE,TRUE),])
[1] "matrix"
> class(m[c(FALSE,TRUE,FALSE),])
[1] "numeric"

The result is disappointing. For a two-row select R returns a matrix (what we would expect). For a single-row select R does us the "favor" of converting the result into a vector. This is a disaster. A single row matrix is similar to a vector, but even R itself does not support the same set of operations and outcomes on vectors as it does on matrices (for example the failure of the "dim()" method). It is not safe to further calculate with these results (without by-hand converting the result back to a single row matrix which R can in fact represent). In my case this created crashing bugs deep in a long running analysis (and was hard to diagnose as the bug was in an "innocent operation" not in a "risky calculation").

All of this has to violate John Chambers' "Prime Directive" for data: "an obligation on all creators of software to program in such a way that the computations can be understood and trusted." Chambers' opinion being relevant as he is the author of the S language (of which R is an open source re-implementation). We continue to recommend R, but we also recommend being exceptionally careful when using it (which unfortunately adds time to projects).

Related posts:

1. R examine objects tutorial
2. Relative returns: a banker versus trader paradox
3. Sorting Used in Anger

Postel’s Law (also called The Robustness Principle) comes from RFC 761 “Transmission Control Protocol” in 1980 and is: “Be conservative in what you do; be liberal in what you accept from others.” (Side note: RFC is the now ironic acronym used to describe Internet standards- the letters stood for “request for comments”).

This idea probably worked best where it started- in the TCP/IP world which is where a lot of hairy details of how computers network are handled. When your goal is the basic establishment of transient communications- success is measured by getting information through and not unnecessarily triggering a failure. It may be okay to tolerate mistakes here- because they don’t live long anyway.

Unfortunately, the law works less well other places where it is applied. The law is a downright hazard in dealing with archiving meaningful data (instead of managing transient signaling protocols). Sometimes the cost of obeying the law far outweighs the potential benefit.

A common arena for Postel’s law is now HTML, the markup language used to represent the content we view in web-browsers. In this arena Postel’s law has had two consequences: one good, one bad.

The good: almost anyone can create a working web-page or even a web-site because modern browsers have been designed to paper around almost every common HTML mistake. It has been pointed out that this ease of creation and “worse is better” (a deep principal due to Richard P. Gabriel see wikipedia: worse is better) has been one of the reasons that HTML out-competed and killed many other ideas. Philip Greenspun’s famous story of a 10-year-old building web site to get his mother medical attention happened in the sloppy world of HTML and could not have happened in the straight jacket of RDF (Resource Description Framework: the darling of the semantic web). I would not wish having to actually read or adhere to the incredibly long and irrelevant standards from w3.org (where the ratio of value to pedantry goes to zero) on an enemy. The web is only interesting due to its content and much of its content was only possible due to low barrier of entry.

The bad: to read HTML you almost have to re-create the entire history of web-browsers. This is a history of many hostile competitors (Microsoft, Netscape, Opera, WebKit, Mozzila, Google) and billions of dollars. Reproducing a significant fraction of this history is a significant (and useless) expense. For the most part I use a permissive parsing library like TagSoup or HTMLTidy but even these miss some things that browsers accept and are far more complicated than the task truly justifies.

Even worse is the cases of XML and RDF. These are often used for archival storage of semantic data. That is you may need to read and understand (not just display) data in XML for a long time. To be liberal in what you accept you have to again master a long set of useless complications (DTDs, namespaces incredibly inept character encoding and escapes) and still get burned by improperly encoded XML (that “used to work” because the bugs in the emitted XML matched the bugs in a library that is now out of date).

It is clear in the case of HTML and XML that Postel’s law’s cost is too high for what it delivers. Or at least half of the law is too expensive: no amount of being generous in what we accept makes up for the original data not have been impounded correctly (not being “conservative in what they do” and not having checked that at the time it was created). Some of this is that the producers of the data have no way of telling they are not being “conservative in what they do” because the “generous in what they accept” libraries they use to debug don’t tell them they are emitting bad data. And lets be honest- most systems are not designed for correctness, they are instead debugged until they seem to work. I would say that in fact HTML is not an example of the power of Postel’s law but of the pernicious influence of “worse is better.” Computer science has not risen to the level of “software engineering” we still are a horrible “fit to finish” industry.

Frankly for many things we need a simpler “fail early” discipline. Tools need to be better and standards need to be simpler so that if you write something that is wrong it is easy to see why it is wrong and easy to fix it. Postel’s law has helped hide the negative impacts of complicated standards, we need to push the cost of complications back on to standards committees. The need to be “generous in what you accept” overly favors large, rich entrenched players who have had the time and resources in incrementally invest in papering around every common mistake.

However, I am not sure if we can throw out half of Postel’s law or even if we want to. When Postel’s law fails it is not clear who to be mad at.

Sun, to kick somebody who is already down, was famous for making elaborate frameworks that correctly and brutally implement many details of RFCs. Sun’s Java includes huge frameworks for XML, UTF8 and email that scrupulously implement page after page of useless standard documentation but fail in the wild due to not being “generous in what they accept.” For example Sun’s GlassFish (which got listed named as one of four or five important assets during Sun’s various acquisition talks much like the fact the car has cup-holder somehow always gets mentioned in spec sheets) is an “open source production-quality enterprise software application server.” A supposedly major component of the GlassFish is its email component which is a huge unwieldy framework that implements many of the email related RFCs and protocols including IMAP. Unfortunately for all its hugeness it can not reliably read email folder names from one of the biggest IMAP servers: Google Mail. Google Mail includes “against standard” characters in the protocol and crashes the GlassFish software.

And here is where Postel’s law fails us: under Postel’s law both sides are at fault (Sun for failing to be generous in what they accepted, Google for failing to be conservative in what they did). We can’t assign only one villain. We have no proscription of who to ask for a fix. Postel’s law seems useful in that if either Google or Sun had followed it the two systems would work. But the law doesn’t pick one side to assign blame and help us to efficiently diagnose and fix the problem. It becomes difficult to find the critical bugs when they are masked by a see of “acceptable” bugs. Take a contrary example: the simpler law “implement the standard or fix the standard” would clearly assign blame to GMail.

Similar pain is encountered in Java’s handling of character encodings like UTF8. It is hard to move up the stack of artificial intelligence (from words, to concepts, to ideas, to reasoning to consciousness) when you can’t even reliably transcribe characters. When faced with bad character sequences (a common occurrence on the web) there is no practical way to get Java “mostly parse it,” Java libraries and frameworks authors seem to extract a perverse joy in throwing a program-killing exception (it does not matter if you catch it the library has already stopped doing what you wanted) because they are concerned that a diacritical mark was not properly encoded (web browsers, on the other hand, lose the mark or show some sort of damage near the mistake and blunder on). And here is were the frustration sets in, how can you make applications that are generous in what they accept when the libraries and frameworks are overly proud and picky? This, at first, seems like an argument for Postel’s law- if everybody else (especially the library authors) were generous in what they accepted your life could be easy. That is certainly one possibility- but I argue it often becomes a matter of semantics to assign blame where there is no pre-existing specification or performance agreement. In the end you will waste more time dealing with errors that should never have made it to you than the time you save emitting the odd error of your own.

The unit testing people have a somewhat better idea: fail early, fail at the factory where it is cheap to fix. Don’t litter all of your code with indecisive statements like:

  Set matches = computeMatches();
  if( matches!=null ) {
     for(String match: matches) {
         ...
     }
  }

Instead: write a unit test to document you expectation that the empty set is expressed in single consistent way:

   Set matches = computeMatches();
   assertNotNull(matches);

And from then on write more confident code:

  for(String match: computeMatches()) {
         ...
  }

This may seem overly optimistic and overly strict- but I have a point. One of the few good principles in computer science (and perhaps one of computer science’s contributions to knowledge, computers are a huge contribution to society- but they were made by engineers) is composition. A plan for getting from A to B followed by (or composed with) a plan for getting from B to C is a plan for getting from A to C. Well a correct plan for getting from A to B when composed with a correct plan for getting from B to C, if each of the plans “is mostly right if the piece after is so nice to fix up a few mistakes” you really don’t know what you have. You may have nothing.

That is my complaint- you can’t put an a priori bound on how expensive attempting to allow both sides of Postel’s law will be. You would like others to paper over your mistakes, but it is becoming too expensive to paper over the mistakes of others. In the end Postel’s law is of little help when cleaning up the inevitable mess.

Related posts:

1. Map Reduce: A Good Idea
2. R annoyances
3. Something I don’t get about business and bailouts

• Statistics:

  “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures

  Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’

  Statistics to English Translation, Part 2b: Calculating Significance

  Good Graphs: Graphical Perception and Data Visualization

  Google AdSense Channels IDs and the Cramer Rao Inequality

  A Demonstration of Data Mining

• Mathematical Finance:

  What is the gambler’s equivalent of Amdahl’s Law?

  Betting Best-Of Series

  Automatic Generation and Testing of Un-Rolls for Profitable Technical Trades

• The R statistical package:

  Survive R

  R examine objects tutorial

• Computer Science:

  The Local to Global Principle

  Sorting Used in Anger

  On The Hysteria Over “The Cloud”

• Philosophy:

  What is Mathematics, Really?

  Hello World: An Instance Of Rhetoric in Computer Science

Related posts:

1. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
2. Statistics to English Translation, Part 2b: Calculating Significance
3. Hello World: An Instance Of Rhetoric in Computer Science

Portfolio allocation is not the “magic predict the future” part of finance, it is the scheme for correctly applying magic predictions of the future. The idea is that if you had an prediction of future returns of a number of assets, the naive thing to do would be to invest everything into the asset with highest predicted return. Portfolio theory, while still taking the predictions at face value, picks an investment pattern that will (in risk-adjusted dollars) outperform the naive strategy even if the predictions are correct and is a bit safer when the predictions are wrong.

Suppose you had different assets you could invest in. For the -th asset there is an expected excess relative return of and an estimated variance of (for a definition of relative return see Relative returns: a banker versus trader paradox and for a definition of variance see A Quick Appreciation of the Sharpe Ratio). Let the vector be such that represents the number of dollars we invest in the -th asset. If is positive then our plan is “to go long” or buy some of the -th asset. If is negative our plan is “to short” or sell some of the -th asset to somebody else (It is called going short as we actually sell something we do not have. This is often allowed in finance; as long as we make the same pay-outs to the buyer that the buyer would receive if we really had the item to sell).

When we appeal to the idea of optimizing the portfolio Sharpe Ratio (again, see A Quick Appreciation of the Sharpe Ratio) then we say a good portfolio is one that doesn’t just maximize expected relative returns (which is ) but maximizes the ratio of expected relative return to standard deviation:

where (for now) is the matrix . This ratio is called a “risk adjusted return” (versus the un-adjusted form ). Also notice that the ratio is homogeneous in (doubling does not change the ratio as it simultaneously doubles the numerator and the denominator) so an optimal solution describes not how much to invest, but what pattern to invest in. This allows us to introduce an important practical constraint: we are only going to allow ourselves to risk a total of dollars (both long and short). That is: we insist . We will ignore this total investment constraint until the end when we can satisfy the constraint by simply re-scaling an partial solution.

To solve for we introduce an old friend: Lagrange Multipliers (or equivalently the Karush-Kuhn-Tucker conditions of optimality). Since the fraction we are trying to optimize is homogeneous in we can convert the denominator into a constraint and arbitrarily insist that without changing the nature of the problem. We are now trying to maximize subject to . The Lagrangian conditions of optimality state at the optimum we must have the gradient of the objective is proportional to the gradient of the constraint or:

for some (to be determined) constant . Pushing the gradient operator through we get:

A similar equation could be gotten by appealing to a Rayleigh Quotient argument.

We do not yet know (that is what we are trying to solve for), so we do not know what is. However, this is just a scalar and since we are just trying to solve up to a multiple we can throw it out and introduce a new multiple and see that it is enough to solve:

where is new (still unknown) scalar. This means we have:

so our desired solution is some re-scaling of .

As we stated earlier we have a total investment constraint of . We can achieve this with the following adjusted solution:

as our desired optimal portfolio allocation. In the end we can solve for the optimal portfolio by merely solving a linear system (we don’t need anything as expensive as a general purpose optimizer in this case).

These are very old results (going back as long as there has been Sharpe Ratios and portfolio theory). A good example reference is: “The Valuation of Risk Assets and the Selection of Risky Investments in Stock Portfolios and Capital Budgets,” John Lintner, The Review of Economics and Statistics (1965) vol. 47 (1) pp. 13-37. These results are the basis for advice like: “diversify.” Without modeling risk you would tend to put all of your money in the predicted highest paying asset. When modeling risk you tend to put some of your money in each high paying asset and as long as they do not all fail at the same time you have some safety. Another (very different) route to diversification is the Kelly Criterion (discussed in What is the gambler’s equivalent of Amdahl’s Law?).

A very important risk we have not yet modeled is that our assets may have a tendency to fail at the same time (meaning we may not have really diversified usefully). The notion of assets may fail at the same time brings us to the ideas of correlation and covariance. When we took we were implicitly assuming (or modeling), without justification, that each possible asset was independent of all the others (that there was no correlation between asset returns). This is, of course, not going to be anywhere near true in practice. Instead we should take to be the Covariance Matrix that represent our estimate of the assent to asset correlations. In this case the solution methods above all work exactly as before. Companies such as MSCI Barra have made complete businesses out of producing and selling estimates of .

Another issue is when we do not allow ourselves to “short” (or take a negative allocation of) assets. In this case we have the additional constraints which complicates our solution. For the special case where the asset variances are assumed to be independent (i.e. ) it is enough to solve as above and merely replace any negative allocations with zero when inspecting and scaling the final step of the solution. When the covariances are non-trivial ( has non-zero off-diagonal entries) this solution may not be optimal. In this case the Karush-Kuhn-Tucker conditions are more complicated and at the point of optimal solution we have the following conditions:

		0
		0
		0
		0

where is the allocation vector we wish to solve for, is an unknown scalar, is a new unknown vector and is the vector with and zeroes elsewhere. Using the Karush-Kuhn-Tucker conditions has allowed us to again almost linearize the problem, but we know have sign constraints on and and what is called a complementarity constraint: . This sort of problem essentially called a “Linear Complementarity Problem” and is about as hard as solving a linear program (the typical solution method is a variation of the simplex method called “Lemke’s algorithm”). (Technically the prevents the problem from being in the right form, but can be inspected out of the problem.) The problem can still be solved, you just need a bit more software. If we can not short assets (or at least simulate shorting assets) we not only eliminate many possible portfolios from consideration (so we likely end up with a less profitable portfolio than we would like) we also make the mathematics and computation a bit harder.

The goal of this writeup has been to show how to systematically convert investment advice like “this stock is going to really take off” into an allocation of assets (which in turn implies a pattern of trades). We take as unexamined premises where to get such advice and whether to use the Sharpe ratio or some other notion of risk and/or utility. The point is that even though it may be complicated, from this point it is just calculation and calculation is easy to automate.

Related posts:

1. A Quick Appreciation of the Sharpe Ratio
2. A Discrete Model Gauging Market Efficiency
3. What is the gambler’s equivalent of Amdahl’s Law?