The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

then the following is a picture of reverse accumulation:

Related posts:

1. Automatic Differentiation with Scala
2. “Easy” Portfolio Allocation
3. A Quick Appreciation of the Sharpe Ratio

Our Example Problem

Our small scale problem is this: we have a number of target points on a map and we want to pick a central point to directly connect to all of these points with wire. Our goal is to minimize the total amount of wire used. This problem is called the “Geometric Median”. So we are trying to find a point that minimizes the sum of distances from our chosen center to every target point. If we were trying to minimize the sum of squared distances from our chosen center to every target point the answer would be obvious: the average or mean (which by Hooke’s law is also the point where a set of identical springs would relax to). The mean is in fact a fairly good guess, but you can do better (which could important if the “wire” is expensive, such as cutting irrigation or drainage ditches). For example given the three target points (20,0), (-1,-1) and (-1,1) the optimal point is (-0.42,0) not the mean (6,0) and the choice of optimal point represents an over 19% savings in total wiring distance (see figure).

This is a substantial saving in cost.

The problem changes as we consider variations. If indirect connections (such as routing one point through another, which may or may not be possible for reasons of capacity or safety) and multiple new centers are allowed we then have an instance of the Steiner Tree Problem which is harder to solve (since it is known to be NP complete). If no new centers are allowed (all routing must be between pre-existing target points) then we have a Spanning Tree Problem- which admits very quick solutions.

We bring up the geometric median as a mere example. We don’t intend for our code to solve only the geometric median problem and we don’t intend to touch on the literature of specialized methods for solving the geometric median problem. Instead we are trying to demonstrate the speed you can develop prototype solutions if you have a few good tools (like various optimizers) available in your toolkit. Numeric optimizers may sound exotic, but they often are the kind of thing you want to experiment with and link directly into your code.

Optimization as General Tool

Now that we have the example problem we can describe a solution strategy. In this case the solution uses code “we wished we had lying around” before we started on the problem. We will pretend we have the tools we want ready to solve our problem and then we will pay our debt and build the required tools. The issue is that there is not an obvious closed form for the solution of the geometric median problem. So we are forced to work a bit harder. In this case harder means we need to solve an optimization problem. Consider the contour plot of the total wiring cost as function of where we choose to place our center. Our optimal point (-0.42,0) had wiring cost of 22.73 and the contour plot given here shows concentric regions of solution positions with higher cost.

In general it is unwise to throw an optimizer at an arbitrary problem and hope to find the globally best solution. But in this case (and in many similar situations) we can prove that a simple local optimizer will in fact find the unique best solution. This is a property of the problem not of the optimizer. The concentric regions shown in the contour plot have a very nice shape: they are convex. That is: they have no intrusions- for any two points drawn from one of these shapes the straight line segment between these points stays inside the given shape. We don’t have to depend on observation- we can actually prove this is always the case for this problem. The wiring cost from a proposed center to any single target point is a convex function of where we choose to place our center (a convex function is a function whose graph never reaches above the secant line drawn between any two points on its graph). The total wiring cost is just the sum of the wiring costs to each target point. And to finish: the sum of a collection of convex functions is itself a convex function. Since the contour plot of a convex function has only convex shapes and we have proven the statement.

But how does this help us? There is a standard technique to find “local minima” of a function by inspecting a function for places where the gradient is zero (points where there is no obvious down hill direction on the contour plot). This technique usually can only be guaranteed to find local minima (places where no small change improves your situation). But there is no guarantee that the local minimum you find is in fact the global minimum (the best possible solution). Except when you are dealing with a convex function. When a function is convex then all of the local minima are always grouped together into a single convex connected shape (if not a line drawn between two remote minima would violate the convexity definition). And if the function is never flat then this set is a single unique point: the unique best solution. Our inspection technique will be a gradient driven optimizer- that is an optimizer that when the gradient is non-zero improves its objective by running down hill and halts when the gradient is zero.

The stated function to minimize is to sum the distance from our proposed center to each target point. We can write this as the sum of the distances:

( which is the traditional Euclidean or L2 distance). This function actually has one one subtle flaw that we will deal with in the appendix (see: Fixing Smoothness).

Using Scala to Apply the Optimization Solution

To find our optimal center placement using Scala we first write our cost or objective as a Scala function:

    val dat:Array[Array[Double]] = Array(
      Array( 20, 0.0),
      Array( -1.0, 1.0),
      Array( -1.0, -1.0)
    )

    def fx(p:Array[Double]):Double = {
      val dim = p.length
      val npoint = dat.length
      var total = 0.0
      for(k <- 0 to (npoint-1)) {
        var term = 0.0
        for(i <- 0 to (dim-1)) {
          val diff = p(i) - dat(k)(i)
          term = term + diff*diff
        }
        total = total + scala.math.sqrt(term)
      }
      total
    }

Scala is succinct and it is a great connivence to have a function definition capture data from its environment. What we would like to do is generate an initial guess as the solution (we use the mean as our initial guess) and then call an optimizer (in this case a conjugate gradient optimizer) to do all the work:

 val p0:Array[Double] = mean(dat)
 val (pF,fpF) = CG.minimize(fx,p0)

At this point we would be done, except the conjugate gradient method (which is superior to gradient descent and many the non-gradient methods) requires a gradient.
We could provide a numeric estimate of the gradient by the following divided difference method:

  def gradientD(f:Array[Double]=>Double,p:Array[Double]):Array[Double] = {
    val xdim = p.length
    val p2 = copy(p)
    val base = f(p2)
    val ret = new Array[Double](xdim)
    val delta = 1.0e-6
    for(i <- 0 to (xdim-1)) {
      p2(i) = p(i) + delta
      val fplus = f(p2)
      p2(i) = p(i)
      val diff = (fplus-base)/delta
      ret(i) = diff
    }
    ret
  }

This numeric divided difference method often outperforms non-derivative optimization methods (like Powell’s Method and the Nelder-Mead Amoeba method). But the technique can run into numeric difficulties. We can remedy this if we are willing to write our function in a slightly more general way. If we re-encode our function in a generic manner we can use automatic differentiation (not to be confused with numeric differentiation or with symbolic differentiation) to produce a reliable gradient for optimization. What we need to do is re-write our function to work over an abstract field of numbers instead of only the machine supplied doubles. In fact what we need to do is specify a generic function that will work over any field, with the field to be determined later. The code to do this in Scala is very similar to the non-generic code:

   val genericFx = new VectorFN {
      def apply[Y <: NumberBase[Y]](p:Array[Y]):Y = {
        val field = p(0).field
        val dim = p.length
        val npoint = dat.length
        var total = field.zero
        for(k <- 0 to (npoint-1)) {
          var term = field.zero
          for(i <- 0 to (dim-1)) {
            val diff = p(i) - field.inject(dat(k)(i))
            term = term + diff*diff
          }
          total = total + smoothSQRT(term)
        }
        total
      }
    }

Notice that code is very similar to the “def fx()” code. The key differences are that we had to define genericFx as extending a trait (a type of Scala interface) called VectorFN and inside this trait extension we defined a parameterized function name apply(). apply() is a generic function that is willing to work over any type Y where Y is at least of type NumberBase[Y] (we will get more into what that means in a moment). The difference in notation is that while the Scala function syntax can not specify a generic function with free type parameters (the incompletely specified Y) the Scala semantics are strong enough to implement this. In fact standard function definitions (such as “def fx()”) are just syntactic sugar for extending the Scala built-in Function1 trait. With a generic objective function in hand all we need is conjugate gradient code that is expecting a VectorFN (and willing to call apply() instead of just using naked function parenthesis) and some type NumberBase[Y] that can compute gradients for us. The Scala compiler can specialize our genericFx() into one version for quick calculation and another for gradients. How this is done is what we will discuss next. From our point of view our problem is solved with the following one line of code:

val (pF,fpF) = CG.minimize(genericFx,p0)

This should always be your goal- build sufficient preparation so your last step is a “obvious one liner.”

What Tools we Wish we Had Lying Around

We supply in our example some workable conjugate gradient code, but that is standard so we will not discuss it. What is of interest (and facilitated by Scala’s parametrized type system) is the implementation of dual numbers as a framework to supply automatic differentiation. An implementation of dual numbers as a NumerBase[DualNumber] type is the core of our demonstration.

Dual numbers are an algebraic structure written as pairs of real numbers “(a,b)”. The arithmetic table for dual numbers is given below:

In a dual number (a,b) “a” is the “large” or “standard” part of the number. You can check from the arithmetic table that the pair of dual numbers (a,0) and (c,0) behave just as we would expect the real numbers a and c to behave. In the dual number (a,b) “b” is the “small” or “ideal” portion of the number. From the multiplication rule above we can observe two rules: (0,b) * (c,0) = (0,b*c) (something small times anything else is small) and (0,b)*(0,d) = (0,0) (two small things become zero when multiplied). Essentially the dual numbers are carrying around the first two terms of a Taylor series: we get as a result both the function value and the function derivative. For a function f() over the real numbers we extend f() to work over the dual number by defining: f((a,b)) = (f(a),b f’(a)) (which is consistent with the previously defined arithmetic). We can check that the dual numbers numbers obey the usual laws of arithmetic (associative, commutative, distributive, identities and inverses). The punchline is that over the dual numbers the divided difference estimate of f’(x) (the derivative of f() evaluated at x) is in fact exact in the sense that f((x,1)) = (f(x),f’(x)) (or f((x,0)+(0,1)) – f((x,0)) = (0, f’(x))). Implementing the DualNumber class is little more than transcribing the above arithmetic table into Scala.

We have already seen how to write code that uses NumberBase[Y] types (genericFx() itself is an example). A more complicated example is the CG.minimize() code which not only accepts a generic function (in the form of VectorFN) but then specializes it to NumberBase[DualNumber] to compute gradients and also specializes to NumberBase[MDouble] for quick calculation during line searches (MDouble is just an adapter for machine Doubles, used for speed). The ability to re-specialize a function is one of the advantages of a parameterized type system. The DualNumbers are an example of forward automatic differentiation. We could also use the same object framework to capture a representation of the computation path and apply more sophisticated methods such as reverse automatic differentiation.

We give a link to a jar containing complete Scala source code including this example, the DualNumber implementation, a conjugate gradient implementation and some JUnit tests (all under a GPLv3 license) and will go on to describe some of the design decisions. The code is the bulky part of this work, so we will move on to discuss something more compact: types.

Types

If code is ever beautiful it is only when it is succinct. Among the most succinct forms of code are individual type signatures and interfaces (though the indiscriminate repetition of type signatures is rightly considered ugly bloat, which Scala works to avoid). Since we are distributing complete source we will describe only types and method signatures. The entry points to the code are the JUnit tests (organized in the ScalaDiff/test source directory and depending on JUnit which was not included) and the demo program in ScalaDiff/src/demo/Demo.scala).

To be a usable arithmetic type (like DualNumber or MDouble) you must extend the following parameterized abstract class:

abstract class NumberBase[NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  // basic arithmetic
  def + (that: NUMBERTYPE):NUMBERTYPE
  def - (that: NUMBERTYPE):NUMBERTYPE
  def unary_-():NUMBERTYPE
  def * (that: NUMBERTYPE):NUMBERTYPE
  def / (that: NUMBERTYPE):NUMBERTYPE  // that not equal to zero
  // more complicated
  def pow(that:Double):NUMBERTYPE
  def exp:NUMBERTYPE
  def log:NUMBERTYPE // this is positive
  // comparison functions
  def > (that: NUMBERTYPE):Boolean
  def >= (that: NUMBERTYPE):Boolean
  def == (that: NUMBERTYPE):Boolean
  def != (that: NUMBERTYPE):Boolean
  def < (that: NUMBERTYPE):Boolean
  def <= (that: NUMBERTYPE):Boolean
  // utility
  def field:Field[NUMBERTYPE]
}

In particular DualNumber extends NumberBase[DualNumber]. This deliberate circular reference has a big purpose: it allows publicly visible contravariant return types (returning nearly the exact type we really are instead of a base type). This allows us to have strict type arguments so that trying to add a MDouble to DualNumber is a type error (even though they both extend the same base class). The automatic differentiation technique encapsulated in the DualNumber class only works if all of the calculation is in the DualNumber types and this strict type enforcement allows the compiler to help prevent results sneaking in and out through other types. All of the methods on NumberBase are obviously related to arithmetic except the field() method. This method gives us access to a Field object which is responsible for carrying around the runtime type information (this is a common problem in Java and Scala, that some type information known at compile type such choice of template types is not easily accessed at runtime). The Field class is as follows:

abstract class Field [NUMBERTYPE <: NumberBase[NUMBERTYPE]] {
  def zero:NUMBERTYPE            // return canonical zero in field
  def one:NUMBERTYPE             // return canonical one in field
  def inject(v:Double):NUMBERTYPE  // return canonical representation of number in field
  def project(v:NUMBERTYPE):Double // return standard-number represented in field
  def array(n:Int):Array[NUMBERTYPE] // return an array of this type

The Field class is where we have factories for numbers (zero, one, arrays, injection from standard Doubles), casting (projection back to standard Doubles).

With these types defined we can actually read intent off some of the method signatures.

For example our conjugate gradient optimizer is accessed through the following method signature:

 def minimize(fn:VectorFN,x0:Array[Double]):(Array[Double],Double) // return x,f(x)

The above can be read as: CG.minimize() requires a VectorFN (our trait representing single argument functions with a free type parameter) and an initial point (in standard Doubles). The code will the return a pair of the optimum point and the function evaluated at the optimum point. From the type signature we can see that CG.minimize() expects to re-specialize the function “fn” to types of its own choosing (else it could have accepted a parameterized argument instead of our custom trait) and will handle all up-conversion and down-conversion between machine Doubles and NumberBase[Y]‘s itself. This sort of type information is hard to express (let alone enforce) in a dynamically typed language.

A slightly more complicated example is the lineMinD() method:

def lineMinD[Y<:NumberBase[Y]](field:Field[Y],
 f:Array[Y]=>Y,
 xm:Array[Double],
 di:Array[Double]):(Array[Double],Double)

Notice it is willing to work with any type parameterized function (which means it is willing to let the caller pick the actual type of NumberBase[Y] and work with that). Most callers will call with Y=MDouble (the wrapper for machine Doubles) and lineMin() will then work with that (without ever really knowing the actual underlying type).

A lot of fans of dynamic languages consider type systems to be mere hairshirt penance. But that is not so. Broken type systems (like Java’s collections before erasure parameters were introduced in Java 1.5) are indeed more trouble than they are worth. Working type systems (like C++ Templates/STL, Java 1.5+ and Scala) allow you to solve problems (and enforce decisions) during the design phase (which is much much cheaper than during the deployment phase). You can’t set your types in stone (you are likely going to have them subtly wrong for the first few iteration). You must be willing to think like a “language lawyer” to find out what parts of your work can be specified and enforced in the language type system. To use an analogy: static types are your blueprint or your underpainting.

Tests

One argument against static types is that you can get much of their benefit from unit tests. My opinion is you never have enough unit tests, so putting more pressure on your test suite is not wise. Static types plus tests are strictly more powerful than static types alone or tests alone.

Even for this example toy-scale project we have include a JUnit test set to pursue a number of goals:

• Confirm our number implementations (DualNumber and MDouble) correctly model machine Doubles (perform parallel calculations and compare).
• Confirm DualNumber obeys expected laws of algebra composition and cancellation including the portions that can not be modeled in machine Doubles.
• Confirm DualNumbers compute gradients.
• Confirm operations of optimizers and optimizer components.

Many of these tests are related, but they don’t all imply each other and give different perspective on the errors they catch. For example no amount of parallel computation between DualNumbers and machine Doubles is going to confirm the infinitesimal portion of the DualNumber is propagating correctly (since this is not a property of machine Doubles). So we add extra tests that expect DualNumber to obey algebraic relations like: a*(b+c) = a*b + a*c hold. It is then another step to confirm that whatever the DualNumbers calculate is not only self-consistent, but also models a truncated Taylor Series or differentiation.

Conclusion

We hope we have demonstrated how the complexity of a mathematical programming problem can be managed by breaking the problem into an objective function that is separate from the optimizer (allowing the optimizer to be both good and hidden) and a static type system (such as Scala) to help enforce required properties of a calculation (such as all numbers being routed though a required representation). With these sort of tools available many formerly hard problems (that are often, unfortunately solved by over-specifying direct inefficient iterative improvement techniques) become “if I can write a reasonable objective function this may already by solved by an optimizer in my library.” The more of these tools you have (either in your code or in your reference library) the more of these problems become easy (this is the topic of my earlier paper: The Local to Global Principle).

Appendix: Fixing Smoothness

Our chosen example objective function is very nice (i.e. convex) but it has a small (but correctable) problem. The derivative or gradient or gradient has some jump discontinuities that could cause an optimizer to exit prematurely (not at the global optimum). Consider the simple form of this for wiring a center to a single point at the origin (even in 1 dimension). The wiring cost function is sqrt(x*x) has a cost graph as shown here.

This is convex- but derivative is not smooth as we see in the included graph of the derivative of sqrt(x*x).

So: in this case if the optimizer stops at one of the target points we can’t be sure that it stopped at the global optimum (it may have stopped due to the discontinuity in the gradient). For some simple problems the optimum is necessarily at a target point. For example on the number line take the target points 0,1 and x. As long as x≥0 and x≤1 the optimum placement will be x itself.

One way to defend against this is to use some sort of smoothed version of sqrt() that essentially decreases a little faster near the origin. Our cost function becomes:

Related posts:

1. Gradients via Reverse Accumulation
2. R examine objects tutorial
3. Survive R

This client already tracked the net profit and loss generated by every customer who used his services, and had begun to analyze his customers by group. He was especially interested in Customer Type A; his gut instinct told him that Type A customers were quite profitable compared to the others (Type B) and he wanted to back up this feeling with numbers.

He found that, on average, Type A customers generate about $92 profit per month, and Type B customers average about $115 per month (The data and figures that we are using in this discussion aren’t actual client data, of course, but a notional example). He also found that while Type A customers make up about 4% of the customer base, they generate less than 4% of the net profit per month. So Type A customers actually seem to be less profitable than Type B customers. Apparently, our client was mistaken.

Or was he?

A little more elementary statistics revealed that the median profit generated by Type A customers is $65 — e.g., half the customers from group A generate more than $65 profit per month. The median for Type B customers is about $4.80 — so half the customers from group B generate less than five dollars profit every month. Maybe our client’s instincts aren’t completely off-base.

Let’s look at the distribution of net profit across both customer populations:

Figure 1: Distribution of net profit for Type A customers (blue) and Type B customers (red). The x-axis gives the net profit or loss, and the y-axis gives the fraction of the population that generates a given net profit.

This pattern is typical among the customers of many businesses. The majority of customers generate relatively moderate profit (or loss); but there is an important minority of large-profit and large-loss customers out on both tails. In this case, the monthly customer value actually ranges from losses in the tens of thousands to profits of several hundred thousands (I clipped the graph, for “clarity”).

I hesitate to call these large magnitude customers “outliers” because that term implies anomalous, possibly erroneous, data. In this case, the “outliers” are relatively rare, but important, customers who can potentially make the difference between a company that is in the black or in the red. Still, they are the exception and their behavior doesn’t necessarily tell you anything about the behavior of your typical customer. Knowing the mean profitability of a given customer group is important, of course, but the estimate will be dominated by your exceptionally profitable or lossy customers in that group, and as we’ve seen, that hides information about the majority of your customers.

You might remember from our Good Graphs article that if you have positive skewed data with a wide dynamic range, graphing the data on a log scale helps you see phenomena across the entire range of data that you might miss on the ordinary graph. Unfortunately, we have data here in the positive and negative range. So let’s split the customers into three groups: profitable, unprofitable, and break-even. About 5-6% of the customer base is break-even, roughly the same proportion in Groups A and B; we’ll ignore them for now, and look at the profitable customers first (over 80% of the customers, in both groups).

Figure 2: Distribution of profit from profitable Type A customers (blue) and Type B customers (red). The x-axis gives net profit on a log 10 scale, so every labelled tick corresponds to a change by a factor of 100 (eg. 10^0 = $1, 10^2 = $100, and so on). The y-axis represents the fraction of the profitable customer base that generates a given profit.

Now we can clearly see that (among profitable customers) the typical Type A customer is in fact more profitable than the typical Type B customer. The mean profit from profitable Type A customers is about $227, and the median profit is about $93 (shown by the dashed blue line). About 2/3 of the profitable Type A customers generate between $21 and $400 in profit, and over 95% of them generate between $5 and $1721 in profit. We can call that 95% the set of “typical” profitable Type A customers. That’s not a standard definition, but it’s an intuitive one, and useful for this discussion.

Approximately 2.5% of Type A customers generate profits greater than $1721; let’s call them the Type A “best-customers,” some of whom generate profits in the tens of thousands. They are responsible for 30% of the profit that comes from profitable Type A customers, and 3% of the profit that comes from all profitable customers (even though they only make up 0.2% of that population).

Profitable Type B customers generate $148 mean profit, and about $7.67 median profit (the red dashed line). A typical profitable Type B customer generates between six cents and $1031 in profit — a lower range than what the typical Type A customer generates, although the very highest-performing Type B customers are competitive with the highest-performing Type A customers (about 130 Type B customers outperform all the Type A customers).

Unfortunately, when Type A customers are unprofitable, they are typically more unprofitable than those of Type B. This is another reason why the mean profit from Type A customers overall was so low. Our client correctly perceived that Type A customers are typically quite profitable, but there is a small population of real clunkers in the group, too.

Figure 3: Distribution of loss from unprofitable Type A customers (blue) and Type B customers (red). The x-axis gives loss on a log 10 scale; further to the right on the graph means a larger loss. An unprofitable Type A customer loses a median of $137 a month, and a mean of $1180. Unprofitable Type B customers lose a median of $4.80, and a mean of $210.

We can do a similar analysis for the entire base of profitable customers. We would find that the typical profitable customer generates between six cents and $1200 in profit every month (median $8.65, mean $153), and that the 2.5% of best-customers generate over 60% of the profits.

The Lognormal Distribution

Figure 4: (Left) Distribution of profitable customers (graph clipped at $10,000). The x-axis gives the net profit, and the y-axis gives the fraction of the population that generates a given net profit. (Right) Distribution of profitable customers plotted on a log scale.

The distribution of highly skewed positive data, like the value of profitable customers, incomes, sales, or stock prices, can often be modelled as a lognormal distribution: that is, the log of the data is distributed in a bell-shaped curve centered (in log space) at the median of the data (remember, for a normal curve, the median and the mean are the same). In our case, both the profits (seen above, in Figure 4) and the losses are distributed approximately lognormally. For lognormal populations, the mean is generally much higher than the median, and the bulk of the contribution towards the mean will be made by a small population of highest-valued data points. If you use the mean as a stand-in for value, you will overstate the value of most of your customers.

If your customer value data is distributed approximately lognormally, then you can quickly estimate the range of values that 95% of your customers will fall into. About 95% of normally distributed data will fall within plus/minus two standard deviations of the mean, and taking logarithms converts multiplication into addition. So: if sd is the standard deviation of the natural log of your customer value data, M is the median profit, and k = exp(sd), then 95% of your customers will fall in the value range (M/(k*k), M*k*k). The 2.5% of customers who generate more than M*k*k profit are your best-customers, who often drive a majority of your profit.

Long Tail Theory

The distribution of customers above sounds a lot like Chris Anderson’s Long Tail Theory of consumer goods. Most of the revenue of (for example) a bookseller or a music store comes from a few “hits”, or blockbusters, with the rest of the merchant’s inventory out along the tail of Figure 5, moving a relatively small volume per title.

Figure 5: (Top) A notional long tail curve. The y-axis represents sales volume, and the x-axis represents goods ranked from most to least popular. The highest selling goods are to the left. Note that this figure represents the sales curve differently from how the distribution of customer value is represented on the left side of Figure 4. (Bottom) The customer value data (top 10,000 customers) from Figure 4, plotted in the style above. The y-axis has been limited to $50,000 for clarity.

Anderson generally assumes that sales of such goods are distributed as a power law distribution, rather than a lognormal; the log of power law data isn’t distributed symmetrically, but actually has a longer tail to the right. This means that even for the log of the data, the mean is higher than the median. In fact, in some cases, the mean of a power law distribution can be infinite. If sales volume is power law distributed, then top-selling hits are responsible for an even larger percentage of total sales volume than would be the case with a lognormal.

The Pareto Distribution, which is one form of a power distribution, has been proposed as an alternative to the lognormal for modelling income distribution and other similar phenomena. Researchers have debated whether lognormal or Pareto is a better model for income distribution since at least the 1950s. Qualitatively, the two distributions have similar behavior. There are certain estimation and forecasting tasks where it does make a difference if your data follows a power law rather than a lognormal, but for the purposes of this discussion, it doesn’t really matter. For those who are interested, Michael Mitzenmacher has a fairly approachable discussion about the difference between power laws and lognormal distributions.

Back to Long Tail Theory. Historically, merchants tend to concentrate on high-volume items, due to space limitations and the cost of holding inventory. Overall, however, the sum total of tail-product sales will add up to a respectable volume, especially for web retailers who have unlimited “floor space” — or so the Long Tail theory goes. A retailer must then decide whether to follow the traditional “hits-oriented” strategy, or a more “tail-oriented” strategy that caters to the numerous niche markets.

If we draw an analogy with customer value, then best-customers are “hits.” Obviously, our client would like to “fire” his unprofitable customers while retaining his best-performing customers, and even attract more customers like them. But what about his little customers — the 95% of customers in the typical range? If his retention and growth strategy focuses primarily on attracting and retaining big customers, he is following a hits-oriented strategy. If his campaign also includes reaching out to little customers, then he is following something analogous to a tail strategy.

Not all business works like a music or book seller; the appropriate strategy will vary. Still, we can think of a few reasons why keeping little customers happy is a good idea.

For one thing, big customers are not only rare, but they are the ones that your competitors covet the most. Little customers, meanwhile, can still add up to a respectable chunk of change (close to 40% of net profit in our example above). A solid cushion of smaller customers may soften the blow to your profit margin, should a few of your bigger customers defect.

Consider computer sales. Microsoft and Dell serve both the corporate and consumer markets. To judge from their past marketing practices, they consider business customers to be the more valuable segment (see here for a rant somewhat related to this topic). But business IT sales have declined in the current moribund economic climate; analysts attribute the growth in computer sales for the last quarter of 2009 primarily to consumer spending. Dell’s market growth for that last quarter was much lower than that of HP, Acer, and Apple, which are more consumer-oriented companies. It’s also worth noting that Microsoft saw a 14% decline in revenue for the quarter ending September 30, 2009, compared to the year-ago quarter (and their earnings were in large part due to sales of the Xbox, a consumer product), while at the same time, consumer-oriented Apple saw a 24% increase in revenue from its year-ago quarter.

Your pool of little customers is also a pool of potential future best-customers. And you can’t always guess which ones. So a wise strategy might be to allocate part of your retention and growth campaign to providing loyalty incentives to smaller customers, and educating them about how your higher-end services or products might benefit them. Those little customers who have the means or opportunity to move on to the next level might very well appreciate your efforts, and stay with you, rather than defecting to a competitor.

Optimizing Sales vs. Optimizing Customers

One last thought about retail hits and high-value customers. McPhee’s Theory of Exposure, which is cited by Anita Elberse in her Harvard Business Review article “Should You Invest in the Long Tail?”, states that the popularity of music, film, TV or books is largely driven by “marginal audience participants” — the casual, or light, consumer. Casual consumers gravitate to already popular products because they have limited exposure to alternatives, and hence limited knowledge of them. Consumers of more obscure products, on the other hand, tend to be heavy (and knowledgable) consumers: voracious readers, dedicated music or film buffs, or enthusiasts of specific genres, like science-fiction or horror.

McPhee’s research was done in 1963, using subjects who had a fairly small range of choices, compared to internet scale. Elberse found, however, that the phenomena McPhee described still held for the internet merchants that she studied. She uses this observation (along with McPhee’s companion theory of Double Jeopardy) to argue that retailers should not substantially alter their traditional hits-based strategies. There is an alternative interpretation:

If your business follows McPhee’s theory, then hit products disproportionately attract low-value (low-volume) customers, and vice-versa.

So an overly hits-oriented strategy will skew you towards a base of low-value customers. Indeed, Seth Godin argues that iTunes and Amazon, who are in a better position to implement a more tail-oriented strategy, are thriving at the expense of physical stores exactly because they have been able to steal the quality (high-volume) customers away.

The moral is that both sales and customer value live in a lognormal world, where blockbuster products are marketed to a large cloud of low revenue customers, and high revenue best-customers are supported by large catalogues of low volume products. Fail to serve one side of this relationship, and you risk losing the other side.

Related posts:

1. Statistics to English Translation, Part 2b: Calculating Significance
2. A Demonstration of Data Mining
3. Good Graphs: Graphical Perception and Data Visualization

As in the last article, we will concentrate on situations where we want to test the difference of means. You should read that previous article first, so you are familiar with the terminology that we use in this one.

A pdf version of this current article can be found here.

How is Significance Determined?

Generally speaking, we calculate significance by computing a test statistic from the data. If we assume a specific null hypothesis, then we know that this test statistic will be distributed in a certain way. We can then compute how likely it is to observe our value of the test statistic, if we assume that the null hypothesis is true.

We’ll explain the use of a test statistic with our Sneetch example from the last installment.

The t-test for Difference of Means

Suppose that the test scores for both Star-Bellies and Plain-Bellies are normally distributed, with the means and standard deviations as given in the table below.

	(number of subjects)	(mean score)	(standard error)
Star-Bellies	50	78	7
Plain-Bellies	40	74	8

Remember from the previous installment that we can estimate the true population means and as normally distributed around the empirical population means and respectively, with variances
and
. This is shown in Figure 1. Informally speaking, there is no significant difference in the two populations if the shaded overlap area in Figure 1 is large.

Figure 1: The estimates of the means for two populations

Calculating this area is somewhat involved. Instead, we calculate the t-statistic:

(1)

where is called the pooled variance of the two populations.

(2)

For our Sneetch example, , and , or the negative of that, depending on which group is Group 1. There are
degrees of freedom.

If the null hypothesis is true, and the two populations are identical, then is distributed according to Student’s distribution with
degrees of freedom. Student’s distribution is sort of a “stretched out” bell curve; as the degrees of freedom increase (
), Student’s distribution approaches the standard normal distribution, ¹.

In other words, if the null hypothesis is true, should be near zero. The probability of seeing a of a certain magnitude or greater under the null hypothesis is given by the area under the tails of Student’s distribution:

Figure 2: The area under the tails for a given

This area is . For the Sneetch example, .

The further out on the tails is, the stronger the evidence that you should reject the null hypothesis. If you know for some reason that the mean of one population will be greater than or equal to the other, than you can use the one-tailed test:

Figure 3: The one-tailed test for a given

This test halves the p-value as compared to the two-tailed test, making a given value twice as significant. When in doubt about which to use, the two-tailed test is more conservative against false positives².

In discussions of t-tests, you will often see statements of the form:

The t-test meets the hypothesis that two means are equal if

for a two-tailed test, or

for a (right-sided) one-tailed test.

The quantities on the right hand side of the two equations above are called the critical values for a given significance level (usually,
) and degrees of freedom. The critical values are the values for which the area of the right hand tail is equal to .

Figure 4: Critical value for a one-tailed test. Reject the null hypothesis if

For a two-tailed test, you must halve the area under a single tail.

Figure 5: Critical value for a two-tailed test. Reject the null hypothesis if

This convention dates back to the time when computational resources were scarce, and researchers had to use pre-computed tables of critical values, rather than calculating directly. Today, general statistical packages such as R or Matlab can compute the CDFs of any number of standard distributions; once you can compute the CDF, directly computing (the area under the tails) is straightforward. Despite this, many tutorials of the t-test (and of the F-test, and other significance tests) still adhere to the convention of comparing test statistics to critical values. This tends to needlessly ritualize the whole process, and make it seem more complicated and mysterious than it actually is, at least in my opinion.

David Freedman was very much against the continued practice of using critical values, rather than reporting the actual p-value. The last chapter of Freedman, Pisani and Purves [FPP07] is worth reading for its discussion of this, and other potential pitfalls of significance tests.

Some standard packages for evaluating t-tests, F-tests, or the ANOVA also present analysis results in terms of critical values. Most of them do usually print the actual p value as well, along with the value of the test statistic and the degrees of freedom. Most researchers rightfully report the test statistics along with the actual significance levels: “we conclude that there is a significant difference in mathematical performance (t(88) = 2.499, p = 0.014)… .” Here, 88 gives the degrees of freedom, is the value of the t-statistic, and is of course the p-value.

Similar comments apply to the F-test, discussed in more detail below.

Assumptions

Strictly speaking, the t-test is only valid for normally distributed data where both populations have equal variance. However, the test is fairly robust to non-normal data [Box53]. You can verify that the sample variances are “equal enough” – that is, they could plausibly both be sampled observations from populations with the same variance, by using the F-test. The F-statistic

is distributed according to the F distribution with
degrees of freedom

Figure 6: The F distribution

In practice, the larger variance is usually put in the numerator, so . The test should still be two-tailed, so you should double the area under the right-hand tail³. In this situation, you want to check if you ƒshould accept the null hypothesis (that
) at a given significance level. If so, then you can go ahead and apply the t-test.

There is a variation of the t-tests for distributions of unequal variance, called Welch’s t-test [Wikc]. In this case, you are only checking if the means are equal, not that the distributions are the same.

The F-test for Analysis of Variance (ANOVA)

ANOVA is an extension of the difference of means test above to the casae of more than two populations. The null hypothesis in this case is that all the sample means are equal – or more strictly, that all the treatment groups are drawn from the same population.

The simplest version of the ANOVA is the one-way ANOVA, where there are treatment groups (populations) with subjects (or repetitions, or replications) each, for a total of subjects. Each population corresponds to a different single factor (a treatment or a condition: for example, a type of medicine, or a Star-Bellied Sneetch vs. a Plain-Bellied Sneetch vs. a Grinch). Two- or three- way ANOVAs correspond to varying two or three different factors combinatorially. For example, we could do a two-way ANOVA of Sneetch math performance by considering both the belly type and the gender of the Sneetchs.

Figure 7: Table for a Two-way ANOVA of Sneetch math performance

We will only discuss one-way ANOVA in this article, since that covers all the relevant ideas about calculating significance.

For a one-way ANOVA, we have the population means and variances . We can also calculate the overall mean , over the entire aggregate population.

The between-groups mean sum of squares, which is an estimate of the between-groups variance, is given by

(3)

is sometimes designated It is a measure of how the population means vary with respect to the grand mean.

The within-group mean sum of squares is an estimate of the within-group variance:

(4)

is sometimes designated . It is a measure of the “average population variance”.

Figure 8: Within-group and between-group variance

If the null hypothesis is true, then

is distributed according to the F distribution wiht
degrees of freedom.

Figure 9: p-value for the one-tailed F-test

That is, under the null hypothesis, the within-group and between-group variances should be about equal:
. If , then some of the treatment groups overlap other groups substantially, so practically speaking, one might as well accept the null hypothesis. Hence, a one-sided F test is good enough. As with the t-test, research papers usually give the value of the F statistic, the degrees of freedom, and the p-value: “
”. In this example, the test statistic value is 6.6, and it was evaluated against the F distribution with (2, 864) degrees of freedom, which means that
. The p-value is 0.0014.

Assumptions

Like the t-test, ANOVA assumes that the data is normally distributed with equal variances. According to Box [Box53], ANOVA is fairly robust to unequal variances when the population sizes are about the same, but you might want to check anyway. If all the populations are the same size (all the are the same), the easiest way to check for equality of variances is an F-test of the statistic
with degrees of freedom[Sac84]. In other cases, you can use Bartlett’s Test [Wika] or Levene’s Test [Wikb]. Bartlett’s test uses a test statistic that is distributed as the distribution, and Levene’s test uses one that is distributed as the F distribution. Levene’s test does not assume normally distributed data.

If the data are not normally distributed, or have unequal variance, often they can be transformed to a form that is closer to obeying the assumptions of ANOVA. The following table of transformations is based on [Sac84, p. 517], and other sources [Hor].

Figure 10: Table of Transformations

Jim Deacon from the University of Edinburgh lists some suggestions as well [Dea]. He also reminds us that running ANOVA on the transformed data will identify significant differences in the transformed data. This is not the same as saying there are significant differences in the original data!

Once the Null Hypothesis is Rejected

If you are able to reject the ANOVA null hypothesis, you will usually want to know which population means are significantly different from the rest. Often, in fact, you are primarily interested in which population had the highest mean. For example, if you are comparing the efficacy of a new medicine A against existing medicines B and C, you are probably not too concerned about whether B and C perform significantly differently from each other, only about whether A is significantly better than both.

If all you care about is whether the highest mean is significantly higher than the others, you can simply test where the statistic

falls on the Student-t distribution with degrees of freedom. Here, is the within-group variance, as calculated in Equation 4, and are the highest and second highest population means, is the total number of samples (
), and is the number of treatment groups.

This test is usually written

where
is the (two-sided) critical value for significance level and is the number of degrees of freedom to use. This quantity is called the least significant difference (LSD) between the highest and second highest means, and the test is usually called the LSD test.

If you want to test all the population differences for significance, (or test the highest value against all of the others explicitly) then you need to take some care with the LSD test. Remember that a significance level of means that with probability you will make a false positive error. To test all possible population differences is = ( choose ) comparisons, or comparisons, if you sort all the means in descending order and compare adjacent ones. Testing the highest mean against all the lower values is also comparisons. This means you have a
probability of making a false positive error. So if you want the overall significance level to be , each individual comparison should use a stricter significance threshold
.

A preferred way to compare multiple means for significance (once the ANOVA null hypothesis has been rejected) is to use a multiple range test [Dea] or Tukey’s method [oST06], rather than the LSD test. Tukey’s method tests all pairwise comparison simultaneously, and the multiple range test starts with the broadest range (the highest and the lowest means), and works its way in until significance is lost.

Conclusion

We’ve skimmed over many complications in this discussion. Hopefully, though, what we have gone over is enough to demystify much of the statistical discussion in research papers. Perhaps, it will demystify the output of standard ANOVA and t-test packages for you, as well.

Chong-ho Yu’s site [hY] gives a brief discussion of some of the issues that I’ve skimmed over. It also lists a few common non-parametric tests. These are tests that do not make assumptions about how the data is distributed, and so they may be more appropriate for data that is very non-normal, or for discrete data. They tend to have less power than parametric tests (that is, they have a lower true positive rate); so if the data is at all normal-like, parametric tests are preferred.

Significance tests are used in other applications beyond testing the difference in means or variances. They are used for testing whether events follow an expected distribution, for testing if there is a correlation between two variables, and for evaluating the coefficients of a regression analysis. We hope to cover some of these applications in future installments of this series.

Bibliography

Box53

G.E.P. Box, Non-normality and tests on variances, Biometrika 40 (1953), no. 3/4, 318-335.

Dea

Jim Deacon, A multiple range test for comparing means in an analysis of variance, http://www.biology.ed.ac.uk/research/groups/jdeacon/statistics/tress7.html.

FPP07

David Freedman, Robert Pisani, and Roger Purves, Statistics, 4th ed., W. W. Norton & Company, New York, 2007.

Hor

Rich Horsley, Transformations, http://www.ndsu.nodak.edu/ndsu/horsley/Transfrm.pdf, Class notes, Plant Sciences 724, North Dakota State University.

hY

Chong ho Yu, Parametric tests, http://www.creative-wisdom.com/teaching/WBI/parametric_test.shtml.

oST06

National Institute of Standards and Technology, Tukey’s method, NIST/SEMATECH e-Handbook of Statistical Methods, 2006, http://itl.nist.gov/div898/handbook/prc/section4/prc471.htm.

Sac84

Lothar Sachs, Applied statistics: A handbook of techniques, 2nd ed., Springer-Verlag, New York, 1984.

Wika

Wikipedia, Bartlett’s test, http://en.wikipedia.org/wiki/Bartlett's_test.

Wikb

—–, Levene’s test, http://en.wikipedia.org/wiki/Levene's_test.

Wikc

—–, Welch’s t test, http://en.wikipedia.org/wiki/Welch's_t_test.

Footnotes

…¹

Remember from the last installment that when you are estimating the mean of a distribution with unknown mean and unknown variance , the 95% confidence interval around your estimate is
. Intuitively speaking, Student’s distribution is what you get if you calculate confidence intervals using the estimated variance instead of the true but unknown variance . The distribution is stretched out compared to the normal distribution to reflect this increased uncertainty.

… positives²

In his textbook Statistics, Freedman tells an anecdote about a study that was published in the Journal of the AMA, claiming to demonstrate that cholesterol causes heart attacks. The treatment group that took a cholesterol reducing drug had “significantly fewer” heart attacks than the control group (
). A closer reading revealed that the researchers used a one-tailed test, which is equivalent to assuming that the treatment group was going to have fewer heart attacks. What if the drug had increased the risk of heart attack? The proper two-tailed significance of their results would have been
, which is higher than JAMA‘s strict significance threshold of 0.05. [FPP07, p. 550]

… tail³

The area to the right of with degrees of freedom is equal to the area to the left of , with degrees of freedom.

Related posts:

1. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
2. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
3. Living in A Lognormal World

As always, a pdf version of this article is available as well.

Does too much salt cause high blood pressure, or doesn’t it? That debate has raged for decades, with a slew of studies finding “yes” and a slew of others finding “no.” Two new studies out today in the journal Hypertension tip the scales in favor of reducing sodium – particularly for those 1 in 4 Americans who have high blood pressure. One study found that reducing salt intake from 9,700 milligrams a day to 6,500 milligrams decreased blood pressure significantly in blacks, Asians, and whites who had untreated mild hypertension. Another study found that switching to a lower-salt diet helped lower blood pressure in folks with treatment-resistant hypertension.
- “10 salt shockers that could make hypertension worse,” U.S. News & World Report [Kot09]

“Great!” you think. “Who needs to spend money on high-blood pressure meds? I can just cut down my salt!” Well, maybe so, maybe not. To come to that conclusion, you need more information than you were given in that paragraph. What was the “significant” decrease in blood pressure? What was the “before” and the “after”? Does “significant” mean important, or useful? And why has there been so much controversy over this?

Let’s discuss the important points with an example.

Suppose that we wanted to test for a difference in intelligence between two groups, say Star-Bellied Sneetches and Plain-Bellied Sneetches². We take a group of 50 Star-Bellies and a group of 40 Plain-Bellies, and give them both a series of tests designed to measure their mathematical, linguistic, and problem-solving abilities. After evaluating the data, we conclude that there is “a significant difference in mathematical performance (t(88) = 2.499, p = 0.014) between the two groups”. The mean mathematics score of the Star-Bellies is 78, with a standard deviation of 7, and the mean mathematics score of the Plain-Bellies is 74, with a standard deviation of 8, for a difference of 4 points³.

Should we interpret this result to mean that Star-Bellied Sneetches are better than Plain-Bellied ones at math? It depends.

How Hypothesis Tests Work

The Sneetch example above and the blood-pressure study cited earlier are both examples of hypothesis tests. In hypothesis testing, researchers set their proposed hypothesis (that there is an effect or a relationship) against the null hypothesis that there is no effect or relationship. In this article, we consider proposed relationships of the form

The mean value of X measured for group A is different from the mean value of X measured for group B.

In this case, the null hypothesis is

The mean value of X is the same for groups A and B, and any difference observed in the data is only by observational chance.

In fact, we are actually testing the stricter null hypothesis:

The distribution of X is the same for groups A and B, and any difference observed is only by observational chance.

A and B are sometimes called treatment groups; this terminology comes from the original applications of hypothesis testing procedures, in agriculture and medicine. In the blood pressure study above, the treatment is daily salt intake. One group ingests about 9,700 milligrams of sodium a day, the other group about 6,500 milligrams a day. The question of interest is: does the difference in sodium intake make a difference in the average blood pressure of the two groups? The null hypothesis is “No.”

Significance

We call an observed difference significant – meaning that a difference as large as we observed is probably not by chance – if the the value is “high enough.” In the Sneetch example, is the significance level of the result. To interpret the p-value, suppose the null hypothesis is true: there is truly no difference between Star-Bellied math scores and Plain-Bellied math scores. If this is so, then there is only a 0.014 (1.4%) chance that the difference in the average scores of the two groups will be 4 points or larger. In other words, if the null hypothesis is true, and we administer this same test to different groups of 50 Star-Bellies and 40 Plain-Bellies a hundred times, then the difference in scores will be 4 points or more only about once or twice.

We interpret the fact that we have seen a difference that should be rare to be evidence that the null hypothesis isn’t true. So we reject the null hypothesis and say that there is a “significant difference” in the performance of the two groups. Alternatively, we could say that Star-Bellied Sneetches performed “significantly better” than Plain-Bellied Sneetches on the math test.

Effect Size

Four points (or about a 5% difference) is the effect size of the comparison. The effect size represents what might be called the “practical significance” of the result. In general, the larger the effect size, the better. In this example, Star-Bellies might truly outperform Plain-Bellies by about four points on average, but if we were to examine the relationship between math scores and real-life math performance (say, how well college-attending Sneetches do in their math and science courses), we might discover that it takes a test score difference of ten points or more to reliably predict which Sneetches will do better. In that case, a four point average difference would not be a practical difference.

Evaluating a Result

When evaluating a result, you should look both for its significance and its effect size. In practice, researchers usually consider a finding to be significant if
. This is actually a pretty large ; it means even if the null hypothesis is true, you would still observe a difference as large as the one that you observed about five times out of every one hundred trials. In fact, Sachs noted that
used to be the commonly used threshold for significance ([Sac84, p. 114]).

Sometimes results are reported using an asterisk convention: (*) means
, (**) means
, and (***) means
. Hopefully, the actual significance level is reported (it isn’t always), as well as the actual effect size (it isn’t always).

The effect size in medical studies is often reported in the popular press with statements like “those who abstained from coffee had triple the risk of contracting colon cancer compared to those who drank three or more cups a day.” Does that mean that all confirmed Lapsang Souchong drinkers and the uncaffeinated should run out and learn to embrace Starbucks? Well, no. First of all, ask yourself: what is the baseline risk of colon cancer? If abstaining from coffee triples the risk from 0.01% to 0.03%, well, it probably isn’t worth worrying about. On the other hand, if the risk triples from 5% to 15%, perhaps that is a reason to take up espressos. You should also see who were the subjects of the study, and how similar they are to you. Suppose the study was done on Caucasian males in the U.S., ages 55-65, with no family history of colon cancer. If you are a young white American male, it’s possible that this study says something about your future health. If you are female or non-Caucasian or not living in the U.S., the finding may or may not be relevant to you. It depends on the mechanism that drives the relationship, and whether or not it applies to you as well as to the subjects of the study.

“Significant” is not the same as “Important”

With a large sample, even a small difference can be “statistically significant”… . This doesn’t necessarily make it important. Conversely, an important difference may not be statistically significant if the sample size is too small.
- Freedman, Pisani and Purves, Statistics [FPP07, p. 550]

The ability of a study to detect a significant difference depends almost entirely on its size. When a researcher designs a study, she has to decide how much risk of error – and what type of error – she is willing to tolerate.

How big a risk [of inventing a difference] between two indistinguishable treatments are we willing to put up with? This risk is known as the significance level . [Sac84, p. 214]

is the probability of rejecting a null hypothesis that should be accepted. This is a Type I error (a false positive). enters the design of the study as the threshold for p-values that the researcher will accept as significant.

How big a risk do we allow of missing a substantial difference between two treatments? … This risk is called . [Sac84, p. 214]

is the probability of accepting a null hypothesis that should have been rejected. This is a Type II error (a false negative). The quantity is known as the power of the test: the probability that the test will correctly reject the null hypothesis when the alternative hypothesis is true.

How small a difference should still be recognized as significant? This difference is called . [Sac84, p. 214]

is the minimum effect size that we are willing to consider “practically significant.”

It is important to consider all three of , , and when determining an appropriate sample size for a trial. The power of a test and the significance of a result both increase as the sample size increases. So if is not specified, any difference can appear significant, with a large enough , even if the difference is really by chance.

The Central Limit Theorem

To see why the above statement is true, we need a few more facts about estimating the mean. Suppose we have a random variable that is normally (or nearly normally) distributed, with a true mean and (unknown) variance . You want to estimate by drawing samples; the sample mean gives you an estimate of . According to the Central Limit Theorem, if you were to repeat this experiment over and over again, you would see that the estimated has a normal distribution, with mean and variance
. So is a good estimate of , one that improves with a larger sample size .

Another fact about normal distributions is that a little over 95% of the probability mass is within standard deviations of the mean. So, for a single experiment, we can reason that the true mean is in the interval
with 95% probability⁴.

Figure 1: Confidence bounds on the estimate of for different values of

So, as gets larger, we zoom in on ⁵.

Now, back to the problem of checking for the difference of means. We’ll take samples from population and from population . Let’s assume for now that the variances are equal.

Figure 2: Confidence bounds overlap; means may not be truly different

With 95% probability,
, and
. If
is small compared to
, then the two confidence intervals overlap substantially, and we cannot reject the null hypothesis that
.

If, on the other hand,
is wide compared to
:

Figure 3: Confidence bounds don’t overlap; means are significantly different

then the confidence intervals are well separated, and we can reject the null hypothesis.

So , the minimum significant distance – the “resolution” of the experiment – is about the distance when the two confidence intervals touch:
, if our desired significance level is 0.05.

Figure 4: Minimum significant distance for a given sample size

If is too large, the experiment may be unable to detect important differences because the confidence intervals overlap too soon. This means that the sample size was too small (the test didn’t have enough power), and the experiment should be repeated with a larger test population.

If is too small, then the experiment will potentially detect statistically significant differences that are, for all practical intents and purposes, meaningless. To go back to the Sneetch example, if the math exam has one hundred questions, then an effect size of two points would correspond to one group answering two additional questions correctly, on average. Practically speaking, that’s probably not a very big difference. But if we made the experiment big enough, about 250 Sneetches in each group, it would be a statistically significant difference, to the 0.05 level. In theory, we could even make a difference of less than one point statistically significant! That is why knowing the effect size of a significant result is important.

“Significant” is not the same as “True”

The power and significance level of a test play similar roles to the sensitivity and specificity of a diagnostic test. You’ll remember from Part 1 of this series⁶that sensitivity and specificity are properties of the test, not how the test performs in a given population. To know the practical accuracy of a screening test, you must know the underlying prevalence of the condition that it is screening for. If it is crucial that the screening not miss any positive cases, then the test will be designed to be highly sensitive, possibly at the cost of specificity. In that case, the test will tend to have a high false positive rate if the condition is relatively rare. And yet, this same screening test will have a lower overall false positive rate when used in a population where the condition is more prevalent.

The same is true for hypothesis tests. The probability that a statistically significant result is actually true depends on the underlying probability that results “of that type” tend to be true in the domain of study. It also depends on whether the researcher was trying to minimize the chance of a false positive error, or a false negative error.

You should also be careful interpreting the results of exploratory work, where the researchers have run a series of several different studies, but only highlight the “significant” ones. Running twenty experiments and having one of them return a significant result to the level is actually not significant at all.

John Ioannides discusses these points (and a few others) in his 2005 essay “Why Most Published Research Findings are False”[Ioa05]. The essay made a few waves at the time of its publication, and it is still available online. We recommend that you read it, along with the 2007 followup article by Moonesinghe, et.al [MKJ07]. Now that you’ve read the first two installments of the Statistics to English translation, both essays should be a breeze!

Some Points to Remember

• “Significant” is a statistical statement that an observed relationship is unlikely to be by chance. It is not an necessarily a statement about the magnitude or the importance (or the truth!) of the relationship.
• Knowing the effect size of a significant result will help you decide if the relationship is “practically significant.”
• With a large enough sample size, any difference in means can appear significant, even when it is by chance.

You now have a general idea what a “statistically significant result” is. The next installment will go into a little more technical detail of how significance is calculated. You should read that installment if you want to decipher statements in research papers like “
” — or if you are simply curious.

Bibliography

FPP07

David Freedman, Robert Pisani, and Roger Purves, Statistics, 4th ed., W. W. Norton & Company, New York, 2007.

Ioa05

John P. A. Ioannidis, Why most published research findings are false, PLoS Med 2 (2005), no. 8, e124, Available as http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124.

Kot09

Deborah Kotz, 10 salt shockers that could make hypertension worse, U.S. News & World Report (2009), Online as http://health.usnews.com/articles/health/heart/2009/07/20/10-salt-shockers-that-could-make-hypertension-worse.html.

MKJ07

Ramal Moonesinghe, Muin J Khoury, and A. Cecile J. W Janssens, Most published research findings are false — but a little replication goes a long way, PLoS Med 4 (2007), no. 2, e28, Available as http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0040028.

Sac84

Lothar Sachs, Applied statistics: A handbook of techniques, 2nd ed., Springer-Verlag, New York, 1984.

SS99

Murray R. Spiegel and Larry J. Stephens, Schaum’s outline of statistics, 4th ed., McGraw-Hill, New York, 1999.

Footnotes

… series¹

http://www.win-vector.com/blog/category/statistics-to-english-translation/

… Sneetches²

“The Sneetchs,” from The Sneetches and Other Stories by Dr. Seuss.
http://www.youtube.com/watch?v=Ln3V0HgW4eM
and http://www.youtube.com/watch?v=s0LgMpfLD1Y

… points³

This example is based on Exercise 10.17 in [SS99]; the original exercise did not, unfortunately, involve Sneetches.

… probability⁴

The correct way to state this is that for a given (unknown) , the estimate falls in the interval
just over 95% of the time. This gets awkward to reason about. Luckily, symmetry arguments let us center the appropriate confidence interval around instead.

…⁵

Of course, we don’t actually know , so we don’t know exactly how fast we zoom in. That doesn’t affect our argument, though, since only changes

… series⁶

http://www.win-vector.com/blog/2009/11/i-dont-think-that-means-what-you-think-it-means-statistics-to-english-translation-part-1-accuracy-measures/

Related posts:

1. Statistics to English Translation, Part 2b: Calculating Significance
2. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
3. R annoyances

The “Statistics to English Translation” series is a new set of articles that we will be posting from time to time, as an attempt to bridge the language gaps. Our goal is to increase statistical literacy: we hope that you will find it easier to read and understand the statistical results in research papers, even if you can’t replicate the analyses. We also hope that you will be able to read popular media accounts of statistical and scientific results more critically, and to recognize common misunderstandings when they occur.

The first installment discusses some different accuracy measures that are commonly used in various research communities, and how they are related to each other. There is also a more legible PDF version of the article here.

The Basics

In informal language and in popular press articles, “accuracy” is often discussed as if it were a one-dimensional property of a diagnostic test or a classifier.

In general though, a single number is not enough. A test or classifier should detect what’s interesting, and ignore what’s not. How well it accomplishes these two tasks is related to the two kinds of mistakes that a test or classifier can make: false negatives, and false positives.

For a classification task, positive means that an instance is labeled as belonging to the class of interest: we may want to automatically gather all news articles about Microsoft out of a news feed, or identify fraudulent credit card transactions. For a screening test, positive means that the test detects whatever it was designed to look for: an HIV test detects the presence of human immunodeficiency virus, for example, while an allergy test detects the presence of an allergic reaction. A negative is obviously the opposite of a positive.

A false positive is concluding that something is positive when it is not. False positives are sometimes called Type I errors. A false negative is concluding that something is negative when it is not. False negatives are sometimes called Type II errors. The terms “Type I error” and “Type II error” are not terribly mnemonic, but they are commonly used, and therefore worth knowing.

For binary classification or binary test procedures, the False Positive Rate, , is the fraction of negative instances that are erroneously misclassified as positive.

(1)

Likewise, the False Negative Rate, , is the fraction of positive instances that are erroneously misclassified as negative.

(2)

The True Positive Rate, , is the fraction of positive instances that are correctly identified as such. It follows from the Definition 2 above that
.

The True Negative Rate, , is the fraction of negative instances that are correctly identified as such. It follows from the Definition 1 above that
.

Sensitivity and Specificity

The terms sensitivity and specificity generally refer to diagnostic or screening procedures, such as an HIV or allergy tests. The sensitivity of a test is its true positive rate; the specificity is its true negative rate, although it can be more intuitive to think of specificity as the complement of the false positive rate: Specificity =
.

The Wikipedia entry on Sensitivity and Specificity [Wik] uses a nice example to illustrate the difference: think of a drug-sniffing dog as a screening test for illicit drugs. If the dog’s nose is highly sensitive to the smell of drugs, then it will detect all the hidden packets of drugs; if it is less sensitive, then it will fail to detect some of the packets. At the same time, the dog should react specifically to drugs, and not, say, jambalaya or doggie biscuits. If the dog is highly specific in its reactions, it will only react to drugs; if it is less specific, then it will react to the occasional care package of yummy home cooking from Mom.

Screening tests may trade off specificity for sensitivity (and vice-versa). To go back to our drug-sniffing example, we might treat every suitcase and bag that comes through the airport as if it contained drugs; this procedure is perfectly sensitive (it will detect every packet of drugs, for sure), but not specific at all. Or, we might assume that no one is carrying drugs. This is perfectly specific (we will never make a false accusation), but not sensitive at all.

A more realistic example, inspired by a discussion of mandatory AIDS testing by Joshua Rosenau [Ros06], is the use of the ELISA screening test to detect HIV-infected blood donations. The ELISA test is designed to be very sensitive: it detects 99.7% of the cases of HIV-infection, which gives a false negative rate of
. On the other hand, it is not very specific: it has a 1.9% false positive rate¹. If you assume that the incidence of HIV-positive individuals in the general population is about 448 out of every 100,000 people [Hig08], then a positive test result is correct only about 19% of the time: one case of true infection out of every five positives. This error rate may be appropriate for screening blood donations, since it is better to discard four perfectly good pints of blood, “just in case”, than to allow a pint of HIV-infected blood into the blood bank. But it is not appropriate to assume that all five of those poor blood donors are HIV-positive, without followup tests to increase the specificity of the screening procedure.

Sensitivity, Specificity, and Prevalence

The example above brings up an important point. Sensitivity and specificity are properties of the test itself, not how the test performs in a given population. The absolute accuracy (as the term is commonly understood) of a screening test will change, depending on the prevalence of the condition that the test is screening for.

Let’s imagine the ELISA test described above as an HIV-screening daemon, who uses two coins to generate uncertainty. When the daemon is shown a pint of infected blood, she flips an unfair quarter. If the quarter comes up heads (which it does 3 times out of every 1000 flips), then she lies and says the blood is uninfected, otherwise she tells the truth. When the daemon is shown a pint of uninfected blood, she flips a silver dollar that comes up heads about 2 times out of every 100 flips. If the silver dollar comes up heads, she lies and says the blood is infected, or else she tells the truth. The quarter and the silver dollar encode ELISA’s sensitivity and specificity, respectively.

Figure 1: The ELISA daemon screening an uninfected pint of blood

Suppose ELISA looks at the blood of 1000 people a day, drawn from the general population. We can expect that about 5 of them are truly infected. That means that ELISA flips her silver dollar 995 times; it will come up heads about 20 times. That’s about twenty false positives a day. She’ll flip the quarter about 5 times, and with high probability, won’t ever see a head. That’s near zero false negatives a day. In total, ELISA will read positive for about 25 pints of blood every day, and she will be wrong for 80% of those cases.

But suppose ELISA looks at the blood of 1000 people from a high-risk population, where one out of four people are infected. Then ELISA flips her silver dollar about 750 times, and it will come up heads about 15 times: 15 false positives. She’ll also flip the quarter 250 times; the coin just might come up heads one time. Let’s say it does. Then ELISA will read positive for 249+15 = 264 pints of blood, and she’ll be wrong for only about 6% of those cases – plus that case of infection that she missed.

Same test, same sensitivity and specificity, but different overall accuracy. The percentage of positives that are actually true positives in a given population is called the positive predictive value ( ) of the test within that population; it is the probability for that population that a positive test result correctly predicts a positive instance.

(3)

where is the probability of a positive instance, or in other words the prevalence of the condition in the population. is the probability of a negative instance, and of course
.²

Likelihood Ratios

Likelihood ratios are another measure of diagnostic test accuracy. The positive likelihood ratio is the true positive rate over the false positive rate:
. For our example ELISA test, the positive likelihood ratio is 0.997/0.019 = 52.47. The negative likelihood ratio is the false negative rate over the true negative rate,
. For our ELISA example, the negative likelihood ratio is 0.003/.981 = 0.003058.

Likelihood ratios are a property of the screening test, independent of the prevalence of the condition in the population. If you know the odds of infection for the population of interest,
, then you can calculate the posterior odds of infection for someone who has tested positive:

and the posterior odds of infection for someone who has tested negative:

It’s been argued that likelihood ratios make it easier for non-statistically-minded practitioners to interpret the results of a test than sensitivity and specificity do [JGS94]. It’s also been argued the other way [PSBtR05]. Which framework makes more sense depends on if you prefer thinking in odds or probabilities. In either case you should be leery of “guidelines” of the sort: “ indicates large and often conclusive increase in the likelihood of the disease.” There is certainly a large increase in the posterior likelihood of infection if , but as the ELISA coin-flipping example should have made clear, this posterior likelihood can still be quite small, if the disease is sufficiently rare.

I occasionally see something called the diagnostic odds ratio. It was developed as “a single indicator of test performance,” and I’ve seen it described as “the odds of the true positive rate divided by the odds of the false positive rate” [HC07]. I could give you the actual formula here, but frankly – it makes no sense. The whole point of having two measures for accuracy is that one is not enough, and if you absolutely must have one number, you are better off using something like the measure that we describe in the next section.

Precision and Recall

Precision and recall are similar (but not identical) to sensitivity and specificity. The measures are popular in the information retrieval and machine learning communities.

Recall is the same as sensitivity, or the true positive rate: the number of true examples correctly classified as such. Precision is defined as the fraction of instances classified as positive that really are positive:

   precision

This is not the same as specificity; it is the same as the positive predictive value, and is a joint property of the classifier and the population that it was evaluated on.

Information retrieval research concerns itself with efficient discovery of relevant documents from document collections, and that domain motivates the definitions of precision and recall. A library patron queries the library catalog for books on a given topic; the catalog’s search engine should return all of the books relevant to her query, and only those books. Recall is a measure of how well the search delivers “all of the relevant books”, and precision is a measure of how well it delivers “only the relevant books”. If recall is poor, then the patron will miss finding many relevant books; if precision is poor, then she will be inundated with a bunch of book suggestions that have nothing to do with her search.

As with diagnostic procedures, classifiers may trade precision for recall, and vice-versa. Suppose our library patron is looking for novels about vampires. She could request all novels with the word “vampire” in the title. This search would have almost perfect precision, since presumably a novel with the word “vampire” in the title is going to be about vampires. It would not have perfect recall, since many novels about vampires – like Dracula, or the books from the Twilight series – don’t announce themselves quite that blatantly. Now suppose she is only interested in novels from Anne Rice’s Vampire Chronicles series. She could request all novels authored by Anne Rice. This search would have perfect recall, but not perfect precision, since Ms. Rice did in fact write several novels that are not about vampires.

These examples show that neither high precision nor high recall guarantee a useful classifier. It is the tension between achieving high precision and high recall that leads to good classifiers.

As we discussed above, the primary difference between precision and specificity is that precision is a property of the algorithm and the population. One could argue that precision is a more appropriate measure than specificity for many classification and machine learning tasks, especially those related to text or natural language. The fundamental assumption, after all, is that such algorithms are trained on data that is representative of the population that the classifier will be deployed in.

If you insist: Single Score Measures

There is another measure called , the harmonic mean of precision and recall:

is near one when both precision and recall are high, and near zero when they are both low. It is a convenient single score to characterize overall accuracy, especially for comparing the performance of different classifiers.

Using to compare classifiers assumes that precision and recall are equally important for the application. If one criterion is more important than the other, then one can also use the weighted geometric mean:

precision   recall   precision + recall

describes how much more important recall is than precision: use if recall is twice as important as precision, if precision is twice as important as recall.

It is still better to have separate target goals for precision and recall that a candidate classifier must meet. Still, and are found in the literature, so they are presented here.

ROC Curves

Not all diagnostic tests or classifiers return a simple “yes-or-no” answer. In fact, most probably don’t. Generally, a classification or diagnostic procedure will return a score along a continuum; ideally, the positive instances score towards one end of the scale, and the negative examples towards the other end. It is up to the scientist or the analyst to set a threshold on that score that separates what is considered a positive result from what is considered a negative result. The Receiver Operating Characteristic Curve, or ROC Curve, is a tool that helps set the best threshold.

Figure 2: Plot of score distributions for positive and negative instances (Class 10 is positive)

Suppose we are trying to classify a set of instances into one of two classes, positive and negative³. We’ve gathered a test set of representative samples, and we’ve developed a scoring procedure to try to separate them. Positives tend to score on the high end of the scale, negatives toward the low end. We want to pick a threshold value.

Figure 2 shows what happens when score the test set. We can see that the scores of the positive instances (class 10) are in a cluster centered just above 7, and the scores of the negatives (class 0) are in a cluster centered near 5. Still, there is an interval where the two clusters overlap substantially. If we pick a threshold to the right of that interval (say, ), almost everything that scores greater than will be truly positive (high precision/specificity), but we miss a lot of positives, too (low recall/sensitivity). If we pick a threshold to the left of that interval (say ), we will catch almost all the positives (high recall/sensitivity), but we will also pick up a lot of negatives (low specificity/precision). So we want the threshold to be somewhere in the overlap interval, but where?

Figure 3: ROC Curve corresponding to Figure 2. Selected thresholds are marked on the curve.

ROC curves plot the false positive rate on the x-axis and the true positive rate on the y-axis, as we vary the threshold. The point corresponds to rejecting everything; the point corresponds to accepting everything. The ideal point is : accept all positive instances and reject all negative instances. The line corresponds to random guessing: that is, a procedure that assigns each instance a score uniformly drawn from (in this example) the interval without even checking if the instance is positive or negative.

The ROC curve represents the tradeoff between true positives and false positives that we make as we increase the threshold from accepting everything to rejecting everything. Figure 3 gives the ROC curve for our example, with a few example thresholds marked on the curve.

The area between the ROC curve and the line can be considered a measure of accuracy; the smaller that area, the more the scoring procedure is like random guessing. The larger the area, the better separated the two classes are. We can use the curve to help us decide how to set a threshold that will give us the most acceptable tradeoff between true positives and false positives. For this example, we would probably want to select a threshold somewhere between and .

In Conclusion

Some points to remember:

• Classifier and diagnostic test performance are not one-dimensional.
• Different fields use different (but related) measures of accuracy.
• Classifier and diagnostic test performance depend on the relative cost of Type I and Type II errors, as well as on the proportion of positive and negative instances in the population of interest.

Bibliography

HC07

Childrens Mercy Hospitals and Clinics, Stats: Meta-analysis for a diagnostic test, http://www.childrens-mercy.org/stats/model/diagnostic.asp, 2007.

Hig08

Liz Highleyman, CDC updates estimates of HIV prevalence in the United States, http://www.hivandhepatitis.com/recent/2008/100708_a.html, 2008.

JGS94

R. Jaeschke, GH Guyatt, and DL Sackett, Users’ guides to the medical literature. III. How to use an article about a diagnostic test. B. What are the results and will they help me in caring for my patients? The evidence-based medicine working group, JAMA 271 (1994), no. 6, 703-707.

Org04

World Health Organization, HIV assays: Operational characteristics (phase 1); Report 15 antigen/antibody ELISAs, http://whqlibdoc.who.int/publications/2004/9241592370.pdf, 2004.

PSBtR05

MA Puhan, J. Steurer, LM Bachmann, and G. ter Riet, “A randomized trial of ways to describe test accuracy: the effect on physicians’ post-test probability estimates”, Annals of Internal Medicine 143 (2005), no. 3, 184-âÄì189.

Ros06

Joshua Rosenau, AIDS testing, http://scienceblogs.com/tfk/2006/08/aids_testing.php, 2006.

Wik

Wikipedia, Sensitivity and specificity, http://en.wikipedia.org/wiki/Sensitivity_and_specificity).

Appendix: Glossary of Accuracy Terms

Basic Terms

Accuracy

Type I error

False Positive: to conclude something is a positive instance when it is not.

Type II error

False Negative: to conclude something is a negative instance when it is not.

False Positive Rate ( )

The fraction of negative instances that are erroneously misclassified as positive.

False Negative Rate ( )

The fraction of positive instances that are erroneously misclassified as negative.

True Positive Rate ( )

The fraction of positive instances that are correctly identified as such.
.

True Negative Rate ( )

The fraction of negative instances that are correctly identified as such.
.

Prevalence

The proportion of positive instances in the population, or the probability that someone drawn from the population at random in a positive instance. is the probability of drawing a negative instance from the population at random. The odds of a positive is the ratio of to .

Accuracy Terms

Terms to describe the accuracy of diagnostic tests are conventionally given in terms of sensitivity and specificity. They have been rephrased here in terms of true positive rate, false positive rate, etc., for clarity.

Single score measure of accuracy. The harmonic mean of precision and recall.

is near 1 for high accuracy, near 0 for low accuracy. Also see Precision, Recall.

Single score measure of accuracy when recall is twice as important as precision. Also see , Precision, Recall.

Single score measure of accuracy when precision is twice as important as recall. Also see , Precision, Recall.

Likelihood-ratio (negative)

. In diagnostic screening, used for calculating the posterior odds of a true positive for a subject who has tested positive:

Likelihood-ratio (positive)

. In diagnostic screening, used for calculating the posterior odds of a positive for a subject who has tested negative:

Positive Predictive Value

Probability (with respect to a specific assumed prevalence rate) that a positive result from a diagnostic or screening procedure is a true positive. Same as precision.

Also see Precision

Precision

In information retrieval, the fraction of returned documents that are actually relevant to the query. In classification, the fraction of all instances classified as class A that are truly in class A. The same as Positive Predictive Value.

   precision

Also see Positive Predictive Value

Recall

In information retrieval, the fraction of relevant documents that are returned by the query. In classification, the fraction of all true instances of class A that are classified into class A. The same as sensitivity.

   recall

Also see Sensitivity

ROC Curve

Plot of true positive rate versus false positive rate for a diagnostic test or binary classifier, as the decision threshold is varied.

Sensitivity

The true positive rate of a diagnostic or screening procedure. Also see Recall.

Specificity

The true negative rate
(or the complement of the false positive rate) of a diagnostic or screening procedure.

Footnotes

… rate¹

The ELISA sensitivity and specificity numbers are from WHO’s report on the operational characteristics of HIV Assays [Org04, p. 18], using the lower bounds of the confidence interval. They are slightly different from the numbers Rosenau uses

….²

The definition of is conventionally given in terms of sensitivity and specificity (similarly for the likelihood ratios discussed in the following section). The definitions in this article are given in terms of false positive rate, etc., since that is clearer for people reading outside their discipline.

… negative³

We are using a classifier in our example, but a diagnostic test would work the same way.

Related posts:

1. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
2. Statistics to English Translation, Part 2b: Calculating Significance
3. Living in A Lognormal World

This paper demonstrates and explains some of the basic techniques used in data mining. It also serves as an example of some of the kinds of analyses and projects Win Vector LLC engages in.

August 19, 2009

John Mount¹

A Demonstration of Data Mining

1  Introduction

A major industry in our time is the collection of large data sets in preparation for the magic of data mining [Loh09,HNP09]. There is extreme excitement about both the possible applications (identifying important customers, identifying medical risks, targeting advertising, designing auctions and so on) and the various methods for data mining and machine learning. To some extent these methods are classic statistics presented in a new bottle. Unfortunately, the concerns, background and language of the modern data-mining practitioner are different than that of the classic statistician- so some demonstration and translation is required. In this writeup we will show how much of the magic of current data mining and machine learning can be explained in terms of statistical regression techniques and show how the statistician’s view is useful in choosing techniques.

Too often data mining is used as a black-box. It is quite possible to clearly use statistics to understand the meaning and mechanisms of data mining.

2  The Example Problem

Throughout this writeup we will work on a single idealized example problem. For our problem we will assume we are working with a company that sells items and that this company has recorded its past sales visits. We assume they recorded how well the prospect matched the product offering (we will call this “match factor”), how much of a discount was offered to the prospect (we will call this “discount factor”) and if the prospect became a customer or not (this is our determination of positive or negative outcome). The goal is to use this past record as “training data” and build a model to predict the odds of making a new sale as a function of the match factor and the discount factor. In a perfect world the historic data would look a lot like Figure 1. In Figure 1 each icon represents a past sales-visit, the red diamonds are non-sales and the green disks are successful sales. Each icon is positioned horizontally to correspond to the discount factor used and vertically to correspond to the degree of product match estimated during the prospective customer visit. This data is literally too good to be true in at least three qualities: the past data covers a large range of possibilities, every possible combination has already been tried in an orderly fashion and the good and bad events “are linearly separable.” The job of the modeler would then be to draw the separating line (shown in Figure 1) and label every situation above and to the right of the separating line as good (or positive) and every situation below and to the left as bad (or negative).

In reality past data is subject to what prospects were available (so you are unlikely to have good range and an orderly layout of past sales calls) and also heavily affected by past policy. An example policy might be that potential customers with good product match factor may never have been offered a significant discount in the past; so we would have no data from that situation. Finally each outcome is a unique event that depends on a lot more than the two quantities we are recording- so it is too much to hope that the good prospects are simply separable from the bad ones.

Figure 1 is a mere cartoon or caricature of the modeling process, but it represents the initial intuition behind data mining. Again: the flaws in Figure 1 represent the implicit hopes of the data miner. The data miner wishes that the past experiments are laid out in an orderly manner, data covers most of the combinations of possibilities and there is a perfect and simple concept ready to be learned.

Frankly, an experienced data miner would feel incredibly fortunate if the past data looked anything like what is shown in Figure 2.

The green disks (representing good past prospects) and the red diamonds (representing bad past prospects) are intermingled (which is bad). There is some evidence that past policy was to lower the discount offered as the match factor increased (as seen in the diagonal spread of the green disks). Finally we see the red diamonds are also distributed differently than the green disks. This is both good and bad. The good is that the center of mass of the red diamonds differs from the center of mass of the green disks. The bad is that the density of red diamonds does not fall any faster as it passes into the green disks than it falls in any other direction. This indicates there is something important and different (and not measured in our two variables) about at least some of the bad prospects. It is the data miner’s job be aware and to press on.

2.1  The Trendy Now

In truth data miners often rush where classical statisticians fear to tread. Right now the temptation is to immediately select from any number of “red hot” techniques, methods or software packages. My short list of super-star method buzzwords includes:

• Boosting[Sch01,Bre00,FISS03]
• Latent Dirichlet Allocation[BNJ03]
• Linear Regression[FPP07,Agr02]
• Linear Discriminant Analysis[Fis36]
• Logistic Regression[Agr02,KM03]
• Kernel Methods[CST00,STC04]
• Maximum Entropy[KM03,Gru05,SC89,DS06]
• Naive Bayes[Lew98]
• Perceptrons[BRS08,DKM05]
• Quantile Regression[Koe05]
• Ridge Regression[BF97]
• Support Vector Machines[CST00]

Based on some of the above referenced writing and analysis I would first pick “logistic regression” as I am confident that, when used properly, it is just about as powerful as any of the modern data mining techniques (despite its somewhat less than trendy status). Using logistic regression I immediately get just about as close to a separating line as this data set will support: Figure 3.

The separating line actually encodes a simple rule of the form: “if 2.2*DiscountFactor + 3.1*MatchFactor ≥ 1 then we have a good chance of a sale.” This is classic black-box data mining magic. The purpose of this writeup is to look deeper how to actually derive and understand something like this.

3  Explanation

What is really going on? Why is our magic formula at all sensible advice, why did this work at all and what motivates the analysis? It turns out regression (be it linear regression or logistic regression) works in this case because it somewhat imitates the methodology of linear discriminant analysis (described in: [Fis36]). In fact in many cases it would be a better idea to perform a linear discriminant analysis or perform an analysis of variance than to immediately appeal to a complicated method. I will first step through the process of linear discriminant analysis and then relate it to our logistic regression. Stepping through understandable stages lets us see where we were lucky in modeling and what limits and opportunities for improvement we have.

Our data initially looks very messy (the good and bad group are fairly mixed together). But if we examine out data in separate groups we can see we are actually incredibly lucky in that the data is easy to describe. As we can see in Figure 4: the data, when separated by outcome (plotting only all of the good green disks or only all of the bad red diamonds), is grouped in simple blobs without bends, intrusions or other odd (and more work to model) configurations.

We can plot the idealizations of these data distributions (or densities) as “contour maps” (as if we are looking down on the elevations of a mountain on a map) which gives us Figure 5.

3.1  Full Bayes Model

From Figure 5 we can see while our data is not separable there are significant differences between the groups. The difference in the groups is more obvious if we plot the difference of the densities on the same graph as in Figure 6. Here we are visualizing the distribution of positive examples as a connected pair of peaks (colored green) and the distribution of negative examples a deep valley (colored red) located just below and to the left of the peaks.

This difference graph is demonstrating how both of the densities or distributions (positive and negative) reach into different regions of the plane. The white areas are where the difference in densities is very small which includes the areas in the corners (where there is little of either distribution) and the area between the blobs (where there is a lot of mass from both distributions competing). This view is a bit closer to what a statistician wants to see- how the distributions of successes and failures different (this is a step to take before even guessing at or looking for causes and explanations).

Figure 6 is already an actionable model- we can predict the odds a new prospect will buy or not at a given discount by looking where they fall on Figure 6 and checking if they fall in a region on strong red or strong green color. We can also recommend a discount for a given potential customer by drawing a line at the height determined by their degree of match and tracing from left to right until we first hit a strong green region. We could hand out a simplified Figure 7 as a sales rulebook.

This model is a full Bayes model (but not a Naive Bayes model, which is oddly more famous and which we will cover later). The steps we took were: first we summarized or idealized our known data into two Gaussian blobs (as depicted in Figure 5). Once we had estimated the centers, widths and orientations of these blobs we could then: for any new point say how likely the point is under the modeled distribution of sales and how likely the point is under the modeled distribution of non-sales. Mathematically we claim we can estimate P(x,y |sale)² and P(x,y | non-sale) (where x is our discount factor and y is our matching factor).³ Neither of these are what we are actually interested in (we want: P(sale | x,y)⁴). We can, however, use these values to calculate what we want to know. Bayes’ law is a law of probability that says if we know P(sale | x,y), P(non-sale | x,y), P(sale) and P(non-sale)⁵ then:

Figure 7 depicts a central hourglass shaped region (colored green) that represents the region of x, y values where P(sale |x,y) is estimated to be at least 0.5 and the remaining (darker red region) are the situations predicted to be less favorable. Here we are using priors of P(sale) = P(non-sale) = 0.5, for different priors and thresholds we would get different graphs.

Even at this early stage in the analysis we have already accidentally introduced what we call “an inductive bias.” By modeling both distributions as Gaussians we have guaranteed that our acceptance region will be an hourglass figure (as we saw in Figure 7). One undesirable consequence of the modeling technique is the prediction sales become unlikely when both match factor and discount factor are very large. This is somewhat a consequence of our modeling technique (though the fact that the negative data does not fall quickly as it passes into the green region also added to this). This un-realistic (or “not physically plausible”) prediction is called an artifact (of the technique and of the data) and it is the statistician’s job to see this, confirm they don’t want it and eliminate it (by deliberately introducing a “useful modeling bias”).

3.2  Linear Discriminant

To get around the bad predictions of our model in the upper-right quadrant we “apply domain knowledge” and introduce a useful modeling bias as follows. Let us insist that our model be monotone: that if moving some direction is good than moving further in the same direction is better. In fact let’s insist that our model be a half-plane (instead of two parabolas). We want a nice straight separating cut, which brings us to linear discriminant analysis. We have enough information to apply Fisher linear discriminant technique and find a separator that maximizes the variance of data across categories while minimizing the variance of data within one category and within the other category. This is called the linear discriminant and it is shown in Figure 8.

The blue line is the linear discriminant (similar to the logistic regression line depicted earlier on the data-slide). Everything above or to the right of the blue line is considered good and everything below or to the left of the blue line is considered bad. Notice that this advice while not quite as accurate as the Bayes Model near the boundary between the two distributions is much more sensible about the upper right corner of the graph.

To evaluate a separator we collapse all variation parallel to the separating cut (as shown in Figure 9). We then see that each distribution becomes a small interval or streak. A separator is good if these resulting streaks are both short (the collapse packs the blobs) and the two centers of the streaks are far apart (and on opposite size of the separator). In Figure 9 the streaks are fairly short and despite some overlap we do have some usable separation between the two centers.

To make the above precise we switch to mathematical notation. For the i-th positive training example form the vector v_+,i and the matrix S_+,i where

where x_i and y_i are the known x and y coordinates for this particular past experience. Define v_−,i, S_−,i similarly for all negative examples. In this notation we have for a direction γ: the distance along the γ direction between the center of positive examples and center of negative examples is: γ^T ( ∑_i v_+,i / n₊ − ∑_i v_−,i / n₋) (where n₊ is the number of positive examples and n₋ is the number of negative examples). We would like this quantity to be large. The degree of spread or variance of the positive examples along the γ direction is γ^T (∑_i S_+,i / n₊) γ. The degree of spread or variance of the negative examples along the γ direction is γ^T (∑_i S_−,i / n₋) γ. We would like the last two quantities to be small. The linear discriminant is picked to maximize:

It is a fairly standard observation (involving the Rayleigh quotient) that this form is maximized when:

As we have said, the linear discriminant is very similar to what is returned by a regression or logistic regression. In fact in our diagrams the regression lines are almost identical to the linear discriminant. A large part of why regression can be usefully applied in classification comes from its close relationship to the linear discriminant.

3.3  Linear Regression

Linear regression is designed to model continuous functions subject to independent normal errors in observation. Linear regression is incredibly powerful at characterizing and elimination correlations between the input variables of a model. While function fitting is different than classification (our example problem) linear regression is so useful whenever there is any suspected correlation (which is almost always the case) that it is an appropriate tool. In our example in the positive examples (those that led to sales) there is clearly a historical dependence between the degree of estimated match and amount of discount offered. Likely this dependence is from past prospects being subject to a (rational) policy of “the worse the match the higher the offered discount” (instead of being arranged in a perfect grid-like experiment as in our first diagram: Figure 1). If this dependence is not dealt with we would under-estimate the value of discount because we would think that discounted customers are not signing up at a higher rate (when these prospects are in fact clearly motivated by discount, once you control for the fact that many of the deeply discounted prospects had a much worse degree of match than average).

For analysis of categorical data linear regression is closely linked to ANOVA (analysis of variance).[Agr02] Recall that variance was a major consideration with the linear discriminant analysis, so we should by now be on familiar ground.

In our notation the standard least-squares regression solution is:

where y_+,i = 1 for all i and y_−,i = −1 for all i.

If we have the same number of positive and negative examples (i.e. n₊ = n₋) then Equation 1 and Equation 2 are identical and we have β = γ. So in this special case the linear discriminant equals the least square linear regression solution. We can even ask how the solutions change if the relative proportions of positive and negative training data changes. The linear discriminant is carefully designed not to move, but the regression solution will tilt to be an angle that is more compatible with the larger of the example classes and shift to cut less into that class. The linear regression solution can be fixed (by re-weighting the data) to also be insensitive to the relative proportions of positive and negative examples but does not behave that way “fresh out of the box.”

3.4  Logistic Regression

While linear regression is designed to pick a function that minimizes the sum of square errors logistic regression is designed to pick a separator that maximizes something called the plausibility of the data. In our case since the data is so well behaved the logistic regression line is essentially the same as the linear regression line. It is in fact an important property of logistic regression that there is always a re-weighting (or choice of re-emphasis) of the data that causes some linear regression to pick the same separator as the logistic regression. Because linear and logistic regression are only identical in specific circumstances it is the job of the statistician to know which of the two is more appropriate for a given data set and given intended use of the resulting model.

4  Other Methods and Techniques

4.1  Kernelized Regression

One way to greatly expand the power of modeling methods is a trick called kernel methods. Roughly kernel methods are those methods that increase the power of machine learning by moving from a simple problem space (like ours in variables x and y) to a richer problem space that may be easier to work in. A lot of ink is spilled about how efficient the kernel methods are (they work in time proportional to the size of the simple space, not the complex one) but this is not their essential feature. The essential feature is the expanded explanation power and this is so important that even the trivial kernel methods (such as directly adjoining additional combinations of variables) pick up most of the power of the method. Kernel methods are also overly associated with Support Vector Machines- but are just as useful when added to Naive Bayes, linear regression or logistic regression.

For instance: Figure 10 shows a bow-tie like acceptance region found by using linear regression over the variables x, y, x², y² and x y (instead of just x and y). Note how this result is similar to the full Bayes model (but comes from a different feature set and fitting technique).

4.2  Naive Bayes Model

We briefly return to the Bayes model to discuss a more common alternative called “Naive Bayes.” A Naive Bayes model is like a full Bayes model except an additional modeling simplification is introduced in assuming that P(x,y|sale) = P(x|sale)P(y|sale) and P(x,y|non-sale) = P(x|non-sale)P(y|non-sale). That is we are assuming that the distributions of the x and y measurements are essentially independent (once we know which outcome happened). This assumption is the opposite of what we do with regression in that we ignore dependencies in the data (instead of modeling and eliminating the dependencies). However, Naive Bayes methods are quite powerful and very appropriate in sparse-data situations (such as text classification). The “naive” assumption that the input variables are independent greatly reduces the amount of data that needs to be tracked (it is much less work to track values of variables instead of simultaneous values of pairs of variables). The curved separator from this Naive Bayes model is illustrated in Figure 11.

The Naive Bayes version of the advice or policy chart is always going to be an axis-aligned parabola as in Figure 12. Notice how both the linear discriminant and the Naive Bayes model make mistakes (places some colors on the wrong side of the curve)- but they are simple, reliable models that have the desirable property of having connected prediction regions.

4.3  More Exotic Methods

Many of the hot buzzword machine learning and data mining methods we listed earlier are essentially different techniques of fitting a linear separator over data. These methods seem very different but they all form a family once you realize many of the details of the methods are determined by:

• Choice of Loss Function

  This is what notion of “goodness of fit” is being used. It can be normalized mean-variance (linear discriminants), un-normalized variance (linear regression), plausibility (logistic regression), L1 distance (support vector machines, quantile regression), entropy (maximum entropy), probability mass and so on.
• Choice of Optimization Technique

  For a given loss function we can optimize in many ways (though most authors make the mistake of binding their current favorite optimization method deep into their specification of technique): EM, steepest descent, conjugate gradient, quasi-Newton, linear programming and quadratic programming to name a few.
• Choice of Regularization Method

  Regularization is the idea of forcing the model to not pick extreme values of parameters to over-fit irrelevant artifacts in training data. Methods include MDL, controlling energy/entropy, Lagrange smoothing, shrinkage, bagging and early termination of optimization. Non-explicit treatment of regularization is one reason many methods completely specify their optimization procedure (to get some accidental regularization).
• Choice of Features/Kernelization

  The richness of the feature set the method is applied to is the single largest determinant of model quality.
• Pre-transformation Tricks

  Some statistical methods are improved by pre-transforming the outcome data to look more normal or be more homoscedastic.⁶

If you think along a few axes like these (instead of evaluating them by their name and lineage) you tend to see different data mining methods more as embodying different trade-offs than as being unique incompatible disciplines.

5  Conclusion

Our goal for this writeup was to fully demonstrate a data mining method and then survey some important data mining and machine learning techniques. Many of the important considerations are “too obvious” to be discussed by statisticians and “too statistical” to be comfortably expressed in terms popular with data miners. The theory and considerations from statistics when combined with the experience and optimism of data-mining/machine-learning truly make possible achieving the important goal of “learning from data.”

This expository writeup is also meant to serve as an example of the
types of research, analysis, software and training supplied by
Win-Vector LLC http://www.win-vector.com . Win-Vector LLC
prides itself in depth of research and specializes in identifying,
documenting and implementing the “simplest technique that can
possibly work” (which is often the most understandable, maintainable,
robust and reliable). Win-Vector LLC specializes in research but
has significant experience in delivering full solutions (including
software solutions and integration with existing databases).

A  Graphs

The majority of the graphs in this writeup were produced using “R”
http://www.r-project.org/ and Deepayan Sarkar’s Lattice
package[Sar08].

Footnotes:

¹
mailto:jmount@win-vector.com
http://www.win-vector.com/
http://www.win-vector.com/blog/

²Read P(A | B) as: “the probability of A will
happen given we know B is true.”

³Technically we are working with densities, not
probabilities, but we will use probability notation for its
intuition.

⁴P(sale | x,y) is the probability of
making a sale as a function of what we know about the prospective
customer and our offer. Whereas P(x,y|sale) was just how likely it is
to see a prospect with the given x and y values, conditioned on knowing we made
a sale to this prospect.

⁵ P(sale) and
P(non-sale) are just the “prior odds” of sales or what
our estimate of our chances of success are before we look at any
facts about a particular customer. We can use our historical
overall success and failure rates as estimates of these quantities.

⁶A situation is homoscedastic if the errors are independent of where we are in the parameter space (our x,y or match factor and discount factor). This property is very important for meaningful fitting/modeling and interpreting significance of fits.

Related posts:

1. The Data Enrichment Method
2. Exciting Technique #1: The “R” language.
3. Good Graphs: Graphical Perception and Data Visualization

Now more than ever we must be familiar with the consequences of these important techniques. Especially if we don’t know if we might already be a victim of them.

“The Data-Enrichment Method” is an absolutely wonderful 1957 tongue in cheek parody of a very tempting method of accidental data falsification. The method presented is spookily plausible and actually anticipates some very important (and correct) methods later used in the EM, Jackknife, Bootstrap and other resampling techniques (for example see: “Bootstrap Methods: Another Look at the Jackknife”, Bradley Efron. Ann. Statist. (1979) vol. 7 (1) pp. 1-26).

The idea is innocently presented with an accompanying data-set: perception of a sound at a different presented decibel levels (loudnesses):

From this table it is obvious that the number of detections is increasing (and the number of failures is decreasing) as the sound is presented louder and louder. This makes sense and puts a quantitative rate to our prior expectation that detection gets easier as loudness increases. For this data the trend is quite obvious and we can easily plot a regression line that accurately models the effect of Source.DB on detection rate:

But we want more. Can we increase our model precision and confidence by incorporating our domain knowledge? If we are only trying to accurately estimate the rate that loudness increases the detection level and we are willing to assume that it really does increase, then: could we not pre-prepare the data to use our domain knowledge?

The method suggested is to add in some contra-factuals that we feel confident about. For example we could (using our assumption that loudness increases detection, just to an unknown degree) notice that the 30 failures at 65 DB certainly would not have been heard if they had been run at 62 DB (even quieter). By the same reasoning we can assume that the 5 detections at 62 DB would have been heard had they been run at 65 DB, 68 DB, 71 DB, 74 Db or 77 DB. In this way we have used our starting “seed data” and our domain knowledge to boost into a much larger data set that shows the expected relation much more strongly.

The above paragraph is, of course, nonsense. I am doing the original paper an injustice by summarizing- because in the original paper the procedure seems perfectly plausible (and useful). It is not until the author works a second example that has a poor initial relation (that actually needs the enrichment) that the joke is revealed.

The second example is coin flipping. The author applies an inductive bias that “clearly standing higher up on a staircase increases the chances of a coin flip coming up heads” and then uses the data enrichment method to enhance the data set. The original data set is indeed too noisy to show the effect and the enhancement is in fact quite dramatic. The original data:

The enhanced data is much more interesting:

It is easier to see what is going on in the following plots (which show measured success rates as a function of number of stairs up the staircase and show a smoothed fit of the relationship). The original data is a noisy mess:

And the enriched data is more trend-like:

In fact the regression line fit onto the raw data even has the wrong sign (points down instead of up):

Now, obviously this is a joke. The enhancement procedure did not so much enhance the data as obliterate it. The procedure makes no sense and it is treating the procedure with undue respect to point out any one feature as being “what is wrong with it.” But the original desire is legitimate: can we use informed assumptions to gain a useful inductive bias? If we do know something should we not need less data?

The answer is yes- but we have to be careful. We must read up on the differences between Bayesian, frequentist and empirical methods and decide which set of methods is best for us. Up until now we have been fitting “by standard methods” which is really just minimizing how far the data is from the model (by moving the model around). That isn’t the only way to fit (see: “Controversies In The Foundation Of Statistics” Bradley Efron, American Mathematical Monthly (1978) vol. 85 (4) pp. 231-246).

For example a Bayesian might say that the goal of model fitting is not to pick a model that is closest to the data (maximizes the data’s plausibility with respect to the model) but to pick a model that simultaneously maximizes the product of the data’s plausibility with respect to the model and the model’s acceptability. For example we could say all models for coin-flips with negative slopes are unacceptable and pick the best model with a non-negative slope. However, assigning of degrees of acceptability (or priors) on every possible model is laborious and may require more knowledge than we have from our “reasonable prior domain knowledge.”

Another method is to use more sophisticated notions. One such method is Quantile Regression ( Roger Koenker, Cambridge University Press 2005). This methodology treats regression as a constrained optimization problem- so it is a simple matter to add in more constraints (like the slope must be positive) without having to assign arbitrary plausibilities to every possible model. Another (huge) advantage is that Quantile Regression is much more stable and even without any entered constraints recognizes that the coin-flip data is likely trend free. Here we plot the Quantile Regression analysis of the coin-data (without having added any prior constraints):

To be honest: the method got lucky- the fit is better than should be expected. But Quantile Regression is the perfect framework for adding in domain-constraints.

So: while The Data Enrichment Method is a fraud, there are ways to to enhance analysis to incorporate domain knowledge into results. Instead of saying “any bias (even useful bias) ruins fitting” one should have a cookbook of methods ready to be applied. These cookbooks hide under names like “Econometric Society Monographs” (in my opinion the econometricians really own the interface between theoretical statistics and hard-nosed applications).

Related posts:

1. A Demonstration of Data Mining
2. Good Graphs: Graphical Perception and Data Visualization
3. Exciting Technique #1: The “R” language.

The Sharpe ratio is a famous measure of “risk adjusted return” and is defined as “the ratio of the expected excess return from an investment divided by standard deviation of the excess return.” It is most easily demonstrated by an example (which we work in pieces).

If an investment is expected to generate a profit of 15% in the next year and an insured bank account would generate 10% profit then the expected excess return invested is 15% – 10% = 5%. A rational investor would never take a risky investment that did not have a positive excess return (else they would expect to make more money at a bank). “Expected” is a technical term which means the average return of the investment averaged over all possible outcomes (weighted by the odds of each outcome), we can explain this by working a couple of examples.

Consider investment “A” which is a generally good idea that returns a 20% profit in half the possible years and a 10% profit in the other half of the years. Investment A has an expected return of 0.5*20% + 0.5*10% = 15%. Investment “A” has 15% – 10% = 5% excess return.

Also consider another investment “B” which is a risky bet that returns 20% profit most years (around 95.8% of them) and goes bankrupt in the other years. The expected return of investment “B” is 0.958*20% + 0.042*(-100%) = 14.96%, or essentially 15%. Investment “B” has 15% – 10% = 5% excess return.

As we can see “expectation” alone can not really tell these two investments apart. That is why the second component of the Sharpe ratio is something called the standard deviation. The standard deviation is defined as the square-root of the squared deviations of the return from the target value of 15%. What we do is measure for each possible outcome how far off the return is from the target of 15%, multiply this number by itself (called squaring it) and then take the square-root of the sum of all such values. Again, this is best explained by an example.

Investment “A” has a standard deviation of:
square-root( 0.5 * (20% – 15%)*(20% – 15%) + 0.5 * (10% – 15%)*(10% – 15%) ) = 5%

And investment “B” has a standard deviation of:
square-root( 0.958 *( 20% – 15%)*( 20% – 15%) + 0.042*(-100% – 15%)*(-100% – 15%) ) = 24%

Just like in the calculation of expectation we are taking every possible situation and summing (weighted by the likelihood) our value of interest (in this case the squared variation).

The standard deviation’s opinion is that investment “B” is about five times riskier than investment “A.” And this is the grace of the Sharpe ratio: it says that investment “A”‘s value is (15% – 10%)/5% = 1 and “B”‘s value is (15% – 10%)/24% = 0.2.

An interesting feature of the Sharpe ratio is that, unlike Wall Street, it does not believe that leveraging increases profitability. A common desperation move is to take an investment that has a moderate return and borrow money to simulate larger returns by having larger exposure. For instance an investment that returns 15% can try to simulate a higher return by borrowing. If for every $1,000 invested we borrow another $1,000 to invest (paying the risk rate of 10% for the money) one can show an apparent rate of return of ($2000*15% – $1000*10%)/$1000 or 20%. However, this is not free money- the investor is taking on twice as much risk for only half as much more return. In fact with sufficient leverage (three times, for times, thirty times) one can convert a safe investment into a risky investment that could even go bankrupt. The Sharpe ratio (by design) is not fooled by this sort of manipulation. Investing $1000 in investment A has the exact same Sharpe ratio as investing $1000 plus $1000 more borrowed at the risk-free rate (this is part of the cleverness of using excess returns instead of un-adjusted returns).

Unfortunately to use the Sharpe ratio you need good estimates of three things:

1) The expected return of the investment.

2) The risk-less available in the market (to compute excess).

3) The standard deviation of the investment.

All three of these facts are about the future, so we don’t really know any of them. The historic returns of an investment are not the same thing as the expected returns in the future, interest rates can change and the standard deviation is especially hard to estimate. However, if you have a model (or at least a theory) of what your investments are supposed to do then you can plug in estimates for these three quantities and use the Sharpe ratio to determine which investments really are best.

If you knew how investment “A” worked and could estimate that it returned 20% about half the time and 10% the other times you could estimate its Sharpe ratio as 1. And if you knew investment “B” was a gamble that almost always paid off at 20% with a single rare event that causes bankruptcy you could estimate its Sharpe ratio as 0.2. Even if your estimates were inaccurate (say you estimate investment “A”‘s Sharpe ratio is 0.7 and investment “B”‘s Sharpe ratio as 0.3) the indication is to stay away from investment “B.”

This is in stark contrast to the conclusion you would draw if you thought of these investments as a “black box” (like a fund of funds does) and looked only at their historic performance. If you looked at around 5 years of historic performance of both investments you would (incorrectly) think the following:

Investment A looks kind of noisy, some years it returns 10% and some years it return 20%. You would estimate (correctly) the return as averaging to 15% and you can even get a historic estimate of its standard deviation that is actually about right (5%)

Investment B looks like easy money. With about 80% chance you would not have seen a bankruptcy, just 5 years of 20% returns. You would mis-estimate the return as being 20% (all you have ever seen) and further mis-estimate the standard deviation as 0%.

Based on historic data alone you would fire the manager of investment “A”, give the manager of investment “B” a huge bonus and invest all of your money. And a few years later you would go bankrupt.

What is going on is very well explained by Nassim Nicholas Taleb as “the turkey paradox.” Domestic turkeys are all killed at about the exact same age (say 60 days). For somebody that understands commercial poultry farming there is not any mystery or uncertainty about it. 60 days before you want to sell a turkey carcass you buy a turkey chick. There is an inevitability and reverse causality- the desire for the turkey’s carcass funds and causes the turkey’s start of life 60 days earlier. Now if the turkey is a statistical empiricist (perhaps with a PhD in machine learning) things look good. The turkey sets up a model of each day having an unknown chance of being good or bad. The turkey figures that each day’s outcome is an independent trial drawn from this single unknown probability. The turkey collects evidence: every day it gets fed. Each day is more evidence that all days will be good. And then on day 60 the turkey gets a nasty surprise. The turkey’s life was a bad investment from day one, all of the “evidence” the turkey collects along the way was irrelevant because the model was wrong. And the model was wrong because the turkey guessed at the model instead of investigating the nature of poultry farming.

Much is the same in many investments. There are investments that look like investment “B” when you open the hood. Many of them involve writing “out of the money options” and “default swaps.” These are essentially selling insurance on events that nobody thinks are likely. Selling insurance that usually is not used is profitable, until the insurance gets used. This is why insurance companies (if they are ethical) don’t treat the entirety of collected payments as profit- but as a stockpile that must be kept to pay the claims that will inevitably some day come true.

It is important to point out the Sharpe ratio will give you incorrect results if you plug bad estimates into it. Overall the Sharpe ratio prefers good investments and diversification but it can be led astray. In fact that is the whole point: no amount of smart math will undo the inevitable consequences of wrong models that are used because “you need something you can solve” (like the turkey) or “everybody else is getting rich using them” (like investment “B”).

Related posts:

1. “Easy” Portfolio Allocation
2. What is the gambler’s equivalent of Amdahl’s Law?
3. What does the Market Think?

Also in (less legible) HTML:

Betting Best-Of Series

John Mount¹

Date: May 27, 2008

Introduction

We use the United States’ Major League Baseball World Series to demonstrate some of the “arbitrage arguments”²used in mathematical finance. This problem is a classic finance puzzle question and is an interesting introduction to some exciting techniques.

“Arbitrage” is the simultaneous buying and selling of a commodity, usually in multiple markets, that returns a risk-free profit. An example would be finding a market where apples are selling for $1 and another where they are selling for $2, and then simultaneously executing a purchase order in the cheap market and a sales order in the expensive market (assuming no significant shipping risks or costs). Typically “arbitrage opportunities” are too much to hope for and to make a profit you must add value, loan money, hold inventory or take on risk. This is just the mathematical finance way of saying “there is no free lunch,” but a number of surprising facts about markets can be proven using this principle.

The Problem

Figure 1: World Series Tree (Win over Loss)

Consider a “first to win ” contest like the United States’ Major League Baseball World Series. The World Series is a “first to win four” contest (sometimes called “best of seven”) where a number of games are played between two teams and the first team to win four games is declared the series winner. Ignoring the possibility of ties this process can take from four to seven games. We can (as in Figure 1) lay out all of the possibilities in to a picture that moves from left to right and then moves up when the first team wins and down when the second team wins.

Any sequence of games is represented by a path through this diagram (starting at the left) that reaches a node with no exit. At each node we have marked in the wins for each team (Team One on top, Team Two on the bottom). The nodes where one team has won four games are where the series ends.

The “arbitrage question” is:

If you had access to a bookie who was willing to take an even-payoff bet (on either side) in each game of the World Series, can you design a schedule of bets on games that simulates an even-payoff one dollar bet on the outcome of the entire World Series?

That is: you wish to make a bet that pays you $1 if your team wins the World Series and costs you $1 if your team is defeated. You can not find anybody to take such a bet- but you have found a bookie who makes the incredibly generous offer of taking bets (at even pay-off) on each and every game in the series. Can you, without any additional risk, simulate a World Series bet by making a series of per-game bets with this bookie?

The Answer

The answer turns out to be that you can simulate a world-series bet. The reason for hope is that both types of bets (the even-payoff bets on games and an even-payoff bet on the whole series) are expressing the same underlying belief: that both teams have an exactly equal chance of winning. The teams may or may not have the equal chances of winning- but offering to take bets on both sides at equal pay-off is equivalent expressing just such a belief.

The principle that the probability you are willing to take bets at expresses your subjective probabilities is a principle goes back to Bruno de Finetti and is the most basic “arbitrage style” argument. The principle is simple but it is useful warm-up to think about. Under the assumption that you are “rational” (in the economic sense, which just means you are not giving money away without a reason) and if denotes your personal estimate of the probability of your team winning then if you are willing to bet $1 that your team wins at even payoff (meaning you collect $1 if your team wins pay $1 if your team loses) then for this bet to make economic sense you must have:

which means
.

Similarly if you are willing (for purely economic reasons) to take the other side of the bet at the same even-payoff bet on the other side (reversing the rolls of winning and losing) then it must be true that
. We then have our conclusion: from an economic point of view you should be willing to take either side of a fair-payoff bet only if your estimate of the probability of winning is .

Figure 2: World Series With Some Values Filled In

We now return to the World Series diagram. If we bet on individual games (instead of making one bet on the whole series) then at each node in the diagram we expect to have some sort of net winnings or net losses. For example at each node where our team has won four games we should be holding , so we will label these nodes with . Similarly at each node where the opposing team has won for games we expect to have lost exactly so we label those nodes with . Our task is to figure out the amount bet at each node and our net holdings at each node. If we can find a schedule of bet amounts that leads to the correct outcomes at the end of the world series and starts with an initial net holdings of $0 then we have solved the problem.

If we look at Figure 2 we see there the node corresponding to each team having won 3 games points to two nodes we know the values of (the World Series ending with either team the winner). We can use the fact that this node points only to nodes with known net holdings to figure out both the bet that must be made at this node and the net holdings this node should have at this point in World Series.

Let be the (unknown) net holdings we have at this node and be the (unknown) amount we bet then to complete the World Series bet we must have the following:

This is enough to notice that (your holdings) must be the average of the two outcomes pointed to and (your bet) must be one half of the difference of the two outcomes. So the “each team has won three games” node (near the very right end of the diagram) should have a net holding of and we should bet . Filling in this node with the net holdings ($0) now means that there are other nodes that point only to nodes with filled-in net holdings. We can, in fact, repeat this process of filling in each node with unknown net holdings with the average of the two known nodes it points to until we complete the diagram as in Figure 3.

Figure 3: World Series All Values Filled In

In the completed figure each node is filled in with the net holdings required to implement our betting schedule. We can see that a diagram like this can always be filled out to completion by looking at the diagram as having layers like an onion and noticing that we start with the right most nodes filled in (they are the nodes where the world series ends). It is obvious that we can fill out every node in the layer of nodes just inside the outer layer if we start at the right most such node and work back. Every layer can be completed one after another until we get to the inner most layer which is just the starting node. To implement the betting strategy, we keep track of where we are in the diagram and always bet one half of the difference between the net holdings of the two nodes pointed to by the node we are at.

If the first node of the diagram was marked with a value other than zero it would mean that the world-series has a net bias for the first team or the second. Since the rules are symmetric this would be a nonsense conclusion, so we can be sure that all of the even-score nodes must be valued at zero.

The filling in of blanks using values ahead of them (from the future) is the heart of the Binomial Pricing Theory for options is based on a very deep idea called Dynamic Programming. The idea is that you may not know which future you will experience- but you may know the valuation of every possible future. It is an amazing fact that even without introducing probabilities or probability estimates of which future you will experience just knowing the value of every possible future is enough to compute the value of a bet in the present time. In our example: you may not know ahead of time the final scores of the world series, but you do know value of a world series bet for each possible ending score.

What is the analogy?

From a finance or betting point of view the problem is solved- we have procedures for building the betting schedule and we have the schedule itself. From a mathematician’s point of view we have only just started- we have some procedures and relations but what are they an analogy of?

Naively one might think that they should bet around one fourth of their desired outcome in each game to simulate a best of four series. However to simulate a total World Series bet of we use an initial bet of
in our schedule. This is almost a third of our desired total bet. This gets us wondering: what is the general form of this first bet?

Let
bet denote the amount of the first bet in the simulation of a “best of ” bet. If we compute
bet (by constructing betting schedules as above) for many values of we see that
bet seems to shrink slower than In fact it seems to shrink at a rate of around
. Even more intriguing if you plot
betbet it converges (very slowly) to
. We can conjecture that for very large the initial bet is:
where is the famous ratio of the ratio of the length of the circumference of a circle to the the length of the diameter of the same circle.

Now
is much larger that (as gets large). So the scheme says to bet a fairly large amount of your budget on the first game, and that winning the first bet is worth a bit more than you would expect (it takes you more than one th of the way to victory).

Figure 4: Weighted Paths

What is going on? We can again apply an arbitrage or de Finetti style argument and say since the whole game was “fair” with expected pay-off zero then we can relate probabilities and payoffs. The net holdings at each node encode how much of an advantage you have at the node (or how much you should pay to take over from another gambler at this point). If we let denote the probability of going on to win the World Series bet after winning the first bet then we must have:

   bet

Or
bet . For the real World Series we had
bet so
. This means we can read-off from the valuation tree that the probability of winning the World Series (for perfectly equally matched teams) rise from to after you win the first game.³ This can be confirmed from Figure 3. It is easy to confirm that a portion of all paths the node where Team One has one the first game end with Team One winning the whole World Series (each path must be weighted by its probability which are
). Instead of computing the bets we could have computed the probability of going on to win the World Series at each node⁴ (and then used the above equivalence principle to read off the required bets).

We can create a new diagram where we start at the node where our team has won the first game and we label all the non-ending nodes with the number of paths that reach the node. For example the two nodes immediately after start can be reach one way each and the next three nodes (“3 games to 0”, “2 games to 1” and “1 games to 2”) can be reached 1,2 and 1 ways respectively. It is a clever trick to notice that the easiest way to count the number of paths to a node is to just add the number of ways found on the previous nodes that point to the our target node. This clever way of counting paths is to use weighted paths (inspired by something called Pascal’s Triangle). Figure 4 shows a few columns of a weighted path diagram (thought he ending nodes are re-written as the sum of the paths reaching them where every path is divided by
which is the probability of following such a path).

The entries of weighted path diagram are identified by how many columns out from the start node they are and how many steps from one side of the row they are. Both identifiers start at zero so the starting node is denoted as
the two nodes just after them are denoted
and
. The three nodes just after these are denoted
,
,
and are (as we said before) equal to 1,2 and 1 respectively. These entries are called “binomial coefficients” and the rules for computing them (for integers ) are as follows:

   bet

A lot is known about Binomial coefficients. In fact by a formal called “Stirling’s approximation” we know

as observed.

Relations

de Finetti used this style of reasoning to provide a foundation for the basic theory of probability. Probability theory has always been somewhat problematic for mathematicians in that it has “content” or “an interpretation” whereas the power of modern mathematics comes from a more axiomatic or content-free way of thinking. The issue is if you are defining the meaning or interpretation of something like probability how do you check or demonstrate that you have the correct meaning without referring to some other pre-existing interpretation? A foundational or first interpretation has trouble looking for prior definitions to show equivalence to.[6]

The arbitrage-free arguments and the binomial arguments in particular are the basis of much of mathematical finance and are the basis for a number of Nobel Prizes in Economics including the Black-Scholes-Merton Option Pricing Model[2] and the Binomial Option Pricing Model.[5]

(the ratio of the circumference of a circle to its diameter) is one of the most famous constants in mathematics. Pascal’s Triangle is one of the oldest and most studied diagrams in mathematics with roots all the way back into ancient China.[4] It is actually remarkable how much Zhu Shijie 1303 diagram: looks like our modern version of Pascal’s Triangle (though they are separated by about 350 years, source Wikipedia): . The two diagram differ only in the notation used to write numbers and both start by filling in two diagonals of ones and all other numbers are the sums of the two numbers nearest and above them.

The arguments that replace paths with counts are a particular example of a technique called “Dynamic Programming” invented by Richard Bellman for mathematical optimization and now one of the core concepts of algorithm design.[1]

The idea of using a set of unknown futures that each have a known value is the key idea in solving a number of hard problems in probability and in optimization in the face of uncertainty. One of the the most famous of these problems is the “two armed bandit” where one must decide how to split ones bets between two slot machines that are thought to pay-off at different rates.[3]

For the two armed bandit problem the concern is how long to experiment with both machines when one machine seems to be paying more. The correct solution depends on seeing that how certain you need to be on the difference in machine vales (which in turn drives how long you experiment on both machines). This is a function of how long you intend to use the information. If you intend to play for a long time you want a long initial research phase to produce a very high confidence ranking of the machines; if you do not intend to play for long you want to switch to the machine you suspect is better sooner and on less evidence. Of course “slot machines” is just a toy-problem standing in for uncertain investments, research spending or even spending on different only advertising phrases.

Conclusions

The finance “no arbitrage” principle is actually a very powerful mathematical tool. It is equivalent to but somewhat more graceful than introducing probabilities when solving some combinatorial problems. In this setting it is equivalent to de Finetti’s principle and converting between probabilities and net holdings is very easy.

Bibliography

1

BELLMAN, R.
Dynamic Programming.
Dover Publications, 2003.

2

BLACK, F., AND SCHOLES, M.
The pricing of options and corporate liabilities.
The Journal of Political Economy 81, 3 (Jun 1973), 637-654.

3

CHERNOFF, H.
Sequential design of experiments.
Ann. Math. Statist. 30, 3 (Feb 1959), 755-770.

4

COULTER, L. O.
What is mathematics? toward a global view.
17.

5

COX, J. C., ROSS, S. A., AND RUBINSTEIN, M.
Option pricing: A simplified approach.
Journal of Financial Economics (Sep 1979), 39.

6

SHAFER, G., GILLETT, P. R., AND SCHERL, R. B.
A new understanding of subjective probability and its generalization to lower and upper prevision.
Game-Theoretic Probability Project (Oct 2002), 62.

---

Footnotes

… Mount¹

http://www.win-vector.com/

… arguments”²

More pedantically we are using the principle of “no arbitrage” or “arbitrage free” argument, but the name is traditional.

… game.³

Again, this if for the unrealistic situation of perfectly matched teams. For teams that have uneven probability the series strongly amplifies the better team’s chance of winning (which is one of the series intents). Also a better could update his subjective probability based on the first outcome which also changes things.

… node⁴

This calculation is in essence summing end outcomes across all possible paths weighted by how likely each path is. There are many possible paths, but the calculation can be performed quite efficiently.

… obvious⁵

“Obvious” is actually a special term in mathematics. To illustrate what it means we repeat a story. A mathematician was giving a lecture and stated that the point just shown was obvious. A student asked if it was really obvious. The mathematician stopped the lecture and paused to think. The mathematician thought some more, and eventually walked out of the room. Forty minutes later the mathematician returned to the lecture hall and informed the student that the last point was indeed obvious.

Related posts:

1. What is the gambler’s equivalent of Amdahl’s Law?
2. Paper on stock trading
3. New Paper

]]> http://www.win-vector.com/blog/2008/05/betting-best-of-series/feed/ 0