Goals of this writeup

1. De-mystify kernel methods and support vector machines
   

   Kernel methods and support vector machines have taken mythological proportions in the machine learning imagination. Partly this is because a number of good ideas are overly associated with them: support/non-support training datums, weighting training data, discounting data, regularization, margin and the bounding of generalization error. My issue is that these are all important enough ideas to stand on their own and are often seen in simpler settings. The observations that inform my view are as follows:

   • Kernel methods and support vector machines are in fact two good ideas. Each is important even without the other: kernels are useful all over and support vector machines would be useful even if we restricted to the trivial identity kernel.
   • Small scale “kernel tricks” are not that different than the classic technique of adding “interaction variables.” Kernels let you escape from the limits of “linear hypotheses” (really by moving to a bigger space where things are again linear but look curved from the point of view of your smaller original space). We demonstrate the linear methods and “primal kernel tricks” in A Demonstration of Data Mining.
   • Support/non-support is the central issue of nearest neighbor classifiers.
   • Weighing training data is most famously shared by logistic regression and boosting.
   • Re-weighting of data by smoothing kernels (different but related use of the work “kernel”) is central to non-parametric statistics (kernel smoothers and splines).
   • Regularization is of supreme importance in modeling in general.
   • Most practitioners get tired of “kernel shopping” and fall back to the identity, cosine or radial/Gaussian kernels.

2. Show concrete examples of what are and what are not kernels.
   Few sources give enough theorems to think about kernels abstractly and fewer still work concrete examples.
3. Use as little math as possible (which, unfortunately, turns out to be quite a bit). We will discuss encodings and stopping conditions (important to understand what is going on) but avoid explaining the optimizers (the most beautiful part of support vector machines, but also the part that is available pre-packaged in libraries).
4. Call out common magical thinking and unreasonable expectations associate with kernel methods and support vector machines. This should help the reader be in a better position to “defend their doubts” regarding machine learning promises.
5. Try to place all techniques in a wider context (if it is usable only one place it is a trick, if it is usable multiple places it is a technique).
6. Discuss margin and its impact on generalization error.
   

   Generalization error is an effect of “over fitting” where a model has learned things that are true about the training examples that do not hold for the overall truth or concept we are trying to learn (i.e. don’t generalize). Generalization error is the excess error rate we observe when scoring new examples versus the error-rate we saw in learning the training data. Margin is in fact a posterior observation. That is: margin is observed after the training data is seen, not known before data is seen (like, for example, VC dimension). Margin is useful as it bounds generalization error but it is not the prior bound it is often portrayed as. So we assert margin estimates are not much more special than simple cross-validation estimates which can also be performed once we have data available.

The goals are not to try to indict or try to cut down kernel methods or support vector machines, but just to dump some of the associated baggage so they can be used fluidly and without anxiety.

Example Problem

Consider the following simple (caricature) machine learning problem: we are given a number of points labeled as circles and square and we want to, given a new point, predict the label. In our example the only input will be two numbers: x and y. This example is simple enough that we can depict the entire situation in Figure 1 below:

Figure 1: Truth and Training Data

We are assuming (for simplicity of exposition) that we have the incredible luck that the label is indeed a deterministic function of x and y. We portray this “ground truth” as the blue parabola- every point found in this region will be considered to be labeled as “circle” and every other point (i.e. the red region) will be labeled as “triangle.” We have overlaid 20 example points (with appropriate shape labels). These points will be our training data. We will try to learn from the observed training data a good approximation of the unseen blue and red regions. This process is called learning or training and the ability to correctly predict the label of new points is called generalization. To make sure our concept is learnable we are going to further guarantee a moat (or margin) the distribution we are picking training and test examples from will never pick a point from the shaded region between the blue and black. That is we will never be asked about an example where x*x is very near y.

To promote visual thinking we are going to avoid formulas until the section titled “precise functions used” (where we will list the formulas used for each graph).

Nearest Neighbor Solution

An interesting model in these days of big data and fast machines is the nearest neighbor model. Such a model colors each point in space blue or red as it thinks the point should be classified. In this case each point is colored the color given by the closest known training point. This induces the type of model seen in Figure 2 whose boundary is piecewise line segments mid-way between training points. The data points that determine segments of the classification boundary in this way are thought to support the boundary and are called “support examples” or “support vectors.” In fact any training point that is not “supporting” part of the boundary is “irrelevant” (could be removed and we would get the exact same model). This split between important and non-important training points is considered one of the important observations from the theory of support vector machines, but we see it even here in the nearest neighbor models.

Figure 2: 1 nearest neighbor model

Notice that the nearest neighbor model in Figure 2 is “not half bad.” The blue region is much too wide, but near the training data the blue mass is roughly in the correct position. If we had infinite data and infinite computational resources this sort of model would be hard to beat (as any point we are likely to be tested on would have a lot of very near examples to work from). We can try to clean up the shape of the model a bit by using a vote of the 2 nearest points to color each point in the plane (see Figure 3) or even the majority of the 3 nearest points (see Figure 4).

Figure 3: 2 nearest neighbor model

The purple region in Figure 3 is a “region of uncertainty” where net-vote over the 2 nearest points is zero (they gave inconsistent advice). We can use this as a hint that predictions take from this region are less reliable. As you may notice, we are not getting real improvements. We should always spend more time getting more features (useful coordinates in addition to x and y) and more data; and spend less time tweaking models. But if your data and features are fixed all you work on is your modeling technique (alternately: your modeling technique is all you can prepare before receiving features and data). So we will continue to discuss features and technique.

Figure 4: 3 nearest neighbor model

Nearest neighbor classifiers are optimal in the sense that with an infinite amount of data the 1-nearest neighbor classifier has an error rate that approaches twice the Bayes error rate (the Bayes error rate being the ideal error observed on identical repetitions, or the theoretical best error rate) and for large k the k-nearest neighbor method approaches the Bayes error rate itself (see for example k-nearest neighbor algorithm ).

However the effectiveness of the nearest neighbor classifiers is coming from the very low dimension (2) of our input variables x and y. If we were attempting to predict from n variables we might need an amount of data exponential in n to get a useful nearest neighbor classifier. This is an inefficient use of data of the data; what we want is to use all of the data we have effectively. To do this we further assume there is some relational reason that the position of the point in the plane determines the shape-label (i.e. that we are learning, generalizing, interpolating and extrapolating- not just memorizing). We set our ambitions above memorizing and move towards functional modeling. We start thinking in terms of change: how the label density of examples changes as we move is a good clue as to how it will continue to change. We can do this either in a parametric or primal formulation (where we attempt to directly infer parameters of an assumed functional form as in regression or logistic regression) or in a non-parametric or dual form (where we attempt to learn relations between points and the training data instead of parameters). We have discussed this before ( A Demonstration of Data Mining ) and expanded on primal methods ( Learn Logistic Regression (and beyond, The Simpler Derivation of Logistic Regression and The equivalence of logistic regression and maximum entropy models).

Functional Solution

Our next idea is: can we use our training data to tease out a functional form for the relation between x,y and shape label? The first function we will try is chosen to be similar to the nearest neighbor model we just demonstrated. For this form we say each training point is the center of a Gaussian hump (say up for blue/circle and down for red/triangle) which we will call a “discount function” (or informally a hump). A “Gaussian” is just a function that falls off exponentially in the square of Euclidian distance from a central point (we can see the simple form of such functions in the “precise functions used” section). The contour lines (or levels as seen from looking down upon) of one such hump or discount function are shown in figure 5 (all graphs from here on in will be looking down at a height represented by color and contour lines, much like topographic maps).

Figure 5: Narrow Gaussian example concept

This particular discount function has its maximum (or peak) centered on a data point and then rapidly falls as we move away from the training point. The only property we will use in this section is that we can evaluate the discount easily (so we don’t at this time need or make use of any restrictive properties like integrability, positive semi-definiteness or even anything like bounded level sets).
We could make our functional model just the sum of all of these up and down humps over all of the training data. Each point gets a hump of the same height and same radius, pointing up for blue/circle and down for red/triangle. The net coloring of this sum function is shown in figure 6.

Figure 6: Narrow Gaussian sum model

The idea is that this would imitate the nearest neighbor model we have already seen. This is because even though we are summing over all data the closest training examples should usually dominate (since the Gaussian humps we picked fall to zero as we get further from their centers). The correspondence to nearest neighbor increases if we tighten the radius of our functions (searching for the right radius is a very important statistical problem called “bandwidth estimation”). Figure 7, for example shows the sum over much narrower Gaussians.

Figure 7: Very narrow Gaussian sum model

Figure 7 looks even more like the nearest neighbor solution. In fact your could think of the discount-weighted sum over all of the data as the natural model (as it is easer to reason about) and the nearest neighbors a more efficient approximation (summing only over near points). We also have seen that we can alter the smoothness of our model (which has consequences on model generalization) by changing the steepness of our discount functions. We can also build intermediate models (such as building a nearest neighbor but using discount weighted sums instead of uniform voting in the neighborhoods). We haven’t yet greatly improved on nearest neighbor, but we have identified a couple of obvious avenues for further improvement: mess with the bandwidths (the obvious idea to try and deal with near/far scale) or re-weight the data (as one of the lessons of the nearest neighbor model is that points that are not near a boundary are less important).

Bandwidth Solution

As a digression lets play with bandwidth a bit first. Suppose simultaneously for each training datum we picked an ideal bandwidth or steepness of the Gaussian.

Figure 8: Simultaneous bandwidth model

This simultaneous bandwidth model depicted in figure 8 was determined by picking a simultaneous assignment of Gaussian widths (each training datum gets its own width) that maximizes the log-sum of the category signed model function on the training data (much like the logistic regression maximum likelihood decoding). This differs from typical bandwidth selection problems where all points are given a single common “best bandwidth.” We instead picked a different bandwidth for each point (bandwidths picked to maximize how much of the correct category portion of the sum was correct on each training example). The blue region is reasonable, but does not match the shape of the true concept. Partly this is because there has not been enough training data to have learned a lot about the true shape (for example there is no empirical evidence for circle/blue in the top-left quadrant). The other reason is a bias inherent to this type of model. If one of the bandwidth functions is wider than all of the others (which is almost certain to occur) then it falls slower than all of the others and for points very far away from the center of the training data this one function is most of what remains. Or equivalently: due to the nature of this modeling technique one of the concepts learned is going to be the union of bounded islands and the other concept will grab all points sufficiently far away from the center of the training data. This is strong (and undesirable) bias, but it is a bias also shared by support vector machines with Gaussian kernels (though support vector machines get it from their so-called “dc-term” b not being zero; this will be discussed later). Our bandwidth correction was successful, but frankly the optimization problem we solved to estimate the optimal bandwidths was nasty and would not be something we would advise for a large amount of data.

Data weighting solution (support vector machine)

So instead of messing with the bandwidths let us consider re-weighting the Gaussians in our sum. We will leave the shape of each Gaussian the same but allow each individual training datum to have a unique weight or importance. Perhaps by picking the right weights we can get a better model. By “best” we will mean “best margin” (to be defined later) because with “best margin” as our objective the optimization problem of solving for the best data weights has a particularly beautiful form that can be reliably solved at great scale. This is called a “support vector model” (or support vector machine) and we will describe these ideas in detail later. But first lets see the result. Figure 9 shows the original narrow Gaussians re-summed according to the weights picked by the support vector method (instead of all being 1 as in figure 6).

Figure 9: Narrow Gaussian support vector model

Figure 9 is again an improvement. The figure is formed by taking a signed sum of our discount functions centered at our training points and weighted by the “support vector machine weights.” The signs are picked with one class encoded as +1 and the other as -1. The colors red and blue are picked if the sum is above or below a constant “b” called “the dc-term” (part of the support vector solution). The solution again looks okay: all the training points are in the correct color region (and you can’t hope to interpolate let alone extrapolate if you can’t even reproduce your training concept). Also, as with the bandwidth model, contours are set sensibly (steep/dense where categories are near each other, shallow/sparse where things are safe). We see the same inductive bias: one of the concepts is the union of bounded islands (this will always going to be the case unless the model picks b=0). This bias is not obvious from the kernel choice, as it is hidden in the dc-term of the support vector fitting equations (it is not part of the kernel, but can be defeated by choosing unbounded kernels). An interesting empirical fact about support vector models is they tend to work well with a larger bandwidth. For example if we use a wider Gaussian we get an even more convincing model (see Figure 10).

Figure 10: Wide Gaussian support vector model

In figure 10 we have also called out one of the important features of support vector machines. In the literature all datums are called “vectors” (as they are represented in coordinates) and a subset of these are called the support vectors. In figure 10 we have drawn the 4 training examples that turn out to be support vectors as large shapes and all other training examples as small shapes. The support vectors are the datums with non negligible weights. The learned model is a function of the support vectors only. So after fitting we can discard the other 16 training points. This is similar to the fact that nearest neighbor also only needs the training examples that are supporting its model boundary.

Notice that at this point the support vector models are not “magically” better, they in fact look less like the truth we are trying to learn than either the nearest neighbor models or the un-weighted sum models. To fix this we need to do some “kernel shopping” or find functions that better respect what we are trying to model. With the right functions support vector machines can do a very good job at learning the concept (but knowing “the right functions” is a huge hint as to the what the concept is. There is nothing wrong with using a hint but we do have to a method to produce the hint or have a reasonable number of hints to try from.

The “sum over everything with the same weight” solutions from the earlier section are very similar to Naive Bayes which also sums over all matching features with no weight adjustment. The support vector “use the same model but pick better weights” stands over these sum models in very much the same way Logistic Regression stands over Naive Bayes. In fact the optimization problems are very related. If we consider weights over features or coordinates as “primal” and weights over data items as “dual” we can informally say something like: support vector machines optimize by working over dual variables and inspecting for primal weights, and logistic regression works over primal variables and inspects for data weights. But this is a bit vague.

At this point we need to get a bit more explicit and precise.

Precise functions used

In general we will write our training data as a sequence of n-vectors. Our points will be named u(1) … u(m). For each i = 1..m we also know which category the training point is labeled with. We will encode this as y(1) … y(m) where each y() is either +1 or -1 depending on the training label. So the example we have been working has n=2 and m=20. Let z represent a n-vector we wish to classify.

Figure 6 was a picture of the sign of the function:

Figure 7 was a picture of the sign of the function:

Figure 8 was a picture of the sign of the function:

Figure 9 was a picture of the sign of -b plus the function:

Figure 10 was a picture of the sign of -b plus of the function:

All of this repetition is to emphasize the commonality of the models. They are all of the form:

About Kernels

A lot of awe and mysticism is associated with kernels, but we think they are not that big a deal. Kernels are a combination of two good ideas, they have one important property and are subject to one major limitation. It is also unfortunate that support vector machines and kernels are tied so tightly together. Kernels are the idea of summing functions that imitate similarity (induce a positive-definite encoding of nearness) and support vector machines are the idea of solving a clever dual problem to maximize a quantity called margin. Each is a useful tool even without the other.

The two good ideas are related and unfortunately treated as if they are the same. The two good ideas we promised are:

1. The “kernel trick.” Adding new features/variables that are functions of your other input variables can change linearly inseparable problems into linearly separable problems. For example if our points were encoded not as u(i) = (x(i),y(i)) but as u(i) = (x(i),y(i),x(i)*x(i),y(i)*y(i),x(i)*y(i))

we could easily find the exact concept ( y(i) > x(i)*x(i) which is now a linear concept encoded as the vector (0,1,-1,0,0).

2. Often you don’t need the coordinates of u(i). You are only interested in functions of distances ||u(i)-u(j)|^2 and in many cases you can get at these by inner products and relations like ||u(i)-u(j)||^2 = <u(i),u(i)> + <u(j),u(j)> – 2<u(i),u(j)> .

We will expand on these issues later.

The important property is that kernels look like inner products in a transformed space. The definition of a kernel is: there exists a magic function phi() such that for all u,v:

.

And this brings us to the major limitations of kernels. The phi() transform can be arbitrarily magic except when transforming one vector it doesn’t know what the other vector is and phi() doesn’t even know which side of the inner product it is encoding. That is kernels are not as powerful as any of the following forms:

• snooping (knowing the other):
  
• positional (knowing which part of inner product mapping to):
  
• fully general:
  

Not everything is a kernel.

Some non-kernels are:

1. k(u,v) = -c for any c>0

A consequence of the fact that no negative number can be written as a sum of squares or as the limit of a sum of squares in the reals.

2. k(u,v) = ||u-v||
   

   This can be shown as follows. Suppose k(u,v) is a kernel with k(u,u) = 0 for all u (which is necessary to try and match ||u-v|| as ||u-u|| = 0 for all u). By the definition
   of kernels k(u,u) = < phi(u), phi(u) > for some real vector valued function phi(.). But, by the properties of the inner real inner product <.,.>, this means phi(u) is
   the zero vector for all u. So k(u,v) = 0 for all u,v and does not match ||u-v|| for any u,v such that u ≠ v.

There are some obvious kernels:

1. k(u,v) = c for any c ≥ 0 (non-negative constant kernels)
2. k(u,v) = < u , v > (the identity kernel)
3. k(u,v) = f(u) f(v) for any real valued function f(.)
4. k(u,v) = < f(u) , f(v) > for any real vector valued function f(.) (again, the definition of a kernel)
5. k(u,v) = transpose(u) B v where B is any symmetric positive semi-definite matrix

And there are several subtle ways to build new kernels from old. If q(.,.) and r(.,.) are kernels then so are:

1. k(u,v) = q(u,v) + r(u,v)
2. k(u,v) = c q(u,v) for any c ≥ 0
3. k(u,v) = q(u,v) r(u,v)
4. k(u,v) = q(f(u),f(v)) for any real vector valued function f(.)
5. k(u,v) = lim_{k-> infinity} q_k(u,v) where q_k(u,v) is sequence of kernels and the limit exists.
6. k(u,v) = p(q(u,v)) where p(.) is any polynomial with all non-negative terms.
7. k(u,v) = f(q(u,v)) where f(.) is any function with an absolutely convergent Taylor series with all non-negative terms.

Most of these facts are taken from the excellent book: John Shawe-Taylor and Nello Cristianini’s “Kernel Methods for Pattern Analysis”, Cambridge 2004. We are allowing
kernels of the form k(u,v) = < phi(u), phi(v) > where phi(.) is mapping into an infinite dimensional vector space (like a series). Most of these facts can be checked by
imagining how to alter the phi(.) function. For example to add two kernels just build a larger vector with enough slots for all of the coordinates for the vectors encoding
the phi(.)’s of the two kernels you are trying to add. To scale a kernel by c multiply all coordinates of phi() by sqrt(c).

Multiplying two kernels is the trickiest. Without loss of generality assume q(u,v) = < f(u) ,f(v) > and r(u,v) = < g(u), g(v) > and f(.) and g(.) are both mapping into the same finite dimensional vector space R^m (any other situation can be simulated or approximated by padding with zeros and/or taking limits). Imagine a new vector function p(.) that maps into R^{m*m} such that p(z)_{m*(i-1) + j} = f(z)_i g(z)_j . It is easy to check that k(u,v) = < p(u) , p(v) > is a kernel and k(u,v) = q(u,v) r(u,v). (The reference proof uses tensor notation and the Schur product, but these are big hammers mathematicians use when they don’t want to mess around with re-encoding indices).

Note that one of the attractions of kernel methods is that you never have to actually implement any of the above constructions. What you do is think in terms of sub-routines (easy for computer scientists, unpleasant for mathematicians). For instance: if you had access to two functions q(.,.) and r(.,.) that claim to be kernels and you wanted the product kernel you would just, when asked to evaluate the kernel, just get the results for the two sub-kernels and multiply (so you never need to see the space phi(.) is implicitly working in).

This implicitness can be important (though far too much is made of it). For example the Gaussians we have been using throughout are kernels, but kernels of infinite dimension, so we can not directly represent the space they live in. To see the Gaussian is a kernel notice the following:

.

And for all c ≥ 0 each of the three terms on the right is a kernel (the first because the Taylor series of exp() is absolutely convergent and non-negative and the second two are the f(u) f(v) form we listed as obvious kernels). The trick is magic, but the idea is to use the fact that Euclidian squared distance breaks nicely into dot-products ( ||u-v||^2 = < u, u &gt + < v, v &gt – 2 < u , v >) and exp() converts addition to multiplication. It is rather remarkable that kernels (which are a generalization of inner products that induce half open spaces) can encode bounded concepts (more on this later when we discuss the VC dimension of the Gaussian).

Back to Support Vector Machines

We can now restate what a support vector machine is: it is a method for picking data weights so that the modeling function for a given kernel has a maximum margin. The margin is the minimal distance between a training example and the model’s boundary between blue and red. Notice the 4 support vectors in figure 10 are all not “tight up against the boundary,” but are all the same (minimal distance) from the boundary. This distance is called the margin and the area near the boundary is obviously a place the model has uncertainty (so it makes sense to keep the boundary as far as possible from the training data). The great benefit of the support vector machine is that with access only to the data labels and the kernel function (in fact only the kernel function evaluated at pairs of training datums) the support vector machine can quickly solve for the optimal margin and data weights achieving this margin.

For fun lets plug in new kernel called the cosine kernel:

This kernel can be thought of as having a phi(.) function that takes a vector z and adds an extra coordinate of sqrt(c) and then projects the resulting vector onto the unit sphere. This kernel induces concepts that look like parabolas, as seen in figure 11.

Figure 11: Cosine example concept

Figure 12 shows the sum-concept (add all discount functions with same weight) model. In this case it is far too broad (averaging of the cosine concepts which are not bounded concepts like the Gaussians) creates an overly wide model. The model not only generalizes poorly (fails to match the shape of the truth diagram) it also gets some of its own training data wrong.

Figure 12: Cosine kernel sum model

And figure 13 shows the excellent fit returned by the support vector machine. Actually an excellent fit determined by 4 support vectors (indicated by larger labels). Also notice the support vector machine using unbounded concepts generated a very good unbounded model (unbounded in the sense that both the blue and red regions are infinite). By changing the kernel or discount functions/concepts we changed the inductive bias. So any knowledge of what sort of model we want (one class bounded or not) should greatly influence our choice of kernel functions (since the support vector machine can only pick weights for the kernel functions, not tune their shapes or bandwidths).

Figure 13: Cosine Support Vector model

Another family of kernels (which I consider afflictions) are the “something for nothing” kernels. These are kernels of the form:

Figure 14: Squared magic kernel support vector model

If you want higher order terms I feel you are much better off performing a primal transform so the terms are available in their most general form. That is re-encode the vector u = (x,y) as the larger vector (x,y,x*y,x*x,y*y). You have to be honest: if you are trying to fit more complicated functions you are searching a larger hypothesis space so you need more data to falsify the bad hypotheses. You can be efficient (don’t add terms you don’t think you can use as they make generalization harder) but you can’t get something for nothing (even with kernel methods).

Back to margin

Large margin (having a large distance between the decision boundary and all training examples) is a good idea. At the simplest level it means anything close to a training data point gets classified correctly (so the model is immune to an amount of fuzz proportional to the margin width).

We get the following remarkable generalization theorem (this one is the hard SVM version, we weaken the result a bit to make it easier to state). Suppose we train a model using kernel k(.,.) on m training examples u(1),…,u(m) with training labels y(1)…y(1). Further assume the hidden true concept is separable with respect to our kernel and data distribution with a margin of at least w (that is for any sample we draw we can find a SVM model that classifies all of the training data correctly and sees a margin of at least w). Or more simply: assume w is behaving like a constant with respect to m (i.e. it is not going down as we increase sample size). Then with probability at least 1-d the generalization error (that is error seen on new test points not seen during training, but drawn from the same distribution as the training examples) is no more than:

.

If we further assume the expected value k(u,u) under the training distribution exists (i.e. we don’t have too many examples of very high kernel norm according to the distribution we are training with respect to) and is stable (not taking a lot of value from rare instances) then we can read off the meaning of this upper bound. Assuming we can always build a model from the training data of margin at least w then the generalization error (excess error we expect to see on classifying similar points) is falling at a rate proportional to the square root of the amount of training data we have. This is in one sense amazing- we have a bound on how well we are going to do on data we have not seen. And this bound is also in some sense “best possible” because we need around this much data to even confirm such an accuracy (see What is a large enough random sample? ). Also notice how the dimension of the implicit features space (which can in fact be infinite) does not enter into the bound.

However, the bound is also some sense to be expected. It depends on the following strong assumptions (usually not remembered or stated):

1. Test and training data must be generated in the same manner (i.e. come from the same distribution, this is usually stated but not obeyed).
2. We must not have collapsing margin as the number of data points goes up (so there must actually be a moat between the positive and negative examples such as portrayed in figure1 where the blurry boundary was a region we never generated training or test examples from).
3. The expected value of k(u,u) must be bounded for examples drawn from our training distribution. For example this is not true even for the identity kernel if x is just numbers drawn from a Cauchy distribution.

Also it is a posterior estimate of expected generalization error. That is, we only get a bound after we plug in some facts found from fitting a model to the training data (the margin and the expected value of k(u,u)). It is not like the famous prior bounds based on VC dimension that depend only on the structure of the kernel k(,) and not the distribution of training data or labels (beyond the concept being expressible by the kernel, which is itself a posterior fact). The thing to remember for all posterior estimates is you can get such estimates at a small expense in statistical efficiency (i.e. you waste some data) and a moderate computational blow-up by cross validation (and with essentially no math and many fewer assumptions). In fact the cross validation and hold-out estimates may be much tighter than calculated bounds (and are certainly easier to explain).

When we say we weakened the result we mean we have added some stated some conditions that are not strictly needed for the bound to be true. One of these assumptions is that the margin is not collapsing (the bound remains true even for a collapsing margin; it is just not useful). We added this assumption because most readings of the bound (or presumed applications) implicitly assume the bound remains useful (which means something near our additional assumptions must also hold). The reader is really lured to think of the margin w as a constant depending only on the kernel when in fact it depends on the kernel, data distribution and number of training examples. For example: consider a very simple one dimensional classification problem: learning the concept x ≥ 0 from labeled training data drawn uniformly from the interval [-0.5,0.5] using a Gaussian kernel. The support vector models are going to be essentially picking two neighboring points in the observed training sample that have opposite labels and saying the concept boundary is between them. In this simple case the margin is collapsing: as m (the number of training examples increases) the expected margin width is O(1/m). This renders the bound calculated above useless (drives it above 1 as the number of training examples increases). The model is good but the bound is not useful. This is because the training distribution is discovering points closer and closer to the concept boundary (a necessary discontinuity) as the number of training examples grows. This is why we had to add the additional (weakening) assumption that not only do we need to assume the concept to be learned is separable with respect to the kernel we are using, but we must further assume the data-distribution (determining where test and training data come from) stays some minimal distance away from the decision surface (i.e. together the data distribution and underlying concept have a moat, as in our figure 1 example). We will phrase this as “the concept is wide margin separable with respect to the training distribution” (implicitly we need to know the kernel, but it is the training distribution that is critical). This is a very strong assumption (not true for all classification problems) but is often true when the classification task is to “un-mix a mixture” (the observed data is coming from two different sources and the training labels are which source they come from, in this case you often do have a margin). Obviously a soft-margin SVM deals better with this (some of the issue is due to our use of hard-margin)- but it is an interesting exercise to think why you would expect your learned model to have a large margin if your underlying concept and training data do not.

The result remains important: support vector machines are in some sense an optimal learning procedure (up to some constants they achieve as tight a generalization error as can even be confirmed for a given training set size). Another strong point of the support vector result is the result applies even for kernels with unbounded VC dimension (such as the Gaussian kernel).

The Gaussian (or radial) kernel again

We said the Gaussian kernel was of infinite VC dimension. But we have not shown why it is true. The fact that the only encoding we could think of off the top of our head was a power-series or an limit does not guarantee there isn’t some more clever encoding that easily demonstrates that the Gaussian kernel is of bounded VC dimension (like the identity kernel and cosine kernels are).

However we can see the Gaussian kernel can not have finite VC dimension because it can place an arbitrary number of bounded patches in the plane. Thus we can build arbitrarily large sets (by placing all points far apart) that can be shattered (thus falsifying any bounded VC dimension).

One of the wonders of the support vector method is that the theory works even in this situation.

Back to SVM

Support vector machines seem nearly magical in their power to pick “best weights.” However, as we have seen, these best weights may not always be much better than typical good weights (for instance: using all uniform weights). Also it is very important to remember the support vector machine is picking the best weighting of a sum of already picked kernels. It is not picking the best kernel shape or adjusting things like bandwidth (you must pick the best kernel ahead of time or try several kernels to find a good one). You must remember support vector machines can only directly adjust dual weights (or relative training data weighting) this can affect some changes in hypothesis shape but the support vector machine can not directly implement arbitrary changes in hypothesis shape like a nearest neighbor or a parameterized primal method can (like logistic regression with the kernel trick of adding interaction variables).

Support vector machines are to be much preferred to other combiners like Boosting. This is because support vector machines have explicit stopping criteria (conditions known to be true at the optimal solution that characterize the optimal solution) and explicit regularization (control of over fitting). Boosting relies a path argument (“do the computation in these stages”) so it is much harder to reason about and frankly the folklore that “early stop prevents over fitting in boosting” is just false (you are much better off explicitly regularizing).

It is also lore that random kernel transformations (like power series over the cosine kernel) are magic. They can make data separable but they are unlikely to meet the (unstated but critical) conditions of the support vector theorem (margin being wide, margin not filling in and expectation of k(.,.) being small).

A comment on optimization

In this writeup we have skipped the most beautiful part of support vector machines- the form of the optimization problem used to solve them. We strongly suggest consulting John Shawe-Taylor and Nello Cristianini’s “Kernel Methods for Pattern Analysis”, Cambridge 2004 for this. The writing is dense but accurate. You can literally paste their formulas into a general solver and they work.

Conclusion

We have shown how kernel methods and support vector machines fit in a sequence of machine learning methods:

1. nearest neighbor models
2. uniform sum models
3. support vector machines.

We also showed a number of examples of kernels and non-kernels. It is good to be able to quickly remember the constant “-1″ is not a kernel and the constant “1″ is a kernel (though not a very useful one).

It is our argument that support vector machines are “dual methods” (as they work in weights over the data instead of weights over coordinates in the features space) and the move from the primal algorithm Naive Bayes to Logistic regression is similar to the move from uniform kernel sums to support vector machines.

Also, if you want to directly manipulate shape (like picking bandwidth) you must in some sense use a primal method (this isn’t what support vector machines are for).

We end with: If you have some feeling of both what a method can and can not do then you have some understanding of the method.

Related posts:

1. A Personal Perspective on Machine Learning
2. Do your tools support production or complexity?
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)

The problem that got me thinking is this:

Given a sequence of n integers x1 through xn and an integer k (1 ≤ k ≤ n), find the mean value of all of the medians of the k-sized selections from x1 through xn. Or as a formula:

where x_s is defined as the sequence of integers whose indices are in the set s (not necessarily a contiguous sequence). The median is the “value in the middle” (a value such that half of the selected data are above it and half are below) and “(n choose k)” is the number of ways to choose k items from a collection of n items (which is just the number: n!/((n-k)! k!)). So our sum is adding up a number of terms and then dividing by the number of such terms to get the average or mean of the terms. We will call this sum a “mean of medians”.

Some obvious special cases are: for k=1 the
expression simplifies to the sum of the x_i divided by n (or just the mean of all the x_i) and for k=n it simplifies to the median of all of the x_i. For intermediate values of k it is not immediately obvious how to efficiently compute the value of the sum. Directly adding all (n choose k) terms (as the sum is written) would be very slow for large n with even moderate sized k. Instead we look to find some method that has the same value of the total sum, without directly computing all of the terms.

This gets us to the ad-hoc side of theoretical computer science. We need a clever idea. In this case the idea is simple. To keep things simple: suppose all of the n integers in our sequence have different values and that k is odd (neither of these conditions are important they just let us avoid some non-essential technicalities). What values can median(x_s) take on? median(x_s) is always x_i form some i in the subset s. In fact our sum is equivalent to:

This new sum has a reasonable number of terms- so we can actually calculate it directly if we knew the values of the terms. Without loss of generality assume the x_i are sorted in increasing order. Then the number of times x_i is the median of some x_s is exactly:

The complete solution calculating the mean of medians for distinct sorted x_i is:

A statistician would recognize this expression as a kind of centrally weighted Winsorized mean. The shape of the graph of weights (in this case the n=10, k=5) is suggestive of
a bounded normal window (though i is a rank, not a free-ranging value):

Likely we have re-invented a data treatment known to statisticians. But the above steps were really just combinatorics. What a theorist does is abstract something down to this sort of problem and think of variations and solutions. The formal side of theoretical computer science is attempting to organize all of the problems in the world into two classes: problems presumed easy and problems presumed hard.

For example- what if we had wanted to know the median of many means instead of the mean of many medians?
It turns out a small variation of the median of means problem is already known to be difficult. The hard version of the reversed problem is called “Kth largest subset” (this is a different K than we have been using up until now). The Kth largest subset problem is: given a sequence of integers and constants K and B do K or more distinct subsets have a sum of no more than B? The Kth largest subset problem is known to be “NP hard” which means that a whole host of other thought to be hard problems could be solved if we had the ability to solve this problem (see “Computers and Intractability: A Guide to the Theory of NP-Completeness” Michael R. Garey and David S. Johnson, 1979). The median of many means is not quite as expressive as the Kth largest subset problem (so we have not proven the median of many means is itself NP hard) but the relation is strong: to even verify that we had the right median we would need to solve a Kth largest subset problem with K=(n choose k)/2 and B= k*the_claimed_median (assuming we modified the subset problem to restrict itself to size k sub-sequences). If fact even checking if a given number is one of the possible means (let alone if it is the median of the means) is equivalent to the NP Complete knapsack problem. This correspondence makes us suspect that something as simple as the trick we devised for the mean of medians problem will not solve the median of means problem. One should not take this argument too seriously though, as the same argument would seem to apply to the pair of problems “min of means” and “mean of mins” both of which are in fact easy. We have not proven the median of means problem to be hard, we have just found relevant algorithms and related problems.

What theorists do is find these analogs and then do the heavy lifting (continue the work even when the solution is not simple) to find a way to either efficiently solve the problem at hand or prove it is in fact as hard as other important problems. This kind of thing is hard work for small gains, but the value accumulates because each of these gains is permanent. Finally additional variations of the problem are tried and characterized, to help check we hare not “leaving money on the table” (missing nearby improvements). Some of this may seem like pointless work on toy problems- but every once in a while one of these solutions or characterizations is exactly what is needed to take a large system (like a search engine) to the next level of performance.

Related posts:

1. Good Graphs: Graphical Perception and Data Visualization
2. The Data Enrichment Method
3. A Demonstration of Data Mining

The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

then the following is a picture of reverse accumulation:

Related posts:

1. Automatic Differentiation with Scala
2. Automatic Detection of Potential Deadlock
3. Must Have Software

Portfolio allocation is not the “magic predict the future” part of finance, it is the scheme for correctly applying magic predictions of the future. The idea is that if you had an prediction of future returns of a number of assets, the naive thing to do would be to invest everything into the asset with highest predicted return. Portfolio theory, while still taking the predictions at face value, picks an investment pattern that will (in risk-adjusted dollars) outperform the naive strategy even if the predictions are correct and is a bit safer when the predictions are wrong.

Suppose you had different assets you could invest in. For the -th asset there is an expected excess relative return of and an estimated variance of (for a definition of relative return see Relative returns: a banker versus trader paradox and for a definition of variance see A Quick Appreciation of the Sharpe Ratio). Let the vector be such that represents the number of dollars we invest in the -th asset. If is positive then our plan is “to go long” or buy some of the -th asset. If is negative our plan is “to short” or sell some of the -th asset to somebody else (It is called going short as we actually sell something we do not have. This is often allowed in finance; as long as we make the same pay-outs to the buyer that the buyer would receive if we really had the item to sell).

When we appeal to the idea of optimizing the portfolio Sharpe Ratio (again, see A Quick Appreciation of the Sharpe Ratio) then we say a good portfolio is one that doesn’t just maximize expected relative returns (which is ) but maximizes the ratio of expected relative return to standard deviation:

where (for now) is the matrix . This ratio is called a “risk adjusted return” (versus the un-adjusted form ). Also notice that the ratio is homogeneous in (doubling does not change the ratio as it simultaneously doubles the numerator and the denominator) so an optimal solution describes not how much to invest, but what pattern to invest in. This allows us to introduce an important practical constraint: we are only going to allow ourselves to risk a total of dollars (both long and short). That is: we insist . We will ignore this total investment constraint until the end when we can satisfy the constraint by simply re-scaling an partial solution.

To solve for we introduce an old friend: Lagrange Multipliers (or equivalently the Karush-Kuhn-Tucker conditions of optimality). Since the fraction we are trying to optimize is homogeneous in we can convert the denominator into a constraint and arbitrarily insist that without changing the nature of the problem. We are now trying to maximize subject to . The Lagrangian conditions of optimality state at the optimum we must have the gradient of the objective is proportional to the gradient of the constraint or:

for some (to be determined) constant . Pushing the gradient operator through we get:

A similar equation could be gotten by appealing to a Rayleigh Quotient argument.

We do not yet know (that is what we are trying to solve for), so we do not know what is. However, this is just a scalar and since we are just trying to solve up to a multiple we can throw it out and introduce a new multiple and see that it is enough to solve:

where is new (still unknown) scalar. This means we have:

so our desired solution is some re-scaling of .

As we stated earlier we have a total investment constraint of . We can achieve this with the following adjusted solution:

as our desired optimal portfolio allocation. In the end we can solve for the optimal portfolio by merely solving a linear system (we don’t need anything as expensive as a general purpose optimizer in this case).

These are very old results (going back as long as there has been Sharpe Ratios and portfolio theory). A good example reference is: “The Valuation of Risk Assets and the Selection of Risky Investments in Stock Portfolios and Capital Budgets,” John Lintner, The Review of Economics and Statistics (1965) vol. 47 (1) pp. 13-37. These results are the basis for advice like: “diversify.” Without modeling risk you would tend to put all of your money in the predicted highest paying asset. When modeling risk you tend to put some of your money in each high paying asset and as long as they do not all fail at the same time you have some safety. Another (very different) route to diversification is the Kelly Criterion (discussed in What is the gambler’s equivalent of Amdahl’s Law?).

A very important risk we have not yet modeled is that our assets may have a tendency to fail at the same time (meaning we may not have really diversified usefully). The notion of assets may fail at the same time brings us to the ideas of correlation and covariance. When we took we were implicitly assuming (or modeling), without justification, that each possible asset was independent of all the others (that there was no correlation between asset returns). This is, of course, not going to be anywhere near true in practice. Instead we should take to be the Covariance Matrix that represent our estimate of the assent to asset correlations. In this case the solution methods above all work exactly as before. Companies such as MSCI Barra have made complete businesses out of producing and selling estimates of .

Another issue is when we do not allow ourselves to “short” (or take a negative allocation of) assets. In this case we have the additional constraints which complicates our solution. For the special case where the asset variances are assumed to be independent (i.e. ) it is enough to solve as above and merely replace any negative allocations with zero when inspecting and scaling the final step of the solution. When the covariances are non-trivial ( has non-zero off-diagonal entries) this solution may not be optimal. In this case the Karush-Kuhn-Tucker conditions are more complicated and at the point of optimal solution we have the following conditions:

		0
		0
		0
		0

where is the allocation vector we wish to solve for, is an unknown scalar, is a new unknown vector and is the vector with and zeroes elsewhere. Using the Karush-Kuhn-Tucker conditions has allowed us to again almost linearize the problem, but we know have sign constraints on and and what is called a complementarity constraint: . This sort of problem essentially called a “Linear Complementarity Problem” and is about as hard as solving a linear program (the typical solution method is a variation of the simplex method called “Lemke’s algorithm”). (Technically the prevents the problem from being in the right form, but can be inspected out of the problem.) The problem can still be solved, you just need a bit more software. If we can not short assets (or at least simulate shorting assets) we not only eliminate many possible portfolios from consideration (so we likely end up with a less profitable portfolio than we would like) we also make the mathematics and computation a bit harder.

The goal of this writeup has been to show how to systematically convert investment advice like “this stock is going to really take off” into an allocation of assets (which in turn implies a pattern of trades). We take as unexamined premises where to get such advice and whether to use the Sharpe ratio or some other notion of risk and/or utility. The point is that even though it may be complicated, from this point it is just calculation and calculation is easy to automate.

Related posts:

1. Fast Portfolio re-Balancing as a Fractional Linear Program
2. Kernel Methods and Support Vector Machines de-Mystified
3. A Quick Appreciation of the Sharpe Ratio

Amdahl’s Law is famous idea due to computer architect Gene Amdahl. It is a simple technique that computer scientists use to re-direct their work back to important parts of problems. Suppose you have a complicated system you wish to speed up. Suppose this system is spending a p-fraction of its time in an important sub-process and that you have an idea that would speed up the sub-process by a factor of k. Should you invest the effort?

Amdahl’s Law says (by simple arithmetic): the speed-up (the ratio of the old run-time over the new run-time) the entire system would achieve if you implemented your improvement is not the factor of k you would hope for, but instead:

For example if p = 1/3 then you can only speed up the over all system by at most a factor of 33%, even your idea is so astoundingly good that you have k=1000.

Amdahl’s Law reminds us that speeding up a component you do not lose much time to is not an important accomplishment. In fact Amdahl’s Law directly prescribes looking at your most expensive components as being the largest opportunities for improvement. Appealing to Amdahl’s Law is an important nerd-tool to end “color of the bike shed” arguments (and concentrate only on the design of systems that actually have an impact on outcomes).

It is clear there are similar principles for managing expenses, revenue, effort and so on (such as the Pareto Principle).

But what is the equivalent statement in the harder and more complicated world of probabilities and gambling systems? There are a lot of candidate statements and theorems (such as “look for horses not for zebras”, the Kraft Inequality, Kullback Leibler Distance, Cross Entropy and the Asymptotic Equipartition Principle) but I think the most powerful and direct analogue is: the Kelly Betting System. The Kelly Betting System is a remarkable system that, like Amdahl’s Law, tells us exactly what to look at (and surprisingly some things to ignore).

Kelly’s original paper: “A New Interpretation of Information Rate” J. L. Jr Kelly, AT&T Technical Journal (1956) phrases the problem as betting at a horse race. The technique applies more generally (other forms of gambling, portfolio management, even explaining the preferences of lab-mice) but the clearest example remains a horse race.

We follow the excellent discussion of the problem from Cover and Thomas “Information Theory” Wiley (1991). Consider a simplified horse race where there is only one payoff offered: picking the winning horse. Suppose the (unknown) true probability of the i-th horse winning is p_i. Further suppose the track publishes a set of payoffs for each horse such that if you bet a dollar on the i-th horse and it wins: you are given o_i dollars back.

Now a gambler that has no estimate of the p_i might put all of their money on “the highest paying horse.” That is picking the i such that o_i is maximal (“going for big score”). A somewhat more informed gambler might put all of their money on the “horse with the best expected return” that is a horse i that maximizes p_i * o_i. But this betting strategy “invites ruin”: you have probability of 1 – p_i of losing all of your money. Kelly starts with the controversial idea of trying to maximize expected log-return (instead of maximizing expected return). Maximizing expected log-return avoids ruin, maximizes the exponential rate your wealth grows and maximizes the median wealth over all outcomes (see: “The Kelly System Maximizes Median Fortune” S N Ethier, Journal of Applied Probability (2004) vol. 41 (4) pp. 1230-1236). Even the observation that you don’t always want to put all of your money in a “favorable bet” (that is one with expectation p_i * o_i >1) is an important one.

To get the next part of Kelly’s system consider the sum of reciprocals of track offered payoffs:

At any real track this sum will be greater than 1 (i.e. the o_i will be small, making the sum large). The larger the sum the more clearly unfair the track’s published payoff schedule is. Let us assume we were at a fantastically generous track where this sum is exactly 1 (admittedly unrealistic, and both the paper and the book work beyond this limitation). In this case we can write r_i = 1/o_i and we know r_i > 0 and the r_i sum to 1. That is we can interpret the r_i as the track’s estimate of the probability of the i-th horse winning. If o_i = 100 (the track is paying off 100:1) we then can infer they think the i-th horse has no more than a 1 in 100 chance of winning (else they could not afford to offer the bet). Kelly’s system gives (and proves correct) the following remarkable advice: if the sum given above is 1 (i.e. the track is paying off at least a fair rate) then you can safely bet all of your money and you should bet a p_i fraction of your money on the i-th horse.

That is: if you decide the track is paying off so much that it is worth your while to gamble then you should then completely ignore the track’s payoff schedule in making your bet. You might use the track’s published payoffs as some of your evidence when trying to estimate the p_i (the probability of each horse winning), but once you have estimated these probabilities you then ignore the track’s payoff rates in designing your bets. In fact your expected rate of winning is exactly proportional to how much closer to the true probabilities your estimate is than the track’s estimate is (Cover/Thomas example 6.1.1, so if unless you know something the track does not know you should not bet). Also you should bet even on unlikely and underpaying horses to help cover the possibilities (this is because you are making a series of bets, not just a single bet- so each bet’s value is computed under the assumption that your other bets have failed). This (provably correct) advice is contrary to many obvious and traditional betting systems.

The Kelly System is simultaneously very precise and broadly applicable. For example: it has be extended to many other games and the stock market (see: “The Kelly Criterion and the Stock Market” Louis M Rotando, Edward O Thorp, The American Mathematical Monthly (1992) vol. 99 (10) pp. 922-931). The Kelly System gives actionable advice (exact amounts to bet or exact amounts of effort to invest) and is very specific in saying what to look at.

Just as Amdahl’s law shows us component speedup is a distraction the Kelly System shows us that published rates of return are siren songs. Thus the Kelly System is the gambler’s equivalent of Amdahl’s Law.

Related posts:

1. Good Graphs: Graphical Perception and Data Visualization
2. New Paper
3. The Data Enrichment Method

When a graph is made, quantitative and categorical information is encoded by a display method. Then the information is visually decoded. This visual perception is a vital link. No matter how clever the choice of the information, and no matter how technologically impressive the encoding, a visualization fails if the decoding fails. Some display methods lead to efficient, accurate decoding, and others lead to inefficient, inaccurate decoding. It is only through scientific study of visual perception that informed judgments can be made about display methods. The display methods of Elements rest on a foundation of scientific enquiry.

— from the preface of The Elements of Graphing Data

A revised edition of The Elements of Graphing Data was published in 1994, along with a companion volume, Visualizing Data, which is oriented towards the implementation and technical details of different graphing techniques. I highly recommend The Elements of Graphing Data as a guidebook for creating graphs, as well as for its excellent survey of several useful techniques. Cleveland, along with other colleagues at Bell Labs, developed the Trellis display system, a framework for the visualization of multivariable databases, using the ideas developed in his texts. Trellis, in turn, influenced Deepayan Sarkar’s Lattice graphics system for R. Lattice implements many of Cleveland’s ideas, and I also recommend Sarkar’s Lattice manual if you do data visualization in R.

It’s important to note here that Cleveland writes for researchers and decision-makers who use graphs to analyze data, or to convey scientific results to colleagues in an (ideally) objective manner. This distinguishes him from Darrell Huff, whose 1954 How to Lie with Statistics considered the use of graphs (and statistics in general) as rhetorical devices for convincing others of one’s point of view. Hence, some of Cleveland’s recommendations and guidelines actually contradict Huff’s. ¹

Edward Tufte also explored the idea that the choice of graphical display should be influenced by the viewer’s cognitive processes, in his 1990 book Envisioning Information. Tufte tends to be more broadly concerned with the gestalt of a graph, beyond its use as an analysis tool; he is also more concerned than Cleveland is with aesthetic considerations.

Cleveland’s philosophy might be summarized as: minimize the mental gymnastics that the viewer must go through to understand the graph. This leads to some obvious advice: avoid clutter and occlusion, make graphing symbols or color-coding unambiguous, use scale-lines on all four sides of the graph, and so on. It also leads to advice that perhaps should be as obvious, but isn’t: make the aspect of the data that you want to analyze as clear as possible. But what does this mean in practice?

Make important differences large enough to perceive

Weber’s Law is a well known observation from the psychophysics literature, which states that the “just noticeable” change in a stimulus is a constant ratio of the original stimulus. Put another way, people are only capable of detecting a change in a stimulus that is greater than a certain percentage k of the original stimulus. Here, “stimulus” can refer to any perceivable physical quantity: weight, intensity, length, orientation. The percentage k will vary with stimulus, and with observer.

Figure 1: From Cleveland, The Elements of Graphing Data

Figure 1 shows the application of Weber’s law to lengths. The bars A and B are of different lengths, but the difference is such a small fraction of the “base” length (say, A’s length, to be specific) that is difficult to tell whether or not they are different, or which is longer. On the right, the bars have been embedded in frames of identical length, and now it is easy to see that B is longer. Why? Because the difference in lengths of the white intervals is a much larger percentage of the white “base” length (say the white A interval). It is easy to see that the white B interval is shorter than the white A interval, and therefore, the black B interval is longer than the black A interval.

The moral is that you always want the viewer to be estimating changes or differences with respect to a short base length. You can do this with reference grids, as demonstrated below.

From Cleveland, The Elements of Graphing Data

Figure 2	Figure 3

Figure 2 shows eight curves. Which one dips to the lowest minimum? Are the high curves approaching the same value, and which one is rising the fastest? Are the low curves dipping to the same minimum? Are they going to the same steady state? Figure 3 shows the same curves, graphed with identical reference grids. The grids shorten the base lengths that are being compared, and it is now much easier to compare highs, lows, and steady state behavior.

But wouldn’t it be better to compare the graphs by superposing them? For two or three curves, perhaps. But in this case, eight curves can clutter the graph, and use up the symbol or color space, making it difficult to distinguish the different datasets — increasing the mental gymnastics.

Reference grids are useful even for a single curve, especially one with slowly varying segments, such as these graphs have. The reference grid makes it easier to answer questions like: does the process return to the initial state, or to a different steady state? Has the process reached steady state, or is it still growing?

Make important shape changes large enough to perceive: Banking to 45 degrees.

The aspect ratio of a graph is important when trying to understand shape. Rate of change information is encoded in the slope of the curve, which the viewer estimates by changes in the orientation of the local tangents at each point of the graph. Weber’s Law tells us that very small changes in this orientation will be difficult to detect. For a given (physical) curve, the local orientation changes will be dependent on the aspect ratio of its graphical presentation, as shown (to an exaggerated degree) in Figure 4. Here, the same curve (two line segments) is plotted at three different aspect ratios, one that centers the graph at 45 degrees, one that forces the curve to be nearly vertical, and another that forces it to be nearly horizontal. In the last two cases, the change in orientation of the two line segments is so small as to be nearly undetectable.

Figure 4: From Cleveland

For two line segments with positive, unequal slopes, a simple geometric argument shows that their absolute difference in orientation is maximized by the aspect ratio that sets their average orientation to 45 degrees (the first graph in Figure 4). Empirical studies by Cleveland and others have indeed verified that a viewer’s ability to judge the relative slopes of line segments on a graph is maximized when the absolute values of the orientations of the segments are centered on 45 degrees.

This result leads to a technique called Banking to 45, whereby the aspect ratio of the graph is chosen so that the average slope of the entire graph is 45 degrees. The details are discussed in Cleveland, and many of the plots in R’s Lattice package also have an option to bank the graph to 45 degrees.

This deliberate exaggeration of slope is something that Darrell Huff deplores. In How to Lie with Statistics, Huff refers to these graphs as “gee-whiz” graphs — and in the context of his discussion of statistics as rhetoric, they are:

Figure 5: From Huff, How to Lie With Statistics

To insist that a graph should always include a zero line and that units be in proportion may be good advice from a rhetorical perspective; but it is poor advice if the purpose of the graph is data analysis. As Figure 6 below demonstrates, we can lose resolution if we always insist on including the zero. Does the trend line in the left graph increase linearly, superlinearly, or sublinearly? The convexity of the curve is more apparent when it is banked to 45, as on the right. Assuming that the scientist reads the axis and is cognizant of the actual magnitude changes involved, the graph on the right conveys more information.

Figure 6: From Cleveland

Make sure all the data is equally well resolved.

It is quite common for positive data — word frequencies, populations, price distributions, just to name a few examples — to be skewed: most of the data is bunched towards low values, the rest of it is spread out on a very long tail. This long tail squashes the majority of the data into a tiny interval of a very narrow dynamic range, as in Figure 7, making it difficult to evaluate the data.

Figure 7: Long-tailed distribution of purchase sizes

Figure 8: Distribution of log(purchase size)

Imagine that Figure 7 represents the distribution of average purchase size across an online merchant’s customers: average purchase size is plotted on the x-axis, and the y-axis represents the fraction of the total customer population whose average purchase size is a given value (the area under the graph integrates to one). According to this graph, most customers make fairly small purchases on average, but there is a long tail of big spenders trailing out into the range of several thousand dollars. Obviously, one would like a little more resolution on the big spike of customers near zero. One could simply “zoom in” on this range, by chopping off some long chunk of the tail, but you may potentially lose sight of some global patterns in the data by doing so.

Graphing the distribution of log(purchase size) enables you to increase the resolution near zero, while preserving the global view. Figure 8 shows the distribution of log(purchase size), revealing two spending populations: a population of high spenders who tend to make purchases in the $3000 range (in log space), and another population whose purchases are centered (in log space) around $60. The existence of these two distinct populations is not apparent in the original graph.

Notice that Figure 8 has two x-axis scales: the top axis is marked in log units, while the bottom axis is marked in absolute dollars, spaced on a log scale. This accords with the principle of minimizing mental gymnastics, since the viewer of the graph will typically be concerned about prices in dollars, not log dollars. In fact, it would have been better yet to have plotted the distribution of log₂ or log₁₀ of the data; the former would allow us to see at a glance the doubling of price ranges, the latter to see price changes in factors of ten.

Figure 9: The 14 most abundant elements in meteorites. From Cleveland

Figure 9 shows another example: the fourteen most abundant elements in meteorites, specifically the average percent of each of the elements. If we graph the percentages directly, as on the left, we cannot easily distinguish the differences in the elements from aluminum on down. Graphing log₂ of the percentages, as on the right, improves the resolution. Again, we have two x-axes on the graph of the log data.

If you want to analyze the difference between two processes, then graph the difference, not the processes (or graph both).

Suppose that we are comparing the two processes f1 and f2 that are shown in Figure 10. As x increases, the two processes appear to be approaching each other — that is, the difference between the two seems to be decreasing. In reality, the difference between the two is constant: f2 = f1+1.

Figure 10: The illusion of convergence

Figure 11: British Imports and Exports. From Cleveland

It turns out that people are good at perceiving the perpendicular difference between two curves, but not the differences in height, which is what we are actually interested in here. When we try to infer the differences from the process graph, we may not only miss key information, we may actually draw incorrect conclusions.

A less toy example is given in Figure 11. Here the imports to and exports from England are graphed over the first 80 years of the 18th century. In the difference graph on the bottom, we can see a local peak in (imports-exports) just after 1760; this is not obvious from simply comparing the two processes (top graph).

If you are interested in rate of change, then graph rate of change.

In Figure 12, we see the population figures for a given community from 1990 to 2009. Obviously, the population is steadily increasing, but how quickly? Is the rate of population growth increasing over time, or is it decreasing? If we are interested in these questions, then simply graphing the population over time is not enough. We need to look at the rate of change directly.

Figure 12

Figure 13

The classic way to do this is by graphing the logarithm of the data. In Figure 13, we have graphed log₂ of the population over time, with the log scale printed on the right hand y-axis, and the actual population numbers printed at a log scale on the left hand axis. Now we can see that the population increased at a constant rate from 1990 to 2000, quadrupling approximately every four years, and then slowed down (to a lower constant rate) after 2000.

Graphs as a research tool

Throughout this discussion, we have considered graphs as a tool for data exploration and initial understanding. It is an iterative process — as questions arise, the data will be reprocessed and re-plotted to highlight the new issues to be examined. A good research graph must display this information directly, with a minimum of mental gymnastics, but — as with any research tool — there can be a learning curve. For example, densityplots (such as those shown in Figures 7 and 8) are in my opinion more useful than histograms for understanding how numerical data is distributed — and I am constantly surprised at the amount of explanation that they require when I show them to people who are unfamiliar with them. A number of very useful graphs that are discussed in Cleveland’s texts meet with the same reaction from people who encounter that style of graph for the first time. This is a disadvantage, relative to using a more fashionable graph, when attempting to communicate results. But the insight into the data that these graphs provide often make it worth spending the time to educate clients or peers on how to read the graph.

Even so, a good graph still may not be a quick read. As Cleveland writes:

While there is a place for rapidly-understood graphs, it is too limiting to make speed a requirement in science and technology, where the use of graphs ranges from detailed in-depth data analysis to quick presentation.
…

The important criterion for a graph is not simply how fast we can see a result; rather it is whether through the use of the graph we can see something that would have been harder to see otherwise or that could not have been seen at all.

- The Elements of Graphing Data, Chapter 2

Related posts:

1. My Favorite Graphs
2. A Demonstration of Data Mining
3. What Did Theorists Do Before The Age Of Big Data?

Now more than ever we must be familiar with the consequences of these important techniques. Especially if we don’t know if we might already be a victim of them.

“The Data-Enrichment Method” is an absolutely wonderful 1957 tongue in cheek parody of a very tempting method of accidental data falsification. The method presented is spookily plausible and actually anticipates some very important (and correct) methods later used in the EM, Jackknife, Bootstrap and other resampling techniques (for example see: “Bootstrap Methods: Another Look at the Jackknife”, Bradley Efron. Ann. Statist. (1979) vol. 7 (1) pp. 1-26).

The idea is innocently presented with an accompanying data-set: perception of a sound at a different presented decibel levels (loudnesses):

From this table it is obvious that the number of detections is increasing (and the number of failures is decreasing) as the sound is presented louder and louder. This makes sense and puts a quantitative rate to our prior expectation that detection gets easier as loudness increases. For this data the trend is quite obvious and we can easily plot a regression line that accurately models the effect of Source.DB on detection rate:

But we want more. Can we increase our model precision and confidence by incorporating our domain knowledge? If we are only trying to accurately estimate the rate that loudness increases the detection level and we are willing to assume that it really does increase, then: could we not pre-prepare the data to use our domain knowledge?

The method suggested is to add in some contra-factuals that we feel confident about. For example we could (using our assumption that loudness increases detection, just to an unknown degree) notice that the 30 failures at 65 DB certainly would not have been heard if they had been run at 62 DB (even quieter). By the same reasoning we can assume that the 5 detections at 62 DB would have been heard had they been run at 65 DB, 68 DB, 71 DB, 74 Db or 77 DB. In this way we have used our starting “seed data” and our domain knowledge to boost into a much larger data set that shows the expected relation much more strongly.

The above paragraph is, of course, nonsense. I am doing the original paper an injustice by summarizing- because in the original paper the procedure seems perfectly plausible (and useful). It is not until the author works a second example that has a poor initial relation (that actually needs the enrichment) that the joke is revealed.

The second example is coin flipping. The author applies an inductive bias that “clearly standing higher up on a staircase increases the chances of a coin flip coming up heads” and then uses the data enrichment method to enhance the data set. The original data set is indeed too noisy to show the effect and the enhancement is in fact quite dramatic. The original data:

The enhanced data is much more interesting:

It is easier to see what is going on in the following plots (which show measured success rates as a function of number of stairs up the staircase and show a smoothed fit of the relationship). The original data is a noisy mess:

And the enriched data is more trend-like:

In fact the regression line fit onto the raw data even has the wrong sign (points down instead of up):

Now, obviously this is a joke. The enhancement procedure did not so much enhance the data as obliterate it. The procedure makes no sense and it is treating the procedure with undue respect to point out any one feature as being “what is wrong with it.” But the original desire is legitimate: can we use informed assumptions to gain a useful inductive bias? If we do know something should we not need less data?

The answer is yes- but we have to be careful. We must read up on the differences between Bayesian, frequentist and empirical methods and decide which set of methods is best for us. Up until now we have been fitting “by standard methods” which is really just minimizing how far the data is from the model (by moving the model around). That isn’t the only way to fit (see: “Controversies In The Foundation Of Statistics” Bradley Efron, American Mathematical Monthly (1978) vol. 85 (4) pp. 231-246).

For example a Bayesian might say that the goal of model fitting is not to pick a model that is closest to the data (maximizes the data’s plausibility with respect to the model) but to pick a model that simultaneously maximizes the product of the data’s plausibility with respect to the model and the model’s acceptability. For example we could say all models for coin-flips with negative slopes are unacceptable and pick the best model with a non-negative slope. However, assigning of degrees of acceptability (or priors) on every possible model is laborious and may require more knowledge than we have from our “reasonable prior domain knowledge.”

Another method is to use more sophisticated notions. One such method is Quantile Regression ( Roger Koenker, Cambridge University Press 2005). This methodology treats regression as a constrained optimization problem- so it is a simple matter to add in more constraints (like the slope must be positive) without having to assign arbitrary plausibilities to every possible model. Another (huge) advantage is that Quantile Regression is much more stable and even without any entered constraints recognizes that the coin-flip data is likely trend free. Here we plot the Quantile Regression analysis of the coin-data (without having added any prior constraints):

To be honest: the method got lucky- the fit is better than should be expected. But Quantile Regression is the perfect framework for adding in domain-constraints.

So: while The Data Enrichment Method is a fraud, there are ways to to enhance analysis to incorporate domain knowledge into results. Instead of saying “any bias (even useful bias) ruins fitting” one should have a cookbook of methods ready to be applied. These cookbooks hide under names like “Econometric Society Monographs” (in my opinion the econometricians really own the interface between theoretical statistics and hard-nosed applications).

Related posts:

1. A Demonstration of Data Mining
2. Good Graphs: Graphical Perception and Data Visualization
3. What Did Theorists Do Before The Age Of Big Data?

The Sharpe ratio is a famous measure of “risk adjusted return” and is defined as “the ratio of the expected excess return from an investment divided by standard deviation of the excess return.” It is most easily demonstrated by an example (which we work in pieces).

If an investment is expected to generate a profit of 15% in the next year and an insured bank account would generate 10% profit then the expected excess return invested is 15% – 10% = 5%. A rational investor would never take a risky investment that did not have a positive excess return (else they would expect to make more money at a bank). “Expected” is a technical term which means the average return of the investment averaged over all possible outcomes (weighted by the odds of each outcome), we can explain this by working a couple of examples.

Consider investment “A” which is a generally good idea that returns a 20% profit in half the possible years and a 10% profit in the other half of the years. Investment A has an expected return of 0.5*20% + 0.5*10% = 15%. Investment “A” has 15% – 10% = 5% excess return.

Also consider another investment “B” which is a risky bet that returns 20% profit most years (around 95.8% of them) and goes bankrupt in the other years. The expected return of investment “B” is 0.958*20% + 0.042*(-100%) = 14.96%, or essentially 15%. Investment “B” has 15% – 10% = 5% excess return.

As we can see “expectation” alone can not really tell these two investments apart. That is why the second component of the Sharpe ratio is something called the standard deviation. The standard deviation is defined as the square-root of the squared deviations of the return from the target value of 15%. What we do is measure for each possible outcome how far off the return is from the target of 15%, multiply this number by itself (called squaring it) and then take the square-root of the sum of all such values. Again, this is best explained by an example.

Investment “A” has a standard deviation of:
square-root( 0.5 * (20% – 15%)*(20% – 15%) + 0.5 * (10% – 15%)*(10% – 15%) ) = 5%

And investment “B” has a standard deviation of:
square-root( 0.958 *( 20% – 15%)*( 20% – 15%) + 0.042*(-100% – 15%)*(-100% – 15%) ) = 24%

Just like in the calculation of expectation we are taking every possible situation and summing (weighted by the likelihood) our value of interest (in this case the squared variation).

The standard deviation’s opinion is that investment “B” is about five times riskier than investment “A.” And this is the grace of the Sharpe ratio: it says that investment “A”‘s value is (15% – 10%)/5% = 1 and “B”‘s value is (15% – 10%)/24% = 0.2.

An interesting feature of the Sharpe ratio is that, unlike Wall Street, it does not believe that leveraging increases profitability. A common desperation move is to take an investment that has a moderate return and borrow money to simulate larger returns by having larger exposure. For instance an investment that returns 15% can try to simulate a higher return by borrowing. If for every $1,000 invested we borrow another $1,000 to invest (paying the risk rate of 10% for the money) one can show an apparent rate of return of ($2000*15% – $1000*10%)/$1000 or 20%. However, this is not free money- the investor is taking on twice as much risk for only half as much more return. In fact with sufficient leverage (three times, for times, thirty times) one can convert a safe investment into a risky investment that could even go bankrupt. The Sharpe ratio (by design) is not fooled by this sort of manipulation. Investing $1000 in investment A has the exact same Sharpe ratio as investing $1000 plus $1000 more borrowed at the risk-free rate (this is part of the cleverness of using excess returns instead of un-adjusted returns).

Unfortunately to use the Sharpe ratio you need good estimates of three things:

1) The expected return of the investment.

2) The risk-less available in the market (to compute excess).

3) The standard deviation of the investment.

All three of these facts are about the future, so we don’t really know any of them. The historic returns of an investment are not the same thing as the expected returns in the future, interest rates can change and the standard deviation is especially hard to estimate. However, if you have a model (or at least a theory) of what your investments are supposed to do then you can plug in estimates for these three quantities and use the Sharpe ratio to determine which investments really are best.

If you knew how investment “A” worked and could estimate that it returned 20% about half the time and 10% the other times you could estimate its Sharpe ratio as 1. And if you knew investment “B” was a gamble that almost always paid off at 20% with a single rare event that causes bankruptcy you could estimate its Sharpe ratio as 0.2. Even if your estimates were inaccurate (say you estimate investment “A”‘s Sharpe ratio is 0.7 and investment “B”‘s Sharpe ratio as 0.3) the indication is to stay away from investment “B.”

This is in stark contrast to the conclusion you would draw if you thought of these investments as a “black box” (like a fund of funds does) and looked only at their historic performance. If you looked at around 5 years of historic performance of both investments you would (incorrectly) think the following:

Investment A looks kind of noisy, some years it returns 10% and some years it return 20%. You would estimate (correctly) the return as averaging to 15% and you can even get a historic estimate of its standard deviation that is actually about right (5%)

Investment B looks like easy money. With about 80% chance you would not have seen a bankruptcy, just 5 years of 20% returns. You would mis-estimate the return as being 20% (all you have ever seen) and further mis-estimate the standard deviation as 0%.

Based on historic data alone you would fire the manager of investment “A”, give the manager of investment “B” a huge bonus and invest all of your money. And a few years later you would go bankrupt.

What is going on is very well explained by Nassim Nicholas Taleb as “the turkey paradox.” Domestic turkeys are all killed at about the exact same age (say 60 days). For somebody that understands commercial poultry farming there is not any mystery or uncertainty about it. 60 days before you want to sell a turkey carcass you buy a turkey chick. There is an inevitability and reverse causality- the desire for the turkey’s carcass funds and causes the turkey’s start of life 60 days earlier. Now if the turkey is a statistical empiricist (perhaps with a PhD in machine learning) things look good. The turkey sets up a model of each day having an unknown chance of being good or bad. The turkey figures that each day’s outcome is an independent trial drawn from this single unknown probability. The turkey collects evidence: every day it gets fed. Each day is more evidence that all days will be good. And then on day 60 the turkey gets a nasty surprise. The turkey’s life was a bad investment from day one, all of the “evidence” the turkey collects along the way was irrelevant because the model was wrong. And the model was wrong because the turkey guessed at the model instead of investigating the nature of poultry farming.

Much is the same in many investments. There are investments that look like investment “B” when you open the hood. Many of them involve writing “out of the money options” and “default swaps.” These are essentially selling insurance on events that nobody thinks are likely. Selling insurance that usually is not used is profitable, until the insurance gets used. This is why insurance companies (if they are ethical) don’t treat the entirety of collected payments as profit- but as a stockpile that must be kept to pay the claims that will inevitably some day come true.

It is important to point out the Sharpe ratio will give you incorrect results if you plug bad estimates into it. Overall the Sharpe ratio prefers good investments and diversification but it can be led astray. In fact that is the whole point: no amount of smart math will undo the inevitable consequences of wrong models that are used because “you need something you can solve” (like the turkey) or “everybody else is getting rich using them” (like investment “B”).

Related posts:

1. “Easy” Portfolio Allocation
2. An Appreciation of Locality Sensitive Hashing
3. Betting Best-Of Series