Goals of this writeup

1. De-mystify kernel methods and support vector machines
   

   Kernel methods and support vector machines have taken mythological proportions in the machine learning imagination. Partly this is because a number of good ideas are overly associated with them: support/non-support training datums, weighting training data, discounting data, regularization, margin and the bounding of generalization error. My issue is that these are all important enough ideas to stand on their own and are often seen in simpler settings. The observations that inform my view are as follows:

   • Kernel methods and support vector machines are in fact two good ideas. Each is important even without the other: kernels are useful all over and support vector machines would be useful even if we restricted to the trivial identity kernel.
   • Small scale “kernel tricks” are not that different than the classic technique of adding “interaction variables.” Kernels let you escape from the limits of “linear hypotheses” (really by moving to a bigger space where things are again linear but look curved from the point of view of your smaller original space). We demonstrate the linear methods and “primal kernel tricks” in A Demonstration of Data Mining.
   • Support/non-support is the central issue of nearest neighbor classifiers.
   • Weighing training data is most famously shared by logistic regression and boosting.
   • Re-weighting of data by smoothing kernels (different but related use of the work “kernel”) is central to non-parametric statistics (kernel smoothers and splines).
   • Regularization is of supreme importance in modeling in general.
   • Most practitioners get tired of “kernel shopping” and fall back to the identity, cosine or radial/Gaussian kernels.

2. Show concrete examples of what are and what are not kernels.
   Few sources give enough theorems to think about kernels abstractly and fewer still work concrete examples.
3. Use as little math as possible (which, unfortunately, turns out to be quite a bit). We will discuss encodings and stopping conditions (important to understand what is going on) but avoid explaining the optimizers (the most beautiful part of support vector machines, but also the part that is available pre-packaged in libraries).
4. Call out common magical thinking and unreasonable expectations associate with kernel methods and support vector machines. This should help the reader be in a better position to “defend their doubts” regarding machine learning promises.
5. Try to place all techniques in a wider context (if it is usable only one place it is a trick, if it is usable multiple places it is a technique).
6. Discuss margin and its impact on generalization error.
   

   Generalization error is an effect of “over fitting” where a model has learned things that are true about the training examples that do not hold for the overall truth or concept we are trying to learn (i.e. don’t generalize). Generalization error is the excess error rate we observe when scoring new examples versus the error-rate we saw in learning the training data. Margin is in fact a posterior observation. That is: margin is observed after the training data is seen, not known before data is seen (like, for example, VC dimension). Margin is useful as it bounds generalization error but it is not the prior bound it is often portrayed as. So we assert margin estimates are not much more special than simple cross-validation estimates which can also be performed once we have data available.

The goals are not to try to indict or try to cut down kernel methods or support vector machines, but just to dump some of the associated baggage so they can be used fluidly and without anxiety.

Example Problem

Consider the following simple (caricature) machine learning problem: we are given a number of points labeled as circles and square and we want to, given a new point, predict the label. In our example the only input will be two numbers: x and y. This example is simple enough that we can depict the entire situation in Figure 1 below:

Figure 1: Truth and Training Data

We are assuming (for simplicity of exposition) that we have the incredible luck that the label is indeed a deterministic function of x and y. We portray this “ground truth” as the blue parabola- every point found in this region will be considered to be labeled as “circle” and every other point (i.e. the red region) will be labeled as “triangle.” We have overlaid 20 example points (with appropriate shape labels). These points will be our training data. We will try to learn from the observed training data a good approximation of the unseen blue and red regions. This process is called learning or training and the ability to correctly predict the label of new points is called generalization. To make sure our concept is learnable we are going to further guarantee a moat (or margin) the distribution we are picking training and test examples from will never pick a point from the shaded region between the blue and black. That is we will never be asked about an example where x*x is very near y.

To promote visual thinking we are going to avoid formulas until the section titled “precise functions used” (where we will list the formulas used for each graph).

Nearest Neighbor Solution

An interesting model in these days of big data and fast machines is the nearest neighbor model. Such a model colors each point in space blue or red as it thinks the point should be classified. In this case each point is colored the color given by the closest known training point. This induces the type of model seen in Figure 2 whose boundary is piecewise line segments mid-way between training points. The data points that determine segments of the classification boundary in this way are thought to support the boundary and are called “support examples” or “support vectors.” In fact any training point that is not “supporting” part of the boundary is “irrelevant” (could be removed and we would get the exact same model). This split between important and non-important training points is considered one of the important observations from the theory of support vector machines, but we see it even here in the nearest neighbor models.

Figure 2: 1 nearest neighbor model

Notice that the nearest neighbor model in Figure 2 is “not half bad.” The blue region is much too wide, but near the training data the blue mass is roughly in the correct position. If we had infinite data and infinite computational resources this sort of model would be hard to beat (as any point we are likely to be tested on would have a lot of very near examples to work from). We can try to clean up the shape of the model a bit by using a vote of the 2 nearest points to color each point in the plane (see Figure 3) or even the majority of the 3 nearest points (see Figure 4).

Figure 3: 2 nearest neighbor model

The purple region in Figure 3 is a “region of uncertainty” where net-vote over the 2 nearest points is zero (they gave inconsistent advice). We can use this as a hint that predictions take from this region are less reliable. As you may notice, we are not getting real improvements. We should always spend more time getting more features (useful coordinates in addition to x and y) and more data; and spend less time tweaking models. But if your data and features are fixed all you work on is your modeling technique (alternately: your modeling technique is all you can prepare before receiving features and data). So we will continue to discuss features and technique.

Figure 4: 3 nearest neighbor model

Nearest neighbor classifiers are optimal in the sense that with an infinite amount of data the 1-nearest neighbor classifier has an error rate that approaches twice the Bayes error rate (the Bayes error rate being the ideal error observed on identical repetitions, or the theoretical best error rate) and for large k the k-nearest neighbor method approaches the Bayes error rate itself (see for example k-nearest neighbor algorithm ).

However the effectiveness of the nearest neighbor classifiers is coming from the very low dimension (2) of our input variables x and y. If we were attempting to predict from n variables we might need an amount of data exponential in n to get a useful nearest neighbor classifier. This is an inefficient use of data of the data; what we want is to use all of the data we have effectively. To do this we further assume there is some relational reason that the position of the point in the plane determines the shape-label (i.e. that we are learning, generalizing, interpolating and extrapolating- not just memorizing). We set our ambitions above memorizing and move towards functional modeling. We start thinking in terms of change: how the label density of examples changes as we move is a good clue as to how it will continue to change. We can do this either in a parametric or primal formulation (where we attempt to directly infer parameters of an assumed functional form as in regression or logistic regression) or in a non-parametric or dual form (where we attempt to learn relations between points and the training data instead of parameters). We have discussed this before ( A Demonstration of Data Mining ) and expanded on primal methods ( Learn Logistic Regression (and beyond, The Simpler Derivation of Logistic Regression and The equivalence of logistic regression and maximum entropy models).

Functional Solution

Our next idea is: can we use our training data to tease out a functional form for the relation between x,y and shape label? The first function we will try is chosen to be similar to the nearest neighbor model we just demonstrated. For this form we say each training point is the center of a Gaussian hump (say up for blue/circle and down for red/triangle) which we will call a “discount function” (or informally a hump). A “Gaussian” is just a function that falls off exponentially in the square of Euclidian distance from a central point (we can see the simple form of such functions in the “precise functions used” section). The contour lines (or levels as seen from looking down upon) of one such hump or discount function are shown in figure 5 (all graphs from here on in will be looking down at a height represented by color and contour lines, much like topographic maps).

Figure 5: Narrow Gaussian example concept

This particular discount function has its maximum (or peak) centered on a data point and then rapidly falls as we move away from the training point. The only property we will use in this section is that we can evaluate the discount easily (so we don’t at this time need or make use of any restrictive properties like integrability, positive semi-definiteness or even anything like bounded level sets).
We could make our functional model just the sum of all of these up and down humps over all of the training data. Each point gets a hump of the same height and same radius, pointing up for blue/circle and down for red/triangle. The net coloring of this sum function is shown in figure 6.

Figure 6: Narrow Gaussian sum model

The idea is that this would imitate the nearest neighbor model we have already seen. This is because even though we are summing over all data the closest training examples should usually dominate (since the Gaussian humps we picked fall to zero as we get further from their centers). The correspondence to nearest neighbor increases if we tighten the radius of our functions (searching for the right radius is a very important statistical problem called “bandwidth estimation”). Figure 7, for example shows the sum over much narrower Gaussians.

Figure 7: Very narrow Gaussian sum model

Figure 7 looks even more like the nearest neighbor solution. In fact your could think of the discount-weighted sum over all of the data as the natural model (as it is easer to reason about) and the nearest neighbors a more efficient approximation (summing only over near points). We also have seen that we can alter the smoothness of our model (which has consequences on model generalization) by changing the steepness of our discount functions. We can also build intermediate models (such as building a nearest neighbor but using discount weighted sums instead of uniform voting in the neighborhoods). We haven’t yet greatly improved on nearest neighbor, but we have identified a couple of obvious avenues for further improvement: mess with the bandwidths (the obvious idea to try and deal with near/far scale) or re-weight the data (as one of the lessons of the nearest neighbor model is that points that are not near a boundary are less important).

Bandwidth Solution

As a digression lets play with bandwidth a bit first. Suppose simultaneously for each training datum we picked an ideal bandwidth or steepness of the Gaussian.

Figure 8: Simultaneous bandwidth model

This simultaneous bandwidth model depicted in figure 8 was determined by picking a simultaneous assignment of Gaussian widths (each training datum gets its own width) that maximizes the log-sum of the category signed model function on the training data (much like the logistic regression maximum likelihood decoding). This differs from typical bandwidth selection problems where all points are given a single common “best bandwidth.” We instead picked a different bandwidth for each point (bandwidths picked to maximize how much of the correct category portion of the sum was correct on each training example). The blue region is reasonable, but does not match the shape of the true concept. Partly this is because there has not been enough training data to have learned a lot about the true shape (for example there is no empirical evidence for circle/blue in the top-left quadrant). The other reason is a bias inherent to this type of model. If one of the bandwidth functions is wider than all of the others (which is almost certain to occur) then it falls slower than all of the others and for points very far away from the center of the training data this one function is most of what remains. Or equivalently: due to the nature of this modeling technique one of the concepts learned is going to be the union of bounded islands and the other concept will grab all points sufficiently far away from the center of the training data. This is strong (and undesirable) bias, but it is a bias also shared by support vector machines with Gaussian kernels (though support vector machines get it from their so-called “dc-term” b not being zero; this will be discussed later). Our bandwidth correction was successful, but frankly the optimization problem we solved to estimate the optimal bandwidths was nasty and would not be something we would advise for a large amount of data.

Data weighting solution (support vector machine)

So instead of messing with the bandwidths let us consider re-weighting the Gaussians in our sum. We will leave the shape of each Gaussian the same but allow each individual training datum to have a unique weight or importance. Perhaps by picking the right weights we can get a better model. By “best” we will mean “best margin” (to be defined later) because with “best margin” as our objective the optimization problem of solving for the best data weights has a particularly beautiful form that can be reliably solved at great scale. This is called a “support vector model” (or support vector machine) and we will describe these ideas in detail later. But first lets see the result. Figure 9 shows the original narrow Gaussians re-summed according to the weights picked by the support vector method (instead of all being 1 as in figure 6).

Figure 9: Narrow Gaussian support vector model

Figure 9 is again an improvement. The figure is formed by taking a signed sum of our discount functions centered at our training points and weighted by the “support vector machine weights.” The signs are picked with one class encoded as +1 and the other as -1. The colors red and blue are picked if the sum is above or below a constant “b” called “the dc-term” (part of the support vector solution). The solution again looks okay: all the training points are in the correct color region (and you can’t hope to interpolate let alone extrapolate if you can’t even reproduce your training concept). Also, as with the bandwidth model, contours are set sensibly (steep/dense where categories are near each other, shallow/sparse where things are safe). We see the same inductive bias: one of the concepts is the union of bounded islands (this will always going to be the case unless the model picks b=0). This bias is not obvious from the kernel choice, as it is hidden in the dc-term of the support vector fitting equations (it is not part of the kernel, but can be defeated by choosing unbounded kernels). An interesting empirical fact about support vector models is they tend to work well with a larger bandwidth. For example if we use a wider Gaussian we get an even more convincing model (see Figure 10).

Figure 10: Wide Gaussian support vector model

In figure 10 we have also called out one of the important features of support vector machines. In the literature all datums are called “vectors” (as they are represented in coordinates) and a subset of these are called the support vectors. In figure 10 we have drawn the 4 training examples that turn out to be support vectors as large shapes and all other training examples as small shapes. The support vectors are the datums with non negligible weights. The learned model is a function of the support vectors only. So after fitting we can discard the other 16 training points. This is similar to the fact that nearest neighbor also only needs the training examples that are supporting its model boundary.

Notice that at this point the support vector models are not “magically” better, they in fact look less like the truth we are trying to learn than either the nearest neighbor models or the un-weighted sum models. To fix this we need to do some “kernel shopping” or find functions that better respect what we are trying to model. With the right functions support vector machines can do a very good job at learning the concept (but knowing “the right functions” is a huge hint as to the what the concept is. There is nothing wrong with using a hint but we do have to a method to produce the hint or have a reasonable number of hints to try from.

The “sum over everything with the same weight” solutions from the earlier section are very similar to Naive Bayes which also sums over all matching features with no weight adjustment. The support vector “use the same model but pick better weights” stands over these sum models in very much the same way Logistic Regression stands over Naive Bayes. In fact the optimization problems are very related. If we consider weights over features or coordinates as “primal” and weights over data items as “dual” we can informally say something like: support vector machines optimize by working over dual variables and inspecting for primal weights, and logistic regression works over primal variables and inspects for data weights. But this is a bit vague.

At this point we need to get a bit more explicit and precise.

Precise functions used

In general we will write our training data as a sequence of n-vectors. Our points will be named u(1) … u(m). For each i = 1..m we also know which category the training point is labeled with. We will encode this as y(1) … y(m) where each y() is either +1 or -1 depending on the training label. So the example we have been working has n=2 and m=20. Let z represent a n-vector we wish to classify.

Figure 6 was a picture of the sign of the function:

Figure 7 was a picture of the sign of the function:

Figure 8 was a picture of the sign of the function:

Figure 9 was a picture of the sign of -b plus the function:

Figure 10 was a picture of the sign of -b plus of the function:

All of this repetition is to emphasize the commonality of the models. They are all of the form:

About Kernels

A lot of awe and mysticism is associated with kernels, but we think they are not that big a deal. Kernels are a combination of two good ideas, they have one important property and are subject to one major limitation. It is also unfortunate that support vector machines and kernels are tied so tightly together. Kernels are the idea of summing functions that imitate similarity (induce a positive-definite encoding of nearness) and support vector machines are the idea of solving a clever dual problem to maximize a quantity called margin. Each is a useful tool even without the other.

The two good ideas are related and unfortunately treated as if they are the same. The two good ideas we promised are:

1. The “kernel trick.” Adding new features/variables that are functions of your other input variables can change linearly inseparable problems into linearly separable problems. For example if our points were encoded not as u(i) = (x(i),y(i)) but as u(i) = (x(i),y(i),x(i)*x(i),y(i)*y(i),x(i)*y(i))

we could easily find the exact concept ( y(i) > x(i)*x(i) which is now a linear concept encoded as the vector (0,1,-1,0,0).

2. Often you don’t need the coordinates of u(i). You are only interested in functions of distances ||u(i)-u(j)|^2 and in many cases you can get at these by inner products and relations like ||u(i)-u(j)||^2 = <u(i),u(i)> + <u(j),u(j)> – 2<u(i),u(j)> .

We will expand on these issues later.

The important property is that kernels look like inner products in a transformed space. The definition of a kernel is: there exists a magic function phi() such that for all u,v:

.

And this brings us to the major limitations of kernels. The phi() transform can be arbitrarily magic except when transforming one vector it doesn’t know what the other vector is and phi() doesn’t even know which side of the inner product it is encoding. That is kernels are not as powerful as any of the following forms:

• snooping (knowing the other):
  
• positional (knowing which part of inner product mapping to):
  
• fully general:
  

Not everything is a kernel.

Some non-kernels are:

1. k(u,v) = -c for any c>0

A consequence of the fact that no negative number can be written as a sum of squares or as the limit of a sum of squares in the reals.

2. k(u,v) = ||u-v||
   

   This can be shown as follows. Suppose k(u,v) is a kernel with k(u,u) = 0 for all u (which is necessary to try and match ||u-v|| as ||u-u|| = 0 for all u). By the definition
   of kernels k(u,u) = < phi(u), phi(u) > for some real vector valued function phi(.). But, by the properties of the inner real inner product <.,.>, this means phi(u) is
   the zero vector for all u. So k(u,v) = 0 for all u,v and does not match ||u-v|| for any u,v such that u ≠ v.

There are some obvious kernels:

1. k(u,v) = c for any c ≥ 0 (non-negative constant kernels)
2. k(u,v) = < u , v > (the identity kernel)
3. k(u,v) = f(u) f(v) for any real valued function f(.)
4. k(u,v) = < f(u) , f(v) > for any real vector valued function f(.) (again, the definition of a kernel)
5. k(u,v) = transpose(u) B v where B is any symmetric positive semi-definite matrix

And there are several subtle ways to build new kernels from old. If q(.,.) and r(.,.) are kernels then so are:

1. k(u,v) = q(u,v) + r(u,v)
2. k(u,v) = c q(u,v) for any c ≥ 0
3. k(u,v) = q(u,v) r(u,v)
4. k(u,v) = q(f(u),f(v)) for any real vector valued function f(.)
5. k(u,v) = lim_{k-> infinity} q_k(u,v) where q_k(u,v) is sequence of kernels and the limit exists.
6. k(u,v) = p(q(u,v)) where p(.) is any polynomial with all non-negative terms.
7. k(u,v) = f(q(u,v)) where f(.) is any function with an absolutely convergent Taylor series with all non-negative terms.

Most of these facts are taken from the excellent book: John Shawe-Taylor and Nello Cristianini’s “Kernel Methods for Pattern Analysis”, Cambridge 2004. We are allowing
kernels of the form k(u,v) = < phi(u), phi(v) > where phi(.) is mapping into an infinite dimensional vector space (like a series). Most of these facts can be checked by
imagining how to alter the phi(.) function. For example to add two kernels just build a larger vector with enough slots for all of the coordinates for the vectors encoding
the phi(.)’s of the two kernels you are trying to add. To scale a kernel by c multiply all coordinates of phi() by sqrt(c).

Multiplying two kernels is the trickiest. Without loss of generality assume q(u,v) = < f(u) ,f(v) > and r(u,v) = < g(u), g(v) > and f(.) and g(.) are both mapping into the same finite dimensional vector space R^m (any other situation can be simulated or approximated by padding with zeros and/or taking limits). Imagine a new vector function p(.) that maps into R^{m*m} such that p(z)_{m*(i-1) + j} = f(z)_i g(z)_j . It is easy to check that k(u,v) = < p(u) , p(v) > is a kernel and k(u,v) = q(u,v) r(u,v). (The reference proof uses tensor notation and the Schur product, but these are big hammers mathematicians use when they don’t want to mess around with re-encoding indices).

Note that one of the attractions of kernel methods is that you never have to actually implement any of the above constructions. What you do is think in terms of sub-routines (easy for computer scientists, unpleasant for mathematicians). For instance: if you had access to two functions q(.,.) and r(.,.) that claim to be kernels and you wanted the product kernel you would just, when asked to evaluate the kernel, just get the results for the two sub-kernels and multiply (so you never need to see the space phi(.) is implicitly working in).

This implicitness can be important (though far too much is made of it). For example the Gaussians we have been using throughout are kernels, but kernels of infinite dimension, so we can not directly represent the space they live in. To see the Gaussian is a kernel notice the following:

.

And for all c ≥ 0 each of the three terms on the right is a kernel (the first because the Taylor series of exp() is absolutely convergent and non-negative and the second two are the f(u) f(v) form we listed as obvious kernels). The trick is magic, but the idea is to use the fact that Euclidian squared distance breaks nicely into dot-products ( ||u-v||^2 = < u, u &gt + < v, v &gt – 2 < u , v >) and exp() converts addition to multiplication. It is rather remarkable that kernels (which are a generalization of inner products that induce half open spaces) can encode bounded concepts (more on this later when we discuss the VC dimension of the Gaussian).

Back to Support Vector Machines

We can now restate what a support vector machine is: it is a method for picking data weights so that the modeling function for a given kernel has a maximum margin. The margin is the minimal distance between a training example and the model’s boundary between blue and red. Notice the 4 support vectors in figure 10 are all not “tight up against the boundary,” but are all the same (minimal distance) from the boundary. This distance is called the margin and the area near the boundary is obviously a place the model has uncertainty (so it makes sense to keep the boundary as far as possible from the training data). The great benefit of the support vector machine is that with access only to the data labels and the kernel function (in fact only the kernel function evaluated at pairs of training datums) the support vector machine can quickly solve for the optimal margin and data weights achieving this margin.

For fun lets plug in new kernel called the cosine kernel:

This kernel can be thought of as having a phi(.) function that takes a vector z and adds an extra coordinate of sqrt(c) and then projects the resulting vector onto the unit sphere. This kernel induces concepts that look like parabolas, as seen in figure 11.

Figure 11: Cosine example concept

Figure 12 shows the sum-concept (add all discount functions with same weight) model. In this case it is far too broad (averaging of the cosine concepts which are not bounded concepts like the Gaussians) creates an overly wide model. The model not only generalizes poorly (fails to match the shape of the truth diagram) it also gets some of its own training data wrong.

Figure 12: Cosine kernel sum model

And figure 13 shows the excellent fit returned by the support vector machine. Actually an excellent fit determined by 4 support vectors (indicated by larger labels). Also notice the support vector machine using unbounded concepts generated a very good unbounded model (unbounded in the sense that both the blue and red regions are infinite). By changing the kernel or discount functions/concepts we changed the inductive bias. So any knowledge of what sort of model we want (one class bounded or not) should greatly influence our choice of kernel functions (since the support vector machine can only pick weights for the kernel functions, not tune their shapes or bandwidths).

Figure 13: Cosine Support Vector model

Another family of kernels (which I consider afflictions) are the “something for nothing” kernels. These are kernels of the form:

Figure 14: Squared magic kernel support vector model

If you want higher order terms I feel you are much better off performing a primal transform so the terms are available in their most general form. That is re-encode the vector u = (x,y) as the larger vector (x,y,x*y,x*x,y*y). You have to be honest: if you are trying to fit more complicated functions you are searching a larger hypothesis space so you need more data to falsify the bad hypotheses. You can be efficient (don’t add terms you don’t think you can use as they make generalization harder) but you can’t get something for nothing (even with kernel methods).

Back to margin

Large margin (having a large distance between the decision boundary and all training examples) is a good idea. At the simplest level it means anything close to a training data point gets classified correctly (so the model is immune to an amount of fuzz proportional to the margin width).

We get the following remarkable generalization theorem (this one is the hard SVM version, we weaken the result a bit to make it easier to state). Suppose we train a model using kernel k(.,.) on m training examples u(1),…,u(m) with training labels y(1)…y(1). Further assume the hidden true concept is separable with respect to our kernel and data distribution with a margin of at least w (that is for any sample we draw we can find a SVM model that classifies all of the training data correctly and sees a margin of at least w). Or more simply: assume w is behaving like a constant with respect to m (i.e. it is not going down as we increase sample size). Then with probability at least 1-d the generalization error (that is error seen on new test points not seen during training, but drawn from the same distribution as the training examples) is no more than:

.

If we further assume the expected value k(u,u) under the training distribution exists (i.e. we don’t have too many examples of very high kernel norm according to the distribution we are training with respect to) and is stable (not taking a lot of value from rare instances) then we can read off the meaning of this upper bound. Assuming we can always build a model from the training data of margin at least w then the generalization error (excess error we expect to see on classifying similar points) is falling at a rate proportional to the square root of the amount of training data we have. This is in one sense amazing- we have a bound on how well we are going to do on data we have not seen. And this bound is also in some sense “best possible” because we need around this much data to even confirm such an accuracy (see What is a large enough random sample? ). Also notice how the dimension of the implicit features space (which can in fact be infinite) does not enter into the bound.

However, the bound is also some sense to be expected. It depends on the following strong assumptions (usually not remembered or stated):

1. Test and training data must be generated in the same manner (i.e. come from the same distribution, this is usually stated but not obeyed).
2. We must not have collapsing margin as the number of data points goes up (so there must actually be a moat between the positive and negative examples such as portrayed in figure1 where the blurry boundary was a region we never generated training or test examples from).
3. The expected value of k(u,u) must be bounded for examples drawn from our training distribution. For example this is not true even for the identity kernel if x is just numbers drawn from a Cauchy distribution.

Also it is a posterior estimate of expected generalization error. That is, we only get a bound after we plug in some facts found from fitting a model to the training data (the margin and the expected value of k(u,u)). It is not like the famous prior bounds based on VC dimension that depend only on the structure of the kernel k(,) and not the distribution of training data or labels (beyond the concept being expressible by the kernel, which is itself a posterior fact). The thing to remember for all posterior estimates is you can get such estimates at a small expense in statistical efficiency (i.e. you waste some data) and a moderate computational blow-up by cross validation (and with essentially no math and many fewer assumptions). In fact the cross validation and hold-out estimates may be much tighter than calculated bounds (and are certainly easier to explain).

When we say we weakened the result we mean we have added some stated some conditions that are not strictly needed for the bound to be true. One of these assumptions is that the margin is not collapsing (the bound remains true even for a collapsing margin; it is just not useful). We added this assumption because most readings of the bound (or presumed applications) implicitly assume the bound remains useful (which means something near our additional assumptions must also hold). The reader is really lured to think of the margin w as a constant depending only on the kernel when in fact it depends on the kernel, data distribution and number of training examples. For example: consider a very simple one dimensional classification problem: learning the concept x ≥ 0 from labeled training data drawn uniformly from the interval [-0.5,0.5] using a Gaussian kernel. The support vector models are going to be essentially picking two neighboring points in the observed training sample that have opposite labels and saying the concept boundary is between them. In this simple case the margin is collapsing: as m (the number of training examples increases) the expected margin width is O(1/m). This renders the bound calculated above useless (drives it above 1 as the number of training examples increases). The model is good but the bound is not useful. This is because the training distribution is discovering points closer and closer to the concept boundary (a necessary discontinuity) as the number of training examples grows. This is why we had to add the additional (weakening) assumption that not only do we need to assume the concept to be learned is separable with respect to the kernel we are using, but we must further assume the data-distribution (determining where test and training data come from) stays some minimal distance away from the decision surface (i.e. together the data distribution and underlying concept have a moat, as in our figure 1 example). We will phrase this as “the concept is wide margin separable with respect to the training distribution” (implicitly we need to know the kernel, but it is the training distribution that is critical). This is a very strong assumption (not true for all classification problems) but is often true when the classification task is to “un-mix a mixture” (the observed data is coming from two different sources and the training labels are which source they come from, in this case you often do have a margin). Obviously a soft-margin SVM deals better with this (some of the issue is due to our use of hard-margin)- but it is an interesting exercise to think why you would expect your learned model to have a large margin if your underlying concept and training data do not.

The result remains important: support vector machines are in some sense an optimal learning procedure (up to some constants they achieve as tight a generalization error as can even be confirmed for a given training set size). Another strong point of the support vector result is the result applies even for kernels with unbounded VC dimension (such as the Gaussian kernel).

The Gaussian (or radial) kernel again

We said the Gaussian kernel was of infinite VC dimension. But we have not shown why it is true. The fact that the only encoding we could think of off the top of our head was a power-series or an limit does not guarantee there isn’t some more clever encoding that easily demonstrates that the Gaussian kernel is of bounded VC dimension (like the identity kernel and cosine kernels are).

However we can see the Gaussian kernel can not have finite VC dimension because it can place an arbitrary number of bounded patches in the plane. Thus we can build arbitrarily large sets (by placing all points far apart) that can be shattered (thus falsifying any bounded VC dimension).

One of the wonders of the support vector method is that the theory works even in this situation.

Back to SVM

Support vector machines seem nearly magical in their power to pick “best weights.” However, as we have seen, these best weights may not always be much better than typical good weights (for instance: using all uniform weights). Also it is very important to remember the support vector machine is picking the best weighting of a sum of already picked kernels. It is not picking the best kernel shape or adjusting things like bandwidth (you must pick the best kernel ahead of time or try several kernels to find a good one). You must remember support vector machines can only directly adjust dual weights (or relative training data weighting) this can affect some changes in hypothesis shape but the support vector machine can not directly implement arbitrary changes in hypothesis shape like a nearest neighbor or a parameterized primal method can (like logistic regression with the kernel trick of adding interaction variables).

Support vector machines are to be much preferred to other combiners like Boosting. This is because support vector machines have explicit stopping criteria (conditions known to be true at the optimal solution that characterize the optimal solution) and explicit regularization (control of over fitting). Boosting relies a path argument (“do the computation in these stages”) so it is much harder to reason about and frankly the folklore that “early stop prevents over fitting in boosting” is just false (you are much better off explicitly regularizing).

It is also lore that random kernel transformations (like power series over the cosine kernel) are magic. They can make data separable but they are unlikely to meet the (unstated but critical) conditions of the support vector theorem (margin being wide, margin not filling in and expectation of k(.,.) being small).

A comment on optimization

In this writeup we have skipped the most beautiful part of support vector machines- the form of the optimization problem used to solve them. We strongly suggest consulting John Shawe-Taylor and Nello Cristianini’s “Kernel Methods for Pattern Analysis”, Cambridge 2004 for this. The writing is dense but accurate. You can literally paste their formulas into a general solver and they work.

Conclusion

We have shown how kernel methods and support vector machines fit in a sequence of machine learning methods:

1. nearest neighbor models
2. uniform sum models
3. support vector machines.

We also showed a number of examples of kernels and non-kernels. It is good to be able to quickly remember the constant “-1″ is not a kernel and the constant “1″ is a kernel (though not a very useful one).

It is our argument that support vector machines are “dual methods” (as they work in weights over the data instead of weights over coordinates in the features space) and the move from the primal algorithm Naive Bayes to Logistic regression is similar to the move from uniform kernel sums to support vector machines.

Also, if you want to directly manipulate shape (like picking bandwidth) you must in some sense use a primal method (this isn’t what support vector machines are for).

We end with: If you have some feeling of both what a method can and can not do then you have some understanding of the method.

Related posts:

1. A Personal Perspective on Machine Learning
2. Do your tools support production or complexity?
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)

Stochastic gradient descent is in fact an appropriate method for big data. For example: if our model complexity is held constant and our data set size is allowed to grow; then stochastic gradient descent will achieve its convergence condition before it even completes a single random order traversal of the data. However, stochastic gradient descent has a control called the learning rate and one can easily imagine a series of problems that require the learning rate to be set arbitrarily slow. For example a data set formed as the union of very many “typical” examples where a given variable is independent of the outcome and small minority of “special” examples where the same variable helps influence the outcome presents a problem. Training on the “typical” examples causes the stochastic gradient descent method to perform a random walk on the given variable coefficient. So the learning rate must be slow enough that the expected drift does not swamp out the rare contributions from the “special” examples (meaning the learning rate must slow roughly proportionally to the square root of the ratio of the typical to special examples).

Not too much must be made of artificial problems designed to slow stochastic gradient descent. The traditional Fisher scoring (or the Newton/Raphston method) can simply be killed by specifying a problem with a great number of levels for categorical variables. In this case traditional methods have to solve a linear system that can in fact be much larger than the entire data set (causing representation, work and numeric stability problems). So it takes little imagination to design problems that kill the traditional methods. Other intermediate complexity methods (like conjugate gradient) avoid the storage size problem; but can require a many more passes through the training data.

There is a common situation where Fisher scoring makes good sense: you are trying to fit a relatively simple model to an enormous amount of data (often to predict a rare event). One could sub-sample the training data to shrink the scale of the problem- but this is a case of the analyst being forced to accede to poor tools. What one would naturally want is a training method that can fit reasonable sized models (that is models with a reasonable number of variables and levels) onto enormous data sets. The software package R can work with fairly large data sets (in the gigabytes range) and has some parallel flavors, but R is mostly an in-memory system. It is appropriate to want a direct method that both “works out of core” (i.e. in the terabytes and petabytes ranges), parallelizes to hundreds of machines (using current typical infrastructure- like a Hadoop cluster) and is exact (without additional parameters like learning rate).

We demonstrate here an example implementation in Java for both single machine “out of core” training (allowing filesystem sized datasets) and MapReduce style parallelism (allowing even larger scale). The method also includes the problem regularization steps discussed in our recent logistic regression article. The code (packaged in: WinVectorLogistic.Hadoop0.20.2.jar ) is being distributed under the GNU Affero General Public License version 3. This is an open source license that (roughly) requires (among other things) redistribution of source code of systems linked against the licensed project to anyone receiving a compiled version or using the system as a network service. The license also promises no warranty or implied fitness. The distribution is a standalone runnable Jar (source code and license inside the jar) and is the minimal object required to run on Hadoop (which is itself a Java project). More advanced versions of the library (with better linear algebra libraries, better problem slice control, unit tests, JDBC bindings and with different license arrangements) can be arranged from the code owners: Win-Vector LLC. This jar was built for Apache Hadoop version 0.20.2 (the latest version Amazon Elastic Map Reduce runs at this time) and we use as many of the newer interfaces as possible (so the code will run against the current Hadoop 0.21.0 if re-built against Hadoop 0.21.0, the jar can not switch versions without being re-built due to how Hadoop calls methods).

For our example we will work on a small data set. The code is designed to pass through data directly from disk, storing only the Fisher structures- which require storage proportional to the square of the number of variables and levels but is independent of the number of data rows. The data format is what we call “naive TSV” or “naive tab separated values.” This is a file where each line has exactly the same number of values (separated by tabs) and the first line of the file is the header line naming each column. This is compatible with Microsoft Excel and R with the proviso that this file format does not allow any sort of escapes, quoting or multiple line fields. Our data set is taken from the UCI machine learning database ( data, description ) and converted into the naive TSV format (split into training and testing subsets: uciCarTrain.tsv, uciCarTest.tsv).

The first few lines of the training file are given here:

buying	maintenance	doors	persons	lug_boot	safety	rating
vhigh	vhigh	2	2	small	med	FALSE
vhigh	vhigh	2	2	med	low	FALSE
vhigh	vhigh	2	2	med	med	FALSE

The first experiment is to use the Java program standalone (without Hadoop) to train a model. The method used is Fisher scoring by multiple passes over the data file. Only the Fisher structures are stored in memory- so in principle the data set could be arbitrarily large. To run the logistic training program download the files WinVectorLogistic.Hadoop0.20.2.jar and uciCarTrain.tsv . You will also need some libraries ( commons-logging-*.jar and commons-logging-api-*.jar , and sometimes hadoop-*-core.jar and log4j-*.jar ) from the appropriate Hadoop distribution. Before running the code you can examine the source (and re-build the project using an IDE like Eclipse) by extracting the code in an empty directory using the Java jar command:

jar xvf WinVectorLogistic.Hadoop0.20.2.jar

To run the code type at the command line (all in a single line, we have inserted line breaks for clarity, we are also assuming you are using a Unix style shell on Linux, OSX or Cygwin on Windows):

java -cp WinVectorLogistic.Hadoop0.20.2.jar:commons-logging-1.0.4.jar:commons-logging-api-1.0.4.jar
   com.winvector.logistic.demo.LogisticTrain
   file:uciCarTrain.tsv "rating ~ buying + maintenance + doors + persons + lug_boot + safety" model.ser

The portion of interest is the last three arguments:

• file:uciCarTrain.tsv : The URI pointing to the file containing the training data.
• “rating ~ buying + maintenance + doors + persons + lug_boot + safety” : The formula specifying that rating will be predicted as a function of buying, maintenance, doors, persons, lug_boot and safety.
• model.ser : Where to write the Java Serialized model result.

After that we can run the scoring procedure on the held-out test data:

java -cp WinVectorLogistic.Hadoop0.20.2.jar:commons-logging-1.0.4.jar:commons-logging-api-1.0.4.jar
   com.winvector.logistic.demo.LogisticScore
   model.ser file:uciCarTest.tsv scored.tsv

In this case the last three arguments are:

• model.ser : Where to read the Java Serialized model from.
• file:uciCarTest.tsv : The URI pointing to the file to make predictions for.
• scored.tsv : Where to write the predictions to.

The first few lines of the result file are:

predict.rating.FALSE	predict.rating.TRUE	buying	maintenance	doors	persons	lug_boot	safety	rating
0.9999999999999392	6.091299561082107E-14	vhigh	vhigh	2	2	small	low	FALSE
0.9999999824028766	1.759712345446162E-8	vhigh	vhigh	2	2	small	high	FALSE

These lines are just lines from the file uciCarTest.tsv (same format is uciCarTrain.tsv) copied over with the addition of the first two columns that show the modeled probabilities of rating acceptable being FALSE or TRUE. The accuracy of the prediction is computed and written into the runlog if the data had the rating outcomes in it (else we just get a file of predictions- which is the usual application of machine learning).

The details of running the Hadoop versions of the same process depend on the configuration of your Hadoop environment. Just unpacking the 0.20.2 version of Hadoop will let you try the single-machine version of the MapReduce Logistic Regression process (which will be much slower than the standalone Java version). To run the training step the Hadoop command line is as follows (notice this time we do not have to specify the logging jars as they are part of the Hadoop environment):

hadoop-0.20.2/bin/hadoop jar WinVectorLogistic.Hadoop0.20.2.jar
   logistictrain
   uciCarTrain.tsv "rating ~ buying + maintinance + doors + persons + lug_boot + safety" model.ser

And the scoring procedure is below:

hadoop-0.20.2/bin/hadoop jar WinVectorLogistic.Hadoop0.20.2.jar
   logisticscore
   model.ser uciCarTest.tsv scoredDir

The only operational differences are that the results are written into the file scoredDir/part-r-00000 (as is Hadoop convention) instead of scored.tsv (and an extra “offset” column is also included) and data is handled in Files (to allow Hadoop Paths to be formed) instead of URIs. The Hadoop training and test steps are able to run in this manner because we have constructed WinVectorLogistic.Hadoop0.20.2.jar as an executable jar file with the class com.winvector.logistic.demo.DemoDriver as the class to execute. This class uses that standard org.apache.hadoop.util.ProgramDriver pattern to run our jobs under the org.apache.hadoop.util.Tool interface. This means that the standard Hadoop generic flags for specifying cluster configuration will be respected.

The big benefit of all of this packaging is: if this command is run on a large Hadoop cluster (instead of on a single machine) then the input file could be split up and processed in parallel on many machines. The easiest way to do this is to use Amazon.com’s Amazon Elastic MapReduce. This service (used in conjunction with S3 storage and EC2 virtual machines) allows the immediate remote provisioning and execution on a version 0.20.* Hadoop cluster. To demonstrate this service we created a new S3 Bucket named wvlogistic. Into wvlogistic we copied our jar of our code compiled against Hadoop 0.20.2 APIs ( WinVectorLogistic.Hadoop0.20.2.jar ) and a moderate sized synthetic training data set ( bigProb.tsv, created by running: java -cp WinVectorLogistic.Hadoop0.20.2.jar com.winvector.logistic.demo.BigExample bigProb.tsv ). Once this has been set up (and you have signed up for the Amazon Elastic MapReduce credentials) you can run the training procedure from the Amazon web UI. In five steps (following the direcitons found in Tutorial: How to Create and Debug an Amazon Elastic MapReduce Job Flow ) the job can be configured and launched.

First: press “Crate New Job Flow” and choose a job name, check “Run your own application” and select “Cusom Jar”.

Step 1/5

Second: specify the location of the jar in your Bucket and give the command line arguments (prepending S3 paths with “s3n://”).

Step 2/5

Third: select the type and number of machine instances you want, run without and EC2 key pair, enable logging and send the log back to your S3 bucket.

Step 3/5

Fourth: add the default bootstrap action of configuring the Hadoop cluster.

Step 4/5

Fifth: confirm and launch the job.

Step 5/5

When the job completes transfer the result ( bigModel.ser ) back to your local system and you have your new map reduced produced logistic model. We can confirm and use the model locally with a Java command similar to our earlier examples:

java -cp WinVectorLogistic.Hadoop0.20.2.jar:commons-logging-1.0.4.jar:commons-logging-api-1.0.4.jar:hadoop-0.20.2-core.jar:log4j-1.2.15.jar
   com.winvector.logistic.demo.LogisticScore
   bigModel.ser bigProb.tsv bigScored.tsv

Be aware that at this tens of megabytes scale there is no advantage in running on a Hadoop cluster (versus using the stand-alone program). At moderate scale parallelism may not even be attempted (due to block size) and the costs of data motion can overcome the benefit of parallel scans. The biggest gain is being able to train many models from many gigabytes of data on a single machine without sub-sampling. While we have the ability to build a logistic model at “web scale” (terabytes or petabytes of data) you would not want to use the MapReduce calling pattern until you had a web-scale amount of training data.

The point of this exercise was to take a solid implementation of regularized logistic regression article and use the decomposition into the ” Statistical Query Model” (as suggested in the NIPS paper “Map-Reduce for Machine Learning on Multicore”) to quickly get an intermediate sophistication machine learning method (more sophisticated than Naive Bayes, less sophisticated than Kernelized Support Vector Machines) working at large (beyond RAM) scale. Briefly: most of the technique is in an interface that considers the mis-fit, gradient if mis-fit and hessian of mis-fit as a linear (summable) function over the data. Or in the “book’s worth of preparation so we can write the result in one line” paradigm: all of the machinery we have been discussing is support so the following summable interface (part of the source code we are distributing) can be used to do all of the work:

Summable Interface

Of course once you have the framework up that makes one non-trivial task easy you have likely made many other non-trivial tasks easy.

We hope this demonstration and examining the source code in our WinVectorLogistic.Hadoop0.20.2.jar will help you find ways to tackle your large data machine learning problems.

Code License:

Packages com.winvector.*, extra.winvector.*
Code for performing logistic regression on Hadoop.
Copyright (C) Win Vector LLC 2010 (contact: John Mount jmount@win-vector.com).
Distributed under GNU Affero General Public License version 3 (2007, see http://www.gnu.org/licenses/agpl.html ).
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, only version 3 of the
License.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see .
(Source code in jar, see also http://www.win-vector.com/blog/2010/12/large-data-logistic-regression-with-example-hadoop-code/ )

usage: com.winvector.logistic.LogisticTrain
 -formula <arg>      formula to fit
 -inmemory           if set data is held in memory during training
 -resultSer <arg>    (optional) file to write seriazlized results to
 -resultTSV <arg>    (optional) file to write TSV results to
 -trainClass <arg>   (optional) alternate class to use for training
 -trainHDL <arg>     XML file to get JDBC connection to training data
                     table
 -trainTBL <arg>     table to use from database for training data
 -trainURI <arg>     URI to get training TSV data from

usage: com.winvector.logistic.LogisticScore
 -dataHDL <arg>      XML file to get JDBC connection to scoring data table
 -dataTBL <arg>      table to use from database for scoring data
 -dataURI <arg>      URI to get scoring data from
 -modelFile <arg>    file to read serialized model from
 -resultFile <arg>   file to write results to

Related posts:

1. The equivalence of logistic regression and maximum entropy models
2. The Simpler Derivation of Logistic Regression
3. Learn Logistic Regression (and beyond)

Hal 9000

There have been vast improvements in artificial intelligence. Much of the improvement has been driven by the engineering effects of Moore’s Law (resulting in my mobile phone’s processor having 12 times the clock speed and over 32 times the memory of an $8 million Cray 1 super computer) and significant machine learning research results. These machine size changes happened during the productive careers of many researchers, so ideas are often evaluated at a series of radically different machine capabilities and data scales.

Cray 1

von Neuman himself commented that scale was a major limiting factor in early computers. He asked the question how you could be expected to achieve anything significant even from a roomful of geniuses if (as with his early computers) all notes, communication and memory were limited to less than a single typed page. von Neuman’s comment stands in contrast to science fiction scientists and early boosters of artificial intelligence who always seem to be in awe of their own creations. Computers are certainly much larger- but we need to be humble and put off deciding if we are yet in the era of large computers (compared to human or animal brains). Everything we are doing now may still just be artificial intelligence’s pre-history and prologue. Feynman in his lectures on computation mentions that RNA transcription can be estimated to take around 100 kT of energy to transcribe a bit while a transistor may easily use 100,000,000 kT energy units to switch states. This means for the amount of heat the human head dissipates (energy supply and heat dissipation are rapidly becoming the most relevant measures of computational power) you could do a million times more work using RNA techniques (if you knew how) than with transistors. So computers may not yet be what we should call large (though they are likely getting there). What we currently call “datacenters” are in fact block sized computers (consuming an enormous amount of energy and dissipating a huge amount of heat).

A datacenter (or a block sized computer)

Not all improvements in machine intelligence have come from (or are to come from) improvements in hardware. Many of the improvements came from machine learning research results and these are what I will outline below.

Early machine learning algorithms were driven by analogy. This led us to perceptrons (1957, fairly early in the history of computer science) and neural nets. These methods have their successes but were largely over used and developed before researchers developed a good list of desirable properties of a machine learning method.

Neural Net diagram

These methods live on but are, in my opinion, not currently competitive. Some of their important ideas and contributions have been revived from time to time, such as the online update rules becoming what we now call stochastic gradients.

A list of (often incompatible) desirable properties of a machine learning algorithm is the following:

• Able to represent complicated functions
• Good generalization performance (quality predictions on data not seen during training)
• Unique optimal model for a given set of data and feature definitions
• Efficient and well characterized solution method
• Consistent summary statistics
• Preference for simple models

We divert from this list for a bit of background and context.

The neural net was largely celebrated for its ability to represent complex functions and the perceived efficiency of its newer back-propagation based training method (related to the efficient calculation of gradients). The downsides were you never knew if your neural net was the right one (even assuming you had the right features, layout and training data) and could not be sure you were biasing towards simple models that might perform well on novel queries. Great effort was expended in extending neural nets based on the supposition they should work as they were an analogy to how we imagined biological neurons might function. An almost mystic hope was derived from the non-linear nature and special properties of the sigmoid curve (which was in fact a curve already known to statisticians).

Other methods than neural nets also had early success. The field of information retrieval (which was not “sexy” prior to the Web) had huge success since the 1960s with Naive Bayes, Rocchio Classification, and TF/IDF methods. The early success of these methods may have in fact delayed research on current hot research areas such as segmentation and author topic models.

Theoretical computer science initially sought to characterize machine learning methods in non-statistical language. In the 1980s a great amount of ink was spilled on “learning boolean functions.” Papers proving nothing was learnable (by picking a function related to cryptography) alternated with papers proving everything was learnable (for example via amplification techniques like boosting). Generalization of models to new data remained a theoretical problem that was dealt with by appeals to model complexity and MDL (minimum description length). A major breakthrough in characterizing generalization performance was the PAC model (probably approximately correct) framework which finally allowed direct treatment of generalization performance.

We now have enough context to discuss some of the current best of breed machine learning techniques (that address many of the desired properties mentioned above):

• Support Vector Machines
• Kernel Methods
• Logistic Regression
• Maximum Entropy Methods
• Regularization
• Graphical Models
• Conditional Random Fields




Typical SVM maximum margin diagram

Not all of these methods are new (Logistic Regression for example dates from 1925 and is itself based on regression which goes back to Gauss). But the concerns these methods address are all much more statistical than artificial intelligence in nature. For example we don’t suppose that there is some cryptographically obscured combination of features that we need to find to make the best prediction. We instead worry about detecting which features are useful and note that it is a significant (though solvable) problem to correctly use combinations of useful features (phrased as statistical concerns: feature to feature dependencies and higher order interactions). Machine learning has always run where statisticians fear to tread. But more and more often we are seeing that the methods and concerns of statisticians are what are needed to achieve many of the listed desired properties of machine learning models.

The methods I have singled out for praise are very effective and achieve a number of our listed desired properties. For example: both logistic regression and maximum entropy have a unique solution that is easy to find. They are also both consistent with all summaries known during training. That is: if 30% of the positive training data has a feature present then 30% of the data also has the feature present when weighted by the model’s score (so the model score shares a lot of properties with training truth). Support Vector Machines also have well understood solutions and a theory (called maximum margin) that directly addresses generalization (good predictions on new data). Kernel Methods (both as used in SVMs and elsewhere) allow controlled introduction of very complex functions. Graphical Models and Conditional Random Fields also allow the controlled introduction of modeled dependencies in the data.

It is now common to call what was previously thought of as artificial intelligence or machine learning: “statistical machine learning.” This reflects that the kind of prediction and characterization we expect from machine learning algorithms are in fact statistical concerns that we can deal with if we have enough data and enough computational resources.

The current important issues for statistical machine learning include:

• Dealing with very large datasets (driving the return of simpler methods like Naive Bayes)
• Dealing with lack of training data (driving interest in clustering and manifold regularization methods)
• Dealing with unstructured data and text mining (driving interest in information extraction and segmentation via generative models)

Just as Wigner famously wrote about “The Unreasonable Effectiveness of Mathematics” in the 1960s Halevy,Norvig and Pereira write about the “Unreasonable Effectiveness of Data.” They argue that we are in the age of big data (or the age of analysts). Or, as Varian observed: “it is a good time to supply a good complementary to data” (i.e. it is a good time to be an analyst). I would temper this with we are likely in the age of unmarked data and unstructured data. Less often are we asked to automate a known prediction and more often we are asked to cluster, characterize and segment wild data. In my opinion the hard problem in machine learning has moved from prediction to characterization. With enough marked training data (that is data for which we know both the observables and desired outcome) it is now quite possible to use standard techniques and libraries to build a very good predictive model. However, it is still hard to characterize, segment or extract useful information from the wealth of unstructured and unmarked data that is upon us. And this is where a lot of the current research in statistical machine learning is directed.

Or course characterization and clustering have their own infamous history. Rota wrote: “… Or a subject is important, but nobody understands what is going on; such is the case with quantum field theory, the distribution of primes, pattern recognition and cluster analysis.” Artificial intelligence may be moving from areas where computer scientists have over-promised to areas where statisticians have over-promised. But this is not a disaster: the most valuable research tends to be done in hectic times in messy fields, not in calm times in neat fields. And the already large scale adoption of statistical machine learning techniques means there is immediate great client value in even seemingly small improvements in understanding, explanation, documentation, training, tools, libraries and techniques.

Classic attempt to add structure to text

(images from Wikipedia)

Related posts:

1. A Demonstration of Data Mining
2. Kernel Methods and Support Vector Machines de-Mystified
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)

The Local to Global Principle

John Mount¹

Date: November 11, 2009

Abstract:

Contents

• Introduction
• The Examples
  • Web Page Link Analysis
  • Natural Language Processing
  • Machine Learning

• Some Methods
  • Local Methods
  • Globalization Methods

• Conclusion
• Bibliography
• Acknowledgement

Introduction

A common vain hope of computer scientists and algorithm designers is that a domain expert has already “boiled down” a problem to a precise, but unsolved, algorithmic core. On this point the mathematician Gian-Carlo Rota wrote:

One of the rarest mathematical talents is the talent for applied mathematics, for picking out of a maze of experimental data the two or three parameters that are relevant, and to discard all other data. This talent is rare. It is taught only at the shop level.[Rot97, ``A Mathematician's Gossip'']

We describe a useful tool for designing algorithmic applications and solutions which we call “the local to global principle.” The local to global principle is the method of deriving applications and solutions by specifying “local” (and deliberately myopic) heuristics, critiques and methods followed by using a powerful general method to “globalize” this specification into a complete solution.

There are many important problem solving prescriptions and methods of thought already systematically described and taught:

• Bacon’s “New Organon” and Mill’s principles of inductive logic.[Mil02]
• Feynman’s genius method.[Rot97, ``Ten Lessons I Wish I Had Been Taught'']
• Reductionism (top down and bottom up).
• Divide and conquer.[CLRS09]
• Forward deduction, backwards induction.
• Root Cause Analysis.
• Polya’s heuristic and conjecture and prove patterns [Pol71,Pol54a,Pol54b]
• Doron Zeilberger’s “Method of Undetermined Generalization and Specialization.” [Zei95]
• Zbigniew Michalewicz and David B. Fogel’s presentation of evolutionary algorithms.[MF00]

The local to global principle is more of an organizational pattern than “computer aided technique” as no one specific species of software or family of notation is required.

The local to global principle can be identified in a number of previous important applications, but it is not currently an identified principle.² The principle is very general, so any succinct description of it is going to be painfully vague. Instead, we explain the principle by discussing some example applications and methods. For each of our example applications we deliberately use a different globalization technique. The effective algorithmist or practitioner must in fact come to each problem already familiar with a reasonably large set of already known local and global techniques, so we conclude with some appropriate fields of study and preparation.

The local to global principle is divided into two parts: local encoding of the problem followed by a globalization step that uses the encoding. The guiding feature of local encodings is that they are usually easy to compute from the data at hand. Any extension that looks like enumeration, search or optimization is best left to the global step. The local step is essentially the translation of your problem into an abstract language that is ready for the globalization step. In contrast globalization methods are often “off the shelf” in that once you abstract and encode the particulars of your problem you can look for pre-existing useful methods or software to finish your solution. The idea of globalization is to find a best overall or global compromise between competing local criteria. The local step does not so much have to avoid conflicts but instead “price them.” There is also an important trade-off that sophisticated local techniques allow the use of simpler globalization methods and more powerful globalization methods allow the use of simpler local techniques.

The Examples

To demonstrate the breadth of the local to global principle we choose a diverse collection of example applications: web page link analysis, natural language processing and machine learning. For each example application we will set up the problem, introduce a reasonable set of local criteria and pick an appropriate globalization technique. We will favor finishing each example without describing the globalization technique in detail, as this would distract from our point and is best left to the given references. These examples are previously solved problems, our contribution is demonstrating the shared underlying principle.

Web Page Link Analysis

For our first example application we demonstrate web page link analysis in the form of the famous PageRank score.[PBMW98]

One of the many good ideas leading up to the early Google search engine was the design of a non-text based measure of importance or interestingness of web pages. A search engine that could fold “interestingness” or popularity into its notion of relevance could better sort important pages into the search user’s view. When the web got so large that there were many pages that were exact matches to any common user query popularity became a critical consideration. A link based notion of popularity exploits what is important about the web (the link structure, for example see [Kle97]) and avoids having to depend on a lot of natural language understanding technology. This technique also uses authority outside of the given page, so has some hope at being resistant (though not immune) to web-spam.

Taken all at once, the task of designing a score of page importance is a daunting task. However, by working in stages (as the local to global principle prescribes) we can quickly derive interesting scores including the famous PageRank score. We start with the idea that popularity (or the amount of web traffic a page receives) is (loosely) correlated with importance. So for our first approximation step we decide to try to estimate popularity (or web traffic) and use this estimate as our importance score. Accurately estimating web traffic is itself a hard problem and a big industry (just a few of the major companies involved in this are: Google/Urchin, Quantcast, Nielsen, comScore, Alexa, Hitwise and LookSmart). For our second approximation step we are going to try and estimate popularity from the link structure⁴ of the web (using no other measurements or historic data) and use this as our score. This link based estimate is unlikely to completely reproduce real web surfing patterns, but it is very interesting in its own right and has been proven in the market to be a useful score.

Now the problem is to try to estimate the popularity of a web page from the link structure of the web. We claim: we can generate a useful (but not necessarily accurate) estimate of web traffic from the web’s link structure alone. Consider Figure 1 where we have a universe of three web pages A,B and C that link to each other in the pattern illustrated by what is called a graph⁵

Figure 1: A set of Mutually Linked Web Pages

In Figure 1 we can consider each link to another page as evidence the other page is interesting or popular. One idea is to simulate a very simple web surfer who clicks on the links on a page uniformly at random. This is called “the random surfer model” and even a model this simple allows us to read some useful information from the link structure of the web. For instance, we could ask what fraction of their time the random surfer spends on each web page, with an eye to the idea that the pages the random surfer visits more often are the more important ones. Let denote the proportion of time the random web surfer spends on page A (and define and similarly). While we do not know any of
or we can derive some relationships between them by inspecting the link graph:

The first equation is just reading from the graph that: all visits on page-A must come from pages B and C, half of the visitors on page-B continue on to A and all of the visitors on page-C continue on to A. The second and third equations are the appropriate summaries of how traffic is routed to pages B and C. We can insist that
as we want these numbers to represent the fraction of time the random web surfer spends on each page. A more sophisticated model would add more features⁶ to get a more useful result.

It turns out we have already encoded enough local rules to completely determine
and . In this example application an algorithmist already familiar with linear algebra [Str76] would recognize these local conditions as “a system of linear equations.” Solving even web-scale systems of linear systems is considered easy with modern techniques and modern computers. For our small example example the solution is:
,
, and
. The role of the local steps was to reduce a new problem (estimating the importance or popularity of web page from the link structure) to something with its already known known techniques (like solving a linear system as illustrated in Figure 2).

Figure 2: Linear Algebra Solution: As Taught in School

So page-A is the most important page by the PageRank measure.

In this example application the local step was setting up the system of linear equalities (which are easy to derive from the web link graph) and the global step was solving the entire system for the final scores (which were not obvious). You spend most of your time encoding the problem and then use a known technique (in this case solving a linear system) to finish the solution.

Natural Language Processing

Our next example application is natural language processing [Cha96,Cha97]. Speech recognition (the alignment or transcription of recognized intelligible segments of sound to written text) is an important problem in natural language processing. An example problem is the need to find the most likely text matching a sequence of sounds such as is shown in Figure 3.

Figure 3: A Sequence of Sounds

Consider Figure 4 (which shows a bad transcription) and Figure 5 (which shows a good transcription).

Figure 4: A Bad Transcription

Figure 5: A Good Transcription

Our claim: we can (given access to training data, and this is the age of data [HNP09]) solve this problem with a local step that is a set of simple criticisms of proposed transcriptions. A good starting point is a database of previous sounds to text transcriptions. This database allows the construction of a series of tables that give the historic frequency (or probability) of all of the following:

• Prior probability of each sound
• Probability of each sound given the immediately previous sound
• Prior probability of each word
• Probability of each word given the immediately previous word
• Which combinations of word fragments are legitimate words
• Probability of each sound being assigned to each word fragment (syllables, phonemes and so on).

These tables encode a “speech model” (the rules involving sounds only), a language model (the rules involving text or words only) and the linkage between the two models. These models are deliberately simple in that they capture only local interactions (like probability of a word given the word before it) but no long range interactions (like subject predicate agreement).

Each box, nested box and arrow on our diagram represents one possible local critique. For each item in our diagram (again, the boxes and arrows) we can use our tables to assign a goodness or plausibility score. For instance bad word to word transitions (like “won”
“won”) will be rare in our historic tables so, just looking up probabilities from the tables (or, better, using the logarithms of probabilities) gives as a “plausibility score” that prefers known patterns of language. Then a score for the overall transcription can be derived by multiplying all of the local scores together. These local scores (though simple) already have encoded enough evidence to prefer the good transcription to the bad transcription without requiring any deep knowledge of speech, text or the meaning of the text. This is because the bad transcription has a series of obvious flaws such as: unlikely sound to word fragment assignments and unlikely word to word transitions.

Figure 6: Naively Extending a Partial Transcription

For example consider Figure 6 where a naive solver is in the process of considering selecting the word “one” as the third word to fill in. The only local critiques they need to consider are:

• how likely the word “one” is in general (call this )
• how likely the word “one” is to follow the word “nine” (call this
  )
• how likely the letter sequence “o” is given the sound “w” (call this
  )
• how likely the letter sequence “ne” is given the sound “n” (call this
     n ).

So the local plausibility of the fill-in word “one” is:
. We will call this the critique of “one” in position 3 and write as
where is the word known to be in position 2. Similarly we can generate all of the possible critiques ,
,
,
and the overall criticize of a sequence
:
from our pre-computed tables of probabilities. Notice for all of these critiques only the immediately previous word and the nearby sounds were used to determine the plausibility of the word we are attempting to fit in. Instead of using these critiques to directly fill in a possible solution (or using search) we will package up these critiques (in the form of the ) and pass them on to a powerful separate globalization step called Dynamic Programming [Bel57].

The globalization or finding of a best overall transcription is not trivial even though our score is simple. This is because the overall best sequence could depend on clever non-local fill-ins (like deliberately picking a less likely first word to allow a later favored transition to a fantastically good third word). Dynamic Programing does not fill in the transcription from left to right, but instead uses a table of scores derived from the left to right arrows and the . In our example Dynamic Programming consists of building a table of information as shown in Figure 7. Let represent the word position we are working looking at (so ranges from 1 to 4) and let be a variable that ranges over every word in the dictionary. Our table is indexed by and and when filled in stores what the highest “plausibility score” of a partial sequence of words where words 1 through have been filled in and the -th word is .

Figure 7: Dynamic Programming: Back Chaining in for a Solution

If we already had this magic table we could find a best possible sequence by “back chaining.” We start by finding a fourth word ( ) such that is maximal (in this case “one”). We then find a best third word ( ) by enumerating all words and picking such that
. We continue back until we had found words and to get a complete best sequence. Notice that we work from right to left (backwards) and except for the starting step we pick each word to match the calculation we are trying to un-roll, not to be the maximal entry in the column. For instance we pick
even though it does not have a the highest score, but because
is the maximal complete chain.

Of course, we don’t start with the table already filled in- so we need a procedure to build it. This procedure is the heart of the Dynamic Programming method (for more examples see: “Introduction to Algorithms” [CLRS09]). Notice that can be filled in for all just by plugging in words and computing the critiques (i.e.
). Once all the are filled in we can fill in the the with the general (and slightly trickier) formula:

as we illustrate for in Figure 8.

Figure 8: Dynamic Programming: Building the Table

The magic of the Dynamic Programing technique is: by being careful to not store too much in the table we avoid an explosion in record keeping that would render the method inefficient. Dynamic Programming exploits the small dependence structure encoded in (each box in our diagram depending on only a few arrows) and as we have shown can find “clever” solutions (such as taking a sub-optimal first word to get better transitions into preferred later words). For those who want more detail on solving this problem we recommend [Cha96] (as our goal is not to fully explain Dynamic Programming, but to demonstrate how it could be applied to the transcription problem as a pre-packaged globalizer).

In this example the local step was the graph link based critiques and the globalization step was Dynamic Programming. The separation of concerns from the local scoring to the globalizing step is a strength of the local to global principle.

Machine Learning

Our final example application is machine learning. Machine learning is loosely defined as computer programs that adapt or learn from data. Thomas Mitchell helps distinguish this activity as a specialty of artificial intelligence that concentrates on “well-posed learning problems.” [Mit97] Trevor Hastie, Robert Tibshirani, Jerome Friedman emphasize the relation to statistics (versus more traditional symbolic AI) [TH09]. A simple demonstration can be found in [Mou09b].

Machine learning is perhaps the strongest example of the local to global principle and is inspired by the work of Kristin P. Bennett and Emilio Parrado-Hernandez [BPH06]. In hindsight many machine learning algorithms (each of which has had a turn at being “the most exciting breakthrough ever” for a while) can be seen as the pairing of a performance criterion (which we call a local criterion as it applies to one specific set of parameter values at a time) and an optimization method (what we have been calling the globalization step). The work of Bennett and Parrado-Hernandez calls this distinction out and shows how it is not productive to present machine learning systems as unique named monolithic units, but instead to consider how to break them into an objective function and an optimizer. This allows both choice of better optimizers (such as replacing the inferior method of gradient descent method wherever it occurs) and for explicit control of important concepts such as hypothesis regularization and control of over-fitting (which some algorithms claim to achieve by deliberately using early exit from a an inferior optimizer).

At a “30,000 feet level” we can build a table of common machine learning techniques and name what is commonly used to implement their local and global steps. When a machine learning algorithm is defined by what conditions are meant to be true at the optimum we are no longer bound by details of the original implementation and can examine fix and improve the components.⁷ Table 1 is a crude summary of a wide selection for machine learning algorithms that may be more likely to offend everybody than just offend somebody. But this is also the point: it is the algorithmist’s job to think fluidly (beyond given names and provenances) and to invent scaffolding to convert partial analogies into practical correspondences.

This table is a necessarily crude summary. For example: notice that several known techniques can not even be distinguished from each other by the local and global columns of the table.

There are a few points we would like to make. Back propagation was considered unique to Neural Nets for quite a while because it was so entwined with the technique it was not recognized as the simple application of Automatic Differentiation [RC96] that it is. Support Vector Machines (SVM) are remarkable for their uniform very good choice of component methods (maximum L1 margin objective regularization, Kernel Methods [STC04] and sophisticated optimization methods [Joa06]). Many of the machine learning methods that SVM outperforms become again competitive when they adopt some of SVM’s technologies (especially using kernel methods to produce synthetic features).

Beyond these points we invoke a “globalizers are pre-packaged” principle and leave the discussion of machine learning and optimization to our reference: [BPH06]. In this example the local step is a per-example score or penalty and the globalization step is optimization.

Some Methods

The application of the local to global principle is similar to the Feynman “genius method.” Feynman’s method is to always have in mind a list of problems and a list of solution methods. The genius step is: anytime you see a new problem or a new solution method to immediately try it against every item from the complementary list. [Rot97, ``Ten Lessons I Wish I Had Been Taught''] This deliberate retention and activity greatly increases your problem solving ability. The power of the local to global principle is itself proportional to the number of local methods times the number of globalization strategies. Of course, to even start: the practitioner must already have available a number of candidate local and globalization methods. We list some methods and some guidance on variation and invention.

Local Methods

Good sources of ideas and analogies for local methods include:

• Introduce a Graph Structure

  A graph structure is a network of nodes connected by edges. Use of graphs was demonstrated both in the natural language processing and web page link analysis examples. We can dress up how we solved these problems and say we used a “Hidden Markov Model”, but the real power was we encoded our problem in a simple graph. Some problems (especially those from logic or those involving time) are essentially solved once they are translated out of their original form and into graph notation (for an example see: [Mou00]).

• Appeal to Physical Conservation Laws

  A good example physical law is Kirchhoff’s law or conservation of flow. All of the web page link analysis’s equations were derived by saying that the attention of at node is essentially the sum of attentions from other nodes (more sophisticated versions of the analysis actually do create and destroy flow, but they do it in a principled way).

• Encode the Problem into an Objective Function

  This method is essentially your declaration that you intend to use an optimizer for the globalization step. In operations research this specific technique has long been the practice (with no disrespect: a very productive part of operations research has been translating different problems into linear programs so the simplex method can be applied, for an example see [Mou09a]).

• Gradient Like Computations

  Includes Gradients, Secants, Lagrangians and other ideas from calculus. Gradients can drive optimizer based globalizers and techniques like Lagrangians are often powerful enough use mere inspection as the globalization step.

• Violation Driven Updates

  This method is particularly effective when your problem is not amenable to continuous optimization. A good example is the Lin-Kernighan heuristic for solving the traveling salesman problem.[LK73] This heuristic looks at subsets of the problem and suggests improving “surgeries” (until no more such improvements are possible).

• Introduction of Symbols

  Often, as with the web page link analysis example, you can not specify specific values for the unknowns, but you can specify relationships. You often can then solve for the symbols or introduce additional conditions and use an optimizer to complete the solution (see for example the maximum entropy method as described in [Ski88]).

• Over Specification

  If we anticipate using a global step like search, enumeration, summation or integration then over specification is a good local idea.

  For example: consider computing the probability that a fair count flipped 10 times comes up with heads exactly 3 times. The easiest way to perform this calculation is to specify exactly which 3 coins come up heads (the local over-specification) and then sum over all choices of 3 out of 10 coins (the global step). In mathematical notation this is:

  exactly 3 heads out of 10 flips

  or just under 12%.

• Under Specification

  One of the core principles of Dynamic Programming is to forget as much as possible about partial solutions, keeping only partial solution cost and just enough information to extend partial solutions. If you anticipate using something like Dynamic Programming as your globalization step then your goal should be to under specify.

• Tables

  A key step of the natural language processing example was the use of tables of past experience to determine which sounds likely corresponded to which words, which words likely followed each other (and so on). Encoding domain knowledge or expertise as probability tables is a very effective problem solving strategy (especially if the globalization strategy is going to be search or Dynamic Programming). In natural language processing examples tables and statistics are much easier to manage than comprehensive rules or grammars.

• Set up as Ranking or Machine Learning Problem

  This tactic is especially appropriate if your solution success metric is counts, frequencies or probabilities (instead of having to always be correct or always be optimal).

Globalization Methods

The universe of possible globalization methods is very diverse (in particular globalization is not always optimization).

• Search / Enumeration

  Search can be slow, but it is always an option to consider. If your problem translates naturally into a graph structure or your solutions are naturally seen as being composed of small pieces search should be considered. One of the big advantages using the local phase to formally encode your problem’s structure and putting search off to the global phase is: you can use advanced search techniques. Once you are freed from your specific problem details it becomes much easier to consider search techniques like branch and bound, A*, game theoretic search and general speed-up techniques like hashing and caching.

• Dynamic Programming

  If your problem has a bit more structure (in that partial solutions summarize and compose easily) then you can likely replace search with Dynamic Programming. The advantage is that Dynamic Programming typically offers an incredible speed up when compared to search.

• Optimization

  If your problem is continuous (involves numbers instead of discrete or categorical decisions), can be encoded as a reasonable objective function (linear, positive definite quadratic) and has reasonable constraints (linear or convex) then you can immediately apply an optimizer as your globalization step. Typical optimization methods include: conjugate gradient, Newton methods, quasi Newton methods, linear programming and quadratic programming.

• Combinatorial Optimization

  If your problem includes a “discrete variables” (that is variables that take on one of fixed set of values instead of values from a numeric range) then you may not be able to apply standard optimization techniques. At this point you may want to use more expensive combinatorial optimization techniques like integer linear programing or constraint satisfaction.

• Fixed Point Methods / Iteration

  Fixed point methods are based on the idea: “incrementally improve until there is no incremental improvement possible.” If the problem is continuous this is similar to steepest descent. If the problem is discrete then this is similar the Lin-Kernighan heuristic.

• Linear Algebra

  The web page link analysis and optimization examples were essentially solved once we reduced them to linear algebra. If you can write your problem as a linear relationship between unknowns or as the fixed-point of a linear operator (i.e. an such that ) then you can immediately use linear algebra to solve the problem at very large scale (e.g. web scale).

• Sampling / Problem Kernels

  A very successful line of attack on large problems is to reduce to a smaller problem containing most of the essential difficulty. David Karger has produced a number of effective algorithms for graph cuts and flows using a theory of sampling [Kar98]. Rod Downey and M. Fellows have demonstrated an effective theory of “problem kernels” that finds solution by focusing on smaller sub-problems (on which we can afford to use more expensive procedures).[DF98]

• Amortized Analysis / Economic Mechanism Methods

  Daniel Sleator and Robert Tarjan’s ideas of amortized analysis [ST85] allow approximation schemes similar to problem kernels. The method is to approximately optimize by pairing a bunch of unavoidable large penalties (conditions we can’t meet) with some accounting credits (say bonuses from other conditions we are meeting very well). We then isolate these paired items and optimize the rest of the problem exactly. The technique often works by showing the approximation can not be too bad because, due to the pairing of large penalties to good credits, there can not be too many large penalties. An informal example is: if it is impossible to pick someplace where all of an office will eat for lunch, perhaps you can solve the problem by paying one person to accept a restaurant they do not like (if the removal of their objection opens up a venue that is acceptable to everybody else).

• Relaxation / Homotopic methods

  These methods involve changing hard constraints to soft penalties (so allowing the constraints to be violated, but at a slowly increasing cost). After such a relaxation the homotopic (or continuous deformation) method is to increase the cost of violation and re-solve to try and get a trajectory of better and better nearly acceptable solutions that point to a possible overall solution.

Conclusion

The purpose of this article has been to make more visible an idea we call the local to global principle. This principle is an organizing tool useful both in designing and analyzing a wide variety of applications. Essentially the whole point of this writeup is to set up enough framework to quickly write down a table of advice such as Table 2 (and for such a table to mean something).

The principle is not universal; not everything can be fit into such a table. For example the local to global decoupling is not a feature of the famous EM algorithm [DLR77], which depends on mixing predictions and corrections.

To conclude: the recipe is as follows. If you come to a problem with a large shopping bag of possible ways to build local criteria and powerful globalization procedures then you stand a very good chance of solving the problem quickly. Also, if you keep the local to global principle in mind you are more likely to identify and retain potential local tricks and globalizers when you see them and thus have a larger more nimble set of tools available to solve problems when the time comes.

Bibliography

AC06

Nir Ailon and Bernard Chazelle, Approximate nearest neighbors and the fast johnson-lindenstrauss transform, STOC (2006).

AI06

Alexandr Andoni and Piotr Indyk, Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions.

BD02

Avrim Blum and John Dunagan, Smoothed analysis of the perceptron algorithm for linear programming, SODA (2002), 11.

Bel57

Richard Bellman, Dynamic programming, Princeton University Press, 1957.

BF97

Leo Breiman and Jerome H Friedman, Predicting multivariate responses in multiple linear regression, Journal of the Royal Statistical Society, Series B (Methodological) 59 (1997), no. 1, 3-54.

BFSO84

Leo Breiman, Jerome Friedman, Charles J. Stone, and R. A. Olshen, Classification and regression trees, Chapman & Hall/CRC, January 1984.

BNJ03

David M Blei, Andrew Y Ng, and Michael I Jordan, Latent dirichlet allocation, Journal of Machine Learning Research 3 (2003), 993-1022.

BPH06

Kristin P. Bennett and Emilio Parrado-Hernandez, The interplay of optimization and machine learning research, Journal of Machine Learning Research 7 (2006), 1265-1281.

Bre00

Leo Breiman, Special invited paper. additive logistic regression: A statistical view of boosting: Discussion, Ann. Statist. 28 (2000), no. 2, 374-377.

BRS91

R Beigel, N Reingold, and D Spielman, The perceptron strikes back, Structure in Complexity Theory Conference 6 (1991), 286-291.

Cha96

Eugene Charniak, Statistical language learning, MIT Press, 1996.

Cha97

to3em, Statistial techniques for natural language parsing, AI Magazine 18 (1997), no. 4, 33-44.

CLRS09

Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein, Introduction to algorithms, MIT Press, 2009.

CSS02

Michael Collins, Robert E Schapire, and Yoram Singer, Logistic regression, adaboost and bregman distances, Machine Learning 48 (2002), no. 1/2/3, 30.

CV05

Rudi Cilibrasi and Paul M.B Vitanyi, Clustering by compression, IEEE Transactions on Information Theory 51 (2005), no. 4, 1523-1545.

DF98

Rod G. Downey and M. R. Fellows, Parameterized complexity, Monographs in Computer Science, Springer, November 1998.

DLR77

A P Dempster, N M Laird, and D B Rubin, Maximum likelihood from incomplete data via the em algorithm, Journal of the Royal Statistical Society, Series B (Methodological) 39 (1977), no. 1, 1-38.

Fis36

Ronald A Fisher, The use of multiple measurements in taxonomic problems, Annals of Eugenics 7 (1936), 179-188.

FS99

Yoav Freund and Robert E Schapire, A short introduction to boosting, Journal of Japanese Society for Artificial Intelligence 14 (1999), no. 5, 771-780.

GD04

Peter D Grunwald and A Philip Dawid, Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory, Ann. Statist. 32 (2004), no. 4, 1367-1433.

Gru00

PD Grunwald, Maximum entropy and the glasses you are looking through, Conference on Uncertainty in Artificial Intelligence (2000), 238-246.

HNP09

Alon Halevy, Peter Norvig, and Fernando Pereira, The unreasonable effectiveness of data, IEEE Intellegent Systems (2009).

Hus99

Dirk Husmeier, Neural networks for conditional probability estimation, Springer, 1999.

IM99

Piotr Indyk and Rajeev Motwani, Approximate nearest neighbors: Towards removing the curse of dimensionality.

Joa98

Thorsten Joachims, Making large-scale svm learning practical, Advances in Kernel Methods – Support Vector Learning (1998).

Joa06

to3em, Training linear svms in linear time, KDD (2006).

Kar98

David R Karger, Randomization in graph optimization problems: A survey, Optima: Mathematical Programming Society Newsletter 58 (1998).

KCVM04

Trausti Kristjansson, Aron Culotta, Paul Viola, and Andrew Kachites McCallum, Interactive information extraction with constrained conditional random fields, AAAI (2004).

Kle97

Jon M Kleinberg, Authoritative souces in a hyperlinked environment, ACM SIAM Symposium on Discrete Algorithms (1997).

Kom08

Paul Komarek, Logistic regression for data mining and high-dimensional classification, CMU CS Thesis (2008), 138.

KWA95

J Kivinen, Manfred K Warmuth, and P Auer, The perceptron algorithm v.s. winnowo: Linear v.s. logarithmic mistake bounds when few input variables are relevant, COLT (1995), 289-296.

Lew98

David D Lewis, Naive (bayes) at forty: The independence assumption in information retrieval, find journal (1998).

LK73

S Lin and BW Kernighan, An effective heuristic algorithm for the traveling-salesman problem, Operations Research (1973), 498-516.

Mar61

M E Maron, Automatic indexing: An experimental inquiry, RAND Technical Report (1961), 404-417.

MF00

Zbigniew Michalewicz and David B. Fogel, How to solve it: Modern heuristics, Springer, 2000.

Mil02

John Stuart Mill, A system of logic, University Press of the Pacific, 2002.

Mit97

Thomas Mitchell, Machine learning, McGraw-Hill, 1997.

MK00

M E Maron and J L Kuhns, On relevance, probabilistic indexing and information retrieval, 1960 (2000), 1-29.

Mou00

John A Mount, Automatic detection of potential deadlock, Dr. Dobbs Journal (2000).

Mou09a

John Mount, Automatic generation and testing of un-rolls for profitable technical trades, http://www.win-vector.com/blog/2007/10/paper-on-stock-trading/, 2009.

Mou09b

to3em, A demonstration of data mining, http://www.win-vector.com/blog/2009/08/a-demonstration-of-data-mining/, 2009.

PBMW98

Lawrence Page, Sergey Brin, Rajeev Motwani, and Tery Winograd, The pagerank citation ranking: Bringing order to the web, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.31.1768 (1998).

Pol54a

G. Polya, Induction and analogy in mathematics, Princeton University Press, 1954.

Pol54b

to3em, Patterns of plausible inference, Princeton University Press, 1954.

Pol71

to3em, How to solve it, Princeton University Press, November 1971.

RC96

Louis B Rall and George F Corliss, An introduction to automatic differentiation, SIAM: Computational Differentiation: Techniques, Applications and Tools (1996), 1-18.

Rot97

Gian-Carlo Rota, Indiscrete thoughts, Birkhauser, 1997.

Ski88

John Skilling, The axioms of maximum entropy, Maximum Entropy and Bayesian Methods in Science and Engineering 1 (1988), no. 173-187.

ST85

Daniel Dominic Sleator and Robert Endre Tarjan, Amortized efficiency of list update and paging rules, Communications of the ACM 28 (1985), no. 2.

STC00

Jown Shawe-Taylor and Nello Cristianini, Support vector machines, Cambridge University Press, 2000.

STC04

to3em, Kernel methods for pattern analysis, Cambridge University Press, 2004.

Str76

Gilbert Strang, Linear algebra and its applications, Academic Press, Inc., 1976.

TH09

Jerome Friedman Trevor Hastie, Robert Tibshirani, The elements of statistical learning: Data mining, inference and prediction, Springer, 2009.

TTV08

Luca Trevisan, Madhur Tulsiani, and Salil Vadhan, Regularity, boosting, and efficiently simulating every high-entropy distribution, Electronic Colloquium on Computational Complexity (2008), 18.

Zei95

Doron Zeilberger, The method of undetermined generalization and specialization illustrated with fred galvin’s amazing proof of the dinitz conjecture, http://arxiv.org/abs/math/9506215, 1995.

Acknowledgement

A thank you to readers who supplied help and comments on earlier drafts.

Footnotes

… Mount¹

email: mailto:jmount@win-vector.com web: http://www.win-vector.com/

… principle.²

The pre-existing practice that comes cloesest to the local o global principle is found in operations research where encoding a problem to be solved by an optimizer is a central technique. We claim the natural statement of the local to global principle is more general than always encoding constraints for a particular optimizer (in particular globalization is not always optimization).

… structure⁴

By “link structure” we mean which web pages link to which other web pages.

… graph⁵

Remember, a graph is diagram consisting of nodes and edges (here depicted as arrows).

… features⁶

For example the model could account for:

• surfers entering and leaving the model
• link odds that vary where they are on a page
• surfers staying on a page proportional to how much text is on the page
• matching known traffic and click behavior where we have such data.

For simplicity we will just stick with the example given example.

… components.⁷

When a system is named and defined as an exact set of procedures the system can, by definition, not be improved. This is because with any change in procedure we have a new system that no longer matches the original definition and therefore requires a new name.

Related posts:

1. Automatic Differentiation with Scala
2. What Did Theorists Do Before The Age Of Big Data?
3. A Personal Perspective on Machine Learning

This paper demonstrates and explains some of the basic techniques used in data mining. It also serves as an example of some of the kinds of analyses and projects Win Vector LLC engages in.

August 19, 2009

John Mount¹

A Demonstration of Data Mining

1  Introduction

A major industry in our time is the collection of large data sets in preparation for the magic of data mining [Loh09,HNP09]. There is extreme excitement about both the possible applications (identifying important customers, identifying medical risks, targeting advertising, designing auctions and so on) and the various methods for data mining and machine learning. To some extent these methods are classic statistics presented in a new bottle. Unfortunately, the concerns, background and language of the modern data-mining practitioner are different than that of the classic statistician- so some demonstration and translation is required. In this writeup we will show how much of the magic of current data mining and machine learning can be explained in terms of statistical regression techniques and show how the statistician’s view is useful in choosing techniques.

Too often data mining is used as a black-box. It is quite possible to clearly use statistics to understand the meaning and mechanisms of data mining.

2  The Example Problem

Throughout this writeup we will work on a single idealized example problem. For our problem we will assume we are working with a company that sells items and that this company has recorded its past sales visits. We assume they recorded how well the prospect matched the product offering (we will call this “match factor”), how much of a discount was offered to the prospect (we will call this “discount factor”) and if the prospect became a customer or not (this is our determination of positive or negative outcome). The goal is to use this past record as “training data” and build a model to predict the odds of making a new sale as a function of the match factor and the discount factor. In a perfect world the historic data would look a lot like Figure 1. In Figure 1 each icon represents a past sales-visit, the red diamonds are non-sales and the green disks are successful sales. Each icon is positioned horizontally to correspond to the discount factor used and vertically to correspond to the degree of product match estimated during the prospective customer visit. This data is literally too good to be true in at least three qualities: the past data covers a large range of possibilities, every possible combination has already been tried in an orderly fashion and the good and bad events “are linearly separable.” The job of the modeler would then be to draw the separating line (shown in Figure 1) and label every situation above and to the right of the separating line as good (or positive) and every situation below and to the left as bad (or negative).

In reality past data is subject to what prospects were available (so you are unlikely to have good range and an orderly layout of past sales calls) and also heavily affected by past policy. An example policy might be that potential customers with good product match factor may never have been offered a significant discount in the past; so we would have no data from that situation. Finally each outcome is a unique event that depends on a lot more than the two quantities we are recording- so it is too much to hope that the good prospects are simply separable from the bad ones.

Figure 1 is a mere cartoon or caricature of the modeling process, but it represents the initial intuition behind data mining. Again: the flaws in Figure 1 represent the implicit hopes of the data miner. The data miner wishes that the past experiments are laid out in an orderly manner, data covers most of the combinations of possibilities and there is a perfect and simple concept ready to be learned.

Frankly, an experienced data miner would feel incredibly fortunate if the past data looked anything like what is shown in Figure 2.

The green disks (representing good past prospects) and the red diamonds (representing bad past prospects) are intermingled (which is bad). There is some evidence that past policy was to lower the discount offered as the match factor increased (as seen in the diagonal spread of the green disks). Finally we see the red diamonds are also distributed differently than the green disks. This is both good and bad. The good is that the center of mass of the red diamonds differs from the center of mass of the green disks. The bad is that the density of red diamonds does not fall any faster as it passes into the green disks than it falls in any other direction. This indicates there is something important and different (and not measured in our two variables) about at least some of the bad prospects. It is the data miner’s job be aware and to press on.

2.1  The Trendy Now

In truth data miners often rush where classical statisticians fear to tread. Right now the temptation is to immediately select from any number of “red hot” techniques, methods or software packages. My short list of super-star method buzzwords includes:

• Boosting[Sch01,Bre00,FISS03]
• Latent Dirichlet Allocation[BNJ03]
• Linear Regression[FPP07,Agr02]
• Linear Discriminant Analysis[Fis36]
• Logistic Regression[Agr02,KM03]
• Kernel Methods[CST00,STC04]
• Maximum Entropy[KM03,Gru05,SC89,DS06]
• Naive Bayes[Lew98]
• Perceptrons[BRS08,DKM05]
• Quantile Regression[Koe05]
• Ridge Regression[BF97]
• Support Vector Machines[CST00]

Based on some of the above referenced writing and analysis I would first pick “logistic regression” as I am confident that, when used properly, it is just about as powerful as any of the modern data mining techniques (despite its somewhat less than trendy status). Using logistic regression I immediately get just about as close to a separating line as this data set will support: Figure 3.

The separating line actually encodes a simple rule of the form: “if 2.2*DiscountFactor + 3.1*MatchFactor ≥ 1 then we have a good chance of a sale.” This is classic black-box data mining magic. The purpose of this writeup is to look deeper how to actually derive and understand something like this.

3  Explanation

What is really going on? Why is our magic formula at all sensible advice, why did this work at all and what motivates the analysis? It turns out regression (be it linear regression or logistic regression) works in this case because it somewhat imitates the methodology of linear discriminant analysis (described in: [Fis36]). In fact in many cases it would be a better idea to perform a linear discriminant analysis or perform an analysis of variance than to immediately appeal to a complicated method. I will first step through the process of linear discriminant analysis and then relate it to our logistic regression. Stepping through understandable stages lets us see where we were lucky in modeling and what limits and opportunities for improvement we have.

Our data initially looks very messy (the good and bad group are fairly mixed together). But if we examine out data in separate groups we can see we are actually incredibly lucky in that the data is easy to describe. As we can see in Figure 4: the data, when separated by outcome (plotting only all of the good green disks or only all of the bad red diamonds), is grouped in simple blobs without bends, intrusions or other odd (and more work to model) configurations.

We can plot the idealizations of these data distributions (or densities) as “contour maps” (as if we are looking down on the elevations of a mountain on a map) which gives us Figure 5.

3.1  Full Bayes Model

From Figure 5 we can see while our data is not separable there are significant differences between the groups. The difference in the groups is more obvious if we plot the difference of the densities on the same graph as in Figure 6. Here we are visualizing the distribution of positive examples as a connected pair of peaks (colored green) and the distribution of negative examples a deep valley (colored red) located just below and to the left of the peaks.

This difference graph is demonstrating how both of the densities or distributions (positive and negative) reach into different regions of the plane. The white areas are where the difference in densities is very small which includes the areas in the corners (where there is little of either distribution) and the area between the blobs (where there is a lot of mass from both distributions competing). This view is a bit closer to what a statistician wants to see- how the distributions of successes and failures different (this is a step to take before even guessing at or looking for causes and explanations).

Figure 6 is already an actionable model- we can predict the odds a new prospect will buy or not at a given discount by looking where they fall on Figure 6 and checking if they fall in a region on strong red or strong green color. We can also recommend a discount for a given potential customer by drawing a line at the height determined by their degree of match and tracing from left to right until we first hit a strong green region. We could hand out a simplified Figure 7 as a sales rulebook.

This model is a full Bayes model (but not a Naive Bayes model, which is oddly more famous and which we will cover later). The steps we took were: first we summarized or idealized our known data into two Gaussian blobs (as depicted in Figure 5). Once we had estimated the centers, widths and orientations of these blobs we could then: for any new point say how likely the point is under the modeled distribution of sales and how likely the point is under the modeled distribution of non-sales. Mathematically we claim we can estimate P(x,y |sale)² and P(x,y | non-sale) (where x is our discount factor and y is our matching factor).³ Neither of these are what we are actually interested in (we want: P(sale | x,y)⁴). We can, however, use these values to calculate what we want to know. Bayes’ law is a law of probability that says if we know P(sale | x,y), P(non-sale | x,y), P(sale) and P(non-sale)⁵ then:

Figure 7 depicts a central hourglass shaped region (colored green) that represents the region of x, y values where P(sale |x,y) is estimated to be at least 0.5 and the remaining (darker red region) are the situations predicted to be less favorable. Here we are using priors of P(sale) = P(non-sale) = 0.5, for different priors and thresholds we would get different graphs.

Even at this early stage in the analysis we have already accidentally introduced what we call “an inductive bias.” By modeling both distributions as Gaussians we have guaranteed that our acceptance region will be an hourglass figure (as we saw in Figure 7). One undesirable consequence of the modeling technique is the prediction sales become unlikely when both match factor and discount factor are very large. This is somewhat a consequence of our modeling technique (though the fact that the negative data does not fall quickly as it passes into the green region also added to this). This un-realistic (or “not physically plausible”) prediction is called an artifact (of the technique and of the data) and it is the statistician’s job to see this, confirm they don’t want it and eliminate it (by deliberately introducing a “useful modeling bias”).

3.2  Linear Discriminant

To get around the bad predictions of our model in the upper-right quadrant we “apply domain knowledge” and introduce a useful modeling bias as follows. Let us insist that our model be monotone: that if moving some direction is good than moving further in the same direction is better. In fact let’s insist that our model be a half-plane (instead of two parabolas). We want a nice straight separating cut, which brings us to linear discriminant analysis. We have enough information to apply Fisher linear discriminant technique and find a separator that maximizes the variance of data across categories while minimizing the variance of data within one category and within the other category. This is called the linear discriminant and it is shown in Figure 8.

The blue line is the linear discriminant (similar to the logistic regression line depicted earlier on the data-slide). Everything above or to the right of the blue line is considered good and everything below or to the left of the blue line is considered bad. Notice that this advice while not quite as accurate as the Bayes Model near the boundary between the two distributions is much more sensible about the upper right corner of the graph.

To evaluate a separator we collapse all variation parallel to the separating cut (as shown in Figure 9). We then see that each distribution becomes a small interval or streak. A separator is good if these resulting streaks are both short (the collapse packs the blobs) and the two centers of the streaks are far apart (and on opposite size of the separator). In Figure 9 the streaks are fairly short and despite some overlap we do have some usable separation between the two centers.

To make the above precise we switch to mathematical notation. For the i-th positive training example form the vector v_+,i and the matrix S_+,i where

where x_i and y_i are the known x and y coordinates for this particular past experience. Define v_−,i, S_−,i similarly for all negative examples. In this notation we have for a direction γ: the distance along the γ direction between the center of positive examples and center of negative examples is: γ^T ( ∑_i v_+,i / n₊ − ∑_i v_−,i / n₋) (where n₊ is the number of positive examples and n₋ is the number of negative examples). We would like this quantity to be large. The degree of spread or variance of the positive examples along the γ direction is γ^T (∑_i S_+,i / n₊) γ. The degree of spread or variance of the negative examples along the γ direction is γ^T (∑_i S_−,i / n₋) γ. We would like the last two quantities to be small. The linear discriminant is picked to maximize:

It is a fairly standard observation (involving the Rayleigh quotient) that this form is maximized when:

As we have said, the linear discriminant is very similar to what is returned by a regression or logistic regression. In fact in our diagrams the regression lines are almost identical to the linear discriminant. A large part of why regression can be usefully applied in classification comes from its close relationship to the linear discriminant.

3.3  Linear Regression

Linear regression is designed to model continuous functions subject to independent normal errors in observation. Linear regression is incredibly powerful at characterizing and elimination correlations between the input variables of a model. While function fitting is different than classification (our example problem) linear regression is so useful whenever there is any suspected correlation (which is almost always the case) that it is an appropriate tool. In our example in the positive examples (those that led to sales) there is clearly a historical dependence between the degree of estimated match and amount of discount offered. Likely this dependence is from past prospects being subject to a (rational) policy of “the worse the match the higher the offered discount” (instead of being arranged in a perfect grid-like experiment as in our first diagram: Figure 1). If this dependence is not dealt with we would under-estimate the value of discount because we would think that discounted customers are not signing up at a higher rate (when these prospects are in fact clearly motivated by discount, once you control for the fact that many of the deeply discounted prospects had a much worse degree of match than average).

For analysis of categorical data linear regression is closely linked to ANOVA (analysis of variance).[Agr02] Recall that variance was a major consideration with the linear discriminant analysis, so we should by now be on familiar ground.

In our notation the standard least-squares regression solution is:

where y_+,i = 1 for all i and y_−,i = −1 for all i.

If we have the same number of positive and negative examples (i.e. n₊ = n₋) then Equation 1 and Equation 2 are identical and we have β = γ. So in this special case the linear discriminant equals the least square linear regression solution. We can even ask how the solutions change if the relative proportions of positive and negative training data changes. The linear discriminant is carefully designed not to move, but the regression solution will tilt to be an angle that is more compatible with the larger of the example classes and shift to cut less into that class. The linear regression solution can be fixed (by re-weighting the data) to also be insensitive to the relative proportions of positive and negative examples but does not behave that way “fresh out of the box.”

3.4  Logistic Regression

While linear regression is designed to pick a function that minimizes the sum of square errors logistic regression is designed to pick a separator that maximizes something called the plausibility of the data. In our case since the data is so well behaved the logistic regression line is essentially the same as the linear regression line. It is in fact an important property of logistic regression that there is always a re-weighting (or choice of re-emphasis) of the data that causes some linear regression to pick the same separator as the logistic regression. Because linear and logistic regression are only identical in specific circumstances it is the job of the statistician to know which of the two is more appropriate for a given data set and given intended use of the resulting model.

4  Other Methods and Techniques

4.1  Kernelized Regression

One way to greatly expand the power of modeling methods is a trick called kernel methods. Roughly kernel methods are those methods that increase the power of machine learning by moving from a simple problem space (like ours in variables x and y) to a richer problem space that may be easier to work in. A lot of ink is spilled about how efficient the kernel methods are (they work in time proportional to the size of the simple space, not the complex one) but this is not their essential feature. The essential feature is the expanded explanation power and this is so important that even the trivial kernel methods (such as directly adjoining additional combinations of variables) pick up most of the power of the method. Kernel methods are also overly associated with Support Vector Machines- but are just as useful when added to Naive Bayes, linear regression or logistic regression.

For instance: Figure 10 shows a bow-tie like acceptance region found by using linear regression over the variables x, y, x², y² and x y (instead of just x and y). Note how this result is similar to the full Bayes model (but comes from a different feature set and fitting technique).

4.2  Naive Bayes Model

We briefly return to the Bayes model to discuss a more common alternative called “Naive Bayes.” A Naive Bayes model is like a full Bayes model except an additional modeling simplification is introduced in assuming that P(x,y|sale) = P(x|sale)P(y|sale) and P(x,y|non-sale) = P(x|non-sale)P(y|non-sale). That is we are assuming that the distributions of the x and y measurements are essentially independent (once we know which outcome happened). This assumption is the opposite of what we do with regression in that we ignore dependencies in the data (instead of modeling and eliminating the dependencies). However, Naive Bayes methods are quite powerful and very appropriate in sparse-data situations (such as text classification). The “naive” assumption that the input variables are independent greatly reduces the amount of data that needs to be tracked (it is much less work to track values of variables instead of simultaneous values of pairs of variables). The curved separator from this Naive Bayes model is illustrated in Figure 11.

The Naive Bayes version of the advice or policy chart is always going to be an axis-aligned parabola as in Figure 12. Notice how both the linear discriminant and the Naive Bayes model make mistakes (places some colors on the wrong side of the curve)- but they are simple, reliable models that have the desirable property of having connected prediction regions.

4.3  More Exotic Methods

Many of the hot buzzword machine learning and data mining methods we listed earlier are essentially different techniques of fitting a linear separator over data. These methods seem very different but they all form a family once you realize many of the details of the methods are determined by:

• Choice of Loss Function

  This is what notion of “goodness of fit” is being used. It can be normalized mean-variance (linear discriminants), un-normalized variance (linear regression), plausibility (logistic regression), L1 distance (support vector machines, quantile regression), entropy (maximum entropy), probability mass and so on.
• Choice of Optimization Technique

  For a given loss function we can optimize in many ways (though most authors make the mistake of binding their current favorite optimization method deep into their specification of technique): EM, steepest descent, conjugate gradient, quasi-Newton, linear programming and quadratic programming to name a few.
• Choice of Regularization Method

  Regularization is the idea of forcing the model to not pick extreme values of parameters to over-fit irrelevant artifacts in training data. Methods include MDL, controlling energy/entropy, Lagrange smoothing, shrinkage, bagging and early termination of optimization. Non-explicit treatment of regularization is one reason many methods completely specify their optimization procedure (to get some accidental regularization).
• Choice of Features/Kernelization

  The richness of the feature set the method is applied to is the single largest determinant of model quality.
• Pre-transformation Tricks

  Some statistical methods are improved by pre-transforming the outcome data to look more normal or be more homoscedastic.⁶

If you think along a few axes like these (instead of evaluating them by their name and lineage) you tend to see different data mining methods more as embodying different trade-offs than as being unique incompatible disciplines.

5  Conclusion

Our goal for this writeup was to fully demonstrate a data mining method and then survey some important data mining and machine learning techniques. Many of the important considerations are “too obvious” to be discussed by statisticians and “too statistical” to be comfortably expressed in terms popular with data miners. The theory and considerations from statistics when combined with the experience and optimism of data-mining/machine-learning truly make possible achieving the important goal of “learning from data.”

This expository writeup is also meant to serve as an example of the
types of research, analysis, software and training supplied by
Win-Vector LLC http://www.win-vector.com . Win-Vector LLC
prides itself in depth of research and specializes in identifying,
documenting and implementing the “simplest technique that can
possibly work” (which is often the most understandable, maintainable,
robust and reliable). Win-Vector LLC specializes in research but
has significant experience in delivering full solutions (including
software solutions and integration with existing databases).

A  Graphs

The majority of the graphs in this writeup were produced using “R”
http://www.r-project.org/ and Deepayan Sarkar’s Lattice
package[Sar08].

Footnotes:

¹
mailto:jmount@win-vector.com
http://www.win-vector.com/
http://www.win-vector.com/blog/

²Read P(A | B) as: “the probability of A will
happen given we know B is true.”

³Technically we are working with densities, not
probabilities, but we will use probability notation for its
intuition.

⁴P(sale | x,y) is the probability of
making a sale as a function of what we know about the prospective
customer and our offer. Whereas P(x,y|sale) was just how likely it is
to see a prospect with the given x and y values, conditioned on knowing we made
a sale to this prospect.

⁵ P(sale) and
P(non-sale) are just the “prior odds” of sales or what
our estimate of our chances of success are before we look at any
facts about a particular customer. We can use our historical
overall success and failure rates as estimates of these quantities.

⁶A situation is homoscedastic if the errors are independent of where we are in the parameter space (our x,y or match factor and discount factor). This property is very important for meaningful fitting/modeling and interpreting significance of fits.

Related posts:

1. Book Review: Ensemble Methods in Data Mining (Seni & Elder)
2. The Data Enrichment Method
3. Your Data is Never the Right Shape