Win-Vector Blog » Tutorials

Six Fundamental Methods to Generate a Random Variable

John Mount — Fri, 20 Jan 2012 19:23:38 +0000

Introduction

To implement many numeric simulations you need a sophisticated source of instances of random variables. The question is: how do you generate them?

The literature is full of algorithms requiring random samples as inputs or drivers (conditional random fields, Bayesian network models, particle filters and so on). The literature is also full of competing methods (pseudorandom generators, entropy sources, Gibbs samplers, Metropolis–Hastings algorithm, Markov chain Monte Carlo methods, bootstrap methods and so on). Our thesis is: this diversity is supported by only a few fundamental methods. And you are much better off thinking in terms of a few deliberately simple composable mechanisms than you would be in relying on some hugely complicated black box “brand name” technique.

We will discuss the half dozen basic methods that all of these techniques are derived from.To our mind all of the famous random variate generation/sampling techniques are derived from combinations of the following six fundamental methods:

Physical sources.
Empirical resampling.
Pseudo random generators.
Simulation/Game-play.
Rejection Sampling.
Transform methods.

The technical fights (such as: “is Gibbs sampling superior to, or even distinguishable from, Markov chain Monte Carlo?”) are all in the details, history and citation conventions. Each field and particular method accretes its own traditions. We will quickly discuss the fundamental methods we listed. As we will see: complexity goes up as we move through the list (so at some point things are no longer fundamental but instead derived, allowing us to end the list).

The Methods

Physical sources

This is the most basic way (though not as practical in the computer age) to generate random variables. Observe the flip of a real coin, shuffle actual cards, mix numbered balls or count the number of ticks from an actual radioactive source. In all of these the randomness comes from physical principles (such chaotic dynamics for coin flips or quantum mechanics for radioactive decay).

These sources are “outside of computer science” so we will say the least about them.

Empirical resampling

This is what used to be called “tables” (which were themselves often generated from physical processes). The observation is: that sometimes
to run a simulation you need access to instances of random variables that are distributed in a very precise way- but you don’t have a usable description of the desired distribution. You would think that in this case you could do nothing. But the principle of empirical resampling is that you can approximately generate new samples by taking samples (with repetition or replacement) from an old sample. This is the cornerstone of Bootstrap methods.

As an example: suppose we were given the sample of numbers 5, 5, 10, 5, 5 which has mean equal to 6. Further suppose we have no
description of how these number were generated but we wanted to know if a mean of at least 8 is likely or unlikely for five more numbers drawn the same way. We can approximate this by drawing many samples of size five from this original sample (allow the same number to be in our new
sample multiple times) and get the bootstrap estimate of the probability of seeing mean of at least 8 as having a probability around 0.6%.

This may seem trivial- but it is very important.

Pseudo random generators

In the computer age, to avoid need for external tables or expensive and slow peripherals we tend to use pseudo random generators. That is the output of deterministic iterative procedures as equivalent to true random sources. The science of pseudo randomness has evolved from cobbled together procedures passing ad-hoc tests (such as in Knuth Volume 2) to more formal pseudo randomness based on important properties (like provably being k-wise independent) or complexity (being computationally indistinguishable from a truly random on a time or space bounded machine). Behind the canned routines of all of the basic “random generators” commonly available is a pseudo random source.

Good references for the modern theory include:

“Pseudorandomness and Cryptographic Applications” Michael Luby 1996.
“Modern Cryptography, Probabilistic Proofs and Pseudorandomness” Oded Goldreich, 1999.

The most basic form of a sequential pseudo random generator is a sequence of states s(1), s(2), s(3) … . Where s(i+1) = g(s(i)) where g() is our deterministic function that maps state to state. The observed random variables are then h(s(i)) where h() is some deterministic function maps state to observables. For example for the linear congruential generator found in glibc we have g(x) = (1103515245*x + 12345) modulo 2^32 and h(x) = x modulo 2^30 (x an integer from 0 to 2^32 – 1). An example application: this generator when divided by (2^30 – 1) might return numbers passably uniformly distributed in the interval [0,1]. Two such variates might be uses as a uniform sample from the unit square.

That a simple iterated deterministic system (like the modulo arithmetic or even a physical system like coin flipping) would even superficially appear random (let alone be safe to use as pseudo random source) turns out to be the main consequence of Ergodic theory (which we will touch on in a later article). The point is: it should not be obvious (without bringing in some more theory) why you should trust pseudo-random sources.

Simulation/Game-play

Another fundamental method is direct simulation or game play. If we wanted a random variable that was 1 with probability equal to the odds of being dealt a full house from a standard shuffled deck of 52 cards (and zero otherwise). We can generate such a variable by simulating shuffling a deck, drawing a hand and returning 1 if the hand draw is a full house (and returning 0 otherwise). Notice in this case we are combining many random variables to get a single result.

One of the most important simulation techniques is Markov chain Monte Carlo methods (related to Gibbs sampling, simulated annealing and many other variations). These method implement a complex procedure over a stream of random inputs to generate a more difficult to achieve sequence of random outputs.

For example: Let T be the set of pairs of non-negative integers x, y such that x + y ≤ 1000. We could implement a Markov chain on this set from a source of coin flips. Given a point (x,y) in T we take three coin flips and move to new point (x’,y’) (also in T) using the following procedure:

Let m = 1 if the first flip is heads and m=0 if the first flip is tails.
Let v = (1,0) if the second flip is heads and v=(0,1) if the second flip is tails.
Let d = +1 if the third flip is heads and d = -1 if the third flip is tails.
If (x,y) + m*d*v is in T let (x’,y’) = (x,y) + m*d*v, otherwise let (x’,y’) = (x,y) (stay put).

Repeating this procedure a large number of times produces a sequence of points (x,y) such that (x,y) is distributed uniformly on S (again this follows from ergodic principles). The correctness of this simulation of or game of following a Markov chain is a very fundamental method in generating more complicated random variates and something we will write more about in an article dealing with the ergodic principle (the relation of connectedness to showing averages over time equal averages over space).

For simple shapes (rectangle, triangles) there are more efficient ways to generate points uniformly at random. For squares we exploit independence and just generate the coordinates independently. For triangles we could rejection sample from a bounding rectangle. Or we could use a tranform method: write down a counting function that indexes all the points in the triangle and generate points by index (for example it is easy to work out there are 501501 points in our example S so if we generate a random integer uniformly from 1 to 501501 can just pick the point with given index as our sample).

For general convex shapes (in high dimensions) these methods become intractible and Markov chain methods are one of the few options remaining.

Rejection Sampling

Rejection sampling is another way to convert one sequence of random variables into another. If we assume we can generate a random variable according to the distribution p(x) we can “rejection sample” to a new distribution using an “acceptance function” q(x) which returns a number in the interval [0,1]. Our procedure is to
repeat the following: generate x with probability p(x), generate a random variable y with uniformly in the interval [0,1] if y ≤ q(x) accept x as
our answer and quit (otherwise draw a new x and repeat).

When the distribution that rejection sampling draws with is such that if x and y had a ratio of being drawn of p(x)/p(y) then under the rejection procedure they have relative odds of (p(x)q(x))/(p(y)q(y)). An important special case is when q() is always 0 or 1, in this case we are drawing with relative odds proportional to p(x) from the subset of x with q(x)=1.

As an example: consider the problem of trying to draw a point (x,y) such that x^2 + y^x < 1 (the open unit disk) uniformly at random. The rejection sampling solution is: repeat the following until you have a success: generate x and y independently uniformly in the interval [-1,1], if x^2 + y^2 < then 1 accept them as our sample (otherwise repeat). This procedure is very fast as the unit disk that represents our acceptance region has area pi and the square we are generating trials from has area 4: so we over a 78% chance of success on each trial or expect to only have to run fewer that 1.28 trials (on average) to get a sample.

Transform methods

A transform method is used when we have the ability to generate instances of a random variable according to one distribution and we would like instances according to another distribution.

One method is used when we have access to the inverse of the cumulative distribution function of the distribution we are trying to generate. In this case we can use this function to convert uniform variants from the interval [0,1] into our target distribution. The commutative distribution function is the function cdf() where cdf(x) is the probability a random variate generated according to our distribution is less than or equal to x. The inverse function function icdf() where icdf(y) is such that cdf(icdf(y)) = y. For example the exponential distribution has an inverse cumulative distribution function icdf(y) = -ln(1-y)/lamda . So if y is
generated uniformly in the interval [0,1] then icdf(y) is a random variable generated according to the exponential distribution with parameter lambda.

A great example of transform methods is generating Gaussian random variables. We could directly use the inverse cumulative distribution function method described above- but to do this we would require a special function library to perform the required calculation of the inverse cummulative distribution (or inverse of erf()). Another way is the polar method: generate x,y uniformly from the open unit disk (by, for example rejection sampling as described earlier), set s = x^2 + y^2 and return x*sqrt(-2 ln(s)/s), y*sqrt(-2 ln(s)/s) as two independent Gaussian random variables. The trick being: the distribution function of r = sqrt(s) is of the form r*e^(-r*r/2) which leads to an elementary cumulative distribution function (unlike the original Gaussian density of the form e^(-r*r/2)) that is easy to invert.

Conclusion

Our thesis is: all major methods to generate random variables use aspects of the six methods we have listed here as fundamental. Or you should at least have a fluid understanding of at least these methods. You should be able to break down big “brand name” methods (like Gibbs sampling) roughly into their constituent parts (so you can reason about them). One example: notice how ratios of probabilities enter into Markov chain Monte Carlo methods (they cause step rejections); from this you can reason if your problem has bounded ratios it is a good candidate for direct application of the technique (and if it does not you need to add some more ideas, as was demonstrated in: “Sampling from Log-Concave Distributions,” Alan Frieze , Ravi Kannan , Nick Polson, Ann. Appl. Prob, 1994 ).

The first two methods we discuss (physical sources and empirical re-sampling) are of the class of solutions “already have the right answer.” Pseudo random generators are the primary way to negate the need for physical sources and resampling techniques. Simulation, rejection sampling and transform methods are the main tools for building new distributions out of old.

It is a matter of taste if a given trick fits into this ad-hoc taxonomy or not. You can invent new and better generation methods- but these methods are easily derived using ideas from the fundamental methods we mentioned here.

What to do when you run out of memory

John Mount — Tue, 06 Dec 2011 12:25:59 +0000

A constant problem for computer science (since its inception) is how to manipulate data that is larger than machine memory. We present here some general strategies for working “out of core” or what you should do when you run out of memory.

Early computers were most limited by their paltry memory sizes. von Neumann himself commented that even a room full of genius mathematicians would not be capable of much if all they could communicate, think upon or remember were the characters on a single type written page (much more memory than the few hundred words available to the Eniac). The most visible portions of early computers are their external memories or secondary stores: card readers, paper tape readers and tape drives.

SDC 920 computer, Computer History Museum, Mountain View CA

Historically computer scientists have concentrated on streaming or online algorithms (that is algorithms that work with the data in the order it is available and use limited memory). For many problems we have found this an insufficient model and it is much better to assume you can re-order and replicate data (such as scattering data to many processors and re-collecting it to sort). The scatter/gather paradigm is ubiquitous and is the underpinning of large scale sorting, databases and Map Reduce. So in one sense databases and Map Reduce different APIs on top of very related technologies (journaling, splitting and merging). Replicating data (or even delaying duplicate elimination) that is already “too large to handle” may seem counterintuitive; but it is exploiting the primary property of secondary storage: that secondary storage tends to be much larger than primary storage (typically by 2 orders of magnitude, compare a 2 terabyte drive to an 8 gigabyte memory stick).In our web age, the typical big data problems are inverting indices (for fast search lookup) and computing term frequencies (for TF/IDF scoring or for things like Naive Bayes classifiers). Since these are over-worked examples we will use a mathematical problem from “Additive Combinatorics”, Terence Tao, Van Vu, (ISBN-13: 9780521853866; ISBN-10: 0521853869)

We take one problem from the field of additive combinatorics: sum sets. For two sets of integers A = {a_1, … a_s} and B {b_1, …, b_t} the sum set is defined as the set (without repetition) A + B = { a_i + b_j | i = 1,…s, j=1…t }. For sets of integers the size of A+B (denoted as |A+B|) can vary from |A| + |B| – 1 to |A| * |B| depending on the relations between the numbers in A and B (or the structure of A and B). If instead of working with integers we work with integers modulo p where p is a prime number (or equivalently we treat all numbers as remainders of division by p) then by the Cauchy-Davenport inequality we have |A + B| ≥ min(|A|+|B|-1,p) (so essentially the same result, except when we run out of possible integers modulo p).

For example we would say (working modulo 19) that [0, 1, 4, 5] + [10, 11, 14, 15] = [0, 1, 10, 11, 12, 14, 15, 16, 18]. In fact there are 19 pairs of sets that add up to [0, 1, 10, 11, 12, 14, 15, 16, 18] ( for instance [5, 6, 9, 10] + [5, 6, 9, 10] is another such pair). Just to move forward assume we were interested in determining how many ways a set can be written as the sum of a pair of sets (each of size 4). For a given sum result we might try search or integer programming to find all possible summands. However, if we want the statistics on all sums simultaneously, we can work much quicker and without need for big gun mathematics.

The straightforward solution is this case is a bit of code like:


for set A from all possible sets of 4 integers from 0 to 18
    for set B from all possible sets of 4 integers from 0 to 18
        let set C = A + B modulo 19
        use set C as a key and add the pair (A,B) to the list associated with C
for all key sets C tracked above
     compute the size of the list of summand pairs found for C
print how many result sets C have a given number of summand pairs

The relations C which have a summand of form A can be collected by any bit of Java code implementing the interface below (just call insertReln(C,(A,B)) to store the relations and then entries() to get them back). A small interface that declares the needed methods is given below:


public interface RelnCollector {
	void insertReln(A a, B b) throws IOException;
	Iterable>> entries() throws IOException, InterruptedException;
	void close() throws IOException;
}

An in-memory relation collector is trivially implemented by a nested map adjusted to declare the above interface, as we see in the next code snippet:

public final class InMemoryRelnCollector
	implements RelnCollector {
	private final DataAdapter adapterA;
	private final DataAdapter adapterB;
	private Map> atoBs;

	public InMemoryRelnCollector(final DataAdapter adapterA,
		final DataAdapter adapterB) {
		this.adapterA = adapterA;
		this.adapterB = adapterB;
		atoBs = new TreeMap>(this.adapterA);
	}

	@Override
	public void insertReln(final A a, final B b) {
		Set set = (Set) atoBs.get(a);
		if(null==set) {
			set = new TreeSet(adapterB);
			atoBs.put(a,set);
		}
		if(!set.contains(b)) {
			set.add(b);
		}
	}

	@Override
	public Iterable>> entries() {
		return atoBs.entrySet();
	}

	@Override
	public void close() {
		atoBs = null;
	}
}

My Favorite Graphs

Nina Zumel — Tue, 06 Dec 2011 00:59:19 +0000

The important criterion for a graph is not simply how fast we can see a result; rather it is whether through the use of the graph we can see something that would have been harder to see otherwise or that could not have been seen at all.

– William Cleveland, The Elements of Graphing Data, Chapter 2

In this article, I will discuss some graphs that I find extremely useful in my day-to-day work as a data scientist. While all of them are helpful (to me) for statistical visualization during the analysis process, not all of them will necessarily be useful for presentation of final results, especially to non-technical audiences.

I tend to follow Cleveland’s philosophy, quoted above; these graphs show me — and hopefully you — aspects of data and models that I might not otherwise see. Some of them, however, are non-standard, and tend to require explanation. My purpose here is to share with our readers some ideas for graphical analysis that are either useful to you directly, or will give you some ideas of your own.

The graphs are all produced in R, using the ggplot2 package. While ggplot2 has a fairly high learning curve, it is the most flexible of the R graphing packages that I have encountered, and I’ve been able to quickly create rich graphics more easily than I would be able to with the R base graphics, or with other graphics packages.

Let’s start with some exploratory analysis. We will use the AdultUCI dataset that is included in the arules package.

library(arules) data("AdultUCI") dframe = AdultUCI[, c("education", "hours-per-week")] colnames(dframe) = c("education", "hours_per_week") # get rid of the annoying minus signs in the column names

We want to compare the distribution of work-week length to education, using a box-and-whisker plot that is overlaid on a jittered scatterplot of the data.

library(ggplot2) ggplot(dframe, aes(x=education, y=hours_per_week)) + geom_point(colour="lightblue", alpha=0.1, position="jitter") + geom_boxplot(outlier.size=0, alpha=0.2) + coord_flip()

The outlier.size=0 argument to geom_boxplot turns off the outlier plotting, and coord_flip switches the coordinate axes (because there are a lot of education levels).

The resulting graph:

Recall that the box of a box-and-whisker plot covers the central 50% of the data distribution; the line in the center marks the median. In this case, the work-week length concentrates so strongly at 40 hours (except for PhDs and those with professional degrees; they are doomed to work longer hours, typically) that most of the boxes appear one-sided; it’s easier to see what is happening with both the scatterplot and box-and-whisker superimposed, than it might be with the box-and-whisker alone. We can also see the relative concentration of the subjects along each educational level.

I’ve found that this superimposed graph is fairly easy to explain in a presentation (easier than a plain box-and-whisker, actually). The primary disadvantage that the scatterplot can get illegible for high volume datasets (this set has about 49 thousand rows). In this case, we have to return to the box-and-whisker plot alone.

Beyond exploratory analysis, we also want plots to evaluate the models that we fit. Win-Vector’s bread-and-butter recently has been logistic regression, so we will start with some visualizations for evaluating binary logistic regression models. We’ll use the heart disease dataset that Hastie, et.al, used in the Elements of Statistical Learning.

path = "http://www-stat.stanford.edu/~tibs/ElemStatLearn/datasets/SAheart.data" saheart = read.table(path, sep=",",head=T,row.names=1) fmla = "chd ~ sbp + tobacco + ldl + adiposity + famhist + typea + obesity" model = glm(fmla, data=saheart, family=binomial(link="logit"), na.action=na.exclude)

We will make a data frame of chd (the true response, coronary heart disease), and the score from the model.

dframe = data.frame(chd=as.factor(saheart$chd), prediction=predict(model, type="response"))

The standard diagnostic plot for logistic models is the ROC curve, which is fine, but personally, I don’t get a visceral feel for the model from looking at the ROC. Also, if you are interested in setting a score threshold on the model for classification purposes, the ROC adds an additional level of indirection, since it essentially integrates the score away. I used to plot the distribution of score (prediction) versus true response, like so:

ggplot(dframe, aes(x=prediction, colour=chd)) + geom_density()

This visualization tells me whether or not the model scores actually separate the response — in this case, the model identifies negative cases (no coronary heart disease) better than positive cases. The graph is hard to explain to a non-technical audience, and it has the disadvantage that both distributions are separately normalized to have unit area, so you get no sense of the relative proportion of positive and negative cases (in this case, about 35% of the population have coronary heart disease).

Here’s an alternate graph:

ggplot(dframe, aes(x=prediction, fill=chd)) + geom_histogram(position="identity", binwidth=0.05, alpha=0.5)

This is two semi-transparent histograms; the blue histogram for chd=1 is “in front” of the the red histogram. Because they are histograms, rather than density plots, we can more clearly see the relative distribution of positive to negative cases, and we have a better sense of how well (or not) the model separates the positive cases from the negative ones. Clearly, for most score thresholds, the model will have a fairly high false positive rate. I use this visualization all the time, but it is also fairly hard to explain, the transparency in particular.

We can also use our friend the box-and-whisker scatterplot.

ggplot(dframe, aes(x=chd, y=prediction)) + geom_point(position="jitter", alpha=0.2) + geom_boxplot(outlier.size=0, alpha=0.5)

The median score for the coronary heart disease cases is pulled away from the median score of the healthy subjects, but the central 50% of the two distributions still overlap.

Finally, let’s look at visualizations for linear regression. We’ll use the prostate cancer data from Elements of Statistical Learning.

fmla = "lpsa ~ lcavol + lweight + age + lbph + svi + lcp + gleason + pgg45" model = lm(fmla, data=prostate.data)

We can just plot(model) for some diagnostic graphs:

par(mfrow = c(2, 2), oma = c(0, 0, 2, 0)) plot(model)

These diagnostics are useful to determine whether or not a linear model is suitable, and to identify outliers; but again, I personally don't get a visceral feel for the model. I prefer to directly plot prediction against true response:

dframe = data.frame(lpsa=prostate.data$lpsa, prediction=predict(model)) title = sprintf("Prostate Cancer model\n R-squared = %1.3f", summary(model)$r.squared) ggplot(dframe, aes(x=lpsa, y=prediction)) + geom_point(alpha=0.2) + geom_line(aes(y=lpsa), colour="blue") + opts(title=title)

This graph gives you the same information as the Residuals vs. Fitted plot, and the Q-Q plot -- in particular, whether there is systematic over- or under-prediction in specific ranges of the data. It will expose outliers, and it is intuitive to explain when presenting your results. Furthermore, it can be used to evaluate other models that predict a continuous response, such as regression trees or polynomial fits.

Which graphs do you find especially useful for your day-to-day work?

Related posts:

The cranky guide to trying R packages

Good Graphs: Graphical Perception and Data Visualization

Learn Logistic Regression (and beyond)

An Appreciation of Locality Sensitive Hashing

John Mount — Mon, 21 Nov 2011 16:41:46 +0000

We share our admiration for a set of results called “locality sensitive hashing” by demonstrating a greatly simplified example that exhibits the spirit of the techniques.Locality sensitive hashing is awe inspiring in its originality, simplicity, beauty and effectiveness. In addition locality sensitive hashing is a remarkable technique as it works even when drastically abridged and simplified. In this paper (link to pdf) we give a description of conditions where the technique works and a heuristic argument why it works (using only elementary math).

Related posts:

A Quick Appreciation of the Sharpe Ratio

The Local to Global Principle

Automatic Differentiation with Scala

Kernel Methods and Support Vector Machines de-Mystified

John Mount — Sat, 08 Oct 2011 00:17:46 +0000

We give a simple explanation of the interrelated machine learning techniques called kernel methods and support vector machines. We hope to characterize and de-mystify some of the properties of these methods. To do this we work some examples and draw a few analogies. The familiar no matter how wonderful is not perceived as mystical.

Goals of this writeup

De-mystify kernel methods and support vector machines

Kernel methods and support vector machines have taken mythological proportions in the machine learning imagination. Partly this is because a number of good ideas are overly associated with them: support/non-support training datums, weighting training data, discounting data, regularization, margin and the bounding of generalization error. My issue is that these are all important enough ideas to stand on their own and are often seen in simpler settings. The observations that inform my view are as follows:

Kernel methods and support vector machines are in fact two good ideas. Each is important even without the other: kernels are useful all over and support vector machines would be useful even if we restricted to the trivial identity kernel.

Small scale “kernel tricks” are not that different than the classic technique of adding “interaction variables.” Kernels let you escape from the limits of “linear hypotheses” (really by moving to a bigger space where things are again linear but look curved from the point of view of your smaller original space). We demonstrate the linear methods and “primal kernel tricks” in A Demonstration of Data Mining.

Support/non-support is the central issue of nearest neighbor classifiers.

Weighing training data is most famously shared by logistic regression and boosting.

Re-weighting of data by smoothing kernels (different but related use of the work “kernel”) is central to non-parametric statistics (kernel smoothers and splines).

Regularization is of supreme importance in modeling in general.

Most practitioners get tired of “kernel shopping” and fall back to the identity, cosine or radial/Gaussian kernels.

Show concrete examples of what are and what are not kernels.
Few sources give enough theorems to think about kernels abstractly and fewer still work concrete examples.

Use as little math as possible (which, unfortunately, turns out to be quite a bit). We will discuss encodings and stopping conditions (important to understand what is going on) but avoid explaining the optimizers (the most beautiful part of support vector machines, but also the part that is available pre-packaged in libraries).

Call out common magical thinking and unreasonable expectations associate with kernel methods and support vector machines. This should help the reader be in a better position to “defend their doubts” regarding machine learning promises.

Try to place all techniques in a wider context (if it is usable only one place it is a trick, if it is usable multiple places it is a technique).

Discuss margin and its impact on generalization error.

Generalization error is an effect of “over fitting” where a model has learned things that are true about the training examples that do not hold for the overall truth or concept we are trying to learn (i.e. don’t generalize). Generalization error is the excess error rate we observe when scoring new examples versus the error-rate we saw in learning the training data. Margin is in fact a posterior observation. That is: margin is observed after the training data is seen, not known before data is seen (like, for example, VC dimension). Margin is useful as it bounds generalization error but it is not the prior bound it is often portrayed as. So we assert margin estimates are not much more special than simple cross-validation estimates which can also be performed once we have data available.

The goals are not to try to indict or try to cut down kernel methods or support vector machines, but just to dump some of the associated baggage so they can be used fluidly and without anxiety.

Example Problem

Consider the following simple (caricature) machine learning problem: we are given a number of points labeled as circles and square and we want to, given a new point, predict the label. In our example the only input will be two numbers: x and y. This example is simple enough that we can depict the entire situation in Figure 1 below:

Figure 1: Truth and Training Data

We are assuming (for simplicity of exposition) that we have the incredible luck that the label is indeed a deterministic function of x and y. We portray this “ground truth” as the blue parabola- every point found in this region will be considered to be labeled as “circle” and every other point (i.e. the red region) will be labeled as “triangle.” We have overlaid 20 example points (with appropriate shape labels). These points will be our training data. We will try to learn from the observed training data a good approximation of the unseen blue and red regions. This process is called learning or training and the ability to correctly predict the label of new points is called generalization. To make sure our concept is learnable we are going to further guarantee a moat (or margin) the distribution we are picking training and test examples from will never pick a point from the shaded region between the blue and black. That is we will never be asked about an example where x*x is very near y.

To promote visual thinking we are going to avoid formulas until the section titled “precise functions used” (where we will list the formulas used for each graph).

Nearest Neighbor Solution

An interesting model in these days of big data and fast machines is the nearest neighbor model. Such a model colors each point in space blue or red as it thinks the point should be classified. In this case each point is colored the color given by the closest known training point. This induces the type of model seen in Figure 2 whose boundary is piecewise line segments mid-way between training points. The data points that determine segments of the classification boundary in this way are thought to support the boundary and are called “support examples” or “support vectors.” In fact any training point that is not “supporting” part of the boundary is “irrelevant” (could be removed and we would get the exact same model). This split between important and non-important training points is considered one of the important observations from the theory of support vector machines, but we see it even here in the nearest neighbor models.

Figure 2: 1 nearest neighbor model

Notice that the nearest neighbor model in Figure 2 is “not half bad.” The blue region is much too wide, but near the training data the blue mass is roughly in the correct position. If we had infinite data and infinite computational resources this sort of model would be hard to beat (as any point we are likely to be tested on would have a lot of very near examples to work from). We can try to clean up the shape of the model a bit by using a vote of the 2 nearest points to color each point in the plane (see Figure 3) or even the majority of the 3 nearest points (see Figure 4).

Figure 3: 2 nearest neighbor model

The purple region in Figure 3 is a “region of uncertainty” where net-vote over the 2 nearest points is zero (they gave inconsistent advice). We can use this as a hint that predictions take from this region are less reliable. As you may notice, we are not getting real improvements. We should always spend more time getting more features (useful coordinates in addition to x and y) and more data; and spend less time tweaking models. But if your data and features are fixed all you work on is your modeling technique (alternately: your modeling technique is all you can prepare before receiving features and data). So we will continue to discuss features and technique.

Figure 4: 3 nearest neighbor model

Nearest neighbor classifiers are optimal in the sense that with an infinite amount of data the 1-nearest neighbor classifier has an error rate that approaches twice the Bayes error rate (the Bayes error rate being the ideal error observed on identical repetitions, or the theoretical best error rate) and for large k the k-nearest neighbor method approaches the Bayes error rate itself (see for example k-nearest neighbor algorithm ).

However the effectiveness of the nearest neighbor classifiers is coming from the very low dimension (2) of our input variables x and y. If we were attempting to predict from n variables we might need an amount of data exponential in n to get a useful nearest neighbor classifier. This is an inefficient use of data of the data; what we want is to use all of the data we have effectively. To do this we further assume there is some relational reason that the position of the point in the plane determines the shape-label (i.e. that we are learning, generalizing, interpolating and extrapolating- not just memorizing). We set our ambitions above memorizing and move towards functional modeling. We start thinking in terms of change: how the label density of examples changes as we move is a good clue as to how it will continue to change. We can do this either in a parametric or primal formulation (where we attempt to directly infer parameters of an assumed functional form as in regression or logistic regression) or in a non-parametric or dual form (where we attempt to learn relations between points and the training data instead of parameters). We have discussed this before ( A Demonstration of Data Mining ) and expanded on primal methods ( Learn Logistic Regression (and beyond, The Simpler Derivation of Logistic Regression and The equivalence of logistic regression and maximum entropy models).

Functional Solution

Our next idea is: can we use our training data to tease out a functional form for the relation between x,y and shape label? The first function we will try is chosen to be similar to the nearest neighbor model we just demonstrated. For this form we say each training point is the center of a Gaussian hump (say up for blue/circle and down for red/triangle) which we will call a “discount function” (or informally a hump). A “Gaussian” is just a function that falls off exponentially in the square of Euclidian distance from a central point (we can see the simple form of such functions in the “precise functions used” section). The contour lines (or levels as seen from looking down upon) of one such hump or discount function are shown in figure 5 (all graphs from here on in will be looking down at a height represented by color and contour lines, much like topographic maps).

Figure 5: Narrow Gaussian example concept

This particular discount function has its maximum (or peak) centered on a data point and then rapidly falls as we move away from the training point. The only property we will use in this section is that we can evaluate the discount easily (so we don’t at this time need or make use of any restrictive properties like integrability, positive semi-definiteness or even anything like bounded level sets).
We could make our functional model just the sum of all of these up and down humps over all of the training data. Each point gets a hump of the same height and same radius, pointing up for blue/circle and down for red/triangle. The net coloring of this sum function is shown in figure 6.

Figure 6: Narrow Gaussian sum model

The idea is that this would imitate the nearest neighbor model we have already seen. This is because even though we are summing over all data the closest training examples should usually dominate (since the Gaussian humps we picked fall to zero as we get further from their centers). The correspondence to nearest neighbor increases if we tighten the radius of our functions (searching for the right radius is a very important statistical problem called “bandwidth estimation”). Figure 7, for example shows the sum over much narrower Gaussians.

Figure 7: Very narrow Gaussian sum model

Figure 7 looks even more like the nearest neighbor solution. In fact your could think of the discount-weighted sum over all of the data as the natural model (as it is easer to reason about) and the nearest neighbors a more efficient approximation (summing only over near points). We also have seen that we can alter the smoothness of our model (which has consequences on model generalization) by changing the steepness of our discount functions. We can also build intermediate models (such as building a nearest neighbor but using discount weighted sums instead of uniform voting in the neighborhoods). We haven’t yet greatly improved on nearest neighbor, but we have identified a couple of obvious avenues for further improvement: mess with the bandwidths (the obvious idea to try and deal with near/far scale) or re-weight the data (as one of the lessons of the nearest neighbor model is that points that are not near a boundary are less important).

Bandwidth Solution

As a digression lets play with bandwidth a bit first. Suppose simultaneously for each training datum we picked an ideal bandwidth or steepness of the Gaussian.

Figure 8: Simultaneous bandwidth model

This simultaneous bandwidth model depicted in figure 8 was determined by picking a simultaneous assignment of Gaussian widths (each training datum gets its own width) that maximizes the log-sum of the category signed model function on the training data (much like the logistic regression maximum likelihood decoding). This differs from typical bandwidth selection problems where all points are given a single common “best bandwidth.” We instead picked a different bandwidth for each point (bandwidths picked to maximize how much of the correct category portion of the sum was correct on each training example). The blue region is reasonable, but does not match the shape of the true concept. Partly this is because there has not been enough training data to have learned a lot about the true shape (for example there is no empirical evidence for circle/blue in the top-left quadrant). The other reason is a bias inherent to this type of model. If one of the bandwidth functions is wider than all of the others (which is almost certain to occur) then it falls slower than all of the others and for points very far away from the center of the training data this one function is most of what remains. Or equivalently: due to the nature of this modeling technique one of the concepts learned is going to be the union of bounded islands and the other concept will grab all points sufficiently far away from the center of the training data. This is strong (and undesirable) bias, but it is a bias also shared by support vector machines with Gaussian kernels (though support vector machines get it from their so-called “dc-term” b not being zero; this will be discussed later). Our bandwidth correction was successful, but frankly the optimization problem we solved to estimate the optimal bandwidths was nasty and would not be something we would advise for a large amount of data.

Data weighting solution (support vector machine)

So instead of messing with the bandwidths let us consider re-weighting the Gaussians in our sum. We will leave the shape of each Gaussian the same but allow each individual training datum to have a unique weight or importance. Perhaps by picking the right weights we can get a better model. By “best” we will mean “best margin” (to be defined later) because with “best margin” as our objective the optimization problem of solving for the best data weights has a particularly beautiful form that can be reliably solved at great scale. This is called a “support vector model” (or support vector machine) and we will describe these ideas in detail later. But first lets see the result. Figure 9 shows the original narrow Gaussians re-summed according to the weights picked by the support vector method (instead of all being 1 as in figure 6).

Figure 9: Narrow Gaussian support vector model

Figure 9 is again an improvement. The figure is formed by taking a signed sum of our discount functions centered at our training points and weighted by the “support vector machine weights.” The signs are picked with one class encoded as +1 and the other as -1. The colors red and blue are picked if the sum is above or below a constant “b” called “the dc-term” (part of the support vector solution). The solution again looks okay: all the training points are in the correct color region (and you can’t hope to interpolate let alone extrapolate if you can’t even reproduce your training concept). Also, as with the bandwidth model, contours are set sensibly (steep/dense where categories are near each other, shallow/sparse where things are safe). We see the same inductive bias: one of the concepts is the union of bounded islands (this will always going to be the case unless the model picks b=0). This bias is not obvious from the kernel choice, as it is hidden in the dc-term of the support vector fitting equations (it is not part of the kernel, but can be defeated by choosing unbounded kernels). An interesting empirical fact about support vector models is they tend to work well with a larger bandwidth. For example if we use a wider Gaussian we get an even more convincing model (see Figure 10).

Figure 10: Wide Gaussian support vector model

In figure 10 we have also called out one of the important features of support vector machines. In the literature all datums are called “vectors” (as they are represented in coordinates) and a subset of these are called the support vectors. In figure 10 we have drawn the 4 training examples that turn out to be support vectors as large shapes and all other training examples as small shapes. The support vectors are the datums with non negligible weights. The learned model is a function of the support vectors only. So after fitting we can discard the other 16 training points. This is similar to the fact that nearest neighbor also only needs the training examples that are supporting its model boundary.

Notice that at this point the support vector models are not “magically” better, they in fact look less like the truth we are trying to learn than either the nearest neighbor models or the un-weighted sum models. To fix this we need to do some “kernel shopping” or find functions that better respect what we are trying to model. With the right functions support vector machines can do a very good job at learning the concept (but knowing “the right functions” is a huge hint as to the what the concept is. There is nothing wrong with using a hint but we do have to a method to produce the hint or have a reasonable number of hints to try from.

The “sum over everything with the same weight” solutions from the earlier section are very similar to Naive Bayes which also sums over all matching features with no weight adjustment. The support vector “use the same model but pick better weights” stands over these sum models in very much the same way Logistic Regression stands over Naive Bayes. In fact the optimization problems are very related. If we consider weights over features or coordinates as “primal” and weights over data items as “dual” we can informally say something like: support vector machines optimize by working over dual variables and inspecting for primal weights, and logistic regression works over primal variables and inspects for data weights. But this is a bit vague.

At this point we need to get a bit more explicit and precise.

Precise functions used

In general we will write our training data as a sequence of n-vectors. Our points will be named u(1) … u(m). For each i = 1..m we also know which category the training point is labeled with. We will encode this as y(1) … y(m) where each y() is either +1 or -1 depending on the training label. So the example we have been working has n=2 and m=20. Let z represent a n-vector we wish to classify.

Figure 6 was a picture of the sign of the function:

Figure 7 was a picture of the sign of the function:

Figure 8 was a picture of the sign of the function:

where the w(i) are non-negative numbers picked so that

is large.

Figure 9 was a picture of the sign of -b plus the function:

where the a(i) and b are picked so “the margin” (discussed later) is large.

Figure 10 was a picture of the sign of -b plus of the function:

where the a(i) and b are picked so the margin is large.

All of this repetition is to emphasize the commonality of the models. They are all of the form:

where the a(i) are non-negative numbers called the “support weights” and k(,) is a function mapping pairs of vectors to numbers and is called a “a kernel.” Unfortunately there are many incompatible uses of the word kernel in mathematics and statistics. Here “kernel” is being used in the sense of positive semi-definiteness or that k(u,u) ≥ 0 for all u. Note that the support vector machine instead of using the sign of f(z) as its decision instead uses which side f(z) is of a constant b as its category decision. A consequence is: for support vector machines if b is non-zero (as it almost surely will be) and the kernels all go to zero as we approach infinity fast enough (as they are designed to do) then exactly one of the learned classes is infinite and the other is a union of islands (regardless if this was true for the training data).

About Kernels

A lot of awe and mysticism is associated with kernels, but we think they are not that big a deal. Kernels are a combination of two good ideas, they have one important property and are subject to one major limitation. It is also unfortunate that support vector machines and kernels are tied so tightly together. Kernels are the idea of summing functions that imitate similarity (induce a positive-definite encoding of nearness) and support vector machines are the idea of solving a clever dual problem to maximize a quantity called margin. Each is a useful tool even without the other.

The two good ideas are related and unfortunately treated as if they are the same. The two good ideas we promised are:

The “kernel trick.” Adding new features/variables that are functions of your other input variables can change linearly inseparable problems into linearly separable problems. For example if our points were encoded not as u(i) = (x(i),y(i)) but as u(i) = (x(i),y(i),x(i)*x(i),y(i)*y(i),x(i)*y(i))

we could easily find the exact concept ( y(i) > x(i)*x(i) which is now a linear concept encoded as the vector (0,1,-1,0,0).

Often you don’t need the coordinates of u(i). You are only interested in functions of distances ||u(i)-u(j)|^2 and in many cases you can get at these by inner products and relations like ||u(i)-u(j)||^2 = + – 2 .

We will expand on these issues later.

The important property is that kernels look like inner products in a transformed space. The definition of a kernel is: there exists a magic function phi() such that for all u,v:

.

This means that k(.,.) is behaving like an inner product in some (possibly unknown) space. The important consequence is the positive semi-definiteness, which implies k(u,u)≥0 for all u (and this just follows from the fact about inner products over the real numbers that ≥0 for all z). This is why optimization problems that use the kernel as their encoding are well formed (such as the optimization problem of maximizing margin which is how support vector machines work). You can “regularize” optimization problems with a kernel penalty because it behaves a lot like a norm. Without the positive semidefinite property all of these optimization problems would be able to “run to negative infinity” or use negative terms (which are not possible from a kernel) to hide high error rates. The limits of the kernel functions (not being able to turn distance penalties into bonuses) help ensure that the result of optimization is actually useful (and not just a flaw in our problem encoding).

And this brings us to the major limitations of kernels. The phi() transform can be arbitrarily magic except when transforming one vector it doesn’t know what the other vector is and phi() doesn’t even know which side of the inner product it is encoding. That is kernels are not as powerful as any of the following forms:

snooping (knowing the other):

positional (knowing which part of inner product mapping to):

fully general:

Not everything is a kernel.

Some non-kernels are:

k(u,v) = -c for any c>0

A consequence of the fact that no negative number can be written as a sum of squares or as the limit of a sum of squares in the reals.

k(u,v) = ||u-v||

This can be shown as follows. Suppose k(u,v) is a kernel with k(u,u) = 0 for all u (which is necessary to try and match ||u-v|| as ||u-u|| = 0 for all u). By the definition
of kernels k(u,u) = < phi(u), phi(u) > for some real vector valued function phi(.). But, by the properties of the inner real inner product <.,.>, this means phi(u) is
the zero vector for all u. So k(u,v) = 0 for all u,v and does not match ||u-v|| for any u,v such that u ≠ v.

There are some obvious kernels:

k(u,v) = c for any c ≥ 0 (non-negative constant kernels)

k(u,v) = < u , v > (the identity kernel)

k(u,v) = f(u) f(v) for any real valued function f(.)

k(u,v) = < f(u) , f(v) > for any real vector valued function f(.) (again, the definition of a kernel)

k(u,v) = transpose(u) B v where B is any symmetric positive semi-definite matrix

And there are several subtle ways to build new kernels from old. If q(.,.) and r(.,.) are kernels then so are:

k(u,v) = q(u,v) + r(u,v)

k(u,v) = c q(u,v) for any c ≥ 0

k(u,v) = q(u,v) r(u,v)

k(u,v) = q(f(u),f(v)) for any real vector valued function f(.)

k(u,v) = lim_{k-> infinity} q_k(u,v) where q_k(u,v) is sequence of kernels and the limit exists.

k(u,v) = p(q(u,v)) where p(.) is any polynomial with all non-negative terms.

k(u,v) = f(q(u,v)) where f(.) is any function with an absolutely convergent Taylor series with all non-negative terms.

Most of these facts are taken from the excellent book: John Shawe-Taylor and Nello Cristianini’s “Kernel Methods for Pattern Analysis”, Cambridge 2004. We are allowing
kernels of the form k(u,v) = < phi(u), phi(v) > where phi(.) is mapping into an infinite dimensional vector space (like a series). Most of these facts can be checked by
imagining how to alter the phi(.) function. For example to add two kernels just build a larger vector with enough slots for all of the coordinates for the vectors encoding
the phi(.)’s of the two kernels you are trying to add. To scale a kernel by c multiply all coordinates of phi() by sqrt(c).

Multiplying two kernels is the trickiest. Without loss of generality assume q(u,v) = < f(u) ,f(v) > and r(u,v) = < g(u), g(v) > and f(.) and g(.) are both mapping into the same finite dimensional vector space R^m (any other situation can be simulated or approximated by padding with zeros and/or taking limits). Imagine a new vector function p(.) that maps into R^{m*m} such that p(z)_{m*(i-1) + j} = f(z)_i g(z)_j . It is easy to check that k(u,v) = < p(u) , p(v) > is a kernel and k(u,v) = q(u,v) r(u,v). (The reference proof uses tensor notation and the Schur product, but these are big hammers mathematicians use when they don’t want to mess around with re-encoding indices).

Note that one of the attractions of kernel methods is that you never have to actually implement any of the above constructions. What you do is think in terms of sub-routines (easy for computer scientists, unpleasant for mathematicians). For instance: if you had access to two functions q(.,.) and r(.,.) that claim to be kernels and you wanted the product kernel you would just, when asked to evaluate the kernel, just get the results for the two sub-kernels and multiply (so you never need to see the space phi(.) is implicitly working in).

This implicitness can be important (though far too much is made of it). For example the Gaussians we have been using throughout are kernels, but kernels of infinite dimension, so we can not directly represent the space they live in. To see the Gaussian is a kernel notice the following:

.

And for all c ≥ 0 each of the three terms on the right is a kernel (the first because the Taylor series of exp() is absolutely convergent and non-negative and the second two are the f(u) f(v) form we listed as obvious kernels). The trick is magic, but the idea is to use the fact that Euclidian squared distance breaks nicely into dot-products ( ||u-v||^2 = < u, u &gt + < v, v &gt – 2 < u , v >) and exp() converts addition to multiplication. It is rather remarkable that kernels (which are a generalization of inner products that induce half open spaces) can encode bounded concepts (more on this later when we discuss the VC dimension of the Gaussian).

Back to Support Vector Machines

We can now restate what a support vector machine is: it is a method for picking data weights so that the modeling function for a given kernel has a maximum margin. The margin is the minimal distance between a training example and the model’s boundary between blue and red. Notice the 4 support vectors in figure 10 are all not “tight up against the boundary,” but are all the same (minimal distance) from the boundary. This distance is called the margin and the area near the boundary is obviously a place the model has uncertainty (so it makes sense to keep the boundary as far as possible from the training data). The great benefit of the support vector machine is that with access only to the data labels and the kernel function (in fact only the kernel function evaluated at pairs of training datums) the support vector machine can quickly solve for the optimal margin and data weights achieving this margin.

For fun lets plug in new kernel called the cosine kernel:

(c ≥ 0).

This kernel can be thought of as having a phi(.) function that takes a vector z and adds an extra coordinate of sqrt(c) and then projects the resulting vector onto the unit sphere. This kernel induces concepts that look like parabolas, as seen in figure 11.

Figure 11: Cosine example concept

Figure 12 shows the sum-concept (add all discount functions with same weight) model. In this case it is far too broad (averaging of the cosine concepts which are not bounded concepts like the Gaussians) creates an overly wide model. The model not only generalizes poorly (fails to match the shape of the truth diagram) it also gets some of its own training data wrong.

Figure 12: Cosine kernel sum model

And figure 13 shows the excellent fit returned by the support vector machine. Actually an excellent fit determined by 4 support vectors (indicated by larger labels). Also notice the support vector machine using unbounded concepts generated a very good unbounded model (unbounded in the sense that both the blue and red regions are infinite). By changing the kernel or discount functions/concepts we changed the inductive bias. So any knowledge of what sort of model we want (one class bounded or not) should greatly influence our choice of kernel functions (since the support vector machine can only pick weights for the kernel functions, not tune their shapes or bandwidths).

Figure 13: Cosine Support Vector model

Another family of kernels (which I consider afflictions) are the “something for nothing” kernels. These are kernels of the form:

or higher powers or other finishing functions. The idea is that if you were to look at the expansion of these kernels you would see lots of higher order terms (powers of u_i and v_j)
and if these terms were available to the support vector machine it could use them. It is further claimed that in addition to getting new features for free you don’t use up degrees
of freedom exploiting them (so you cross-validate as well as for simpler models). Both these claims are fallacious- you can’t fully use the higher order terms because they are entangled with other terms that are not orthogonal the outcome and the complexity of a kernel is not quite as simple as degrees of freedom (proofs about support vector machines are stated in terms of margin, not in terms of degrees of freedom or even in terms of VC dimension). The optimizer in the SVM does try hard to make a good hypothesis using the higher order terms- but due to the forced relations among the terms you get best fits like figure 14.

Figure 14: Squared magic kernel support vector model

If you want higher order terms I feel you are much better off performing a primal transform so the terms are available in their most general form. That is re-encode the vector u = (x,y) as the larger vector (x,y,x*y,x*x,y*y). You have to be honest: if you are trying to fit more complicated functions you are searching a larger hypothesis space so you need more data to falsify the bad hypotheses. You can be efficient (don’t add terms you don’t think you can use as they make generalization harder) but you can’t get something for nothing (even with kernel methods).

Back to margin

Large margin (having a large distance between the decision boundary and all training examples) is a good idea. At the simplest level it means anything close to a training data point gets classified correctly (so the model is immune to an amount of fuzz proportional to the margin width).

We get the following remarkable generalization theorem (this one is the hard SVM version, we weaken the result a bit to make it easier to state). Suppose we train a model using kernel k(.,.) on m training examples u(1),…,u(m) with training labels y(1)…y(1). Further assume the hidden true concept is separable with respect to our kernel and data distribution with a margin of at least w (that is for any sample we draw we can find a SVM model that classifies all of the training data correctly and sees a margin of at least w). Or more simply: assume w is behaving like a constant with respect to m (i.e. it is not going down as we increase sample size). Then with probability at least 1-d the generalization error (that is error seen on new test points not seen during training, but drawn from the same distribution as the training examples) is no more than:

.

(we take this from Theorem 7.22 of “Kernel Methods for Pattern Analysis, page 215).

If we further assume the expected value k(u,u) under the training distribution exists (i.e. we don’t have too many examples of very high kernel norm according to the distribution we are training with respect to) and is stable (not taking a lot of value from rare instances) then we can read off the meaning of this upper bound. Assuming we can always build a model from the training data of margin at least w then the generalization error (excess error we expect to see on classifying similar points) is falling at a rate proportional to the square root of the amount of training data we have. This is in one sense amazing- we have a bound on how well we are going to do on data we have not seen. And this bound is also in some sense “best possible” because we need around this much data to even confirm such an accuracy (see What is a large enough random sample? ). Also notice how the dimension of the implicit features space (which can in fact be infinite) does not enter into the bound.

However, the bound is also some sense to be expected. It depends on the following strong assumptions (usually not remembered or stated):

Test and training data must be generated in the same manner (i.e. come from the same distribution, this is usually stated but not obeyed).

We must not have collapsing margin as the number of data points goes up (so there must actually be a moat between the positive and negative examples such as portrayed in figure1 where the blurry boundary was a region we never generated training or test examples from).

The expected value of k(u,u) must be bounded for examples drawn from our training distribution. For example this is not true even for the identity kernel if x is just numbers drawn from a Cauchy distribution.

Also it is a posterior estimate of expected generalization error. That is, we only get a bound after we plug in some facts found from fitting a model to the training data (the margin and the expected value of k(u,u)). It is not like the famous prior bounds based on VC dimension that depend only on the structure of the kernel k(,) and not the distribution of training data or labels (beyond the concept being expressible by the kernel, which is itself a posterior fact). The thing to remember for all posterior estimates is you can get such estimates at a small expense in statistical efficiency (i.e. you waste some data) and a moderate computational blow-up by cross validation (and with essentially no math and many fewer assumptions). In fact the cross validation and hold-out estimates may be much tighter than calculated bounds (and are certainly easier to explain).

When we say we weakened the result we mean we have added some stated some conditions that are not strictly needed for the bound to be true. One of these assumptions is that the margin is not collapsing (the bound remains true even for a collapsing margin; it is just not useful). We added this assumption because most readings of the bound (or presumed applications) implicitly assume the bound remains useful (which means something near our additional assumptions must also hold). The reader is really lured to think of the margin w as a constant depending only on the kernel when in fact it depends on the kernel, data distribution and number of training examples. For example: consider a very simple one dimensional classification problem: learning the concept x ≥ 0 from labeled training data drawn uniformly from the interval [-0.5,0.5] using a Gaussian kernel. The support vector models are going to be essentially picking two neighboring points in the observed training sample that have opposite labels and saying the concept boundary is between them. In this simple case the margin is collapsing: as m (the number of training examples increases) the expected margin width is O(1/m). This renders the bound calculated above useless (drives it above 1 as the number of training examples increases). The model is good but the bound is not useful. This is because the training distribution is discovering points closer and closer to the concept boundary (a necessary discontinuity) as the number of training examples grows. This is why we had to add the additional (weakening) assumption that not only do we need to assume the concept to be learned is separable with respect to the kernel we are using, but we must further assume the data-distribution (determining where test and training data come from) stays some minimal distance away from the decision surface (i.e. together the data distribution and underlying concept have a moat, as in our figure 1 example). We will phrase this as “the concept is wide margin separable with respect to the training distribution” (implicitly we need to know the kernel, but it is the training distribution that is critical). This is a very strong assumption (not true for all classification problems) but is often true when the classification task is to “un-mix a mixture” (the observed data is coming from two different sources and the training labels are which source they come from, in this case you often do have a margin). Obviously a soft-margin SVM deals better with this (some of the issue is due to our use of hard-margin)- but it is an interesting exercise to think why you would expect your learned model to have a large margin if your underlying concept and training data do not.

The result remains important: support vector machines are in some sense an optimal learning procedure (up to some constants they achieve as tight a generalization error as can even be confirmed for a given training set size). Another strong point of the support vector result is the result applies even for kernels with unbounded VC dimension (such as the Gaussian kernel).

The Gaussian (or radial) kernel again

We said the Gaussian kernel was of infinite VC dimension. But we have not shown why it is true. The fact that the only encoding we could think of off the top of our head was a power-series or an limit does not guarantee there isn’t some more clever encoding that easily demonstrates that the Gaussian kernel is of bounded VC dimension (like the identity kernel and cosine kernels are).

However we can see the Gaussian kernel can not have finite VC dimension because it can place an arbitrary number of bounded patches in the plane. Thus we can build arbitrarily large sets (by placing all points far apart) that can be shattered (thus falsifying any bounded VC dimension).

One of the wonders of the support vector method is that the theory works even in this situation.

Back to SVM

Support vector machines seem nearly magical in their power to pick “best weights.” However, as we have seen, these best weights may not always be much better than typical good weights (for instance: using all uniform weights). Also it is very important to remember the support vector machine is picking the best weighting of a sum of already picked kernels. It is not picking the best kernel shape or adjusting things like bandwidth (you must pick the best kernel ahead of time or try several kernels to find a good one). You must remember support vector machines can only directly adjust dual weights (or relative training data weighting) this can affect some changes in hypothesis shape but the support vector machine can not directly implement arbitrary changes in hypothesis shape like a nearest neighbor or a parameterized primal method can (like logistic regression with the kernel trick of adding interaction variables).

Support vector machines are to be much preferred to other combiners like Boosting. This is because support vector machines have explicit stopping criteria (conditions known to be true at the optimal solution that characterize the optimal solution) and explicit regularization (control of over fitting). Boosting relies a path argument (“do the computation in these stages”) so it is much harder to reason about and frankly the folklore that “early stop prevents over fitting in boosting” is just false (you are much better off explicitly regularizing).

It is also lore that random kernel transformations (like power series over the cosine kernel) are magic. They can make data separable but they are unlikely to meet the (unstated but critical) conditions of the support vector theorem (margin being wide, margin not filling in and expectation of k(.,.) being small).

A comment on optimization

In this writeup we have skipped the most beautiful part of support vector machines- the form of the optimization problem used to solve them. We strongly suggest consulting John Shawe-Taylor and Nello Cristianini’s “Kernel Methods for Pattern Analysis”, Cambridge 2004 for this. The writing is dense but accurate. You can literally paste their formulas into a general solver and they work.

Conclusion

We have shown how kernel methods and support vector machines fit in a sequence of machine learning methods:

nearest neighbor models

uniform sum models

support vector machines.

We also showed a number of examples of kernels and non-kernels. It is good to be able to quickly remember the constant “-1″ is not a kernel and the constant “1″ is a kernel (though not a very useful one).

It is our argument that support vector machines are “dual methods” (as they work in weights over the data instead of weights over coordinates in the features space) and the move from the primal algorithm Naive Bayes to Logistic regression is similar to the move from uniform kernel sums to support vector machines.

Also, if you want to directly manipulate shape (like picking bandwidth) you must in some sense use a primal method (this isn’t what support vector machines are for).

We end with: If you have some feeling of both what a method can and can not do then you have some understanding of the method.

Related posts:

A Personal Perspective on Machine Learning

Do your tools support production or complexity?

Book Review: Ensemble Methods in Data Mining (Seni & Elder)

The equivalence of logistic regression and maximum entropy models

John Mount — Fri, 23 Sep 2011 16:21:09 +0000

Nina Zumel recently gave a very clear explanation of logistic regression ( The Simpler Derivation of Logistic Regression ). In particular she called out the central role of log-odds ratios and demonstrated how the “deviance” (that mysterious
quantity reported by fitting packages) is both a term in “the pseudo-R^2″ (so directly measures goodness of fit) and is the quantity that is actually optimized during the fitting procedure. One great point of the writeup was how simple everything is once you start thinking in terms of derivatives (and that it isn’t so much the functional form of the sigmoid that is special but its relation to its own derivative that is special).

We adapt these presentation ideas to make explicit the well known equivalence of logistic regression and maximum entropy models.In our new writeup: The equivalence of logistic regression and maximum entropy models we move to multi-category modeling and demonstrate how one invents something as remarkable as logistic regression.

Related posts:

The Simpler Derivation of Logistic Regression

Learn Logistic Regression (and beyond)

Large Data Logistic Regression (with example Hadoop code)

The Simpler Derivation of Logistic Regression

Nina Zumel — Wed, 14 Sep 2011 15:36:37 +0000

Logistic regression is one of the most popular ways to fit models for categorical data, especially for binary response data. It is the most important (and probably most used) member of a class of models called generalized linear models. Unlike linear regression, logistic regression can directly predict probabilities (values that are restricted to the (0,1) interval); furthermore, those probabilities are well-calibrated when compared to the probabilities predicted by some other classifiers, such as Naive Bayes. Logistic regression preserves the marginal probabilities of the training data. The coefficients of the model also provide some hint of the relative importance of each input variable.

While you don’t have to know how to derive logistic regression or how to implement it in order to use it, the details of its derivation give important insights into interpreting and troubleshooting the resulting models. Unfortunately, most derivations (like the ones in [Agresti, 1990] or [Hastie, et.al, 2009]) are too terse for easy comprehension. Here, we give a derivation that is less terse (and less general than Agresti’s), and we’ll take the time to point out some details and useful facts that sometimes get lost in the discussion.

To make the discussion easier, we will focus on the binary response case. We assume that the case of interest (or “true”) is coded to 1, and the alternative case (or “false”) is coded to 0.

The logistic regression model assumes that the log-odds of an observation y can be expressed as a linear function of the K input variables x:

Here, we add the constant term b₀, by setting x₀ = 1. This gives us K+1 parameters. The left hand side of the above equation is called the logit of P (hence, the name logistic regression).

Let’s take the exponent of both sides of the logit equation.

This immediately tells us that logistic models are multiplicative in their inputs (rather than additive, like a linear model), and it gives us a way to interpret the coefficients. The value exp(b_j) tells us how the odds of the response being “true” increase (or decrease) as x_j increases by one unit, all other things being equal. For example, suppose b_j = 0.693. Then exp(b_j) = 2. If x_j is a numerical variable (say, age in years), then every year’s increase in age doubles the odds of the response being true — all other things being equal. If x_j is a binary variable (say, sex, with female coded as 1 and male as 0), then if the subject is female, then the response is two times more likely to be true than if the subject is male, all other things being equal.

We can also invert the logit equation to get a new expression for P(x):

The right hand side of the top equation is the sigmoid of z, which maps the real line to the interval (0, 1), and is approximately linear near the origin. A useful fact about P(z) is that the derivative P’(z) = P(z) (1 – P(z)). Here’s the derivation:

Later, we will want to take the gradient of P with respect to the set of coefficients b, rather than z. In that case, P’(z) = P(z) (1 – P(z))z‘, where ‘ is the gradient taken with respect to b.

The solution to a Logistic Regression problem is the set of parameters b that maximizes the likelihood of the data, which is expressed as the product of the predicted probabilities of the N individual observations.

(X, y) is the set of observations; X is a K+1 by N matrix of inputs, where each column corresponds to an observation, and the first row is 1; y is an N-dimensional vector of responses; and (x_i, y_i) are the individual observations.

It’s generally easier to work with the log of this expression, known (of course) as the log-likelihood.

Maximizing the log-likelihood will maximize the likelihood. As a side note, the quantity −2*log-likelihood is called the deviance of the model. It is analogous to the residual sum of squares (RSS) of a linear model. Ordinary least squares minimizes RSS; logistic regression minimizes deviance. A useful goodness-of-fit heuristic for a logistic regression model is to compare the deviance of the model with the so-called null deviance: the deviance of the constant model that returns only the global response probability for every data point. One minus the ratio of deviance to null deviance is sometimes called pseudo-R², and is used the way one would use R² to evaluate a linear model.

Traditional derivations of Logistic Regression tend to start by substituting the logit function directly into the log-likelihood equations, and expanding from there. The derivation is much simpler if we don’t plug the logit function in immediately. To maximize the log-likelihood, we take its gradient with respect to b:

where P_i is shorthand for P(x_i). The maximum occurs where the gradient is zero.

We can expand this equation further, when we remember that P’ = P(1-P):

The last line merges the two cases (y_i = 1 and y_i = 0) into a single sum. We can now cancel terms and set the gradient to zero. This gives us the set of simultaneous equations that are true at the optimum:

Notice that the equations to be solved are in terms of the probabilities P (which are a function of b), not directly in terms of the coefficients b themselves. This means that logistic models are coordinate-free: for a given set of input variables, the probabilities returned by the model will be the same even if the variables are shifted, combined, or rescaled. Only the values of the coefficients will change.

The other thing to notice from the above equations is that the sum of probability mass across each coordinate of the x_i vectors is equal to the count of observations with that coordinate value for which the response was true. For example, suppose the jth input variable is 1 if the subject is female, 0 if the subject is male. Then

In other words, the summed probability mass for the female subjects equals the count of female subjects with the response “true”. It is also true that the sum of all the probability mass over the entire training set will equal the number of “true” responses in the training set. This is what we mean when we say that logistic regression preserves the marginal probabilities of the training data.

Solving for the Coefficients

The most straightforward way to solve for the coefficients b is Newton’s method. The Fisher scoring method that is used in most off-the-shelf implementations is a more general variation of Newton’s method; it works on the same principles. We will describe solving for the coefficients using Newton’s method.

Suppose you have a vector valued function f: y = f(b). You want to find the value b_opt such that f(b)_opt = 0. Assuming that we start with an initial guess b₀, we can take the Taylor expansion of f around b₀:

Here, f‘ is a matrix; it is the Jacobean of first derivatives of f with respect to b. Setting the left hand side to zero, we can solve for Δ as

We then update our estimate for b:

and iterate until convergence.

In our case, f is the gradient of the log-likelihood, and its Jacobean is the Hessian (the matrix of second derivatives) of the log-likelihood function.

where W is a diagonal matrix of the derivatives P’_i, and the ith column of X corresponds to the ith observation. So we can solve for Δ at each iteration as

where W is the current matrix of derivatives, y is the vector of observed responses, and P_k is the vector of probabilities as calculated by the current estimate of b.

Compare this to the solution of a linear regression:

Comparing the two, we can see that at each iteration, Δ is the solution of a weighted least square problem, where the “response” is the difference between the observed response and its current estimated probability of being true. This is why the technique for solving logistic regression problems is sometimes referred to as iteratively re-weighted least squares. Generally, the method does not take long to converge (about 6 or so iterations).

Thinking of logistic regression as a weighted least squares problem immediately tells you a few things that can go wrong, and how. For example, if some of the input variables are correlated, then the Hessian H will be ill-conditioned, or even singular. This will result in large error bars (or “loss of significance”) around the estimates of certain coefficients. It can also result in coefficients with excessively large magnitudes, and often the wrong sign. If an input perfectly predicts the response for some subset of the data (but not all), then the term P_i (1 – P_i) will be driven to zero for that subset, which will drive the coefficient for that input to infinity (if the input perfectly predicted all the data, then the residual (y – P_k) has already gone to zero, which means that you are already at the optimum).

On the other hand, the least squares analogy also gives us the solution to these problems: regularized regression, such as lasso or ridge. Regularized regression penalizes excessively large coefficients, and keeps them bounded. If you are implementing your own logistic regression procedure, rather than using a package, then it is straightforward to implement a regularized least squares for the iteration step (as Win-Vector has done). But even if you are using an off-the-shelf implementation, the above discussion will help give you a sense of how to interpret the coefficients of your model, and how to recognize and troubleshoot some issues that might arise.

Conclusion

Here is what you should now know from going through the derivation of logistic regression step by step:

- Logistic regression models are multiplicative in their inputs.

- The exponent of each coefficient tells you how a unit change in that input variable affects the odds ratio of the response being true.

- Logistic regression is coordinate-free: translations, rotations, and rescaling of the input variables will not affect the resulting probabilities.

- Logistic regression preserves the marginal probabilities of the training data.

- Overly large coefficient magnitudes, overly large error bars on the coefficient estimates, and the wrong sign on a coefficient could be indications of correlated inputs.

- Coefficients that tend to infinity could be a sign that an input is perfectly correlated with a subset of your responses. Or put another way, it could be a sign that this input is only really useful on a subset of your data, so perhaps it is time to segment the data.

- Pseudo-R² is a useful goodness-of-fit heuristic.

References

[Agresti, 1990] Agresti, A. (1990). Categorical Data Analysis.

[Hastie, et.al, 2009] Hastie, T., R. Tibshirani, and J. Friedman (2009). The Elements of Statistical Learning, 2nd Edition.

Related posts:

The equivalence of logistic regression and maximum entropy models

Learn Logistic Regression (and beyond)

Large Data Logistic Regression (with example Hadoop code)

Programmers Should Know R

John Mount — Sat, 06 Aug 2011 15:29:22 +0000

Programmers should definitely know how to use R. I don’t mean they should switch from their current language to R, but they should think of R as a handy tool during development.Again and again I find myself working with Java code like the following.

public class SomeBigProject1 { public static double logStirlingApproximation(final int n) { return n*(Math.log(n)-1) + 0.5*Math.log(2*Math.PI*n); } public static double logFactorial(final int n) { double r = 0.0; for(int i=n;i>1;--i) { r += Math.log(i); } return r; } public static void main(final String[] args) { int nbad = 0; for(int n=1000;n<10000;++n) { if(Math.abs(logFactorial(n)-logStirlingApproximation(n))>=1.0e-5) { ++nbad; } } System.out.println("nbad: " + nbad); } }

Imagine that this is some humongous project to use Stirling’s Approximation as a replacement for factorial. All the code up until main is great. But the unfortunate developer has hard-coded an acceptance test into main(). If they run their big project all they get out is:

nbad: 7334

The developer needs to re-code and re-build to diagnose the failure, tweak their acceptance criteria or add more measurements.

I strongly recommend a different work pattern. Instead of bringing criteria into the code, bring the data out:

public class SomeBigProject2 { public static void main(final String[] args) { System.out.println("n" + "\t" + "logFactorial" + "\t" + "logStirlingApproximation"); for(int n=1000;n<10000;++n) { System.out.println(String.valueOf(n) + "\t" + SomeBigProject1.logFactorial(n) + "\t" + SomeBigProject1.logStirlingApproximation(n)); } } }

Capture this output in a file named “data.tsv” and both Microsoft Excel and R can open it. Naturally I prefer to use R (so that is what I will demonstrate). To read the results into R you start up an R and type in a command like the following:

> d <- read.table('data.tsv', header=T,sep='\t',quote='',as.is=T, stringsAsFactors=F,comment.char='',allowEscapes=F)

Most of the arguments controlling the style of file R is to expected (what the field separator is, weather to expect escapes and quotes and so on). The settings I suggest here are the “ultra hardened” settings. If you make sure none of your fields have a tab or line-break in them when you print then it is guaranteed R can read the data (not matter what whacky symbols are in it). On the java side that usually means making sure any varying text fields are run through .replaceAll("\\s+"," ") “just in case.” At this point you can already look at your data with the summary() command:

> summary(d)

n logFactorial logStirlingApproximation Min. :1000 Min. : 5912 Min. : 5912 1st Qu.:3250 1st Qu.:23034 1st Qu.:23034 Median :5500 Median :41870 Median :41870 Mean :5500 Mean :42536 Mean :42536 3rd Qu.:7749 3rd Qu.:61653 3rd Qu.:61653 Max. :9999 Max. :82100 Max. :82100

This immediately hints that you should have been thinking in terms of relative error instead of absolute error (since insisting on high absolute accuracy on large results does not always make sense).

You also have access to standard statistical measures of agreement like correlation:

> with(d,cor(logFactorial,logStirlingApproximation))

result: 1

You can see where your failures were:

> library(ggplot2) > d$bad <- with(d,abs(logFactorial-logStirlingApproximation)>=1.0e-5) > ggplot(d) + geom_point(aes(x=n,y=bad))

Yields the graph:

You can see all your failures are in the initial interval. You can then drill in:

> ggplot(d) + geom_point(aes(x=n,y=logFactorial-logStirlingApproximation)) + scale_y_log10()

And here we see some things (that are in general true for Stirling’s approximation):

It is very accurate.

It is always an under estimate.

It gets better as n gets larger.

Essentially by poking around with graphs in R you can figure out the nature of your errors (telling you what to fix) and generate findings that tell you how to fix your criteria (perhaps your code is working- but your test wasn’t sensible). The “dump everything and then use R” technique is also particularly good for generating reports on code timings using either geom_histogram or geom_density.

For example, if we had data with a field runTimeMS then it is a simple one-liner to get plot like the following:

> ggplot(t) + geom_density(aes(x=runTimeMS))

From this graph we can immediately see:

Most of our run-times are very fast.

We have a heavy right-tail (evidence of “contagion” or one slow-down causing others, like CPU or IO contention).

Data is truncated at 100MS (could be something “censoring” the measurement, an exception being thrown or an abort).

There is a spike at 30MS (something is true and slow for some subset of the data that isn’t present in the majority).

This is a lot more that would be seen in a mean-only or mean and standard deviation summary. We may even being seeings signs of two different bugs (the truncation and the spike).

In all cases the key is to dump a lot of data in machine readable form and then come back to to analyze. This is far more flexible than hoping to code in the right summaries and then further hoping the summaries don’t miss something important (or that you at least get a chance to notice if they do miss something). Being able to do exploratory statistics on dumps from your code (both results and timing) gives you incredible measurement, tuning and debugging powers. The scriptability of R means any later analysis is as easy as cut and paste.

Related posts:

Automatic Differentiation with Scala

R examine objects tutorial

Survive R

Your Data is Never the Right Shape

John Mount — Sun, 31 Jul 2011 20:27:00 +0000

One of the recurring frustrations in data analytics is that your data is never in the right shape. Worst case: you are not aware of this and every step you attempt is more expensive, less reliable and less informative than you would want. Best case: you notice this and have the tools to reshape your data.

There is no final “right shape.” In fact even your data is never right. You will always be called to re-do your analysis (new variables, new data, corrections) so you should always understand you are on your “penultimate analysis” (always one more to come). This is why we insist on using general methods and scripted techniques, as these methods are much much easier to reliably reapply on new data than GUI/WYSWYG techniques.

In this article we will work a small example and call out some R tools that make reshaping your data much easier. The idea is to think in terms of “relational algebra” (like SQL) and transform your data towards your tools (and not to attempt to adapt your tools towards the data in an ad-hoc manner).Take a simple example where you are designing a new score called “score2” to predict or track an already known value called “score1.” The typical situation is score1 is a future outcome (such as the number of dollars profit on a transaction) and score2 is a prediction (such as the estimated profit before the transaction is attempted). Training data is usually assembled by performing a large number of transactions, recording what was known before the transaction and then aligning or joining this data with measured results when they become available. For this example we are not interested in the inputs driving the model (a rare situation, but we are trying to make our example as simple as possible) but only examining the quality of score2 (which is defined as how well it tracks score1).

All of this example will be in R, but the principles are chosen apply more generally. First let us enter some example data:

> d <- data.frame(id=c(1,2,3,1,2,3),score1=c(17,5,6,10,13,7),score2=c(13,10,5,13,10,5)) > d

This gives us our example data. Each row is numbered (1 through 6) has an id and both our scores:

id score1 score2 1 1 17 13 2 2 5 10 3 3 6 5 4 1 10 13 5 2 13 10 6 3 7 5

We said our only task was to characterize how well score2 works at predicting score1 (or how good a substitute score2 is for score1). We could compute correlation, RMS error, info-gain or some such. But instead lets look at this graphically. We will prepare a graph showing how well score1 is represented by score2. For this we choose to place score1 on the y-axis (as it is the outcome) and score2 on the x-axis (as it is the driver).

> library(ggplot2) > ggplot(d) + geom_point(aes(x=score2,y=score1))

Figure 1: score1 as a function of score2.

This does not look good. We would liked to have seen all of the dots falling on the line “y=x.” This plot shows score2 is not predicting score1 very well. Part of this is that we missed an important feature of the data (and because we missed it the feature becomes a problem): the ids repeat. First we re-order by id to make this more obvious.

> dsort <- d[order(d$id),] > dsort

id score1 score2 1 1 17 13 4 1 10 13 2 2 5 10 5 2 13 10 3 3 6 5 6 3 7 5

This is a very common situation. The original score is not completely a function of the known inputs. We are using “id” to abstract represent all of the inputs, two rows in our example have the same id if and only if all known inputs are exactly the same. The repeating ids are the same experiment run at different times (a good idea) and the variation in score1 could be the effect of an un-modeled input that changed value or something simple like a “noise term” (a random un-modeled effect). Notice that score2 is behaving as a function of id- all rows with the same id have the same value for score2. If score2 is a model then it has to be a function of the inputs (or more precisely if it is not a function of the inputs you have done something wrong). So any variation of score1 between rows with identical id is “unexplainable variation” (unexplainable from the point of view of currently tracked inputs). You should know about, characterize and report this variation (why it is good to have some repeated experiments). But this variation is not the model’s fault, if we want to know how good a job we did constructing the model (which we now see can be a slightly different question than how well the model works at prediction) we need to see how much of the explainable variation the model accounts for.

If we assume (as is traditional) the unexplained variation is from a “unbiased noise source” then we can lessen the impact of the noise source by replacing score1 with a value averaged over rows with the same id. This assumption is traditional because an unbiased noise source is present in many problems and assuming anything more requires more research into the problem domain. You would eventually fold such research into your model- so your goal is always have all effects or biases in your model and hope what is left over is unbiased. This is usually not strictly true, but not accounting for the unexplained variation at all is in many cases even worse than modeling the unexplained variation as being bias-free.

And now we find our data is the “wrong shape.” To replace score1 with the appropriate averages we need to do some significant data manipulation. We need to group sets of rows and add new columns. We could do this imperatively (write some loops and design some variables to track and manipulate state) or declaratively (find a path of operations from what you have to what you need through R’s data manipulation algebra). Even though the declarative method is more work the first time (you could often write the code in less time than it takes to read the manuals) it is the right way to go (as it is more versatile and powerful in the end).

Luckily we don’t have to use raw R. There are a number of remarkable packages (all by Hadley Wickham who is also the author of the ggplot2 package we use to prepare our figures) that really improve R’s ability to coherently manage data. The easiest (on us) way do fix up our data is to make the computer work hard and use the powerful melt/cast technique. These functions are found in the libraries reshape and plyr (which were automatically loaded with we loaded ggplot2 library).

melt is a bit abstract. What it does convert your data into a “narrow” format where rows are split into many rows each carrying just one result column of the original row. For example we can melt our data by id as follows:

> dmelt <- melt(d,id.vars=c('id')) > dmelt

Which yields the following:

id variable value 1 1 score1 17 2 1 score1 10 3 2 score1 5 4 2 score1 13 5 3 score1 6 6 3 score1 7 7 1 score2 13 8 1 score2 13 9 2 score2 10 10 2 score2 10 11 3 score2 5 12 3 score2 5

Each of the two facts (score1, score2) from our original row is split into its own row. The id column plus the new variable column are now considered to be keys. This format is not used directly but used because it is easy to express important data transformations in terms of it. For instance we wanted our table to have duplicate rows collected and score1 replaced by its average (to attempt to remove the unexplainable variation). That is now easy:

> dmean <- cast(dmelt,fun.aggregate=mean) > dmean

id score1 score2 1 1 13.5 13 2 2 9.0 10 3 3 6.5 5

We used cast() in its default mode, where it assumes all columns not equal to “value” are the keyset. It then collects all rows with identical keying and combines them back into wide rows using mean or average as the function to deal with duplicates. Notice score1 is now the desired average, and score2 is as before (as it was a function of the keys or inputs, so it is not affected by averaging). With this new smaller data set we can re-try our original graph:

> ggplot(dmean) + geom_point(aes(x=score2,y=score1))

Figure 2: mean(score1) as a function of score2.

This doesn’t look so bad. A lot of the error or variation in the first plot was unexplainable variation. score2 isn’t bad given its inputs. If you wanted to do better than score2 you would be advised to find more modeling inputs (versus trying more exotic modeling techniques).

Of course a client or user is not interested if score2 is “best possible.” They want to know if it is any good. To do this we should show them (either by graph or by quantitative summary statistics like we mentioned earlier) at least 3 things:

How well the model predicts overall (the very first graph we presented).

How much of the explainable variation the model predicts (the second graph).

The nature of the unexplained variation (which we will explore next).

We said earlier we are hoping the unexplained variation is noise (or if it is not noise it would be nice if it is a clue to new important modeling features). So the unexplained variation must not go unexamined. We will finish by showing how to characterize the unexplained variation. As before will will just make a graph, but the data preparation steps would be exactly the same if we were using a quantitive summary (like correlation, or any other). And, of course, our data is still not the right shape for this step. Luckily there is another tool ready to fix this: join().

> djoin <- join(dsort,dsort,'id') > fixnames <- function(cn) { n <- length(cn); for(i in 2:((n+1)/2)) { cn[i] <- paste('a',cn[i],sep='') }; for(i in ((n+3)/2):n) { cn[i] <- paste('b',cn[i],sep='') }; cn } > colnames(djoin) <- fixnames(colnames(djoin)) > djoin

which produces:

id ascore1 ascore2 bscore1 bscore2 1 1 17 13 17 13 2 1 17 13 10 13 3 1 10 13 17 13 4 1 10 13 10 13 5 2 5 10 5 10 6 2 5 10 13 10 7 2 13 10 5 10 8 2 13 10 13 10 9 3 6 5 6 5 10 3 6 5 7 5 11 3 7 5 6 5 12 3 7 5 7 5

All of the work was done by the single line “djoin <- join(dsort,dsort,'id')” the rest was just fixing the column names (as self-join is not the central use case of join). What we have now is a table that is exactly right for studying unexplained variation. For each id we have each row with the same id matched. This blows every id from having 2 rows in dsort to 4 rows in djoin. Notices this gives us every pair of score1 values seen for the same id (which will let us examine unexplained variation) and score2 is still constant over all rows with the same id (as it has always been throughout our analysis). With this table we can now plot how score1 varies for rows with the same id:

> ggplot(djoin) + geom_point(aes(x=ascore1,y=bscore1))

Figure 3: score1 as a function of score1.

And we can see, as we expected, the unexplained variation in score1 is about as large as the mismatch between score1 and score2 in our original plot. The important thing is this is all about score1 (score2 is now literally out of the picture). The analyst’s job would now be to try and tie bits of the unexplained variation to new inputs (that can be folded into a new score2) and/or characterize the noise term (so the customer knows how close they should expect repeated experiments to be).

What we are trying to encourage with the use of “big hammer tools” is an ability and willingness to look at and transform your data in meaningful steps. It often seems easier and more efficient to build one more piece of data tubing, but a lot of data tubes become an unmanageable collection of spaghetti code. The analyst should, in some sense, always be looking at data and not looking at coding details. For these sort of analyses we encourage analysts to think in terms of “data shape” and transforms. This discipline leaves more of the analysts energy and attention to think productively about the data and actual problem domain.

Note:

For the third plot showing the variation of score1 across different rows (but same ids) it may be appropriate to use a slightly more complicated join() procedure than we showed. The join shown produced rows of artificial agreement where both values of score1 came from the same row (thus had no chance of being different, so in some sense deserve no credit). This is also the only way any non-duplicated evaluations could make it to the plot. To eliminate these uninteresting agreements from the plot do the following:

> d$rowNumber <- 1:(dim(d)[1]) > djoin <- join(d,d,'id') > colnames(djoin) <- fixnames(colnames(djoin)) > djoin <- djoin[djoin$arowNumber!=djoin$browNumber,] > djoin

This gives us a table that shows only values of score1 from different rows:

id ascore1 ascore2 arowNumber bscore1 bscore2 browNumber 2 1 17 13 1 10 13 4 4 2 5 10 2 13 10 5 6 3 6 5 3 7 5 6 7 1 10 13 4 17 13 1 9 2 13 10 5 5 10 2 11 3 7 5 6 6 5 3

And only plots points on the diagonal if “you have really earned them”:

So while the direct join() may not be the immediate perfect answer it is still a good intermediate to form as what you want is only simple data transformation away from it.

Related posts:

A Demonstration of Data Mining

Good Graphs: Graphical Perception and Data Visualization

The Data Enrichment Method

What is a large enough random sample?

John Mount — Sun, 26 Jun 2011 18:54:14 +0000

With the well deserved popularity of A/B testing computer scientists are finally becoming practicing statisticians. One part of experiment design that has always been particularly hard to teach is how to pick the size of your sample. The two points that are hard to communicate are that:

The required sample size is essentially independent of the total population size.

The required sample size depends strongly on the strength of the effect you are trying to measure.

These things are only hard to explain because the literature is overly technical (too many buzzwords and too many irrelevant concerns) and these misapprehensions can’t be relieved unless you spend some time addressing the legitimate underlying concerns they are standing in for. As usual explanation requires common ground (moving to shared assumptions) not mere technical bullying.

We will try to work through these assumptions and then discuss proper sample size.

The problem of population size.

Many technical people (including some good physical scientists) have a hard time understanding that to test the effect of a treatment on a population the sample size you need does not depend on the size of the overall population. You see vestiges of this discomfort when for a treatment or an opinion poll you seen something like the percent of the population polled listed as an important feature. The math says this is wrong, but there is absolutely no point in trotting out the math until we move to common ground (or a shared set of assumptions). After all math is just the pushing of axioms to their consequences- so if you don’t share the same axioms the math is irrelevant.

The statistical definition of a good sample from a population usually insists on independence or exchangeability. That is: if we draw a sample of 100 people from a population of 1,000,000 that each person in the sample had an equal chance of being in the sample independent of who else was in the sample or who else was left out of the sample. Or (in a slightly different form): if we had a good sample and then picked one of the sample members uniformly at random and exchanged them with one of the non sample members (also picked uniformly at random) we end up with a different but equally good sample.

The ideal notion of a sample is what is at odds with experience. People know that their sampling procedure is often unlikely to rise to the standard demanded by statistics. A doctor knows that the patients that walk into his office are not an exchangeable sample of the total population (his or her patients tend to live near them, may be grouped in age and share many other common traits). Political polls (which are typically done on land line telephones) have been famously bad samples dating back from “Dewey Beats Truman” mis-prediction (when telephones were more concentrated among the rich) to the current blindness of land-line telephone polls to the younger cell phone only sub-population. Another example is small molecule drug discovery where experiments are inevitably run where each molecule is related to previous experiments and earlier experiments have a strong bias towards cheaper reagent cost and low molecular weight.

These real world situations are non-uniform, non-stationary, not independent and non-exchangeable (to use the statistical terms). Since the usual statistical requirements are not met it should not be surprising the desired statistical theories do not apply. This isn’t a paradox it is just the real world failing to meet the desired assumptions. The intuition that if your sample is not in a trustable order that you must increase sample size to cover the entire population is essentially justified. But that misses the often cheaper alternative of fixing your sampling procedure so you don’t need to push to large sample sizes.

This is why statisticians are so concerned with experimental design and experimental procedure. If you can design your experiment so the standard statistical assumptions are met: you get the enormous benefits of the theory.

The problem of effect strength.

The basic problem of effect strength is simple, so in this case part of the confusion is actually from not having thought hard about it. The (incorrect) common intuition is that the size of an experimental sample needed is independent of the type of effect you are trying to measure. You see this implicitly when studies are commissioned that have no hope of success. Typically this would be A/B testing a minor change on a web page on a small population (perhaps too small to even see the change) or even running an expensive clinical trial with too few patients. These recurring errors come from the mistaken intuition that the effect strength isn’t important in designing your experiment.

To try and set your intuition consider the extreme cases. Suppose you are trying to test if an insecticide is “very very very deadly” to fleas. If the insecticide is strong then the only confounding effect would be natural death of fleas. You need an experiment size big enough that it is unlikely most of your fleas died of natural causes. So maybe use 5 fleas instead of 1. The measurement is: dose the fleas, if they all die you can conclude the insecticide is very deadly. If only half die: the experiment fails (the insecticide is deadly, but not very very deadly). The key to good statistical experiment design is committing to what you are trying to test before starting the experiment. This is the important thing. After that you can improve your technique by adding such methods as “control groups” (groups of untreated fleas to try to get an estimate of the base mortality rate) and repetitions (trying to control for systematic error, all the fleas in one cage dying due to some other cause, like spilled cleaning solution).

Now consider the same experiment and suppose you were trying to establish the same insecticide “kills all fleas and their eggs.” The word “all” made this something that can not be proven by empirical experiment. This is too strong a statement to be proven empirically. The only way to prove a categorical absolute is through logic (not empiricism or statistics). This sort of proof looks something like:

Major Premise: Sulfuric acid completely dissolves organic matter.

Minor Premise: Fleas are organic matter.

Conclusion:Sulfuric acid completely dissolves fleas.

That is, you don’t run an experiment to prove an absolute- you combine other things you already accept as absolutes to infer additional absolutes. (We dodge the question how you acquire these initial absolutes, we just point out you can usually only create new absolutes by combining other absolutes).

One last variation: suppose we are asked to prove the insecticide “kills at least 99.9% of all fleas.” This can be done by sampling- but it requires a big sample as we are trying to prove that very very few fleas survive (or in technical terms- trying to measure a rare event).

This is deliberately silly, but the point is: easy effects can use small samples, hard effects require big samples and absolutes can’t be measured by sampling.

There is a second, very subtle problem regarding effect strengths: they cost more than you would reasonably suppose. Roughly this is because when you try to measure weaker (more subtle) effects you also increase your need for precision. That is it might make sense to try and measure a disease that affects 10% of the population to +-5% absolute error (that is accept any number from 15% through 5% as being a good enough estimate). But when you try to measure a disease that affects 1% of the population this wide interval ( 6% to -4%) no longer makes sense. You would likely be required to use a +-0.5% interval. You would need 10 times as much data to see the same number of affected patients (1% versus 10% incidence rate) and on top of that you are now requiring a high precision ( +- 0.5% instead of +-5%). You end up needing 100 times the data to get a similar confidence on the smaller measurement range (+-0.5%). In my opinion: this is always going to feel wrong- you expect to need 10 times the data but you actually need 100 times the data (due to your change in measurement range).

The rule of thumb.

The typical requirements of a random experiment are: you are given a very small number d>0 and a small number e>0 and you want a sample such that if p is the (unknown) true proportion of the world that has the property you are trying to measure then with probability no more than d your measured proportion q is bad (i.e. |p-q| > e). If somebody asks for d or e to be zero then you are pretty much forced to test the whole population (sampling won’t work). The larger d and e get the easier sampling becomes.

The rules to know in designing experiment sizes are:

d is easy to lower (it doesn’t take much work to increase confidence).

e is hard to lower (it takes a lot of work to increase precision).

A good formula is: if your sample is picked uniformly at random and is of size at least:

then with probability at least 1-d your sample measured proportion q is no further away than e from the unknown true population proportion p (what you are trying to estimate). This formula is technical consequence of Hoeffding’s inequality (it is a convenient form, you can get a slightly better bound by directly using the Chernoff bounds). The point is: you can plug this into a calculator. Need to estimate with 99% certainty the unknown true population rate to +- 10% precision? That is d=0.01, e=0.1 and a sample of size 265 should be enough. A stricter precision of +-0.05 causes the estimated sample size to increase to 1060, but increasing the certainty requirement to 99.5% only increases the estimated sample size to 300. This is what me mean by confidence is cheap and precision is expensive.

Notice the effect strength doesn’t directly enter into the formula- it only comes in the increase in precision needed to measure weak effects. This is why you want to avoid attempting to empirically measure weak effects (see Analytics Sabotage).

Also notice the faster than linear dependence on 1/e (a square in this case). This is very expensive (liner would be better) and unintuitive. The undesirable super-linear rate follows from the fact that if you flip a fair coin (50/50 odds of heads tails and independent outcomes on each flip) n-times the observed number of heads tends to be more than +-sqrt(n) away from the expected value of n/2. To get a linear rate it would have to be usually within a constant from n/2 (independent of n!) which is too perfect and way too much to hope for (also relevant is the law of the iterated logarithm). This is “hand wavy” but it is intended to help you reset your intuition (intuiting being a less precise, but less brittle way of reasoning than doing all the math).

A very technical paper on sampling from very large populations.

A long time ago we took some of the ideas here to the extreme. We considered the situation of “stream processing”- that is when you have so much data flowing by you that you do not even have enough storage to remember a significant fraction of it. This might arise in a networking application, a scientific instrument or a map-reduce node. The question is: can you still estimate rates in this situation? The answer is you can. The trick is even though you don’t have enough memory to store a significant fraction of what you saw (or even enough memory to store your sample design!) you can store the specification of a pseudorandom sample. A pseudorandom sample is like a sample except it has a very succinct description (a description small enough to store). The idea is to store only the description of the pseudorandom sample (not the sample itself) and the intersection of this sample and the stream (or mark “failure” if this intersection gets too large).

There are some additional technical details in that you have to simultaneously work with many guesses of the unknown true population rate (because each sample design only works properly for a narrow range of this unknown rate as you either miss too much or run out of memory when you use the wrong design). But overall the ideas in this paper are just the type of bounds we discussed here (but in a very compressed form for publication).

This sort of paper is what we theorists call “fun” (or really just sharpening our knives between real world problems):

Estimating the range of a function in an online setting, J. A. Mount, Information Processing Letters 72 31–35 (1999) .

Related posts:

Large Data Logistic Regression (with example Hadoop code)

Volunteers in Large Clubs: The Theorist’s View

Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’

Win-Vector Blog » Tutorials

Six Fundamental Methods to Generate a Random Variable

Introduction

The Methods

Physical sources

Empirical resampling

Pseudo random generators

Simulation/Game-play

Rejection Sampling

Transform methods

Conclusion

What to do when you run out of memory

My Favorite Graphs

An Appreciation of Locality Sensitive Hashing

Kernel Methods and Support Vector Machines de-Mystified

Goals of this writeup

Example Problem

Nearest Neighbor Solution

Functional Solution

Bandwidth Solution

Data weighting solution (support vector machine)

Precise functions used

About Kernels

Back to Support Vector Machines

Back to margin

The Gaussian (or radial) kernel again

Back to SVM

A comment on optimization

Conclusion

The equivalence of logistic regression and maximum entropy models

The Simpler Derivation of Logistic Regression

Programmers Should Know R

Your Data is Never the Right Shape

What is a large enough random sample?

The problem of population size.

The problem of effect strength.

The rule of thumb.

A very technical paper on sampling from very large populations.