There is no final “right shape.” In fact even your data is never right. You will always be called to re-do your analysis (new variables, new data, corrections) so you should always understand you are on your “penultimate analysis” (always one more to come). This is why we insist on using general methods and scripted techniques, as these methods are much much easier to reliably reapply on new data than GUI/WYSWYG techniques.

In this article we will work a small example and call out some R tools that make reshaping your data much easier. The idea is to think in terms of “relational algebra” (like SQL) and transform your data towards your tools (and not to attempt to adapt your tools towards the data in an ad-hoc manner).Take a simple example where you are designing a new score called “score2” to predict or track an already known value called “score1.” The typical situation is score1 is a future outcome (such as the number of dollars profit on a transaction) and score2 is a prediction (such as the estimated profit before the transaction is attempted). Training data is usually assembled by performing a large number of transactions, recording what was known before the transaction and then aligning or joining this data with measured results when they become available. For this example we are not interested in the inputs driving the model (a rare situation, but we are trying to make our example as simple as possible) but only examining the quality of score2 (which is defined as how well it tracks score1).

All of this example will be in R, but the principles are chosen apply more generally. First let us enter some example data:



> d <- data.frame(id=c(1,2,3,1,2,3),score1=c(17,5,6,10,13,7),score2=c(13,10,5,13,10,5))

> d


This gives us our example data. Each row is numbered (1 through 6) has an id and both our scores:

  id score1 score2
1  1     17     13
2  2      5     10
3  3      6      5
4  1     10     13
5  2     13     10
6  3      7      5

We said our only task was to characterize how well score2 works at predicting score1 (or how good a substitute score2 is for score1). We could compute correlation, RMS error, info-gain or some such. But instead lets look at this graphically. We will prepare a graph showing how well score1 is represented by score2. For this we choose to place score1 on the y-axis (as it is the outcome) and score2 on the x-axis (as it is the driver).



> library(ggplot2)

> ggplot(d) + geom_point(aes(x=score2,y=score1))


Figure 1: score1 as a function of score2.

This does not look good. We would liked to have seen all of the dots falling on the line “y=x.” This plot shows score2 is not predicting score1 very well. Part of this is that we missed an important feature of the data (and because we missed it the feature becomes a problem): the ids repeat. First we re-order by id to make this more obvious.



> dsort <- d[order(d$id),]

> dsort


  id score1 score2
1  1     17     13
4  1     10     13
2  2      5     10
5  2     13     10
3  3      6      5
6  3      7      5

This is a very common situation. The original score is not completely a function of the known inputs. We are using “id” to abstract represent all of the inputs, two rows in our example have the same id if and only if all known inputs are exactly the same. The repeating ids are the same experiment run at different times (a good idea) and the variation in score1 could be the effect of an un-modeled input that changed value or something simple like a “noise term” (a random un-modeled effect). Notice that score2 is behaving as a function of id- all rows with the same id have the same value for score2. If score2 is a model then it has to be a function of the inputs (or more precisely if it is not a function of the inputs you have done something wrong). So any variation of score1 between rows with identical id is “unexplainable variation” (unexplainable from the point of view of currently tracked inputs). You should know about, characterize and report this variation (why it is good to have some repeated experiments). But this variation is not the model’s fault, if we want to know how good a job we did constructing the model (which we now see can be a slightly different question than how well the model works at prediction) we need to see how much of the explainable variation the model accounts for.

If we assume (as is traditional) the unexplained variation is from a “unbiased noise source” then we can lessen the impact of the noise source by replacing score1 with a value averaged over rows with the same id. This assumption is traditional because an unbiased noise source is present in many problems and assuming anything more requires more research into the problem domain. You would eventually fold such research into your model- so your goal is always have all effects or biases in your model and hope what is left over is unbiased. This is usually not strictly true, but not accounting for the unexplained variation at all is in many cases even worse than modeling the unexplained variation as being bias-free.

And now we find our data is the “wrong shape.” To replace score1 with the appropriate averages we need to do some significant data manipulation. We need to group sets of rows and add new columns. We could do this imperatively (write some loops and design some variables to track and manipulate state) or declaratively (find a path of operations from what you have to what you need through R’s data manipulation algebra). Even though the declarative method is more work the first time (you could often write the code in less time than it takes to read the manuals) it is the right way to go (as it is more versatile and powerful in the end).

Luckily we don’t have to use raw R. There are a number of remarkable packages (all by Hadley Wickham who is also the author of the ggplot2 package we use to prepare our figures) that really improve R’s ability to coherently manage data. The easiest (on us) way do fix up our data is to make the computer work hard and use the powerful melt/cast technique. These functions are found in the libraries reshape and plyr (which were automatically loaded with we loaded ggplot2 library).

melt is a bit abstract. What it does convert your data into a “narrow” format where rows are split into many rows each carrying just one result column of the original row. For example we can melt our data by id as follows:



> dmelt <- melt(d,id.vars=c('id'))

> dmelt


Which yields the following:

   id variable value
1   1   score1    17
2   1   score1    10
3   2   score1     5
4   2   score1    13
5   3   score1     6
6   3   score1     7
7   1   score2    13
8   1   score2    13
9   2   score2    10
10  2   score2    10
11  3   score2     5
12  3   score2     5

Each of the two facts (score1, score2) from our original row is split into its own row. The id column plus the new variable column are now considered to be keys. This format is not used directly but used because it is easy to express important data transformations in terms of it. For instance we wanted our table to have duplicate rows collected and score1 replaced by its average (to attempt to remove the unexplainable variation). That is now easy:



> dmean <- cast(dmelt,fun.aggregate=mean)

> dmean


  id score1 score2
1  1   13.5     13
2  2    9.0     10
3  3    6.5      5

We used cast() in its default mode, where it assumes all columns not equal to “value” are the keyset. It then collects all rows with identical keying and combines them back into wide rows using mean or average as the function to deal with duplicates. Notice score1 is now the desired average, and score2 is as before (as it was a function of the keys or inputs, so it is not affected by averaging). With this new smaller data set we can re-try our original graph:



> ggplot(dmean) + geom_point(aes(x=score2,y=score1))


Figure 2: mean(score1) as a function of score2.

This doesn’t look so bad. A lot of the error or variation in the first plot was unexplainable variation. score2 isn’t bad given its inputs. If you wanted to do better than score2 you would be advised to find more modeling inputs (versus trying more exotic modeling techniques).

Of course a client or user is not interested if score2 is “best possible.” They want to know if it is any good. To do this we should show them (either by graph or by quantitative summary statistics like we mentioned earlier) at least 3 things:

1. How well the model predicts overall (the very first graph we presented).
2. How much of the explainable variation the model predicts (the second graph).
3. The nature of the unexplained variation (which we will explore next).

We said earlier we are hoping the unexplained variation is noise (or if it is not noise it would be nice if it is a clue to new important modeling features). So the unexplained variation must not go unexamined. We will finish by showing how to characterize the unexplained variation. As before will will just make a graph, but the data preparation steps would be exactly the same if we were using a quantitive summary (like correlation, or any other). And, of course, our data is still not the right shape for this step. Luckily there is another tool ready to fix this: join().



> djoin <- join(dsort,dsort,'id')

> fixnames <- function(cn) {
n <- length(cn);
for(i in 2:((n+1)/2)) { cn[i] <- paste('a',cn[i],sep='') };
for(i in ((n+3)/2):n) { cn[i] <- paste('b',cn[i],sep='') };
cn
}

> colnames(djoin) <- fixnames(colnames(djoin))

> djoin


which produces:

   id ascore1 ascore2 bscore1 bscore2
1   1      17      13      17      13
2   1      17      13      10      13
3   1      10      13      17      13
4   1      10      13      10      13
5   2       5      10       5      10
6   2       5      10      13      10
7   2      13      10       5      10
8   2      13      10      13      10
9   3       6       5       6       5
10  3       6       5       7       5
11  3       7       5       6       5
12  3       7       5       7       5

All of the work was done by the single line “djoin <- join(dsort,dsort,'id')” the rest was just fixing the column names (as self-join is not the central use case of join). What we have now is a table that is exactly right for studying unexplained variation. For each id we have each row with the same id matched. This blows every id from having 2 rows in dsort to 4 rows in djoin. Notices this gives us every pair of score1 values seen for the same id (which will let us examine unexplained variation) and score2 is still constant over all rows with the same id (as it has always been throughout our analysis). With this table we can now plot how score1 varies for rows with the same id:



> ggplot(djoin) + geom_point(aes(x=ascore1,y=bscore1))


Figure 3: score1 as a function of score1.

And we can see, as we expected, the unexplained variation in score1 is about as large as the mismatch between score1 and score2 in our original plot. The important thing is this is all about score1 (score2 is now literally out of the picture). The analyst’s job would now be to try and tie bits of the unexplained variation to new inputs (that can be folded into a new score2) and/or characterize the noise term (so the customer knows how close they should expect repeated experiments to be).

What we are trying to encourage with the use of “big hammer tools” is an ability and willingness to look at and transform your data in meaningful steps. It often seems easier and more efficient to build one more piece of data tubing, but a lot of data tubes become an unmanageable collection of spaghetti code. The analyst should, in some sense, always be looking at data and not looking at coding details. For these sort of analyses we encourage analysts to think in terms of “data shape” and transforms. This discipline leaves more of the analysts energy and attention to think productively about the data and actual problem domain.

For the third plot showing the variation of score1 across different rows (but same ids) it may be appropriate to use a slightly more complicated join() procedure than we showed. The join shown produced rows of artificial agreement where both values of score1 came from the same row (thus had no chance of being different, so in some sense deserve no credit). This is also the only way any non-duplicated evaluations could make it to the plot. To eliminate these uninteresting agreements from the plot do the following:



> d$rowNumber <- 1:(dim(d)[1])

> djoin <- join(d,d,'id')

> colnames(djoin) <- fixnames(colnames(djoin))

> djoin <- djoin[djoin$arowNumber!=djoin$browNumber,]

> djoin


This gives us a table that shows only values of score1 from different rows:

   id ascore1 ascore2 arowNumber bscore1 bscore2 browNumber
2   1      17      13          1      10      13          4
4   2       5      10          2      13      10          5
6   3       6       5          3       7       5          6
7   1      10      13          4      17      13          1
9   2      13      10          5       5      10          2
11  3       7       5          6       6       5          3

And only plots points on the diagonal if “you have really earned them”:

So while the direct join() may not be the immediate perfect answer it is still a good intermediate to form as what you want is only simple data transformation away from it.

Related posts:

1. A Demonstration of Data Mining
2. Good Graphs: Graphical Perception and Data Visualization
3. The Data Enrichment Method

Hal 9000

There have been vast improvements in artificial intelligence. Much of the improvement has been driven by the engineering effects of Moore’s Law (resulting in my mobile phone’s processor having 12 times the clock speed and over 32 times the memory of an $8 million Cray 1 super computer) and significant machine learning research results. These machine size changes happened during the productive careers of many researchers, so ideas are often evaluated at a series of radically different machine capabilities and data scales.

Cray 1

von Neuman himself commented that scale was a major limiting factor in early computers. He asked the question how you could be expected to achieve anything significant even from a roomful of geniuses if (as with his early computers) all notes, communication and memory were limited to less than a single typed page. von Neuman’s comment stands in contrast to science fiction scientists and early boosters of artificial intelligence who always seem to be in awe of their own creations. Computers are certainly much larger- but we need to be humble and put off deciding if we are yet in the era of large computers (compared to human or animal brains). Everything we are doing now may still just be artificial intelligence’s pre-history and prologue. Feynman in his lectures on computation mentions that RNA transcription can be estimated to take around 100 kT of energy to transcribe a bit while a transistor may easily use 100,000,000 kT energy units to switch states. This means for the amount of heat the human head dissipates (energy supply and heat dissipation are rapidly becoming the most relevant measures of computational power) you could do a million times more work using RNA techniques (if you knew how) than with transistors. So computers may not yet be what we should call large (though they are likely getting there). What we currently call “datacenters” are in fact block sized computers (consuming an enormous amount of energy and dissipating a huge amount of heat).

A datacenter (or a block sized computer)

Not all improvements in machine intelligence have come from (or are to come from) improvements in hardware. Many of the improvements came from machine learning research results and these are what I will outline below.

Early machine learning algorithms were driven by analogy. This led us to perceptrons (1957, fairly early in the history of computer science) and neural nets. These methods have their successes but were largely over used and developed before researchers developed a good list of desirable properties of a machine learning method.

Neural Net diagram

These methods live on but are, in my opinion, not currently competitive. Some of their important ideas and contributions have been revived from time to time, such as the online update rules becoming what we now call stochastic gradients.

A list of (often incompatible) desirable properties of a machine learning algorithm is the following:

• Able to represent complicated functions
• Good generalization performance (quality predictions on data not seen during training)
• Unique optimal model for a given set of data and feature definitions
• Efficient and well characterized solution method
• Consistent summary statistics
• Preference for simple models

We divert from this list for a bit of background and context.

The neural net was largely celebrated for its ability to represent complex functions and the perceived efficiency of its newer back-propagation based training method (related to the efficient calculation of gradients). The downsides were you never knew if your neural net was the right one (even assuming you had the right features, layout and training data) and could not be sure you were biasing towards simple models that might perform well on novel queries. Great effort was expended in extending neural nets based on the supposition they should work as they were an analogy to how we imagined biological neurons might function. An almost mystic hope was derived from the non-linear nature and special properties of the sigmoid curve (which was in fact a curve already known to statisticians).

Other methods than neural nets also had early success. The field of information retrieval (which was not “sexy” prior to the Web) had huge success since the 1960s with Naive Bayes, Rocchio Classification, and TF/IDF methods. The early success of these methods may have in fact delayed research on current hot research areas such as segmentation and author topic models.

Theoretical computer science initially sought to characterize machine learning methods in non-statistical language. In the 1980s a great amount of ink was spilled on “learning boolean functions.” Papers proving nothing was learnable (by picking a function related to cryptography) alternated with papers proving everything was learnable (for example via amplification techniques like boosting). Generalization of models to new data remained a theoretical problem that was dealt with by appeals to model complexity and MDL (minimum description length). A major breakthrough in characterizing generalization performance was the PAC model (probably approximately correct) framework which finally allowed direct treatment of generalization performance.

We now have enough context to discuss some of the current best of breed machine learning techniques (that address many of the desired properties mentioned above):

• Support Vector Machines
• Kernel Methods
• Logistic Regression
• Maximum Entropy Methods
• Regularization
• Graphical Models
• Conditional Random Fields




Typical SVM maximum margin diagram

Not all of these methods are new (Logistic Regression for example dates from 1925 and is itself based on regression which goes back to Gauss). But the concerns these methods address are all much more statistical than artificial intelligence in nature. For example we don’t suppose that there is some cryptographically obscured combination of features that we need to find to make the best prediction. We instead worry about detecting which features are useful and note that it is a significant (though solvable) problem to correctly use combinations of useful features (phrased as statistical concerns: feature to feature dependencies and higher order interactions). Machine learning has always run where statisticians fear to tread. But more and more often we are seeing that the methods and concerns of statisticians are what are needed to achieve many of the listed desired properties of machine learning models.

The methods I have singled out for praise are very effective and achieve a number of our listed desired properties. For example: both logistic regression and maximum entropy have a unique solution that is easy to find. They are also both consistent with all summaries known during training. That is: if 30% of the positive training data has a feature present then 30% of the data also has the feature present when weighted by the model’s score (so the model score shares a lot of properties with training truth). Support Vector Machines also have well understood solutions and a theory (called maximum margin) that directly addresses generalization (good predictions on new data). Kernel Methods (both as used in SVMs and elsewhere) allow controlled introduction of very complex functions. Graphical Models and Conditional Random Fields also allow the controlled introduction of modeled dependencies in the data.

It is now common to call what was previously thought of as artificial intelligence or machine learning: “statistical machine learning.” This reflects that the kind of prediction and characterization we expect from machine learning algorithms are in fact statistical concerns that we can deal with if we have enough data and enough computational resources.

The current important issues for statistical machine learning include:

• Dealing with very large datasets (driving the return of simpler methods like Naive Bayes)
• Dealing with lack of training data (driving interest in clustering and manifold regularization methods)
• Dealing with unstructured data and text mining (driving interest in information extraction and segmentation via generative models)

Just as Wigner famously wrote about “The Unreasonable Effectiveness of Mathematics” in the 1960s Halevy,Norvig and Pereira write about the “Unreasonable Effectiveness of Data.” They argue that we are in the age of big data (or the age of analysts). Or, as Varian observed: “it is a good time to supply a good complementary to data” (i.e. it is a good time to be an analyst). I would temper this with we are likely in the age of unmarked data and unstructured data. Less often are we asked to automate a known prediction and more often we are asked to cluster, characterize and segment wild data. In my opinion the hard problem in machine learning has moved from prediction to characterization. With enough marked training data (that is data for which we know both the observables and desired outcome) it is now quite possible to use standard techniques and libraries to build a very good predictive model. However, it is still hard to characterize, segment or extract useful information from the wealth of unstructured and unmarked data that is upon us. And this is where a lot of the current research in statistical machine learning is directed.

Or course characterization and clustering have their own infamous history. Rota wrote: “… Or a subject is important, but nobody understands what is going on; such is the case with quantum field theory, the distribution of primes, pattern recognition and cluster analysis.” Artificial intelligence may be moving from areas where computer scientists have over-promised to areas where statisticians have over-promised. But this is not a disaster: the most valuable research tends to be done in hectic times in messy fields, not in calm times in neat fields. And the already large scale adoption of statistical machine learning techniques means there is immediate great client value in even seemingly small improvements in understanding, explanation, documentation, training, tools, libraries and techniques.

Classic attempt to add structure to text

(images from Wikipedia)

Related posts:

1. A Demonstration of Data Mining
2. Kernel Methods and Support Vector Machines de-Mystified
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)

This paper demonstrates and explains some of the basic techniques used in data mining. It also serves as an example of some of the kinds of analyses and projects Win Vector LLC engages in.

August 19, 2009

John Mount¹

A Demonstration of Data Mining

1  Introduction

A major industry in our time is the collection of large data sets in preparation for the magic of data mining [Loh09,HNP09]. There is extreme excitement about both the possible applications (identifying important customers, identifying medical risks, targeting advertising, designing auctions and so on) and the various methods for data mining and machine learning. To some extent these methods are classic statistics presented in a new bottle. Unfortunately, the concerns, background and language of the modern data-mining practitioner are different than that of the classic statistician- so some demonstration and translation is required. In this writeup we will show how much of the magic of current data mining and machine learning can be explained in terms of statistical regression techniques and show how the statistician’s view is useful in choosing techniques.

Too often data mining is used as a black-box. It is quite possible to clearly use statistics to understand the meaning and mechanisms of data mining.

2  The Example Problem

Throughout this writeup we will work on a single idealized example problem. For our problem we will assume we are working with a company that sells items and that this company has recorded its past sales visits. We assume they recorded how well the prospect matched the product offering (we will call this “match factor”), how much of a discount was offered to the prospect (we will call this “discount factor”) and if the prospect became a customer or not (this is our determination of positive or negative outcome). The goal is to use this past record as “training data” and build a model to predict the odds of making a new sale as a function of the match factor and the discount factor. In a perfect world the historic data would look a lot like Figure 1. In Figure 1 each icon represents a past sales-visit, the red diamonds are non-sales and the green disks are successful sales. Each icon is positioned horizontally to correspond to the discount factor used and vertically to correspond to the degree of product match estimated during the prospective customer visit. This data is literally too good to be true in at least three qualities: the past data covers a large range of possibilities, every possible combination has already been tried in an orderly fashion and the good and bad events “are linearly separable.” The job of the modeler would then be to draw the separating line (shown in Figure 1) and label every situation above and to the right of the separating line as good (or positive) and every situation below and to the left as bad (or negative).

In reality past data is subject to what prospects were available (so you are unlikely to have good range and an orderly layout of past sales calls) and also heavily affected by past policy. An example policy might be that potential customers with good product match factor may never have been offered a significant discount in the past; so we would have no data from that situation. Finally each outcome is a unique event that depends on a lot more than the two quantities we are recording- so it is too much to hope that the good prospects are simply separable from the bad ones.

Figure 1 is a mere cartoon or caricature of the modeling process, but it represents the initial intuition behind data mining. Again: the flaws in Figure 1 represent the implicit hopes of the data miner. The data miner wishes that the past experiments are laid out in an orderly manner, data covers most of the combinations of possibilities and there is a perfect and simple concept ready to be learned.

Frankly, an experienced data miner would feel incredibly fortunate if the past data looked anything like what is shown in Figure 2.

The green disks (representing good past prospects) and the red diamonds (representing bad past prospects) are intermingled (which is bad). There is some evidence that past policy was to lower the discount offered as the match factor increased (as seen in the diagonal spread of the green disks). Finally we see the red diamonds are also distributed differently than the green disks. This is both good and bad. The good is that the center of mass of the red diamonds differs from the center of mass of the green disks. The bad is that the density of red diamonds does not fall any faster as it passes into the green disks than it falls in any other direction. This indicates there is something important and different (and not measured in our two variables) about at least some of the bad prospects. It is the data miner’s job be aware and to press on.

2.1  The Trendy Now

In truth data miners often rush where classical statisticians fear to tread. Right now the temptation is to immediately select from any number of “red hot” techniques, methods or software packages. My short list of super-star method buzzwords includes:

• Boosting[Sch01,Bre00,FISS03]
• Latent Dirichlet Allocation[BNJ03]
• Linear Regression[FPP07,Agr02]
• Linear Discriminant Analysis[Fis36]
• Logistic Regression[Agr02,KM03]
• Kernel Methods[CST00,STC04]
• Maximum Entropy[KM03,Gru05,SC89,DS06]
• Naive Bayes[Lew98]
• Perceptrons[BRS08,DKM05]
• Quantile Regression[Koe05]
• Ridge Regression[BF97]
• Support Vector Machines[CST00]

Based on some of the above referenced writing and analysis I would first pick “logistic regression” as I am confident that, when used properly, it is just about as powerful as any of the modern data mining techniques (despite its somewhat less than trendy status). Using logistic regression I immediately get just about as close to a separating line as this data set will support: Figure 3.

The separating line actually encodes a simple rule of the form: “if 2.2*DiscountFactor + 3.1*MatchFactor ≥ 1 then we have a good chance of a sale.” This is classic black-box data mining magic. The purpose of this writeup is to look deeper how to actually derive and understand something like this.

3  Explanation

What is really going on? Why is our magic formula at all sensible advice, why did this work at all and what motivates the analysis? It turns out regression (be it linear regression or logistic regression) works in this case because it somewhat imitates the methodology of linear discriminant analysis (described in: [Fis36]). In fact in many cases it would be a better idea to perform a linear discriminant analysis or perform an analysis of variance than to immediately appeal to a complicated method. I will first step through the process of linear discriminant analysis and then relate it to our logistic regression. Stepping through understandable stages lets us see where we were lucky in modeling and what limits and opportunities for improvement we have.

Our data initially looks very messy (the good and bad group are fairly mixed together). But if we examine out data in separate groups we can see we are actually incredibly lucky in that the data is easy to describe. As we can see in Figure 4: the data, when separated by outcome (plotting only all of the good green disks or only all of the bad red diamonds), is grouped in simple blobs without bends, intrusions or other odd (and more work to model) configurations.

We can plot the idealizations of these data distributions (or densities) as “contour maps” (as if we are looking down on the elevations of a mountain on a map) which gives us Figure 5.

3.1  Full Bayes Model

From Figure 5 we can see while our data is not separable there are significant differences between the groups. The difference in the groups is more obvious if we plot the difference of the densities on the same graph as in Figure 6. Here we are visualizing the distribution of positive examples as a connected pair of peaks (colored green) and the distribution of negative examples a deep valley (colored red) located just below and to the left of the peaks.

This difference graph is demonstrating how both of the densities or distributions (positive and negative) reach into different regions of the plane. The white areas are where the difference in densities is very small which includes the areas in the corners (where there is little of either distribution) and the area between the blobs (where there is a lot of mass from both distributions competing). This view is a bit closer to what a statistician wants to see- how the distributions of successes and failures different (this is a step to take before even guessing at or looking for causes and explanations).

Figure 6 is already an actionable model- we can predict the odds a new prospect will buy or not at a given discount by looking where they fall on Figure 6 and checking if they fall in a region on strong red or strong green color. We can also recommend a discount for a given potential customer by drawing a line at the height determined by their degree of match and tracing from left to right until we first hit a strong green region. We could hand out a simplified Figure 7 as a sales rulebook.

This model is a full Bayes model (but not a Naive Bayes model, which is oddly more famous and which we will cover later). The steps we took were: first we summarized or idealized our known data into two Gaussian blobs (as depicted in Figure 5). Once we had estimated the centers, widths and orientations of these blobs we could then: for any new point say how likely the point is under the modeled distribution of sales and how likely the point is under the modeled distribution of non-sales. Mathematically we claim we can estimate P(x,y |sale)² and P(x,y | non-sale) (where x is our discount factor and y is our matching factor).³ Neither of these are what we are actually interested in (we want: P(sale | x,y)⁴). We can, however, use these values to calculate what we want to know. Bayes’ law is a law of probability that says if we know P(sale | x,y), P(non-sale | x,y), P(sale) and P(non-sale)⁵ then:

Figure 7 depicts a central hourglass shaped region (colored green) that represents the region of x, y values where P(sale |x,y) is estimated to be at least 0.5 and the remaining (darker red region) are the situations predicted to be less favorable. Here we are using priors of P(sale) = P(non-sale) = 0.5, for different priors and thresholds we would get different graphs.

Even at this early stage in the analysis we have already accidentally introduced what we call “an inductive bias.” By modeling both distributions as Gaussians we have guaranteed that our acceptance region will be an hourglass figure (as we saw in Figure 7). One undesirable consequence of the modeling technique is the prediction sales become unlikely when both match factor and discount factor are very large. This is somewhat a consequence of our modeling technique (though the fact that the negative data does not fall quickly as it passes into the green region also added to this). This un-realistic (or “not physically plausible”) prediction is called an artifact (of the technique and of the data) and it is the statistician’s job to see this, confirm they don’t want it and eliminate it (by deliberately introducing a “useful modeling bias”).

3.2  Linear Discriminant

To get around the bad predictions of our model in the upper-right quadrant we “apply domain knowledge” and introduce a useful modeling bias as follows. Let us insist that our model be monotone: that if moving some direction is good than moving further in the same direction is better. In fact let’s insist that our model be a half-plane (instead of two parabolas). We want a nice straight separating cut, which brings us to linear discriminant analysis. We have enough information to apply Fisher linear discriminant technique and find a separator that maximizes the variance of data across categories while minimizing the variance of data within one category and within the other category. This is called the linear discriminant and it is shown in Figure 8.

The blue line is the linear discriminant (similar to the logistic regression line depicted earlier on the data-slide). Everything above or to the right of the blue line is considered good and everything below or to the left of the blue line is considered bad. Notice that this advice while not quite as accurate as the Bayes Model near the boundary between the two distributions is much more sensible about the upper right corner of the graph.

To evaluate a separator we collapse all variation parallel to the separating cut (as shown in Figure 9). We then see that each distribution becomes a small interval or streak. A separator is good if these resulting streaks are both short (the collapse packs the blobs) and the two centers of the streaks are far apart (and on opposite size of the separator). In Figure 9 the streaks are fairly short and despite some overlap we do have some usable separation between the two centers.

To make the above precise we switch to mathematical notation. For the i-th positive training example form the vector v_+,i and the matrix S_+,i where

where x_i and y_i are the known x and y coordinates for this particular past experience. Define v_−,i, S_−,i similarly for all negative examples. In this notation we have for a direction γ: the distance along the γ direction between the center of positive examples and center of negative examples is: γ^T ( ∑_i v_+,i / n₊ − ∑_i v_−,i / n₋) (where n₊ is the number of positive examples and n₋ is the number of negative examples). We would like this quantity to be large. The degree of spread or variance of the positive examples along the γ direction is γ^T (∑_i S_+,i / n₊) γ. The degree of spread or variance of the negative examples along the γ direction is γ^T (∑_i S_−,i / n₋) γ. We would like the last two quantities to be small. The linear discriminant is picked to maximize:

It is a fairly standard observation (involving the Rayleigh quotient) that this form is maximized when:

As we have said, the linear discriminant is very similar to what is returned by a regression or logistic regression. In fact in our diagrams the regression lines are almost identical to the linear discriminant. A large part of why regression can be usefully applied in classification comes from its close relationship to the linear discriminant.

3.3  Linear Regression

Linear regression is designed to model continuous functions subject to independent normal errors in observation. Linear regression is incredibly powerful at characterizing and elimination correlations between the input variables of a model. While function fitting is different than classification (our example problem) linear regression is so useful whenever there is any suspected correlation (which is almost always the case) that it is an appropriate tool. In our example in the positive examples (those that led to sales) there is clearly a historical dependence between the degree of estimated match and amount of discount offered. Likely this dependence is from past prospects being subject to a (rational) policy of “the worse the match the higher the offered discount” (instead of being arranged in a perfect grid-like experiment as in our first diagram: Figure 1). If this dependence is not dealt with we would under-estimate the value of discount because we would think that discounted customers are not signing up at a higher rate (when these prospects are in fact clearly motivated by discount, once you control for the fact that many of the deeply discounted prospects had a much worse degree of match than average).

For analysis of categorical data linear regression is closely linked to ANOVA (analysis of variance).[Agr02] Recall that variance was a major consideration with the linear discriminant analysis, so we should by now be on familiar ground.

In our notation the standard least-squares regression solution is:

where y_+,i = 1 for all i and y_−,i = −1 for all i.

If we have the same number of positive and negative examples (i.e. n₊ = n₋) then Equation 1 and Equation 2 are identical and we have β = γ. So in this special case the linear discriminant equals the least square linear regression solution. We can even ask how the solutions change if the relative proportions of positive and negative training data changes. The linear discriminant is carefully designed not to move, but the regression solution will tilt to be an angle that is more compatible with the larger of the example classes and shift to cut less into that class. The linear regression solution can be fixed (by re-weighting the data) to also be insensitive to the relative proportions of positive and negative examples but does not behave that way “fresh out of the box.”

3.4  Logistic Regression

While linear regression is designed to pick a function that minimizes the sum of square errors logistic regression is designed to pick a separator that maximizes something called the plausibility of the data. In our case since the data is so well behaved the logistic regression line is essentially the same as the linear regression line. It is in fact an important property of logistic regression that there is always a re-weighting (or choice of re-emphasis) of the data that causes some linear regression to pick the same separator as the logistic regression. Because linear and logistic regression are only identical in specific circumstances it is the job of the statistician to know which of the two is more appropriate for a given data set and given intended use of the resulting model.

4  Other Methods and Techniques

4.1  Kernelized Regression

One way to greatly expand the power of modeling methods is a trick called kernel methods. Roughly kernel methods are those methods that increase the power of machine learning by moving from a simple problem space (like ours in variables x and y) to a richer problem space that may be easier to work in. A lot of ink is spilled about how efficient the kernel methods are (they work in time proportional to the size of the simple space, not the complex one) but this is not their essential feature. The essential feature is the expanded explanation power and this is so important that even the trivial kernel methods (such as directly adjoining additional combinations of variables) pick up most of the power of the method. Kernel methods are also overly associated with Support Vector Machines- but are just as useful when added to Naive Bayes, linear regression or logistic regression.

For instance: Figure 10 shows a bow-tie like acceptance region found by using linear regression over the variables x, y, x², y² and x y (instead of just x and y). Note how this result is similar to the full Bayes model (but comes from a different feature set and fitting technique).

4.2  Naive Bayes Model

We briefly return to the Bayes model to discuss a more common alternative called “Naive Bayes.” A Naive Bayes model is like a full Bayes model except an additional modeling simplification is introduced in assuming that P(x,y|sale) = P(x|sale)P(y|sale) and P(x,y|non-sale) = P(x|non-sale)P(y|non-sale). That is we are assuming that the distributions of the x and y measurements are essentially independent (once we know which outcome happened). This assumption is the opposite of what we do with regression in that we ignore dependencies in the data (instead of modeling and eliminating the dependencies). However, Naive Bayes methods are quite powerful and very appropriate in sparse-data situations (such as text classification). The “naive” assumption that the input variables are independent greatly reduces the amount of data that needs to be tracked (it is much less work to track values of variables instead of simultaneous values of pairs of variables). The curved separator from this Naive Bayes model is illustrated in Figure 11.

The Naive Bayes version of the advice or policy chart is always going to be an axis-aligned parabola as in Figure 12. Notice how both the linear discriminant and the Naive Bayes model make mistakes (places some colors on the wrong side of the curve)- but they are simple, reliable models that have the desirable property of having connected prediction regions.

4.3  More Exotic Methods

Many of the hot buzzword machine learning and data mining methods we listed earlier are essentially different techniques of fitting a linear separator over data. These methods seem very different but they all form a family once you realize many of the details of the methods are determined by:

• Choice of Loss Function

  This is what notion of “goodness of fit” is being used. It can be normalized mean-variance (linear discriminants), un-normalized variance (linear regression), plausibility (logistic regression), L1 distance (support vector machines, quantile regression), entropy (maximum entropy), probability mass and so on.
• Choice of Optimization Technique

  For a given loss function we can optimize in many ways (though most authors make the mistake of binding their current favorite optimization method deep into their specification of technique): EM, steepest descent, conjugate gradient, quasi-Newton, linear programming and quadratic programming to name a few.
• Choice of Regularization Method

  Regularization is the idea of forcing the model to not pick extreme values of parameters to over-fit irrelevant artifacts in training data. Methods include MDL, controlling energy/entropy, Lagrange smoothing, shrinkage, bagging and early termination of optimization. Non-explicit treatment of regularization is one reason many methods completely specify their optimization procedure (to get some accidental regularization).
• Choice of Features/Kernelization

  The richness of the feature set the method is applied to is the single largest determinant of model quality.
• Pre-transformation Tricks

  Some statistical methods are improved by pre-transforming the outcome data to look more normal or be more homoscedastic.⁶

If you think along a few axes like these (instead of evaluating them by their name and lineage) you tend to see different data mining methods more as embodying different trade-offs than as being unique incompatible disciplines.

5  Conclusion

Our goal for this writeup was to fully demonstrate a data mining method and then survey some important data mining and machine learning techniques. Many of the important considerations are “too obvious” to be discussed by statisticians and “too statistical” to be comfortably expressed in terms popular with data miners. The theory and considerations from statistics when combined with the experience and optimism of data-mining/machine-learning truly make possible achieving the important goal of “learning from data.”

This expository writeup is also meant to serve as an example of the
types of research, analysis, software and training supplied by
Win-Vector LLC http://www.win-vector.com . Win-Vector LLC
prides itself in depth of research and specializes in identifying,
documenting and implementing the “simplest technique that can
possibly work” (which is often the most understandable, maintainable,
robust and reliable). Win-Vector LLC specializes in research but
has significant experience in delivering full solutions (including
software solutions and integration with existing databases).

A  Graphs

The majority of the graphs in this writeup were produced using “R”
http://www.r-project.org/ and Deepayan Sarkar’s Lattice
package[Sar08].

Footnotes:

¹
mailto:jmount@win-vector.com
http://www.win-vector.com/
http://www.win-vector.com/blog/

²Read P(A | B) as: “the probability of A will
happen given we know B is true.”

³Technically we are working with densities, not
probabilities, but we will use probability notation for its
intuition.

⁴P(sale | x,y) is the probability of
making a sale as a function of what we know about the prospective
customer and our offer. Whereas P(x,y|sale) was just how likely it is
to see a prospect with the given x and y values, conditioned on knowing we made
a sale to this prospect.

⁵ P(sale) and
P(non-sale) are just the “prior odds” of sales or what
our estimate of our chances of success are before we look at any
facts about a particular customer. We can use our historical
overall success and failure rates as estimates of these quantities.

⁶A situation is homoscedastic if the errors are independent of where we are in the parameter space (our x,y or match factor and discount factor). This property is very important for meaningful fitting/modeling and interpreting significance of fits.

Related posts:

1. Book Review: Ensemble Methods in Data Mining (Seni & Elder)
2. The Data Enrichment Method
3. Your Data is Never the Right Shape