Fortunately Apple does supply the user with incredibly powerful tools such as AppleScript. This will allow us to add an always present single click (no control or right-click and no menu) action to our Finder toolbar.

To add the ability to open a new finder as a click all one has to do is the following (we are not distributing this as finished code as it is not our work and we don’t wan’t to encourage people to download and use unexamined code).

1. Open the AppleScript Editor
2. Select File->New
3. Paste the the following into the AppleScript Editor (from: dtomasch (Daniel T) on Macword Mac OSX hints
   

   -- From: http://hints.macworld.com/article.php?story=20080108144434753
   try
   	tell application "Finder"
   		activate
   		set this_folder to (the target of the front window) as alias
   		set {x1, y1} to position of front window
   		make new Finder window to this_folder
   		set position of front window to {(x1 + 50), (y1 + 150)} --This offsets the new window more than the average Finder tiling does
   	end tell
   end try
   

4. Press the compile hammer.
5. Select File->Save As, choose file format Applicaton and save as “OpenFinder.app”.
6. (optional, for a nice icon)
   With a finder right click on the new application and select Show Package Contents. Then replace Contents/Resources/applet.icns with Pieter Stroink’ finder icon (the file Finder_without_Bag.icns from the downloadable zip file Finder_Icon_by_eggy.zip).
7. On a Finder right click on some blank space in the upper toolbar and select Customize Toolbar. Drop the new application in the toolbar.

Now all of your finders show the new application as an icon in the upper toolbar. When you click on this icon you get a new finder in the current context.

But you can do more, you can repeat this pattern with other applications. In my case I really like being able to open the command line terminal in the current finder directory. One of the really cool things about OSX is it has a good command line terminal. If you already are a terminal user you will like the following.

You can get the terminal in on this game by using TomWoozle (Tom Anthony)’s Macword Mac OSX hint in the same patter as above to create a button that open a new terminal shell at the current finder location. You just use this block of AppleScript code:

-- from: http://hints.macworld.com/article.php?story=20020426093503563
on run
	tell application "Finder"
		try
			activate
			set frontWin to folder of front window as string
			set frontWinPath to (get POSIX path of frontWin)
			tell application "Terminal"
				activate
				do script with command "cd \"" & frontWinPath & "\""
			end tell
		on error error_message
			beep
			display dialog error_message buttons ¬
				{"OK"} default button 1
		end try
	end tell
end run

and get the icons from /Applications/Utilities/Terminal.app/Contents/Resources/Terminal.icns.

The absolute icing on the cake is OSX’s built in "open" command. You can open a Finder pointing to the current terminal’s working directory by typing "open ." in the terminal (completing the circle).

Note: we have seen the open terminal app “stutter” or sometimes open two terminals.

Related posts:

1. Must Have Software

Telling stereotype machine learning examples and methods include:

• The XOR problem as an important violation of linear seperability: Minsky, Marvin Lee, and Seymour Papert. Perceptrons: an introduction to computational geometry. ,1st ed. Cambridge, Massachusetts: MIT Press, 1969.
• The “two spirals problem” found in Kevin J. Lang and Michael J, Witbrock, “Learning to Tell Two Spirals Apart”, in Proceedings of the 1988 Connectionist Models Summer School, Morgan Kaufmann, 1988.
• The continuing fascination and re-discovery of biologically inspired learning techniques (in the hope that they may work even if we don’t yet know why): perceptrons, self organizing maps and cellular automata.
• Logical methods like version spaces; and other Novum Organum inspired methods (like resolution and classic symbolic AI planning).
• The complementary relation of cryptography and machine learning (at most one of them can be an easy endeavor): Cryptography and Machine Learning by Ronald L. Rivest. Proceedings ASIACRYPT ’91 (Springer 1993), 427–439.

There is an identifiable theme that all of the data is before us and it is just a matter of finding its secrets using either well founded methods or arcane methods. Even if none of the variables or measurements initially available are immediately useful perhaps some combination of them will be. The first order of business is to find the right combination or transformation. It is just a matter of sufficiently clever computation.

The initial activity of a machine learning practitioner is often to choose among sophisticated representations, model forms and tools. And having such powerful tools machine learning practitioners rush where statisticians traditionally fear to tread.

Statisticians, on the other hand, have very good descriptions of what often goes wrong in even observing simple data.

For example: artificial intelligence and machine learning ideas such as version spaces, linear separability and logical entailment all depend on data without a single transcription error. A single positive example deep in the center of the mass of positive examples can kill these methods if it is mis-transcribed as a negative. These issues can be solved (for example, the emergence of soft margin classifiers to deal with error). But it is telling that error and data distribution were not the first concerns.

Statisticians produce a lot of results describing data quality issues like:

• Error/Noise. As we mentioned above- we must assume their may be data in error. That is why methods like logistic regression use maximum likelihood (try to be consistent with as much of the mass of the data as possible) instead of ideas like margin and separability.
• Collinarity. An example of this is when variables that individually are useful fail to perform even better when used together (as they correlate or are collinear with each other, so they each have reduced marginal value once you have some of these variables in your model).
• Simpson’s paradox. Where a treatment can look better in all sub-experiments, yet look worse overall.
• Nuisance variables. Variables that predict the outcome, but not in a useful or controllable way. A simple example would be the day of week’s impact on web traffic. You can’t control the day of the week (pay to have more Mondays) but if you don’t deal with its influence you may mistakingly assign some of its influence to a treatment that overlapped more Mondays than an alternative.

The statistical practitioner usually starts by examining single variable effects. They test if variables are reliable and test to what extent they remain useful after adding more variables. The statistician doesn’t expect some clever combination of variables to out-perform all of its constituent parts, but to build an ensemble of variables such that each variable is not performing much worse than when it was used alone (so the quality of the model is nearly additive in the number of chosen variables).

It is one of our maxims that the major source of deep statistical problems is poor record keeping (or experimental design). With perfect records you would not need a lot of the more powerful statistical tools. If you had sufficiently detailed records of intermediate states you would not need big statistical tools like Bayesian networks or conditional random fields. However, much of what we are calling “bad record keeping” is the excusable failure to have records of important unobservable states (though the statistics gets just as hard when things that could have been recorded are accidentally mixed, aggregated, truncated or censored).

The statistician tends to use sophisticated methods to validate and repair data, not to decode complex hidden relations. A friend of mine characterizes the statistical view as “you have to get up pretty early in the morning to beat linear regression.”

The machine learning practitioner tends to have much better tools for dealing with difficult relations (they are not forced to think in linear terms, especially with the use of kernel methods to allow richer model forms, though the statistician does have their own methods like generalized additive models).

To know which view is more advantageous for a given problem you just need to think clearly are you more worried about functional form (so you should look to machine learning) or issues of measurement (so you should look to statistics).

Related posts:

1. A Personal Perspective on Machine Learning
2. Why you can not to use statistics to dispute magic
3. Statistics to English Translation, Part 2b: Calculating Significance

library(ggplot2)
ggplot(data=data.frame(x=c(1:16))) + geom_point(aes(x=x,y=x,shape=x))

Or if you are feeling more daring:

ggplot(data=data.frame(x=c(1:16))) + geom_point(aes(x=x,y=x,shape=x)) +
   facet_wrap(~x,scales='free')

Update 4-25-2012

As Idris kindly pointed out in the comments the above no longer works. I suspect what changed is something in the transition to R2.15.0. The following hack does work:

sum <- ggplot()
for(i in 1:16) {
   sum <- sum +
      geom_point(data=data.frame(x=c(i)),aes(x=x,y=x),shape=i)
}
sum

The trick is outside the aes() it looks like an integer works and at least 16 values work (instead of only around 6 inside the aes()). Frankly we are doing a lot of work to dance around R’s delayed evaluation and variable binding rules in this case. The ggplot2 examples have always plugged factors into the shapes- but I had always assumed that was to densify the indexing (get all the shape numbers into an interval) and not a requirement for using the shape parameter. The online documentation of shape has always been a line of text of the form:

Which to my mind doesn’t specify if you have to supply a factor or not. But you really should buy Professor Hadley Wickham’s excellent book: ggplot2: Elegant Graphics for Data Analysis (not an affiliate link). My copy (like almost everything else I own) just happens to be packed in a box right now, so I can not easily consult it.

And it looks like we do get some nice bug fixes with this update. The missing scales on some of the facet_wrap panes appear to be fixed (yey!):

ggplot(data=data.frame(x=c(1:16))) + geom_point(aes(x=x,y=x)) +
    facet_wrap(~x,scales='free')

Related posts:

1. Your Data is Never the Right Shape
2. The cranky guide to trying R packages
3. My Favorite Graphs

Related posts:

1. Setting expectations in data science projects
2. Sorting Used in Anger
3. My Favorite Graphs

Data science projects vary from “executive dashboards” through “automate what my analysts are already doing well” to “here is some data, we would like some magic.” That is you may be called to produce visualizations, analytics, data mining, statistics, machine learning, method research or method invention. Given the wide range of wants, diverse data sources, required levels of innovation and methods it often feels like you can not even set goals for data science projects.

Many of these projects either fail or become open ended (become unmanageable).

As an alternative we describe some of our methods for setting quantifiable goals and front-loading risk in data science projects.

The typical situation

Data science projects are often considered untrackable because either there is “some magic” expected, there is no prior bound on how dirty the incoming data is or there is no prior definition of what a good result would look like. An example might be “invent a method to use website visit data to predict who is likely to purchase from us.”

When magic is expected you are really talking about invention and not a data science deployment. The research that more naturally fits into a data science project is both: learning the nature of the domain, problem and data; and traditional literature research (are there known methods that help with our situation?). You can schedule intervals of invention spikes into a larger project; but you can not really specify outcomes of these spikes (“magic method solves problem X by February 7th”). So these should be seen as tasks in a larger project that may or may not help. The entire project should not fully depend on them. The project must be able to succeed even when the invention spikes fail.

Additional red flags include lack of description of the input data and no concrete definition of the outcome we are trying to predict (Purchase ever? Purchase in the next month? Spend at least $100?).

Often the first fix to the project ask happen as: “we better at least quantify expected performance: let’s insist on an accuracy of 95%.” This often happens in a business meeting late in the project launch when it is noticed that what is likely a large and important project has absolutely no acceptance criteria. Unfortunately this “bar” is often set without any research if accuracy is even the correct measure, if 95% is easy or hard or even if the enterprise will be profitable at this accuracy. This is a good intent, but the arbitrary goal (that nobody will really be held to) is a step backwards.

What we need to do is: schedule dedicated time to learn about the domain and data before writing project goals and scope. This itself can be part of a small concrete expectation setting project. To complete the expectation setting project we need reusable methods to set useful, realistic goals that really measure if a data science project is on track (i.e. that a data science project can be held to). We outline a few methods to generate prior estimates for two of the important data science project measures: model performance and business utility.

Minimal components to safely ensure success

At a minimum a project must have an observable quantifiable measure of success. So it makes sense to work on setting this expectation first. That does not mean the success criterion needs to be set in stone- as this is often not possible. Instead it means you often have to commission an initial research project to quantify what sort of outcome is even possible and if such an outcome would make sense for the business. This unknown result determines if the project even has a chance to succeed, so it makes sense to try and eliminate the hidden project risk it represents by determining success criteria as a separate project. The overall modeling project often should not even be commissioned until the expectation setting project is complete. The result may indicate no further data science work is appropriate until features are added to engineering systems or the business. But this is good: nobody wants to start doomed projects, they instead want to know what to changes to implement to allow a successful project to be later launched.

You can in fact run data science projects as you would run any development project (all projects have risks and unknowns- so these problems are not in fact unique to data science projects). It is just that you can not, unfortunately, run data science projects in parallel with developing initial measurement and feedback systems. This is one case where starting a bridge from two shores to meet in the middle does not decrease project time. Specific measurement, control and feedback in a data science project requires running a few cars across the bridge (but won’t require all lanes be ready at the start).

Methods for expectation setting

The expectation setting part of a data science project is to estimate how well a very good model would perform without paying the time and cost of producing the model. This may seem impossible, but there are methods that estimate to be performance and utility with moderate effort. To show the flavor of this idea we list a couple of methods to estimate performance and a couple of methods to estimate utility.

Methods to prior estimate model performance

You need to know what prediction performance to commit to. Some ways to prior estimate this are given here:

• Current performance

  Data scientists don’t work in a vacuum. Usually we are trying to build a model that will be used to improve a business process. This implies there is already some process in place. It could be something as simple as “offer all return visitors a recommendation” or even a hand tuned set of business rules. Do not to be too proud, too polite or too rushed to measure the current system’s performance as if it were a classifier. For example if the current policy is “offer all return visitors a recommendation” measure what percentage of them buy (getting at precision) and what percentage of first time visitors buy (getting at recall).

  If the current system looks like a very high performance classifier (near perfect precision and recall) then you are not going to be able to usefully improve on it (so nobody should want the task of improving on it). You may want the task of automating it (if parts of it are human driven)- but you now have guidance where to go for rules and training data. If the current system looks like a low performance classifier then you should not agree to develop a very high performance replacement as your immediate project. If the business is running with 50% accuracy, then it is plausible the business will run better with 70% accuracy and it does not make sense to propose hitting 95% accuracy as a first project (such a project may be unrealistic or may just take longer).

• 3 by 5 card estimation

  This is an especially useful technique in trying to automate high quality human judgements (often done so they can be applied at a larger scale or higher speed). Get some time with the experts you are trying to extend the work of. If you can’t get the time for the initial project- then you already know any larger project would be doomed, so this itself is a good up-front test. Then ask them to do their job on paper. For example: suppose the task is to send a coupon to somebody likely to use it; prepare by sending a lot of coupons at random and recording who used the coupons (again this is a good gatekeeper or risk that should be front-loaded; if an organization is not ready for quick measurements and A/B tests adding these capabilities if far more important than any model construction). Have the experts pick (using all information available) who should have gotten coupons. Ask what information they used. User your known hidden outcomes to evaluate performance. Prepare 3 by 5 cards with only the chosen information and see if they can indeed predict at their historic rate with the limited information. If they can do it a model may be able to do it (and the less you need to put into the model the less engineering is needed) if they can not do it a model may not be able to do it.

• Bayes limit estimation

  One property shared by most models is that they report the same prediction given two identical examples. This “same input produces same output” observation puts an upper bound on classifier performance called the Bayes limit. Even a perfect model can not outperform the Bayes limit. You can design a cross-validation study on your training data to estimate the Bayes limit without building a real deployable model. Generate pairs of training examples with what you consider to be identical or nearly identical input patterns. You then see how well the known outcome from one example is at predicting the known example of the other. This is an instance of a permutation test or resampling simulation. We do not have to pair all of the training data, just find nearest neighbors for an appropriate random sample to estimate training data variation (with respect to the whole data set, this can be done with a single table scan).

  An actual implementation of this estimate as a final method would be a nearest neighbor model. Which may are may not be advised for actual implementation. Two of the downsides of nearest neighbor algorithms are their computational cost and poor generalization. The computational cost is equal to the number of queries times the training set size (which is okay for scoring an example sample but unacceptable in production, meaning you often must bring in some sort of sophisticated dimension reduction technique). Poor generalization (or over fitting) is why nearest neighbor algorithms tend not to perform as well on truly novel data as they do during naive cross validation.

  Think of the Bayes limit as an estimated upper bound, you probably won’t do better than it and you likely will (intentionally) do worse (as you trade performance on historic data for better performance on new data and efficiency). If your initial Bayes limit is poor, then no amount of modeling will help- you need more features and feature engineering (a different sort of project).

Methods to prior estimate business utility

What the business really needs to know is if promised increase in classifier performance leads to a desired increase in business (customers, revenue or anything as long as it is specific and measurable). From your performance calibration exercises you should have a reasonable target classification or modeling improvement in mind. You want of quantify the expected business impact of that the proposed amount of classification improvement on the business. This is where we are converting from statistical significance (is the math on our side) to clinical significance (will it drive an appreciable change in outcome).

• Retrospective simulation

  From the historic business data choose uniformly at random a sample of interactions that meet your target classifier performance. That is if you thing your achievable goal is a 10% increase and precision and a 2% decrease in recall (often you want to or are forced to trade precision and recall): generate a sample of customers from historic offers that meet this pattern. This doesn’t require a model- just access to historic data. Then measure the change in revenue if these had been your entire customer set versus a unbiased sample of the same size.

• Secant line method

  This can be a prospective or active study. If you feel you can build a model which increases precision by 10% then get permission to degrade the running site precision by 10% (for example: make 10% more offers at random to degrade the current model or procedure). If the degradation has no effect then probably the improvement will have no effect. You want to run this study on a small sub-population as it confirms utility by losing money. The idea is: it is easier to break things than improve them and the rate of change in one direction isn’t a bad approximation of the rate of change in the opposite direction.

• Wizard of Oz method

  Deploy a simulation of a better model by more expensive means (with intention of doing the engineering work to get a reasonable implementation if the experiment yields good results). If you goal is to build a machine that “produces color combinations for you as good as a designer” simulate the effect of success by paying a designer to work with a small subset of your customers. If the designer doesn’t improve revenue and/or customer satisfaction than even an algorithm as smart as the designer will also so fail. It is important that nobody confuses this experiment with a sustainable method that will be left in place.

Review of Purpose

In all cases you are trying to front load getting specific and risk. You are trying to fill in unknowns (what is our current performance?, what is our sensitivity? what do we need?) and front load risk (do we have the data?, does the data even differ between good and bad prospects?). The results of a project like this can serve both as a gatekeeper to and source of specification for a follow-up actual data-science research and implementation project. The first project itself should have specific description like: “be confident in the estimates of the following measures of possible model quality, data availability and probably business impact by this date.” After that those values can be used as part of a project scoping exercise for an actual data science implementation.

Related posts:

1. Hello World: An Instance Of Rhetoric in Computer Science
2. What Did Theorists Do Before The Age Of Big Data?
3. Your Data is Never the Right Shape

Related posts:

1. SQL Screwdriver
2. Lanchester’s Law: why small advantages swell in StarCraft
3. Gradients via Reverse Accumulation

A signed value we have often been asked to characterize or predict is: profit and loss (often called P&L). One natural model for P&L would be as the difference between a revenue and an expense. If the revenue and expense were both normally distributed then their difference would also be normal (and we would not need any stabilizing transform). However, if the revenue and expense were both log-normally distributed (say both proportional to task size or some other log-normal parameter) then the difference is not normal (it retains the propensity for extreme values or heavy tails of the original distributions). And for many financial size measures (company size, contract size and so on) the log-normal distribution is a much more realistic model than the normal distribution. In some situations P&L’s are formed from completely observed revenues and expenses (so we can model everything without sign problems), in other situations the signed P&L from an unobserved (or unrecorded) underling process and we are forced to deal with signed quantities.

For signed data we suggest the following transformation (code in R):


pseudoLog10 <- function(x) { asinh(x/2)/log(10) }


asinh() is a somewhat ugly function that is the inverse of sinh(). sinh() is defined as:


sinh(x) = (e^x - e^(-x))/2


The important point is for x such that |x| is large 2*sinh(x) rapidly approaches sign(x)*e^(|x|). Thus we should expect asinh(x/2) to look a lot like sign(x)*log(|x|) (which is why we call it a signed pseudo logarithm). For pseudoLog10() we take the previous function divided by log(10) to ensure that we are in log-10 like units (i.e. pseudoLog10(100) is nearly 2, pseudoLog10(1000) is nearly 3 and so on). Business audiences tend to have an easier time with log-10 (or dB) units (which can be explained as counting the number of decimal digits) than natural log or log-e units.

So for large positive numbers pseudoLog10() pretty much behaves like log10() (itself a standard transform). In fact pseudoLog10() has the following nice properties:

1. pseudoLog10(x) is defined for all real x.
2. pseudoLog10(0) = 0.
3. pseudoLog10(-x) = -pseudoLog10(x).
4. pseudoLog10(x) is monotone in x.
5. For x such that |x| is large: pseudoLog10(x) is very near sign(x)*log10(|x|).

We strongly recomend trying this transformation before feeding heavy tail data into a linear or logistic model.

However, we can not recommend the transformation for presentation. Consider the simple case of plotting the distribution or density of normal data with mean zero and standard-deviation 10 (see My Favorite Graphs for description of a density plot):

The density plot shows what we would expect- a near normal distribution (most points towards the center and mass falling off quickly as we move away). However, the plot of the pseudoLog10 transformed data is not what we would hope:

The data density (falsely) appears bimodal! This is because the pseudoLog10() transform is compressing ranges more and more violently as we move away from the origin (and not compressing near the origin). So as we move away from origin: the product of the real data density times the degree of range compression climbs, achieves a maximum and then falls. This phenomena (which is just a “change of variables” for densities) gives us the bimodal appearance for unimodal distributions that have significant mass outside of the range [-10,10]. The bimodal appearance is mostly a fact about the transform not really a feature of the underlying data.

We see value in examining at the relative sizes and centers of these two modes for asymmetric distributions (such as the profit and loss statement for a set of accounts that are mostly losing money). The position and relative sizes of the modes gives us an initial hint what to look for (helps with questions like: “are total losses driven by many accounts or by few accounts” and so on). We can not, however, recommend the pseudoLog10() transform for presentation. The most striking feature of the graph is almost always the bimodal appearance of the data; and the bimodal appearance is almost always an artifact of the transform (not a real feature of the data). You can not in good conscious push a presentation where the most prominent and exciting observation is not in fact in the data.

We do still recommend trying the pseudoLog10() transform when building a linear or logistic model with wide ranged data. The transformation usefully compresses range which allows the modeled coefficients to be a function of most of the data and not a function of a few extreme values. Models that depend on most of their data (or on central estimates from their data) tend to be safer, achieve higher statistical significance and cross-validate more reliably. Models that are dominated by a few extreme values tend to be unsafe, not achieve statistical significance and not cross-validate reliably. The bimodal artifact can work in the favor of modeling as it tends to compress a transformed variable into “typical positive example” and “typical negative example” while still allowing magnitudes to enter the model in some form.

Used with care the pseudoLog10() or arcsinh() transform can be an important data preparation step for signed data with large range. Many financial summaries (such as P&L) meet these conditions and often profit from the transform.

Related posts:

1. Your Data is Never the Right Shape

I strongly disagree. I find the pain of having to type or read through extra declarations is small (especially if you know how to copy-paste or use a modern IDE). And certainly much smaller than the pain of the dynamic language driven anti-patterns of: lurking bugs, harder debugging and more difficult maintenance. Debugging is one of the most expensive steps in software development- so you want incur less of it (even if it is at the expense of more typing). To be sure, there is significant cost associated with static typing (I confess: I had to read the book and post a question on Stack Overflow to design the type interfaces in Automatic Differentiation with Scala; but this is up-front design effort that has ongoing benefits, not hidden debugging debt).

There is, of course, no prior reason anybody should immediately care if I do or do not like dynamic typing. What I mean by saying this is I have some experience and observations about problems with dynamic typing that I feel can help others.

I will point out a couple of example bugs that just keep giving. Maybe you think you are too careful to ever make one of these mistakes, but somebody in your group surely will. And a type checking compiler finding a possible bug early is the cheapest way to deal with a bug (and static types themselves are only a stepping stone for even deeper static code analysis).For my examples I will pick on the programming language R (which we have used and written about in the past).

One of the supposed advantages of dynamically typed languages is that “everything is a macro.” That is you write a function and it is really a template that specializes and works over many different data types. For example: suppose we decided to write our own function to compute sample variance in R:

variance <- function(x) {
   n <- length(x)
   sumX <- sum(x)
   sumXX <- sum(x*x)
   (n/(n-1))*(sumXX/n - (sumX/n)*(sumX/n))
}

This works great and even matches the built-in funciton var():

> variance(c(1000000,2000000,3000000,4000000,5000000))
[1] 2.5e+12
> var(c(1000000,2000000,3000000,4000000,5000000))
[1] 2.5e+12

That is it works until we (either knowingly or unknowingly) apply the function to data of a different type:

> variance(as.integer(c(1000000,2000000,3000000,4000000,5000000)))
[1] NA
Warning message:
In x * x : NAs produced by integer overflow

Our macro specialized to calculate over the integers when given integer arguments and then fails due to overflow. Here it is obvious, but in a dynamically typed language we don’t always know the type of what we are passing in as we may have gotten the value from somewhere else. If we define variance() as a function over doubles in a statically typed language then the language would force either an explicit (programmer supplied) or implicit (language supplied) coercion when attempting to use the function on a vector of integers. The problem is: it is a bigger responsibility to write a correct macro (as the macro has to work over more possible types than a simple function). The dynamic language pushes this onto us and sometimes we get burnt and sometimes everything is okay. This sort of consideration is one of the reasons functional programing advocates prefer anonymous functions to declaring on the fly classes: less is possible so it is easier to safely implement what is implied.

Some of the problem can be dispelled with test driven development. I am proponent of test driven development, so much so that I don’t want to waste my valuable test budget testing for things that a decent type system can defend against. Also, by starting broad (assuming it is fair to re-use a function on many different types of arguments) you have entered into a bad bargain where you either have to document what subset of arguments the function works properly on (which is essentially declaring types!), add extra defensive code to cast the arguments on the way in (a waste, and needlessly defensive coding brings in its own problems) or write enough tests to document proper function on a whole bunch of types you don’t actually care about (char, byte, short int …)). Unexpected properties of real world data will throw you enough testing and debugging challenges (for example: the effect of unexpected constant data in bad quicksort implementations) that you don’t need additional hidden challenges that a static type system could exclude.

My second complaint is that most dynamically typed languages go further and force the horrible anti-pattern of automatic (or zero-declaration) variables on us. Since we are not, in a dynamically typed language, required to declare type- it is considered a waste to force the user to declare variables at all (statements like “var colTypeClass“). This argument is seductive because another supposed advantage of dynamically typed languages is conciseness, and variable declarations appear to have little value if you are not declaring types. However consider the following code:

sqlColType <- function(colTypeName) {
   colTypeClass <- 'unhandled'
   if(colTypeName %in% list('smallint','integer','bigint','decimal','numeric','real','double precision','serial','bigserial','money')) {
      colTypeClass <- 'numeric'
   } else if(colTypeName %in% list('character varying','character','text','boolean')) {
      colTypeClass <- 'categorical'
   } else if(colTypeName %in% list('interval','date')) {
      colTypeGlass <- 'temporal'
   } else if(length(grep('time',colTypeName))>0) {
      colTypeClass <- 'temporal'
   }
   colTypeClass
}

This code (for better or for worse, and at some point we all have to write or use something this ugly) is attempting to map specific SQL column type names into broad classes of types (numeric, categorical and temporal). However there is a typo-bug in the above code that is only possible in a language with automatic variable declaration. Consider the following to applications of sqlColType():

> sqlColType('integer')
[1] "numeric"
> sqlColType('date')
[1] "unhandled"

The first result is as designed and the second is wrong. What happened is in the if-block where “date” should have been identified we accidentally spelled “Class” with a “G” and the result we meant to return was trapped in a shiny new automatic variable that never escapes the function. You may consider this particular bug unlikely, but in a language without automatic variable declaration it is literally impossible. And you don’t even have to actually have this bug in your code to suffer from it. This mistake is something you have to check for when inspecting/debugging faulty code (because you have not pre-guarantee it can not happen).

My third complaint is the common lack of significant refactoring tools for dynamically typed languages. The ability to automatically apply larger scale meaningful code changes (such as when using Eclipse’s Java development environment) is big. Dynamic type advocates would argue that most of the successful refactorings are just the IDE shepherding around type cruft that is not present in a dynamic language. This is not true. In addition to the trivial code motion and package management there are significant code transformations: method extraction, method signature alteration and safe variable renaming just to name three. It is a real luxury to work with a system that can safely rename a variable (and all of its references) even when there are other strings and variables using the same token. It is also a luxury to work in teams where nobody can say “yeah, we wanted to remove that argument from the method- but nobody has time to update and test all of the consumers.” Most dynamic languages don’t even have the very clever “poor man’s refactoring” (change the method declaration, attempt a re-compile and then insert changes everyplace the compiler flags an error). When changing a method signature in a typical dynamically typed language you are typically left with the lurking worry that some bit of code somewhere is still attempting to use the old signature and will exhibit a runtime error when the exact set of circumstances required to execute the bad path happen in production (i.e. that you won’t be lucky enough to find it in a test). IDEs have a somewhat dirty reputation as being a crutch (somewhat due to horrible interface builders and large boilerplate systems), but the treatment of code as an object subject to a series of meaningful transformations is game changing (and is most commonly associated with statically typed languages, somewhat by historic accident but also likely due to the presence of extra declaration blocks often in statically typed languages and not due to the actual type system itself).

To sum up: dynamic typing allows more expressive code and saves space. But we pay a large cost downstream in more expensive debugging and much weaker ability to refactor or analyze. I favor the compromise where most code is statically typed and either only language supplied functions are capable of dynamic typing or there are user escapes out (like templating). While there is some doubt as to whether you can design a language as powerful as Scheme or Python without dynamic typing (some attempts have failed and some attempts are still evolving) I still prefer static typing. Or (more accurately) I prefer to deal with statically typed code (and am willing to put up with some expense to have it). Initial coding is not the only phase of the software lifecycle.

Related posts:

1. Programmers Should Know R
2. Automatic Differentiation with Scala
3. My Favorite Graphs

The ergodic theorem (http://en.wikipedia.org/wiki/Ergodic theory (link)) is an important principle of recurrence and averaging in dynamical systems. However, there are some inconsistent uses of the term, much of the machinery is intended to work with deterministic dynamical systems (not probabilistic systems, as is often implied) and often the conclusion of the theory is mis-described as its premises.

By “interested computer scientists” we mean people who know math and work with probabilistic systems1, but know not to accept mathematical definitions without some justification (actually a good attitude for mathematicians also).Please click through to read Ergodic Theory for Interested Computer Scientists.

Related posts:

1. Hello World: An Instance Of Rhetoric in Computer Science
2. Six Fundamental Methods to Generate a Random Variable
3. What Did Theorists Do Before The Age Of Big Data?

To implement many numeric simulations you need a sophisticated source of instances of random variables. The question is: how do you generate them?

The literature is full of algorithms requiring random samples as inputs or drivers (conditional random fields, Bayesian network models, particle filters and so on). The literature is also full of competing methods (pseudorandom generators, entropy sources, Gibbs samplers, Metropolis–Hastings algorithm, Markov chain Monte Carlo methods, bootstrap methods and so on). Our thesis is: this diversity is supported by only a few fundamental methods. And you are much better off thinking in terms of a few deliberately simple composable mechanisms than you would be in relying on some hugely complicated black box “brand name” technique.

We will discuss the half dozen basic methods that all of these techniques are derived from.To our mind all of the famous random variate generation/sampling techniques are derived from combinations of the following six fundamental methods:

1. Physical sources.
2. Empirical resampling.
3. Pseudo random generators.
4. Simulation/Game-play.
5. Rejection Sampling.
6. Transform methods.

The technical fights (such as: “is Gibbs sampling superior to, or even distinguishable from, Markov chain Monte Carlo?”) are all in the details, history and citation conventions. Each field and particular method accretes its own traditions. We will quickly discuss the fundamental methods we listed. As we will see: complexity goes up as we move through the list (so at some point things are no longer fundamental but instead derived, allowing us to end the list).

The Methods

Physical sources

This is the most basic way (though not as practical in the computer age) to generate random variables. Observe the flip of a real coin, shuffle actual cards, mix numbered balls or count the number of ticks from an actual radioactive source. In all of these the randomness comes from physical principles (such chaotic dynamics for coin flips or quantum mechanics for radioactive decay).

These sources are “outside of computer science” so we will say the least about them.

Empirical resampling

This is what used to be called “tables” (which were themselves often generated from physical processes). The observation is: that sometimes
to run a simulation you need access to instances of random variables that are distributed in a very precise way- but you don’t have a usable description of the desired distribution. You would think that in this case you could do nothing. But the principle of empirical resampling is that you can approximately generate new samples by taking samples (with repetition or replacement) from an old sample. This is the cornerstone of Bootstrap methods.

As an example: suppose we were given the sample of numbers 5, 5, 10, 5, 5 which has mean equal to 6. Further suppose we have no
description of how these number were generated but we wanted to know if a mean of at least 8 is likely or unlikely for five more numbers drawn the same way. We can approximate this by drawing many samples of size five from this original sample (allow the same number to be in our new
sample multiple times) and get the bootstrap estimate of the probability of seeing mean of at least 8 as having a probability around 0.6%.

This may seem trivial- but it is very important.

Pseudo random generators

In the computer age, to avoid need for external tables or expensive and slow peripherals we tend to use pseudo random generators. That is the output of deterministic iterative procedures as equivalent to true random sources. The science of pseudo randomness has evolved from cobbled together procedures passing ad-hoc tests (such as in Knuth Volume 2) to more formal pseudo randomness based on important properties (like provably being k-wise independent) or complexity (being computationally indistinguishable from a truly random on a time or space bounded machine). Behind the canned routines of all of the basic “random generators” commonly available is a pseudo random source.

Good references for the modern theory include:

• “Pseudorandomness and Cryptographic Applications” Michael Luby 1996.
• “Modern Cryptography, Probabilistic Proofs and Pseudorandomness” Oded Goldreich, 1999.

The most basic form of a sequential pseudo random generator is a sequence of states s(1), s(2), s(3) … . Where s(i+1) = g(s(i)) where g() is our deterministic function that maps state to state. The observed random variables are then h(s(i)) where h() is some deterministic function maps state to observables. For example for the linear congruential generator found in glibc we have g(x) = (1103515245*x + 12345) modulo 2^32 and h(x) = x modulo 2^30 (x an integer from 0 to 2^32 – 1). An example application: this generator when divided by (2^30 – 1) might return numbers passably uniformly distributed in the interval [0,1]. Two such variates might be uses as a uniform sample from the unit square.

That a simple iterated deterministic system (like the modulo arithmetic or even a physical system like coin flipping) would even superficially appear random (let alone be safe to use as pseudo random source) turns out to be the main consequence of Ergodic theory (which we will touch on in a later article). The point is: it should not be obvious (without bringing in some more theory) why you should trust pseudo-random sources.

Simulation/Game-play

Another fundamental method is direct simulation or game play. If we wanted a random variable that was 1 with probability equal to the odds of being dealt a full house from a standard shuffled deck of 52 cards (and zero otherwise). We can generate such a variable by simulating shuffling a deck, drawing a hand and returning 1 if the hand draw is a full house (and returning 0 otherwise). Notice in this case we are combining many random variables to get a single result.

One of the most important simulation techniques is Markov chain Monte Carlo methods (related to Gibbs sampling, simulated annealing and many other variations). These method implement a complex procedure over a stream of random inputs to generate a more difficult to achieve sequence of random outputs.

For example: Let T be the set of pairs of non-negative integers x, y such that x + y ≤ 1000. We could implement a Markov chain on this set from a source of coin flips. Given a point (x,y) in T we take three coin flips and move to new point (x’,y’) (also in T) using the following procedure:

1. Let m = 1 if the first flip is heads and m=0 if the first flip is tails.
2. Let v = (1,0) if the second flip is heads and v=(0,1) if the second flip is tails.
3. Let d = +1 if the third flip is heads and d = -1 if the third flip is tails.
4. If (x,y) + m*d*v is in T let (x’,y’) = (x,y) + m*d*v, otherwise let (x’,y’) = (x,y) (stay put).

Repeating this procedure a large number of times produces a sequence of points (x,y) such that (x,y) is distributed uniformly on S (again this follows from ergodic principles). The correctness of this simulation of or game of following a Markov chain is a very fundamental method in generating more complicated random variates and something we will write more about in an article dealing with the ergodic principle (the relation of connectedness to showing averages over time equal averages over space).

For simple shapes (rectangle, triangles) there are more efficient ways to generate points uniformly at random. For squares we exploit independence and just generate the coordinates independently. For triangles we could rejection sample from a bounding rectangle. Or we could use a tranform method: write down a counting function that indexes all the points in the triangle and generate points by index (for example it is easy to work out there are 501501 points in our example S so if we generate a random integer uniformly from 1 to 501501 can just pick the point with given index as our sample).

For general convex shapes (in high dimensions) these methods become intractible and Markov chain methods are one of the few options remaining.

Rejection Sampling

Rejection sampling is another way to convert one sequence of random variables into another. If we assume we can generate a random variable according to the distribution p(x) we can “rejection sample” to a new distribution using an “acceptance function” q(x) which returns a number in the interval [0,1]. Our procedure is to
repeat the following: generate x with probability p(x), generate a random variable y with uniformly in the interval [0,1] if y ≤ q(x) accept x as
our answer and quit (otherwise draw a new x and repeat).

When the distribution that rejection sampling draws with is such that if x and y had a ratio of being drawn of p(x)/p(y) then under the rejection procedure they have relative odds of (p(x)q(x))/(p(y)q(y)). An important special case is when q() is always 0 or 1, in this case we are drawing with relative odds proportional to p(x) from the subset of x with q(x)=1.

As an example: consider the problem of trying to draw a point (x,y) such that x^2 + y^x < 1 (the open unit disk) uniformly at random. The rejection sampling solution is: repeat the following until you have a success: generate x and y independently uniformly in the interval [-1,1], if x^2 + y^2 < then 1 accept them as our sample (otherwise repeat). This procedure is very fast as the unit disk that represents our acceptance region has area pi and the square we are generating trials from has area 4: so we over a 78% chance of success on each trial or expect to only have to run fewer that 1.28 trials (on average) to get a sample.

Transform methods

A transform method is used when we have the ability to generate instances of a random variable according to one distribution and we would like instances according to another distribution.

One method is used when we have access to the inverse of the cumulative distribution function of the distribution we are trying to generate. In this case we can use this function to convert uniform variants from the interval [0,1] into our target distribution. The commutative distribution function is the function cdf() where cdf(x) is the probability a random variate generated according to our distribution is less than or equal to x. The inverse function function icdf() where icdf(y) is such that cdf(icdf(y)) = y. For example the exponential distribution has an inverse cumulative distribution function icdf(y) = -ln(1-y)/lamda . So if y is
generated uniformly in the interval [0,1] then icdf(y) is a random variable generated according to the exponential distribution with parameter lambda.

A great example of transform methods is generating Gaussian random variables. We could directly use the inverse cumulative distribution function method described above- but to do this we would require a special function library to perform the required calculation of the inverse cummulative distribution (or inverse of erf()). Another way is the polar method: generate x,y uniformly from the open unit disk (by, for example rejection sampling as described earlier), set s = x^2 + y^2 and return x*sqrt(-2 ln(s)/s), y*sqrt(-2 ln(s)/s) as two independent Gaussian random variables. The trick being: the distribution function of r = sqrt(s) is of the form r*e^(-r*r/2) which leads to an elementary cumulative distribution function (unlike the original Gaussian density of the form e^(-r*r/2)) that is easy to invert.

Conclusion

Our thesis is: all major methods to generate random variables use aspects of the six methods we have listed here as fundamental. Or you should at least have a fluid understanding of at least these methods. You should be able to break down big “brand name” methods (like Gibbs sampling) roughly into their constituent parts (so you can reason about them). One example: notice how ratios of probabilities enter into Markov chain Monte Carlo methods (they cause step rejections); from this you can reason if your problem has bounded ratios it is a good candidate for direct application of the technique (and if it does not you need to add some more ideas, as was demonstrated in: “Sampling from Log-Concave Distributions,” Alan Frieze , Ravi Kannan , Nick Polson, Ann. Appl. Prob, 1994 ).

The first two methods we discuss (physical sources and empirical re-sampling) are of the class of solutions “already have the right answer.” Pseudo random generators are the primary way to negate the need for physical sources and resampling techniques. Simulation, rejection sampling and transform methods are the main tools for building new distributions out of old.

It is a matter of taste if a given trick fits into this ad-hoc taxonomy or not. You can invent new and better generation methods- but these methods are easily derived using ideas from the fundamental methods we mentioned here.

Related posts:

1. What is a large enough random sample?
2. Kernel Methods and Support Vector Machines de-Mystified
3. Book Review: Ensemble Methods in Data Mining (Seni & Elder)