The purpose of our article is to explain reverse accumulation automatic differentiation clearly (and to release some sample code and timing results). A side effect of the article is to make sense of the following two diagrams:

If the following is picture of standard or forward differentiation:

then the following is a picture of reverse accumulation:

Related posts:

1. Automatic Differentiation with Scala
2. “Easy” Portfolio Allocation
3. A Quick Appreciation of the Sharpe Ratio

Portfolio allocation is not the “magic predict the future” part of finance, it is the scheme for correctly applying magic predictions of the future. The idea is that if you had an prediction of future returns of a number of assets, the naive thing to do would be to invest everything into the asset with highest predicted return. Portfolio theory, while still taking the predictions at face value, picks an investment pattern that will (in risk-adjusted dollars) outperform the naive strategy even if the predictions are correct and is a bit safer when the predictions are wrong.

Suppose you had different assets you could invest in. For the -th asset there is an expected excess relative return of and an estimated variance of (for a definition of relative return see Relative returns: a banker versus trader paradox and for a definition of variance see A Quick Appreciation of the Sharpe Ratio). Let the vector be such that represents the number of dollars we invest in the -th asset. If is positive then our plan is “to go long” or buy some of the -th asset. If is negative our plan is “to short” or sell some of the -th asset to somebody else (It is called going short as we actually sell something we do not have. This is often allowed in finance; as long as we make the same pay-outs to the buyer that the buyer would receive if we really had the item to sell).

When we appeal to the idea of optimizing the portfolio Sharpe Ratio (again, see A Quick Appreciation of the Sharpe Ratio) then we say a good portfolio is one that doesn’t just maximize expected relative returns (which is ) but maximizes the ratio of expected relative return to standard deviation:

where (for now) is the matrix . This ratio is called a “risk adjusted return” (versus the un-adjusted form ). Also notice that the ratio is homogeneous in (doubling does not change the ratio as it simultaneously doubles the numerator and the denominator) so an optimal solution describes not how much to invest, but what pattern to invest in. This allows us to introduce an important practical constraint: we are only going to allow ourselves to risk a total of dollars (both long and short). That is: we insist . We will ignore this total investment constraint until the end when we can satisfy the constraint by simply re-scaling an partial solution.

To solve for we introduce an old friend: Lagrange Multipliers (or equivalently the Karush-Kuhn-Tucker conditions of optimality). Since the fraction we are trying to optimize is homogeneous in we can convert the denominator into a constraint and arbitrarily insist that without changing the nature of the problem. We are now trying to maximize subject to . The Lagrangian conditions of optimality state at the optimum we must have the gradient of the objective is proportional to the gradient of the constraint or:

for some (to be determined) constant . Pushing the gradient operator through we get:

A similar equation could be gotten by appealing to a Rayleigh Quotient argument.

We do not yet know (that is what we are trying to solve for), so we do not know what is. However, this is just a scalar and since we are just trying to solve up to a multiple we can throw it out and introduce a new multiple and see that it is enough to solve:

where is new (still unknown) scalar. This means we have:

so our desired solution is some re-scaling of .

As we stated earlier we have a total investment constraint of . We can achieve this with the following adjusted solution:

as our desired optimal portfolio allocation. In the end we can solve for the optimal portfolio by merely solving a linear system (we don’t need anything as expensive as a general purpose optimizer in this case).

These are very old results (going back as long as there has been Sharpe Ratios and portfolio theory). A good example reference is: “The Valuation of Risk Assets and the Selection of Risky Investments in Stock Portfolios and Capital Budgets,” John Lintner, The Review of Economics and Statistics (1965) vol. 47 (1) pp. 13-37. These results are the basis for advice like: “diversify.” Without modeling risk you would tend to put all of your money in the predicted highest paying asset. When modeling risk you tend to put some of your money in each high paying asset and as long as they do not all fail at the same time you have some safety. Another (very different) route to diversification is the Kelly Criterion (discussed in What is the gambler’s equivalent of Amdahl’s Law?).

A very important risk we have not yet modeled is that our assets may have a tendency to fail at the same time (meaning we may not have really diversified usefully). The notion of assets may fail at the same time brings us to the ideas of correlation and covariance. When we took we were implicitly assuming (or modeling), without justification, that each possible asset was independent of all the others (that there was no correlation between asset returns). This is, of course, not going to be anywhere near true in practice. Instead we should take to be the Covariance Matrix that represent our estimate of the assent to asset correlations. In this case the solution methods above all work exactly as before. Companies such as MSCI Barra have made complete businesses out of producing and selling estimates of .

Another issue is when we do not allow ourselves to “short” (or take a negative allocation of) assets. In this case we have the additional constraints which complicates our solution. For the special case where the asset variances are assumed to be independent (i.e. ) it is enough to solve as above and merely replace any negative allocations with zero when inspecting and scaling the final step of the solution. When the covariances are non-trivial ( has non-zero off-diagonal entries) this solution may not be optimal. In this case the Karush-Kuhn-Tucker conditions are more complicated and at the point of optimal solution we have the following conditions:

		0
		0
		0
		0

where is the allocation vector we wish to solve for, is an unknown scalar, is a new unknown vector and is the vector with and zeroes elsewhere. Using the Karush-Kuhn-Tucker conditions has allowed us to again almost linearize the problem, but we know have sign constraints on and and what is called a complementarity constraint: . This sort of problem essentially called a “Linear Complementarity Problem” and is about as hard as solving a linear program (the typical solution method is a variation of the simplex method called “Lemke’s algorithm”). (Technically the prevents the problem from being in the right form, but can be inspected out of the problem.) The problem can still be solved, you just need a bit more software. If we can not short assets (or at least simulate shorting assets) we not only eliminate many possible portfolios from consideration (so we likely end up with a less profitable portfolio than we would like) we also make the mathematics and computation a bit harder.

The goal of this writeup has been to show how to systematically convert investment advice like “this stock is going to really take off” into an allocation of assets (which in turn implies a pattern of trades). We take as unexamined premises where to get such advice and whether to use the Sharpe ratio or some other notion of risk and/or utility. The point is that even though it may be complicated, from this point it is just calculation and calculation is easy to automate.

Related posts:

1. A Quick Appreciation of the Sharpe Ratio
2. A Discrete Model Gauging Market Efficiency
3. What is the gambler’s equivalent of Amdahl’s Law?

Amdahl’s Law is famous idea due to computer architect Gene Amdahl. It is a simple technique that computer scientists use to re-direct their work back to important parts of problems. Suppose you have a complicated system you wish to speed up. Suppose this system is spending a p-fraction of its time in an important sub-process and that you have an idea that would speed up the sub-process by a factor of k. Should you invest the effort?

Amdahl’s Law says (by simple arithmetic): the speed-up (the ratio of the old run-time over the new run-time) the entire system would achieve if you implemented your improvement is not the factor of k you would hope for, but instead:

For example if p = 1/3 then you can only speed up the over all system by at most a factor of 33%, even your idea is so astoundingly good that you have k=1000.

Amdahl’s Law reminds us that speeding up a component you do not lose much time to is not an important accomplishment. In fact Amdahl’s Law directly prescribes looking at your most expensive components as being the largest opportunities for improvement. Appealing to Amdahl’s Law is an important nerd-tool to end “color of the bike shed” arguments (and concentrate only on the design of systems that actually have an impact on outcomes).

It is clear there are similar principles for managing expenses, revenue, effort and so on (such as the Pareto Principle).

But what is the equivalent statement in the harder and more complicated world of probabilities and gambling systems? There are a lot of candidate statements and theorems (such as “look for horses not for zebras”, the Kraft Inequality, Kullback Leibler Distance, Cross Entropy and the Asymptotic Equipartition Principle) but I think the most powerful and direct analogue is: the Kelly Betting System. The Kelly Betting System is a remarkable system that, like Amdahl’s Law, tells us exactly what to look at (and surprisingly some things to ignore).

Kelly’s original paper: “A New Interpretation of Information Rate” J. L. Jr Kelly, AT&T Technical Journal (1956) phrases the problem as betting at a horse race. The technique applies more generally (other forms of gambling, portfolio management, even explaining the preferences of lab-mice) but the clearest example remains a horse race.

We follow the excellent discussion of the problem from Cover and Thomas “Information Theory” Wiley (1991). Consider a simplified horse race where there is only one payoff offered: picking the winning horse. Suppose the (unknown) true probability of the i-th horse winning is p_i. Further suppose the track publishes a set of payoffs for each horse such that if you bet a dollar on the i-th horse and it wins: you are given o_i dollars back.

Now a gambler that has no estimate of the p_i might put all of their money on “the highest paying horse.” That is picking the i such that o_i is maximal (“going for big score”). A somewhat more informed gambler might put all of their money on the “horse with the best expected return” that is a horse i that maximizes p_i * o_i. But this betting strategy “invites ruin”: you have probability of 1 – p_i of losing all of your money. Kelly starts with the controversial idea of trying to maximize expected log-return (instead of maximizing expected return). Maximizing expected log-return avoids ruin, maximizes the exponential rate your wealth grows and maximizes the median wealth over all outcomes (see: “The Kelly System Maximizes Median Fortune” S N Ethier, Journal of Applied Probability (2004) vol. 41 (4) pp. 1230-1236). Even the observation that you don’t always want to put all of your money in a “favorable bet” (that is one with expectation p_i * o_i >1) is an important one.

To get the next part of Kelly’s system consider the sum of reciprocals of track offered payoffs:

At any real track this sum will be greater than 1 (i.e. the o_i will be small, making the sum large). The larger the sum the more clearly unfair the track’s published payoff schedule is. Let us assume we were at a fantastically generous track where this sum is exactly 1 (admittedly unrealistic, and both the paper and the book work beyond this limitation). In this case we can write r_i = 1/o_i and we know r_i > 0 and the r_i sum to 1. That is we can interpret the r_i as the track’s estimate of the probability of the i-th horse winning. If o_i = 100 (the track is paying off 100:1) we then can infer they think the i-th horse has no more than a 1 in 100 chance of winning (else they could not afford to offer the bet). Kelly’s system gives (and proves correct) the following remarkable advice: if the sum given above is 1 (i.e. the track is paying off at least a fair rate) then you can safely bet all of your money and you should bet a p_i fraction of your money on the i-th horse.

That is: if you decide the track is paying off so much that it is worth your while to gamble then you should then completely ignore the track’s payoff schedule in making your bet. You might use the track’s published payoffs as some of your evidence when trying to estimate the p_i (the probability of each horse winning), but once you have estimated these probabilities you then ignore the track’s payoff rates in designing your bets. In fact your expected rate of winning is exactly proportional to how much closer to the true probabilities your estimate is than the track’s estimate is (Cover/Thomas example 6.1.1, so if unless you know something the track does not know you should not bet). Also you should bet even on unlikely and underpaying horses to help cover the possibilities (this is because you are making a series of bets, not just a single bet- so each bet’s value is computed under the assumption that your other bets have failed). This (provably correct) advice is contrary to many obvious and traditional betting systems.

The Kelly System is simultaneously very precise and broadly applicable. For example: it has be extended to many other games and the stock market (see: “The Kelly Criterion and the Stock Market” Louis M Rotando, Edward O Thorp, The American Mathematical Monthly (1992) vol. 99 (10) pp. 922-931). The Kelly System gives actionable advice (exact amounts to bet or exact amounts of effort to invest) and is very specific in saying what to look at.

Just as Amdahl’s law shows us component speedup is a distraction the Kelly System shows us that published rates of return are siren songs. Thus the Kelly System is the gambler’s equivalent of Amdahl’s Law.

Related posts:

1. A Quick Appreciation of the Sharpe Ratio
2. Betting Best-Of Series
3. How Market Designs Set Prices

When a graph is made, quantitative and categorical information is encoded by a display method. Then the information is visually decoded. This visual perception is a vital link. No matter how clever the choice of the information, and no matter how technologically impressive the encoding, a visualization fails if the decoding fails. Some display methods lead to efficient, accurate decoding, and others lead to inefficient, inaccurate decoding. It is only through scientific study of visual perception that informed judgments can be made about display methods. The display methods of Elements rest on a foundation of scientific enquiry.

— from the preface of The Elements of Graphing Data

A revised edition of The Elements of Graphing Data was published in 1994, along with a companion volume, Visualizing Data, which is oriented towards the implementation and technical details of different graphing techniques. I highly recommend The Elements of Graphing Data as a guidebook for creating graphs, as well as for its excellent survey of several useful techniques. Cleveland, along with other colleagues at Bell Labs, developed the Trellis display system, a framework for the visualization of multivariable databases, using the ideas developed in his texts. Trellis, in turn, influenced Deepayan Sarkar’s Lattice graphics system for R. Lattice implements many of Cleveland’s ideas, and I also recommend Sarkar’s Lattice manual if you do data visualization in R.

It’s important to note here that Cleveland writes for researchers and decision-makers who use graphs to analyze data, or to convey scientific results to colleagues in an (ideally) objective manner. This distinguishes him from Darrell Huff, whose 1954 How to Lie with Statistics considered the use of graphs (and statistics in general) as rhetorical devices for convincing others of one’s point of view. Hence, some of Cleveland’s recommendations and guidelines actually contradict Huff’s. ¹

Edward Tufte also explored the idea that the choice of graphical display should be influenced by the viewer’s cognitive processes, in his 1990 book Envisioning Information. Tufte tends to be more broadly concerned with the gestalt of a graph, beyond its use as an analysis tool; he is also more concerned than Cleveland is with aesthetic considerations.

Cleveland’s philosophy might be summarized as: minimize the mental gymnastics that the viewer must go through to understand the graph. This leads to some obvious advice: avoid clutter and occlusion, make graphing symbols or color-coding unambiguous, use scale-lines on all four sides of the graph, and so on. It also leads to advice that perhaps should be as obvious, but isn’t: make the aspect of the data that you want to analyze as clear as possible. But what does this mean in practice?

Make important differences large enough to perceive

Weber’s Law is a well known observation from the psychophysics literature, which states that the “just noticeable” change in a stimulus is a constant ratio of the original stimulus. Put another way, people are only capable of detecting a change in a stimulus that is greater than a certain percentage k of the original stimulus. Here, “stimulus” can refer to any perceivable physical quantity: weight, intensity, length, orientation. The percentage k will vary with stimulus, and with observer.

Figure 1: From Cleveland, The Elements of Graphing Data

Figure 1 shows the application of Weber’s law to lengths. The bars A and B are of different lengths, but the difference is such a small fraction of the “base” length (say, A’s length, to be specific) that is difficult to tell whether or not they are different, or which is longer. On the right, the bars have been embedded in frames of identical length, and now it is easy to see that B is longer. Why? Because the difference in lengths of the white intervals is a much larger percentage of the white “base” length (say the white A interval). It is easy to see that the white B interval is shorter than the white A interval, and therefore, the black B interval is longer than the black A interval.

The moral is that you always want the viewer to be estimating changes or differences with respect to a short base length. You can do this with reference grids, as demonstrated below.

From Cleveland, The Elements of Graphing Data

Figure 2	Figure 3

Figure 2 shows eight curves. Which one dips to the lowest minimum? Are the high curves approaching the same value, and which one is rising the fastest? Are the low curves dipping to the same minimum? Are they going to the same steady state? Figure 3 shows the same curves, graphed with identical reference grids. The grids shorten the base lengths that are being compared, and it is now much easier to compare highs, lows, and steady state behavior.

But wouldn’t it be better to compare the graphs by superposing them? For two or three curves, perhaps. But in this case, eight curves can clutter the graph, and use up the symbol or color space, making it difficult to distinguish the different datasets — increasing the mental gymnastics.

Reference grids are useful even for a single curve, especially one with slowly varying segments, such as these graphs have. The reference grid makes it easier to answer questions like: does the process return to the initial state, or to a different steady state? Has the process reached steady state, or is it still growing?

Make important shape changes large enough to perceive: Banking to 45 degrees.

The aspect ratio of a graph is important when trying to understand shape. Rate of change information is encoded in the slope of the curve, which the viewer estimates by changes in the orientation of the local tangents at each point of the graph. Weber’s Law tells us that very small changes in this orientation will be difficult to detect. For a given (physical) curve, the local orientation changes will be dependent on the aspect ratio of its graphical presentation, as shown (to an exaggerated degree) in Figure 4. Here, the same curve (two line segments) is plotted at three different aspect ratios, one that centers the graph at 45 degrees, one that forces the curve to be nearly vertical, and another that forces it to be nearly horizontal. In the last two cases, the change in orientation of the two line segments is so small as to be nearly undetectable.

Figure 4: From Cleveland

For two line segments with positive, unequal slopes, a simple geometric argument shows that their absolute difference in orientation is maximized by the aspect ratio that sets their average orientation to 45 degrees (the first graph in Figure 4). Empirical studies by Cleveland and others have indeed verified that a viewer’s ability to judge the relative slopes of line segments on a graph is maximized when the absolute values of the orientations of the segments are centered on 45 degrees.

This result leads to a technique called Banking to 45, whereby the aspect ratio of the graph is chosen so that the average slope of the entire graph is 45 degrees. The details are discussed in Cleveland, and many of the plots in R’s Lattice package also have an option to bank the graph to 45 degrees.

This deliberate exaggeration of slope is something that Darrell Huff deplores. In How to Lie with Statistics, Huff refers to these graphs as “gee-whiz” graphs — and in the context of his discussion of statistics as rhetoric, they are:

Figure 5: From Huff, How to Lie With Statistics

To insist that a graph should always include a zero line and that units be in proportion may be good advice from a rhetorical perspective; but it is poor advice if the purpose of the graph is data analysis. As Figure 6 below demonstrates, we can lose resolution if we always insist on including the zero. Does the trend line in the left graph increase linearly, superlinearly, or sublinearly? The convexity of the curve is more apparent when it is banked to 45, as on the right. Assuming that the scientist reads the axis and is cognizant of the actual magnitude changes involved, the graph on the right conveys more information.

Figure 6: From Cleveland

Make sure all the data is equally well resolved.

It is quite common for positive data — word frequencies, populations, price distributions, just to name a few examples — to be skewed: most of the data is bunched towards low values, the rest of it is spread out on a very long tail. This long tail squashes the majority of the data into a tiny interval of a very narrow dynamic range, as in Figure 7, making it difficult to evaluate the data.

Figure 7: Long-tailed distribution of purchase sizes

Figure 8: Distribution of log(purchase size)

Imagine that Figure 7 represents the distribution of average purchase size across an online merchant’s customers: average purchase size is plotted on the x-axis, and the y-axis represents the fraction of the total customer population whose average purchase size is a given value (the area under the graph integrates to one). According to this graph, most customers make fairly small purchases on average, but there is a long tail of big spenders trailing out into the range of several thousand dollars. Obviously, one would like a little more resolution on the big spike of customers near zero. One could simply “zoom in” on this range, by chopping off some long chunk of the tail, but you may potentially lose sight of some global patterns in the data by doing so.

Graphing the distribution of log(purchase size) enables you to increase the resolution near zero, while preserving the global view. Figure 8 shows the distribution of log(purchase size), revealing two spending populations: a population of high spenders who tend to make purchases in the $3000 range (in log space), and another population whose purchases are centered (in log space) around $60. The existence of these two distinct populations is not apparent in the original graph.

Notice that Figure 8 has two x-axis scales: the top axis is marked in log units, while the bottom axis is marked in absolute dollars, spaced on a log scale. This accords with the principle of minimizing mental gymnastics, since the viewer of the graph will typically be concerned about prices in dollars, not log dollars. In fact, it would have been better yet to have plotted the distribution of log₂ or log₁₀ of the data; the former would allow us to see at a glance the doubling of price ranges, the latter to see price changes in factors of ten.

Figure 9: The 14 most abundant elements in meteorites. From Cleveland

Figure 9 shows another example: the fourteen most abundant elements in meteorites, specifically the average percent of each of the elements. If we graph the percentages directly, as on the left, we cannot easily distinguish the differences in the elements from aluminum on down. Graphing log₂ of the percentages, as on the right, improves the resolution. Again, we have two x-axes on the graph of the log data.

If you want to analyze the difference between two processes, then graph the difference, not the processes (or graph both).

Suppose that we are comparing the two processes f1 and f2 that are shown in Figure 10. As x increases, the two processes appear to be approaching each other — that is, the difference between the two seems to be decreasing. In reality, the difference between the two is constant: f2 = f1+1.

Figure 10: The illusion of convergence

Figure 11: British Imports and Exports. From Cleveland

It turns out that people are good at perceiving the perpendicular difference between two curves, but not the differences in height, which is what we are actually interested in here. When we try to infer the differences from the process graph, we may not only miss key information, we may actually draw incorrect conclusions.

A less toy example is given in Figure 11. Here the imports to and exports from England are graphed over the first 80 years of the 18th century. In the difference graph on the bottom, we can see a local peak in (imports-exports) just after 1760; this is not obvious from simply comparing the two processes (top graph).

If you are interested in rate of change, then graph rate of change.

In Figure 12, we see the population figures for a given community from 1990 to 2009. Obviously, the population is steadily increasing, but how quickly? Is the rate of population growth increasing over time, or is it decreasing? If we are interested in these questions, then simply graphing the population over time is not enough. We need to look at the rate of change directly.

Figure 12

Figure 13

The classic way to do this is by graphing the logarithm of the data. In Figure 13, we have graphed log₂ of the population over time, with the log scale printed on the right hand y-axis, and the actual population numbers printed at a log scale on the left hand axis. Now we can see that the population increased at a constant rate from 1990 to 2000, quadrupling approximately every four years, and then slowed down (to a lower constant rate) after 2000.

Graphs as a research tool

Throughout this discussion, we have considered graphs as a tool for data exploration and initial understanding. It is an iterative process — as questions arise, the data will be reprocessed and re-plotted to highlight the new issues to be examined. A good research graph must display this information directly, with a minimum of mental gymnastics, but — as with any research tool — there can be a learning curve. For example, densityplots (such as those shown in Figures 7 and 8) are in my opinion more useful than histograms for understanding how numerical data is distributed — and I am constantly surprised at the amount of explanation that they require when I show them to people who are unfamiliar with them. A number of very useful graphs that are discussed in Cleveland’s texts meet with the same reaction from people who encounter that style of graph for the first time. This is a disadvantage, relative to using a more fashionable graph, when attempting to communicate results. But the insight into the data that these graphs provide often make it worth spending the time to educate clients or peers on how to read the graph.

Even so, a good graph still may not be a quick read. As Cleveland writes:

While there is a place for rapidly-understood graphs, it is too limiting to make speed a requirement in science and technology, where the use of graphs ranges from detailed in-depth data analysis to quick presentation.
…

The important criterion for a graph is not simply how fast we can see a result; rather it is whether through the use of the graph we can see something that would have been harder to see otherwise or that could not have been seen at all.

- The Elements of Graphing Data, Chapter 2

Related posts:

1. A Demonstration of Data Mining
2. Living in A Lognormal World
3. The Data Enrichment Method

Now more than ever we must be familiar with the consequences of these important techniques. Especially if we don’t know if we might already be a victim of them.

“The Data-Enrichment Method” is an absolutely wonderful 1957 tongue in cheek parody of a very tempting method of accidental data falsification. The method presented is spookily plausible and actually anticipates some very important (and correct) methods later used in the EM, Jackknife, Bootstrap and other resampling techniques (for example see: “Bootstrap Methods: Another Look at the Jackknife”, Bradley Efron. Ann. Statist. (1979) vol. 7 (1) pp. 1-26).

The idea is innocently presented with an accompanying data-set: perception of a sound at a different presented decibel levels (loudnesses):

From this table it is obvious that the number of detections is increasing (and the number of failures is decreasing) as the sound is presented louder and louder. This makes sense and puts a quantitative rate to our prior expectation that detection gets easier as loudness increases. For this data the trend is quite obvious and we can easily plot a regression line that accurately models the effect of Source.DB on detection rate:

But we want more. Can we increase our model precision and confidence by incorporating our domain knowledge? If we are only trying to accurately estimate the rate that loudness increases the detection level and we are willing to assume that it really does increase, then: could we not pre-prepare the data to use our domain knowledge?

The method suggested is to add in some contra-factuals that we feel confident about. For example we could (using our assumption that loudness increases detection, just to an unknown degree) notice that the 30 failures at 65 DB certainly would not have been heard if they had been run at 62 DB (even quieter). By the same reasoning we can assume that the 5 detections at 62 DB would have been heard had they been run at 65 DB, 68 DB, 71 DB, 74 Db or 77 DB. In this way we have used our starting “seed data” and our domain knowledge to boost into a much larger data set that shows the expected relation much more strongly.

The above paragraph is, of course, nonsense. I am doing the original paper an injustice by summarizing- because in the original paper the procedure seems perfectly plausible (and useful). It is not until the author works a second example that has a poor initial relation (that actually needs the enrichment) that the joke is revealed.

The second example is coin flipping. The author applies an inductive bias that “clearly standing higher up on a staircase increases the chances of a coin flip coming up heads” and then uses the data enrichment method to enhance the data set. The original data set is indeed too noisy to show the effect and the enhancement is in fact quite dramatic. The original data:

The enhanced data is much more interesting:

It is easier to see what is going on in the following plots (which show measured success rates as a function of number of stairs up the staircase and show a smoothed fit of the relationship). The original data is a noisy mess:

And the enriched data is more trend-like:

In fact the regression line fit onto the raw data even has the wrong sign (points down instead of up):

Now, obviously this is a joke. The enhancement procedure did not so much enhance the data as obliterate it. The procedure makes no sense and it is treating the procedure with undue respect to point out any one feature as being “what is wrong with it.” But the original desire is legitimate: can we use informed assumptions to gain a useful inductive bias? If we do know something should we not need less data?

The answer is yes- but we have to be careful. We must read up on the differences between Bayesian, frequentist and empirical methods and decide which set of methods is best for us. Up until now we have been fitting “by standard methods” which is really just minimizing how far the data is from the model (by moving the model around). That isn’t the only way to fit (see: “Controversies In The Foundation Of Statistics” Bradley Efron, American Mathematical Monthly (1978) vol. 85 (4) pp. 231-246).

For example a Bayesian might say that the goal of model fitting is not to pick a model that is closest to the data (maximizes the data’s plausibility with respect to the model) but to pick a model that simultaneously maximizes the product of the data’s plausibility with respect to the model and the model’s acceptability. For example we could say all models for coin-flips with negative slopes are unacceptable and pick the best model with a non-negative slope. However, assigning of degrees of acceptability (or priors) on every possible model is laborious and may require more knowledge than we have from our “reasonable prior domain knowledge.”

Another method is to use more sophisticated notions. One such method is Quantile Regression ( Roger Koenker, Cambridge University Press 2005). This methodology treats regression as a constrained optimization problem- so it is a simple matter to add in more constraints (like the slope must be positive) without having to assign arbitrary plausibilities to every possible model. Another (huge) advantage is that Quantile Regression is much more stable and even without any entered constraints recognizes that the coin-flip data is likely trend free. Here we plot the Quantile Regression analysis of the coin-data (without having added any prior constraints):

To be honest: the method got lucky- the fit is better than should be expected. But Quantile Regression is the perfect framework for adding in domain-constraints.

So: while The Data Enrichment Method is a fraud, there are ways to to enhance analysis to incorporate domain knowledge into results. Instead of saying “any bias (even useful bias) ruins fitting” one should have a cookbook of methods ready to be applied. These cookbooks hide under names like “Econometric Society Monographs” (in my opinion the econometricians really own the interface between theoretical statistics and hard-nosed applications).

Related posts:

1. A Demonstration of Data Mining
2. Good Graphs: Graphical Perception and Data Visualization
3. Exciting Technique #1: The “R” language.

The Sharpe ratio is a famous measure of “risk adjusted return” and is defined as “the ratio of the expected excess return from an investment divided by standard deviation of the excess return.” It is most easily demonstrated by an example (which we work in pieces).

If an investment is expected to generate a profit of 15% in the next year and an insured bank account would generate 10% profit then the expected excess return invested is 15% – 10% = 5%. A rational investor would never take a risky investment that did not have a positive excess return (else they would expect to make more money at a bank). “Expected” is a technical term which means the average return of the investment averaged over all possible outcomes (weighted by the odds of each outcome), we can explain this by working a couple of examples.

Consider investment “A” which is a generally good idea that returns a 20% profit in half the possible years and a 10% profit in the other half of the years. Investment A has an expected return of 0.5*20% + 0.5*10% = 15%. Investment “A” has 15% – 10% = 5% excess return.

Also consider another investment “B” which is a risky bet that returns 20% profit most years (around 95.8% of them) and goes bankrupt in the other years. The expected return of investment “B” is 0.958*20% + 0.042*(-100%) = 14.96%, or essentially 15%. Investment “B” has 15% – 10% = 5% excess return.

As we can see “expectation” alone can not really tell these two investments apart. That is why the second component of the Sharpe ratio is something called the standard deviation. The standard deviation is defined as the square-root of the squared deviations of the return from the target value of 15%. What we do is measure for each possible outcome how far off the return is from the target of 15%, multiply this number by itself (called squaring it) and then take the square-root of the sum of all such values. Again, this is best explained by an example.

Investment “A” has a standard deviation of:
square-root( 0.5 * (20% – 15%)*(20% – 15%) + 0.5 * (10% – 15%)*(10% – 15%) ) = 5%

And investment “B” has a standard deviation of:
square-root( 0.958 *( 20% – 15%)*( 20% – 15%) + 0.042*(-100% – 15%)*(-100% – 15%) ) = 24%

Just like in the calculation of expectation we are taking every possible situation and summing (weighted by the likelihood) our value of interest (in this case the squared variation).

The standard deviation’s opinion is that investment “B” is about five times riskier than investment “A.” And this is the grace of the Sharpe ratio: it says that investment “A”‘s value is (15% – 10%)/5% = 1 and “B”‘s value is (15% – 10%)/24% = 0.2.

An interesting feature of the Sharpe ratio is that, unlike Wall Street, it does not believe that leveraging increases profitability. A common desperation move is to take an investment that has a moderate return and borrow money to simulate larger returns by having larger exposure. For instance an investment that returns 15% can try to simulate a higher return by borrowing. If for every $1,000 invested we borrow another $1,000 to invest (paying the risk rate of 10% for the money) one can show an apparent rate of return of ($2000*15% – $1000*10%)/$1000 or 20%. However, this is not free money- the investor is taking on twice as much risk for only half as much more return. In fact with sufficient leverage (three times, for times, thirty times) one can convert a safe investment into a risky investment that could even go bankrupt. The Sharpe ratio (by design) is not fooled by this sort of manipulation. Investing $1000 in investment A has the exact same Sharpe ratio as investing $1000 plus $1000 more borrowed at the risk-free rate (this is part of the cleverness of using excess returns instead of un-adjusted returns).

Unfortunately to use the Sharpe ratio you need good estimates of three things:

1) The expected return of the investment.

2) The risk-less available in the market (to compute excess).

3) The standard deviation of the investment.

All three of these facts are about the future, so we don’t really know any of them. The historic returns of an investment are not the same thing as the expected returns in the future, interest rates can change and the standard deviation is especially hard to estimate. However, if you have a model (or at least a theory) of what your investments are supposed to do then you can plug in estimates for these three quantities and use the Sharpe ratio to determine which investments really are best.

If you knew how investment “A” worked and could estimate that it returned 20% about half the time and 10% the other times you could estimate its Sharpe ratio as 1. And if you knew investment “B” was a gamble that almost always paid off at 20% with a single rare event that causes bankruptcy you could estimate its Sharpe ratio as 0.2. Even if your estimates were inaccurate (say you estimate investment “A”‘s Sharpe ratio is 0.7 and investment “B”‘s Sharpe ratio as 0.3) the indication is to stay away from investment “B.”

This is in stark contrast to the conclusion you would draw if you thought of these investments as a “black box” (like a fund of funds does) and looked only at their historic performance. If you looked at around 5 years of historic performance of both investments you would (incorrectly) think the following:

Investment A looks kind of noisy, some years it returns 10% and some years it return 20%. You would estimate (correctly) the return as averaging to 15% and you can even get a historic estimate of its standard deviation that is actually about right (5%)

Investment B looks like easy money. With about 80% chance you would not have seen a bankruptcy, just 5 years of 20% returns. You would mis-estimate the return as being 20% (all you have ever seen) and further mis-estimate the standard deviation as 0%.

Based on historic data alone you would fire the manager of investment “A”, give the manager of investment “B” a huge bonus and invest all of your money. And a few years later you would go bankrupt.

What is going on is very well explained by Nassim Nicholas Taleb as “the turkey paradox.” Domestic turkeys are all killed at about the exact same age (say 60 days). For somebody that understands commercial poultry farming there is not any mystery or uncertainty about it. 60 days before you want to sell a turkey carcass you buy a turkey chick. There is an inevitability and reverse causality- the desire for the turkey’s carcass funds and causes the turkey’s start of life 60 days earlier. Now if the turkey is a statistical empiricist (perhaps with a PhD in machine learning) things look good. The turkey sets up a model of each day having an unknown chance of being good or bad. The turkey figures that each day’s outcome is an independent trial drawn from this single unknown probability. The turkey collects evidence: every day it gets fed. Each day is more evidence that all days will be good. And then on day 60 the turkey gets a nasty surprise. The turkey’s life was a bad investment from day one, all of the “evidence” the turkey collects along the way was irrelevant because the model was wrong. And the model was wrong because the turkey guessed at the model instead of investigating the nature of poultry farming.

Much is the same in many investments. There are investments that look like investment “B” when you open the hood. Many of them involve writing “out of the money options” and “default swaps.” These are essentially selling insurance on events that nobody thinks are likely. Selling insurance that usually is not used is profitable, until the insurance gets used. This is why insurance companies (if they are ethical) don’t treat the entirety of collected payments as profit- but as a stockpile that must be kept to pay the claims that will inevitably some day come true.

It is important to point out the Sharpe ratio will give you incorrect results if you plug bad estimates into it. Overall the Sharpe ratio prefers good investments and diversification but it can be led astray. In fact that is the whole point: no amount of smart math will undo the inevitable consequences of wrong models that are used because “you need something you can solve” (like the turkey) or “everybody else is getting rich using them” (like investment “B”).

Related posts:

1. “Easy” Portfolio Allocation
2. What is the gambler’s equivalent of Amdahl’s Law?
3. What does the Market Think?