I have finally written up and released a paper in PDF: Automatic Generation and Testing of Trades describing a lot of the statistics and optimization methods used when I was technical trading on a Banc of America Securities proprietary program trading desk.  It was a very exciting time.

I have also included a less legible HTML version:

Automatic Generation and Testing of Un-Rolls for Profitable Technical Trades

John Mount¹

Date: September 9, 2007

Introduction

In this paper we discuss some of the basic steps in developing successful technical trading strategies. The method involves identifying an inefficiency or irregularity in the market and then using rigorous statistical methods to track and exploit this single feature of the market. We show how to automatically generate and test optimal un-rolls or trades that undo (at a profit) automatically triggered technical trades. That is to say, if the first half of technical trade is specified we show how to find the other half.

Our technique is to use standard tools, such as kernel methods[8] and Markov chains[4], to model both the efficient and the inefficient portions of the US stock markets.[6]

The author traded profitably using some of these techniques while part of a program trading desk at Banc of America Securities.

Technical Trading

Technical trading is a popular universe of security-trading strategies that trade using only the so-called technical data which are price graphs, volumes, bid/ask books and other data commonly available in market feeds.²Input sources can also include external triggers based on news, RSS feeds, on-line information and corporate announcements.³These strategies are very attractive in that that are quantifiable, easy to implement and easy to back-test on historic data. A major weakness of technical trading strategies is that they ignore deeper knowledge or analysis of the companies that are behind the securities being traded. Systems of technical trading are used both by large sophisticated hedge funds and by a varying population of day-traders.

Typical technical variables include price, time, volume and moving averages. It is important to know that many of these variables are really just analogies and not essential features of the market. For example: none of the variables current price, time, velocity, acceleration or inertia are real market quantities. What is traditionally called current price is actually the price of the last trade, which is in the past and may or may not ever be seen again. The fundamental variables of state of US stock markets are bid (best purchase price and quantity currently offered), ask (best sale price and quantity currently offered), and last trade (price and quantity). Each change of these variables is called a tick and can happen at any time. More detailed views include detailed bid and ask books from multiple market participants and estimates of inventory imbalance of various market makers and specialists.

In addition to working with the proper variables a sound strategy must also have at least two important components that we call foundation and empirical correctness. Without these components there is a large danger self-delusion and an unreliable strategy.

By foundation we mean that there are a priori reasons to believe that some variation of the strategy should be profitable. By ignoring the nature of the companies underling the securities being traded technical trading starts on shaky ground. In fact it is tempting to appeal to an efficient market hypothesis and claim that no technical trading strategy should be profitable. In some sense this is true- trades made in true ignorance expose a trader to significant risk, trading costs and pointless payment of the so-called bid-ask gap. Founded technical trading strategies are based on violations of the efficient market hypothesis- identifying situations where the market is in fact not efficient and trading into these situations. If there is no reason to suspect a market inefficiency there really is no reason to perform a technical trade. Testing numerous un-founded trading strategies is more likely to discover irrelevant anomalies in past data or discover flaws in one’s statistical procedures than it is likely to discover new valuable trading rules.[3]

Possible market irregularities include (but are not limited to):

• Market Open
• External News
• Earnings Reports
• M&A news
• Unusual Volume
• Inferred state of Market Maker / Specialist state
• Detailed Bid/Ask book.

By empirical correctness we mean that strategy can be validated and proven on historic market data. A technical strategy can have as much mathematical pedigree as you like, but it does not make sense if it can not be mechanically implemented and proven on historic data. Many technical features are popular due to their familiarity or the quality of graphs they produce- but the true measure is how well strategies generate specific executable actions and the quantified outcomes of those actions.

Given an irregularity it remains to develop the trading strategy. Typically this involves an initial trade (a buy or a sell) triggered by evidence of the irregularity/inefficiency followed somewhat later by a reversal or un-rolling of the trade (selling back against an initial buy or buying back against an initial sell). If markets were perfectly efficient and instantaneous in incorporating external events this should not work- so it is important to test that there really is a repeatable market inefficacy.

Possible initial trading strategies could include:

• Selling stock into an unusual price spike (a contrarian strategy).
• Buying stock immediately on news (a superior connection strategy).
• Selling stock into a perceived specialist imbalance (a superior knowledge strategy).

It would be naive to expect that a strategy that starts on a trigger and then reverses its trade blindly (say some fixed time after the trigger) is fully efficient. We must assume that other players in the market have seen effects of the trigger we traded and that their actions introduce biases and uncertainty into the market. Modeling these effects will allow us to produce a systematic unrolling strategy that can complete any entry strategy into a complete round-trip system. This systematic unrolling strategy is the subject of this writeup.

First Model

The Efficient Market Hypothesis

The efficient market hypothesis is a useful tool, even when you are attempting to find inefficient market situations. It represents the baseline you feel you have found a useful deviation from. The efficient market hypothesis has many variants but the essential content is that the market is full of informed players so any information is already factored in to the price. For example if there is publicly available information that gives a reasonable expectation that a stock should rise in the future then informed investors would purchase the stock early to be in a position to benefit from this increase. These purchases actually cause their own price-increase (by the simple laws of supply and demand) and have the effect of reducing the value of the information- as they move the price increase back in time (from the expected future change in value to the time of the anticipatory buying). This is what is meant by the phrase “already factored in.”

Figure 1: Dell 10-13-2006 Tick Data.

There is a mathematical concept that captures the idea of already factored in: Martingales. The Martingale condition is a concept that says the expected future value is the current value. For example betting a dollar on the flip of a fair coin is a Martingale of value (the odds of winning and losing a dollar balance out). The future value may be higher or lower- but when the Martingale condition is met the average of all these value weighted by their likelihood of occurrence is equal to the current value. The already factored in example mentioned above shows how the many players in the market tend to establish a near-Martingale by trading in such a way to move the current price to be the expected value of the future price.

Figure 2: Graph of a market-like random walk.

If market prices were the sum of many individual traders each with bounded-budgets who traded independently then we could apply the central limit theorem or law of large numbers and say that the market is indeed a random walk like the famous Brownian motion from physics. In fact on first inspection the market price histories (as in Figure 1) indeed look very similar to graphs generated by such a random process (as in Figure 2).

As we have said: it is no coincidence that the market looks nearly like a Brownian motion. Informed trading effects tend to impart Martingale like tendencies (once the overall increase factor of the value of holding wealth is factored out). Also, if the variance of the market were much larger than that of a similar Brownian motion this would itself attract channel traders who would benefit by trading in and out of the excess wiggling. The point is that an efficient market is usually pretty well described by random processes that have the Martingale property (like Brownian motions or Markov chains), so these are appropriate modeling tools.

If the market process really were such a random walk than there would be little point in technical trading. The whole theory of Martingales was developed to precisely describe situations where bets based on collecting historic information can not work. This is often called the no gambling system principle and it can be actually proven for systems like Martingales, unbiased Markov chains, drift-free Brownian motion and was even used as an foundational concept to define randomness by von Mises.[7] However, traders have a large number of pervasive dependencies. Dependencies can be shared information, herd mentality or shared trading practices. There are also some traders with very large budgets, so the conditions commonly needed to apply the law of large numbers do not apply and it is not inevitable that the market is indeed a Brownian motion. In fact one can show that even though the market overall looks very much like a Brownian motion it has too many events that would be considered very rare in this model (crashes, run-ups, events correlated in time) to have plausibly been generated by such a model.

Exploiting Inefficiency

A basic rule of thumb is: without a good reason to believe contrary you are not too far off assuming the market is efficient. So we decided to model the morning market as being nearly memoryless. That is we modeled it as if future prices depend only on the most recent price and not on the detailed history of prices. We will, however, condition the model on the bias introduced by the presence initial trade trigger.

The most basic memoryless model is the Markov Chain. In this model the world has finite number of situations called states. For example we could say the stock price being near each a number of price differences from the previous day’s close is a state. We could take our states to be: , , , , , . If our strategy involved an end of day sale followed by a next-day buy-back then knowing which state we are in allows us to assign a value to buying back the stock while in that state. This would be the negative of the relative change in stock price (price decreases work for us) times the value of the stock sold the day before (minus trading costs). If we modeled round-trip trading costs as and assume our triggered trade purchased a total value of of Dell then we could map buying back in each possible state to a net dollar value of the round trip. For instance buying back in the state would represent a net-loss of . We actually want to make the states a bit more detailed by adding a notion of time. If we modeled time in 5-minute intervals and (for the sake of diagram clarity) assumed that we only move up or down one state-level the Markov that modeled the first 15 minutes of the market could be represented in a diagram as in Figure 3.

Figure 3: Markov Chain Model

Each circle represents a state and each arrow represents a transition from state to state. We would use historic market data to find for every stock in this situation the relative frequency each transition is taken. For instance we would measure in our historic data what fraction of the time a stock that is 5 minutes and in the state moves to the state at the 10 minute mark. These learned state transition probabilities can be made to depend on factors from the previous day close (% increase, volume, market-capitalization) . In the diagram we are going to assume all transitions are equally likely except for the arrows with square bases which we each take to be twice as likely as each regular arrow leaving the same state. The success of our strategy depends on finding situations where our model predicts these sort of advantageous asymmetric conditions. Without these asymmetries (greater net propensity for price decrease than for price increase) we would be in a gambling situation where no strategy could possibly have net-positive value.

The diagram also encodes another assumption of the problem- we have a deadline for buying back the stock. In this case the diagram indicates a forced buy-back at time +15 minutes if a buy-back has not been made before that time. In reality many more levels and many more time intervals are modeled. Also note we have made the top row (representing maximal loss) absorbing. This is introducing a deliberate pessimistic flaw into the model (or equivalently adds a stop-loss condition to the strategy). We do not want the maximal loss states to have a reflected barrier (like the maximal profit states do) as this would make the model overly optimistic. Instead we force the model to be pessimistic and chose enough levels so that the maximum loss bound is not often achieved and therefor does not have large effect on the model.

Figure 4: Valuing Interior States

What we want to know is the net-value of being short (having sold) the stock the evening before. This is represented by the left-most circle which does not yet have a known value. The value of this state depends both on the transition odds of the states and on the trading strategy used to buy back the stock. There is, for example, no value in reaching the price-drop states if our strategy doesn’t take advantage and buy back while in these states. So the value of the states depends both on the uncertain future behavior of the market and of the currently unspecified buy-back strategy. The neat thing about this sort of diagram and treatment is that the forced-liquidation states at the end make it possible to simultaneously find the optimal trading strategy and assign values to all of the states. For example in the next diagram we see that the value of allowing the middle state at +10 minutes to ride (i.e. waiting instead of buying the stock back at this time) is equal to the properly weighted average of the ending states it connects to, in this case:
. The value of buying-back in this states is so the optimal strategy is to take our chances in the next time interval (see Figure 4).

We can repeat this sort of argument for each state in the second to last column and determine the net-value of each state under the optimal trading strategy. States whose optimal strategy is to stop (perform the buy-back immediately) are indicated by not having any outgoing arrows (see Figure 5).

Figure 5: Propagating the Valuation

The procedure moves from right to left using known states to fill in decisions and values for unknown states. In fact the calculation is so simple and orderly we can encode the entire filling-in procedure in a spreadsheet table:

.

This is in fact the same type dynamic programming[1] method used to value options under the binomial model.

The completed diagram is shown in Figure 6.

Figure 6: Complete Valuation

For our (made up) example the net-value of round trip trade is an expected value profit.

What remains is to choose a set of conditions to base a model estimates on. We then only trade situations that have an acceptable predicted risk and reward profile.

To build the state transition models we collect all the historic trade data and then segregate it into groups of data that match each possible trigger condition we wish to use to help bias our system. There is a trade-off: the more detailed the list of trigger conditions the more powerful biases we can detect (things are less smeared together) but we have less data available for each possible combination of conditions and lower reliability in modeling. To address this we advocate using non-parametric or kernel methods here to average data that nearly fits the conditions to get estimates that are both detailed and reliable.

For example our estimate is of the form:

A usable
can be gotten from the law of conditional probability (
), so we use
. Under empirical re-sampling each training example is treated as equally likely (more common situations are accounted by the fact they yield more examples in the training set) so we can use
.

For
we can just estimate the frequency of when we are in a near how often do we see a next-state such that
is approximately .

For both of these estimates is pays to blur things a bit during the estimation procedure replacing sums of the form:

with softer forms like:

A Second Model

One thing we one might want is to use a much more detailed model of time. One way to do this is just to add more time-states to the model. This can cause problems as we now have many more transition probabilities to estimate.⁴Suppose we wanted to switch our model from being indexed by time to being indexed by tick. Bid, Ask and Trade ticks can happen at any time and any rate so even with a trading deadline, so there is uncertainty in how many more ticks there are before the trade deadline. We can work at the tick level (without introducing too many states) by introducing a new model that has cycles in the arrow diagram (see Figure 7).

Figure 7: Recurrent Model (With Cycles)

The short vertical arrows represent the odds of moving from price-state to price-state in the same time column. The left to right dotted arrows represent the odds of being the tick that moves to the next time column. We can now estimate the transition odds from a great quantity of per-tick data giving us very reliable transition odds. We would like to fill in the values of all the states of this model (like we did in the earlier diagrams)- but the fill-in procedure will not work in the presence of cycles. States we need to fill in our given state do not yet have known values because they themselves depend on the state we are trying to value.

Linear Program Treatment

The standard way to deal with unknown quantities that simultaneously depend on each other is to introduce variables and write down a set of simultaneous inequalities.

If we introduce the variables ,
,
and
to represent all of the unknown values in our last diagram we can quickly write down many relations we know to be true for them.

For example for the set of variables through we know that each state is worth at lest as much as the value of stopping in that state. This can be written as:

Each state (except deadline and stop-loss states) is also worth at least the expected value of continuing one more step, which can be written as:

		escape
		escape
		escape
		escape

This can be re-written into matrix form where we have

and our vector of unknowns is

In matrix form we say . We are assuming we have estimates for all of the entries of and – so the only unknowns are the entries of . If these were equalities (instead of inequalities) we would call this a set of simultaneous equations and we could use linear algebra to solve for the unknown values. Because they are inequalities we will have to instead solve what is known as a linear program.[5] It turns out the optimal values for
are given by solving:

s.t.

This has an admittedly strange form (the objective condition
seems very arbitrary and one would at first think the likely form is
where is the vector probabilities of getting into each -state). There is also the issue that we merely wrote down inequalities that we knew would be true for the optimal solution to the stopping problem, but we have not guaranteed that there are not more conditions we have not thought of (i.e. these conditions are necessary, but we have not yet established that they are sufficient).

We show (in the appendix) that this is in fact the right procedure for solving for all of the -values. Each of these linear programs can be quickly solved using standard software. We can also see that the same type of procedure can then be applied to the -values (which depend only on -values, which are by this point known). In fact we can substitute back (using linear programs instead of filling-in) until we know the expected value (under the model) of the entire round-trip trade.

More on the Transition Probability Estimate

We can augment our state to carry more information that just the current ask-price relative to our previous night’s sale

If we are in of our Markov model we can modify
to be:
(to do this we build an estimated transition matrix for from only the trajectory of today’s stock and then evaluate how likely the trajectory the training example from the past is under this model, much smoothing/blurring is required to make this calculation usable). Even better: we can group training data and use Bayes’ law:

This allows us to group the training examples (on a few criteria, like less than a month old or not, trading volume, volatility …) and use a group of examples to build a model to evaluate today’s moves against (aggregated data to form model to check today’s single trajectory). As is traditional in Bayes estimates we ignore the denominator as it does not vary as a function of training group.

Conclusion

We have demonstrated some of the methods of using standard statistical and optimization techniques to automatically generate and back-test un-roll trades that turn properly conditioned technical trades into profitable round-trip trades. What we have presented is the technical machinery for building the second half of a profitable trade pair where the first half is some technical signal such as price or a market external trigger.

Bibliography

1

BELLMAN, R.
Dynamic Programming.
Dover Publications, 2003.

2

BREIMAN, L.
Stopping Rule Problems.
John Wiley & sons, 1964, ch. Applied Combinatorial Mathematics.

3

IOANNIDS, J. P. A.
Why most published research findings are false.
PLOS Medicine 2, 8 (Aug 2005), 0697-0701.

4

KEMENY, J. G., AND SNELL, J. L.
Finite Markov Chains.
Springer, 1960.

5

SCHRIJVER, A.
Theory of Linear and Integer Programming.
John Wiley & sons, 1986.

6

SHARPE, W., ALEXANDER, G. J., AND BAILLY, J. W.
Investments, 6 ed.
Prentice Hall, 1998.

7

VON MISES, R.
Probability, Statistics and Truth.
Dover Publications, 1981.

8

WASSERMAN, L.
All of Nonparametric Statistics.
Springer, 2006.

Why the Linear Program Solution is Correct

How do we know the linear program solves the original problem?

• Because there are a lot of formulas?
• Linear program looks kind-of right?
• Works on a few examples?

To actually prove correctness we need to derive and compare to some representations of the optimal solution. All of the inequalities we wrote must be true for the optimal solution- but we have no prior guarantee that these are the only conditions. Their could be additional conditions that we forgot to model.

Breiman[2] presented a clever argument technique that exploits the particularly nice structure of solutions of this problem. He noticed that solutions have both a lattice like structure (you can combine solutions by taking minimums) and an operator structure (applying the probability transition matrix and stopping rules to a solution yields a solution). It turns out this is too much well behaved structure for any non-trivial solution set to have and it lets us show that optimal solutions are essentially unique which in turn lets us show the linear program solution solves the actual trading problem.

Theorem 1   Assume that every state in the Markov chain has a path to a forced stopping state. Let be a maximal optimal set of stopping nodes and define the vector such that
. Let be an optimal feasible solution to the linear program:

		0

where is the identity matrix, is the matrix of transition odds of the Markov chain and is the vector of stopping values.

Then .

The theorem says if is an optimal solution for the original valuation problem (that we may or may not know how to calculate) and is an optimal feasible solution to the linear program (which is now written in a slightly different but equivalent form) then . So, as hoped, solving the linear program is equivalent to solving the original stopping problem. The extra condition of every state being able to eventual reach a forced stopping state is true in our formulation due to the trading deadline.

The proof gets a little involved but the essential ideas are as follows:

• Check an optimal stopping solution would obey the linear program inequalities (so they are necessary, still need to show they are sufficient).
• Show that the linear program solution even if it did differ from the optimal stopping solution can not be less than the optimal stopping solution in any coordinate (this is the lattice minimum step).
• Use the fact that every state has a path to a forced stopping state to show that the linear programing solution can not hide any excess value above best possible stopping value away from the rest of the system (this is the operator step).

Proof. [Proof of Theorem 1] The theory of linear programming duality says that there is a dual problem to our linear program and this dual is:
where

		0

The point of the dual is it is known that for all feasible we have
. And for optimal we have
.

Take as an optimal solution to the linear program and the dual.

One can check itself must obey all of the conditions of the linear program so duality theory tells us
.

Define a vector such that
. also obeys the primal linear program inequalities, so we know
. Now
so we have
. Each entry of is and
for all which can only mean that . This means entry by entry we have
.

Now define the vector function such that
. For the true solution we have . The linear program solution also has . Now if we suppose then there exists an such that is maximal and state points to at least one state such that
). For this particular we claimed

So either way we have for this particular :
. But we must have
because
,
for all and
for at least one . So
and we see is essentially a contraction on the segment between and . Since a contraction on a bounded interval can not have two distinct fixed points our supposition that is untenable and we know .

We are done- we have shown there is essentially only one optimal solution to the stopping problem (the only possible variation is rules that differ in what they do for states- such that
). We also should by now have some insight as to why we used a linear program like
s.t. : the linear program is solving for the minimum value at each state that does not dip below the expected value of neighboring states.

Footnotes

… Mount¹

http://www.mzlabs.com/

… feeds.²

To emphasize; by technical trades we mean trades based on market data (as opposed to fundamental analysis) we do not include popular culture uses of the term such as candlesticks, Eliot waves and so on.

… announcements.³

We are assuming that these triggers can be made automatic by using a labeled information service or natural language processing techniques.

… estimate.⁴

The explosion of states can be managed by adding some regularity conditions on how transition probability estimates are allowed to change over time. This serves to reduce the complexity or rank of the estimation problem and improves the generalization ability of the model.

Related posts:

1. New Paper
2. Betting Best-Of Series
3. What does the Market Think?

Nina and I just finished up our analysis of some of the statistical difficulties encountered by users of Google AdSense. It came out a bit long- but we found the right statistical reference to prove that there are real barriers to understanding in this market. The paper is most legible in PDF, but we also include an HTML version so the blog entry can be skimmed.

Comparing Apples and Oranges: Two Examples of the Limits of Statistical Inference, With an Application to Google Advertising Markets

John Mount¹, Nina Zumel²

Date: July 6, 2007

Overview

Bad experimental situations are often a source of great statistical puzzles. We are going to describe an example of this sort of situation using what one author observed while watching a few different companies using the Google AdSense and AdWords products.

The points we argue will be obvious to statisticians – in fact, they are actually elementary exercises. We will show that the measurements allowed in the Google AdSense markets are insufficient to allow accurate tracking of a large number of different revenue sources.

Our goal is to explain a well known limit on inference to a larger non-specialist audience. This is a bit of a challenge as most mathematical papers can only be read by people who could have written the paper themselves. By “non-specialist audience” we mean analytically minded people that may not have seen this sort of math before, or those who have seen the theory but are interested in seeing a complete application. We will include in this writeup the notes, intents, side-thoughts and calculations that mathematicians produce to understand even their own work but, as Gian-Carlo Rota wrote, we are compelled to delete for fear our presentation and understanding won’t appear as deep as everyone else’s.[4]

The counter-intuitive points that we wish to emphasize are:

• The difficulty of estimating the variance of individuals from a small number of aggregated measurements.
• The difficulty of estimating the averages of many groups from a small number of aggregated measurements.

These points will be motivated as they apply in the Google markets and we will try to examine their consequences in a simplified setting.

Contents

• Overview
• The Google Markets
  • Introduction
  • Information Limits
  • Channel Identifiers

• The Statistics
  • The Variance is Not Measurable
  • Trying to Undo a Mixture

• Other Solution Methods
• Conclusion
• Bibliography
• Appendix
  • Derivation That a Single Mean is Easy to Estimate
  • Fisher Information and the Cramer-Rao Inequality

The Google Markets

Introduction

Google both buys and sells a large number of textual advertisements through programs called Google AdSense and Google AdWords.[2] What is actually purchased and sold is “clicks.” Web sites that agree to display Google AdSense are paid when users click on these ads, and advertisers who place advertisements into Google AdWords pay Google when their advertisements are clicked on. The key item in these markets is the “search term” that the advertiser chooses to bid on advertising clicks for. “Search terms” are short phrases for which an advertiser is willing to pay, in order to get a visit from a web surfer who has performed a search on that phrase. For instance a company like Panasonic might consider clicks on the search term “rugged laptop” (and the attention of the underlying web surfer) to be worth $2 to them.

Because Google both buys and sells advertisements they are essentially making a market. There are some unique aspects to this market in that it is not the advertisements or even page-views that are being traded, but clicks. Both Google and its affiliates serve the advertisements for free and then exchange payment only when a web surfer clicks on an advertisement. A website can “resell” advertisements by simultaneously placing ads through AdWords, and serving ads through AdSense. When a user clicks into the website via an advertisement, this costs the web site money; if, however, the user is then shown a number of other advertisements, he or she may then click out on one of them of their own free will, recouping money or perhaps even making a profit for the site. There is significant uncertainty in attempting resale and arbitrage in these advertisement markets, as the user who must be behind all the clicks can just “evaporate” during an attempted resale. Direct reselling of clicks (such as redirecting a web surfer from one advertisement to another) would require a method called “automatic redirection” to move the surfer from one advertisement to a replacement advertisement. Automatic redirection is not allowed by Google’s terms of service.

An interesting issue is that each click on a given search term is a unique event with a unique cost. One click for “rugged laptop” may cost $1 and another may cost $0.50. The differing costs are determined by the advertiser’s bid, available placements for the key phrase, what other advertisers are bidding in the market, how many web surfers are available, and Google’s sorting of bids. The sorting of bids by Google depends on the rank of advertiser’s bid times an adjustment factor managed by Google. The hopeful assumption is that all of the potential viewers and clickers for the same search term are essentially exchangeable in that they all have a similar (unknown) cost and similar probabilities of later actions, such as buying something from a web site. The concept of exchangeability is what allows information collected on one set of unique events to inform predictions about new unique events (drawn from the same exchangeable population).

Whatever the details are, these large advertisement markets have given Google an income of $12 billion, $3.5 billion in profit and 70% year to year growth in 2006.[5] This scale of profit is due in part to the dominant position of Google in forming markets for on-line advertising.

The reasons for Google’s market domination are various and include the superior quality of the Google matching and bidding service, missteps by competitors and the network effects found in a good market – the situation whereby sellers attract buyers and buyers attract sellers. The cost of switching markets (implementation, information handling and staffing multiple relationships) are also significant factors.

In our opinion, Google’s profit margins are also helped by the limits on information available to most of the other market participants. In the next section, we will discuss some of the information limits or barriers to transparency in the Google market.

Information Limits

Google deals are typically set up as revenue sharing arrangements in which Google agrees to pay a negotiated portion of the revenues received by Google to the AdSense hosting web site. As noted above, advertisement click-through values vary from as little as $0.05 to over $40.0 per click. It is obvious that web site operators who receive a commission to serve advertisements on behalf of the Google AdSense program need detailed information about which advertisements are paying at what rate. This is necessary both to verify that Google is sharing the correct amount on valuable advertisements and to adjust and optimize the web site hosting the advertisements.

However, Google does not provide AdSense participants with a complete breakdown of revenues paid. There are a number of possible legitimate reasons for this. First, there is a concern that allowing web sites complete detailed reconciliation data would allow them to over-optimize or perform so-called “keyword arbitrage” where sites buy precisely the keywords they can profitably serve advertisements on instead of buying keywords for which the site actually has useful information or services. In addition, the quantity of data is very large, so there are some technical challenges in providing a detailed timely reconciliation. There can also be reasons favorable to Google.

Channel Identifiers

Google’s current solution to the conflicting informational needs defines the nature of the market and is in itself quite interesting. Google allows the AdSense customer a number of measurements called “channels.” The channels come with identifiers and the AdSense customer is allowed to attach a number of identifiers to every advertisement clicked-out on. Google in turn reports not the detailed revenue for every click-out but instead just the sum of revenue received on clicks-out containing each channel identifier.

For example: if a web site operator wanted to know the revenue from a particular search term (say “head cold”) they could attach a single channel identifier to all click-outs associated with “head cold” and to no other search term. Under this scheme, Google would then be reporting the revenue for the search term as a channel summary. This simple scheme uses up an entire channel-id for a single search term. This would not be a problem except that an AdSense partner is typically limited (by Google) to a few hundred channel identifiers and is often attempting to track tens of thousands of search terms (and other conditions such as traffic source and time of day). It is obvious to any statistician that these limited number of channels are not sufficient to eliminate many degrees of uncertainty in the revenue attribution problem.

Google does allow each click-out to have multiple channel identifiers attached to it. At first this seems promising – for instance one can easily come up with schemes where 30 channel ids would be sufficient to give over a billion unique search terms each a unique pattern of channel identifiers. However, Google does not report revenue for each pattern of channel identifiers; in this case they would only report the total for each of the 30 channels. Each channel total would be the sum of all revenue given for all clicks-out that included the given channel-id. Under this scheme we would have a lot of double counting in that any click-out with multiple channel identifiers attached is necessarily simultaneously contributing to multiple totals. Anyone familiar with statistics or linear algebra will quickly recognize that 30 channels can really only reliably measure about 30 facts about an ad campaign. There is provably no super clever scheme capable of decoding these confounded measurements into a larger number of reliable outcomes.

Let us go back to the points that we promised to discuss at the beginning of this paper:

• The difficulty of estimating the variance of individuals from a small number of aggregated measurements.

  In terms of Google AdSense, this means that we can tell the average (mean) value of a click in a given channel, but we cannot tell how widely the click values in the channel vary from this average value.

• The difficulty of estimating the averages of many groups from a small number of aggregated measurements.

  This means that if we assign multiple search terms into each of our available channels, we cannot separate out the values of each individual search term using only the aggregate channel measurements.

It is an interesting exercise to touch on the theory of why these facts are true.

The Statistics

One thing the last section should have made obvious is that even describing the problem is detailed and tedious. It may be better to work in analogy to avoid real-world details and non-essential complications. Let’s replace advertisement clicks-out with fruit, and channels with weighings of baskets.

Suppose we are dealing with apples and our business depends on knowing the typical weight of each fruit. We assume that all apples are exchangeable: they may each have a different weight (and value) but they all are coming from a single source. We further assume that we have a limited number of times that we are allowed to place our apples into a basket and weigh them on a scale.

The Variance is Not Measurable

The Mean

The first example, the happy one, is when we have a single basket filled with many different items of one type of fruit. For instance suppose we had a single basket with 5 apples in it and we were told the basket contents have a total weight of 1.3 pounds. The fact that we were given only a single measurement for the entire basket (instead of being allowed to weigh each apple independently) does not interfere in any way with accurately deducing that the average (or mean) of this type of apple weighs a little more than 1/4 pound. If we had apples in the basket, and we called the total weight of the contents of the basket , we could estimate the average or mean weight of individual apples as being . If we use to denote the (unknown universal) average weight of individual apples we would denote our estimate of this average as and we have just said that our estimate is
.

However, we are missing the opportunity to learn at least one important thing: how much does the weight of these apples vary? This could be an important fact needed to run our business (apples below a given weight may be unsellable, or other weight considerations may apply). We may need to know how inaccurate is it to use the mean or average weight of the apples in place of individual weights.

If we were allowed 5 basket weighings we could put one apple in each basket and directly see how much the typical variation in weight is for the type of apples we have. Let’s call this Experiment-A. Suppose in this case we find the 5 apples to weigh
respectively. This detailed set of measurements helps inform us on how this type of apple varies in weight.

One of the simplest methods to summarize information about variation is a statistical notion called “variance.” Variance is defined as the expected squared distance of an random individual from the population average. Variance is written as
where is a “random variable” denoting the weight of a single apple drawn uniformly and independently at random (from the unknown larger population) and the notation denotes “expectation.”
is the value that somebody who knew the value of would say is the average value of
over very many repetitions of drawing a single apple and recording its individual weight as . For example if all apples had the exact same weight the variance would be zero.

For the basket above,
is calculated as:

(the itself the average of the 5 apples weights). The interpretation is that for a similar apple with unknown weight we would expect
or for to not be too far outside the interval to (applying the common rule of thumb “2 standard deviations” which is 4 variances). As we see all of the original 5 apples fell in this interval.

Now the 5 apple weights we know are not actually all the possible apples in the world, they are merely the apples in our sample. There are some subtleties about using the variance found in a sample to estimate the variance of the total population, but for this discussion we will use the naive assumption that they are nearly the same. If we use the symbol to denote the (unknown) true variance of individual apple weights (so
) we can use it to express the fact is actually an excellent estimate of .

Specifically: if we were to repeat the experiment of taking a basket of randomly selected apples ( apples in the basket) over and over again, estimating the mean apple weight each time, then
– the expected square error between our estimate of the average apple weight and the true average apple weight – will go to zero as the sample-size is increased. In fact, we can show
, which means that our estimate of the mean gets more precise as is increased. This fact that large samples are very good estimates of unknown means is basic- but for completeness we include its derivation in the appendix.

Trying to Estimate the Variance

We introduced the variance of individual apples (denoted by ) as an unknown quantity that aided reasoning. We know that even with only one measurement of the total weight of all apples that is an estimate of the mean whose error goes to zero as the (the number of apples or the sample size) gets large.

However, the variance of individual apples is so useful that we would like to have an actual estimate ( ) of it. It would be very useful to know if is near zero (all apples have nearly identical weight) or if is large (apples vary wildly in weight). If we were allowed to weigh each apple as in Experiment-A (i.e. if we had an unlimited number of basket weighings or channels), we could estimate the variance by the calculations in the last section. If we were allowed only one measurement we would really have almost no information about the variance as we have only seen one aggregated measurement- so we have no idea how individual apple weights vary. The next question is: can we create a good estimate when we are allowed only two measurements but the sample size ( ) is allowed to grow?

Lets consider Experiment-B: If we have a total of apples ( in each basket) and is the total weight of the first basket and is the total weight of the second basket then some algebra would tell us that
is an unbiased estimate of (the variance in weight of individual apples)³.

It turns out, however, that is actually a bad estimate of the variance. That is, the expected distance of from the unknown true value of the variance (written
) does not shrink beyond a certain bound as the number of apples in each basket ( ) is increased. This “variance of variance estimate” result is in stark contrast to the nice behavior we just saw in estimating the average . With some additional assumptions and algebra (not shown here) we can show that for our estimate
we have
. There is a general reason this is happening, and we will discuss this in the next section.

Cramer-Rao: Why we can not estimate the variance of individual Apples

Of course showing one particular calculation fails is not the same as showing that the variance of individual apples can not be estimated from the two total weighings and . There could be other, better, estimates⁴.

There is a well known statistical law that states no unbiased estimator works well in this situation. The law is called the Cramer-Rao inequality.[1] The Cramer-Rao inequality is a tool for identifying situations where all unbiased estimators have large variance. The Cramer-Rao inequality is typically a calculation so we will add a few more (not necessarily realistic) assumptions to ease calculation. We assume apple weights are distributed normally with mean and variance .⁵

There is a quantity depending only on the experimental set up that reads off how difficult estimation is. By “depending only on the experimental set up” we mean that the quantity does not depend on any specific outcomes of , and does not depend on any specific estimation procedure or formula. This quantity is called “Fisher Information” and is denoted as .

The Cramer-Rao inequality[1] says for any unbiased estimator , the variance of is at least . Written in formulas the conclusion of the Cramer-Rao inequality is:

Since we have now assumed a model for the weight distribution of apples, we can derive (see appendix) the following:

Applying the Cramer-Rao inequality lets us immediately say:

This means that there is no unbiased estimation procedure for which can we expect the squared-error to shrink below
even as the number of items in each basket ( ) is increased. So not only does our proposed variance estimate fail to have the (expected) good behavior we saw when estimating the mean, but in fact no unbiased estimating scheme will work. In general we can show that the quality of the variance estimate is essentially a function of the number of measurements we are allowed⁶ – so any scheme using a constant number of measurements will fail.

Trying to Undo a Mixture

Suppose we are willing to give up on estimating the variance (a dangerous concession). We are still blinded by the limited number of channels if we attempt to estimate more than one individual mean.

In our analogy let’s introduce a second fruit (oranges) to the problem. Call an assignment of fruit to baskets a “channel design.” For example if we were allowed two basket measurements and wanted to know the mean weight of apples and the mean weight of oranges we could assign all apples to one basket and all oranges to the other. This “design” would give us very good estimates of both the mean weight of apples and the mean weight of oranges.

Let’s consider a simple situation where due to the limited number of channels we are attempting to measure something that was not considered in the original channel design. This is very likely because the number of simultaneous independent measurements is limited to the number of channels and it is very likely that one will have important questions that were not in any given experimental design. For example (going back to AdSense), suppose we had 26 channels and we used them all to group our search phrases by first letter of the English alphabet and we later wanted to break down older data by length of phrase.⁷ We would consider ourselves lucky if the first-letter design was even as good as random assignment of channel ids in measuring the effect of search term length.

To work this example we continue to ignore most of the details and suppose we really are trying to estimate the mean weight of apples and the mean weight of oranges at the same time. Due to the kind of bad luck described above we have data from an experiment that was not designed for this purpose. Let’s try the so-called easy case where we have a random experiment. For Experiment-C let’s suppose we have two baskets of fruit and each basket was filled with -items of fruit by repeating the process of flipping a fair coin and placing an apple if the coin came up heads and an orange if the coin came up tails. This admittedly silly process is simulating the situation where we are forced to use measurements that potentially could solve our problem- but were not designed to solve it.⁸ We can measure the total weight of the contents of each basket. So the information at our disposal this time is
(the number of apples in the first basket, the number of oranges in the first basket and the total weight of the first basket) and
(the number of apples in the second basket, the number of oranges in the second basket and the total weight of the second basket). What we want to estimate are and the unknown mean weights of the types of apples and types of oranges we are dealing with.

To simplify things a bit let’s treat the number of apples and oranges in each basket,
, as known constants set at “typical values” that we would expect from the coin flipping procedure. It turns out the following values of
are typical:

We call these values typical because in any experiment where the distribution of items in a collection is chosen by fair coin flips we expect to see a nearly even distribution (due to the fairness of the coin) but not too even (due to the randomness). In fact we really do expect any one of these values to be at least
away from most of the time and closer than
most of the time. So these are typical values, good but not too good.

We illustrate how to produce an unbiased (though in the end unfortunately unusable) estimate for and . The general theory says the estimate will be unreliable- but there is some value in seeing how an estimate is formed and having a specific estimate to experiment with. The fact that we know the count of each fruit in each basket, and each basket’s weight, gives us a simultaneous system of equations:

represents the average value of over imagined repeated experiments where apples and oranges are placed in a basket and weighed (similarly for
). The subscripts are indicating we are only considering experiments where the number of apples and oranges are known to be exactly
. We do not actually know
and
but we can use the specific basket total weighs we saw in our single experiment as stand-ins. In other words, may not equal
but is an unbiased estimator of
(this is a variation on the old “typical family with 2.5 children” joke). So we rewrite the previous system as estimates:

We can the rewrite this system into a “solved form”:

And this gives us the tempting estimates and

and are indeed unbiased estimates of and .

The problem is: even though these are unbiased estimates- they are not good estimates. With some calculation one can show that as (the number of pieces of fruit in each basket) increases that
and
do not approach zero. Our estimates have a certain built-in error bound that does not shrink even as the sample size is increased.

Cramer-Rao: Why we can’t separate Apples from Oranges

What is making estimation difficult has been the same in all experiments: most of what we want to measure is being obscured. As we mentioned earlier, in a typical case all of
will be relatively near a common value. Any estimation procedure is going to depend on separations among these values, which are unfortunately not that big. This is what makes estimation difficult.

Let us assume apple weights are distributed normally with mean and variance and orange weights are distributed normally with mean and variance .

Since we have now assumed a model for the weight distribution of apples and oranges we can derive (calculating as shown in [1]) the following:

What we are really interested in is the inverse of
, which (for or typical values of
) is:

The theory says that the diagonal entries of this matrix are essentially lower bounds on the squared error in the estimates of the apple and orange weights, respectively. The off-diagonal terms describe how an error in the estimate of the mean apple weight affects the estimate of the mean orange weight, and vice-versa. So what we would like is for all the entries of
to approach zero as increases. In our case, however, the entries of
all tend to the constant
as grows, meaning that the errors in the estimates are also bounded away from zero and stop improving as the sample size increases.

The above discussion assumes that the distribution of apples and oranges in each basket is the same (in this case, random and uniform). If there is some constructive bias in the process forming
, such as apples being a bit more likely in the first basket and oranges a bit more likely in the second basket, then the demonstrated estimate is good (with error decreasing as grows) and is actually useful. But the degree of utility of the estimate depends on how much useful bias we have- if there is not much useful bias then the errors shrink very slowly and we need a lot more data than one would first expect to get a good measurement. Finally, we would like to remind the reader that it is impossible for a channel design with a limited number of channels to simultaneously have an independent large useful bias on very many measurements.

As an example of the application of useful bias suppose that our coin has probability of coming up heads, and that the first basket is filled by placing an apple every time the coin is heads, and an orange every time the coin is tails. The second basket is filled the opposite way – apple for tails, orange for heads. Again, let’s treat the number of apples and oranges in each basket,
, as known constants set at “typical values” that we would expect from the coin flipping procedure.

(as long as
the terms are dominated by the bias and can be ignored).

If – the coin always comes up heads – then the first basket is only apples, and the second basket is only oranges, and obviously, we can find good estimates of and , by the arguments in Section 3.1.1. If
, then we are in the situation that we already discussed, with approximately equal numbers of apples and oranges in each basket. But suppose were some other value besides or
, say,
. In that case, the first basket would be primarily oranges, and the second one primarily apples, and we can show that

and all of the entries of
do go to zero as gets larger. This can be shown to be true in general, for any
. This means the Cramer-Rao bound does not prevent estimation. Another calculation (not shown here) confirms that our proposed estimate does indeed have shrinking error (as increases).

Other Solution Methods

We did not discuss solution methods that involve more data, such as repeated experiments, or significantly deeper knowledge, such as factor models. What we discussed were the limits of the basic modeling step, which itself would be a component of the more sophisticated solutions. Here however, we will briefly touch on other procedures that could be used to try to improve the situation discussed above.

Repeated measurements could be implemented by taking data over many days, reassigning the channel identifiers so that each search term participates in different combinations of channel identifiers over the course of the measurements. Essentially, this is setting up a much larger system of simultaneous equations, from which a larger number of variables can be estimated. There are mathematical procedures for this sort of iterative estimation (such as the famous Kalman filter), but the number of quantities a web site would wish to estimate is so much larger than the number of measurements available that the procedure will require many reconciliation rounds to converge. In addition, this model assumes that the values of the variables being measured do not change over time (or change very slowly). This is not an assumption that is necessarily true in the AdWords domain, due to seasonality and other effects.

A factor model is a model where one has researched a small number of causes or factors that explain the expected value of search phrases in a very simple manner. For example it would be nice if the value of a search phrase were the sum of a value determined by the first letter plus an independent value determined by the second letter. In such a case we would only need channels (to track the factors) and we would then be able to apply our model to many different search phrases. Factor models are a good solution, and are commonly used in other industries, such as finance, but one needs to invest in developing factors much better than the example factors we just mentioned.

Conclusion

The last section brings us to the point of this writeup. Having data from a limited number of channels is a fundamental limit on information in the Google click-out market. You can not get around it by mere calculation. You need other information sources or aggregation schemes which may or may not be available.

The points we have touched on are:

• You can not estimate the variance of individuals from a constant number of aggregated measurements.

  This is bad because this interferes with detailed estimates of risk.

• You can not always undo bad channel assignments by calculation after the fact.

  This is bad because this interferes with detailed assignments and management of value.

In a market information is money. To the extent you buy or sell in ignorance you leak money to any counter-parties that know the things that you do not. Even if there are no such informed counter-parties there are distinct disadvantages in not being able to un-bundle mixed measurements. This means it is difficult to un-bundle mixed sales. For example we may be making a profit on a combination purchase of advertisements and we are not able to quickly determine which advertisements in the combination are profitable and which are unprofitable.⁹

The capital markets (stocks, bonds, index funds, ) have evolved and progressed forward from initial disorganized arrangements to open outcry markets and then to detailed information environments. The demands and expectations of these modern markets include a number of features including:

• Complete reconciliation and publicly available detailed records of the past.
• Transparent “books” or listings of all current bids and bidders.

Not all of these are appropriate for a non-capital market and Google’s on-line advertising markets are just that: Google’s. It is interesting that before 2007 Yahoo/Overture offered a research interface that did expose the bidding book. It will be interesting to see how the on-line advertising markets evolve and if this feature survives in the newer “more like Google” Overture market.

The actual lesson we learned in watching others work with on-line advertising markets are the following. It is not necessary to be able to perform any of the calculations mentioned here to run a successful business. It is important, however, to have a statistician’s intuition as to what is risky, what can be estimated and what can not be estimated. The surprise to the first author that his initial intuition was wrong, even though he considers himself a mathematician. It wasn’t until we removed the non-essential details from the problem and found the appropriate statistical references that we was finally able to fully convince ourselves that these estimation problems are in fact difficult.¹⁰

Bibliography

1

COVER, T. M., AND THOMAS, J. A.
Elements of Information Theory.
John Wiley & sons, 1991.

2

GOOGLE.
Google advertising programs.
http://www.google.com/ads/.

3

METROPOLIS, N., AND ULAM, S.
The Monte Carlo method.
335-341.

4

ROTA, G.
Indiscrete Thoughts.
Birkhäuser, Boston, 1997.

5

YAHOO!
Google key statistics.
http://finance.yahoo.com/q/ks?s=GOOG.

Appendix

Derivation That a Single Mean is Easy to Estimate

To show
we introduce the symbols to denote the random variables representing the apples in our basket and work forward.

To calculate we will need to use some of the theory of the expectation notation . Simple facts about the notation are used to reduce complicated expressions into known quantities. For example if is a random variable and is a constant than
. If is a random variable that is independent of then
. And we have for any quantities ,
(even when they are not independent).¹¹

Starting our calculation:

			(1)
			(2)
			(3)
			(4)
			(5)
			(6)
			(7)
			(8)

Most of the lines of the derivation are just substitutions or uses of definition (for example the last substitution on line 8 is of
). A few of the lines use some cute facts about statistics. For example line 4
line 5 is using the fact that
, which under our independent drawing assumption is enough to show
when (hence all these terms can be ignored). The line 5
line 6 substitution uses the fact that each of the apples was drawn using an identical process, so we expect the same amount of error in each trial (and there are trials in total).

The conclusion of the derivation is that the expected squared error
is a factor of smaller than
. This means our estimate is getting better and better (closer to the true ) as we increase the sample size .

Fisher Information and the Cramer-Rao Inequality

Discussion

What is Fisher information? Is it like the other mathematical quantities that go by the name of information?

There are a lot of odd quantities related to information each with its own deep theoretical framework. For example there are Clausius entropy, Shannon information and Kolmogorov-Chaiten complexity. Each of these has useful applications, precise mathematics and deep meaning. They also have somewhat confused and incorrect pseudo-philosophical popularizations.

Fisher information is not really famous outside of statistics. Textbooks motivate it in different ways and often introduce an auxiliary function called “score” that quickly makes the calculations work out. The definition of“score” uses the fact that
to switch from likelihoods to relative likelihoods. The entries of the Fisher information matrix are terms of the form

where is our vector of parameters (set at their unknown true values that we are trying to estimate) , ranges over all possible measurements and
reads off the likelihood of observing the measurement given the parameter .

Fisher information is actually a simpler concept than the other forms of information. The entries in the Fisher information matrix are merely the expected values of the effect of each pair of parameters on the relative likelihood of different observations. In this case, it is showing how alterations in the unknown parameters would change the relative likelihood of different observed outcomes. It is then fairly clever (but not too surprising) that its inverse can then read off how changes in observed outcome influence estimates of the unknown parameters. The Cramer-Rao inequality is using Fisher information to describe properties of an inverse (recovering parameters from observed data) without needing to know the specific inversion process (how we performed the estimate).

Calculating Cramer-Rao on the Variance of Variance Estimate

When attempting to measure the variance of individual apples (Experiment-B) our data was two sums of random variables (each or representing a single apple):

can be any positive integers.

Under our assumption that the weight of apples is normally distributed with mean-weight and variance we can write down the odds-density for any pair of measurements as:

To apply the Cramer-Rao inequality we need the Fischer information of this distribution which is defined as:

The first step is to use the fact that

and write

where is the standard single variable normal density:

The first term is the 4th moment of the normal and it is known that:

It is also a standard fact about the normal density that

So we have

Finally we have the Fisher Information
We can then apply the Cramer-Rao inequality which says that
for any unbiased estimator (no matter how we choose and ) of (unbiased meaning
). The theory is telling us that the unknown parameter has such a sloppy contribution to the likelihood of our observations that it is in fact difficult to pin down the value from any one set of observations. In our case we have just shown that
, which means no estimation procedure that uses just a single instance of the total can reliably estimate the variance of individual apple weights.

Calculating Cramer-Rao Inequality on Multiple Mean Estimates

In Experiment-C we again have two baskets of fruit- but they contain apples and oranges in the proportions given by
. Our assumption that the individual fruit weights are normally distributed with means and common variance lets us us write the joint probability of the total measurements in terms of the normal-density ( ).

For our problem where the variables are the sums and we have two parameters (the two unknown means ) and a single per-fruit variance we will use the two dimensional normal density:

We concentrate on the variables and will abbreviate this density (leaving implicit the important parameters ) as
.

From this we can read off the difficulty in estimating individual apple weight:

The first step is using the fact that

The last step is using a number fundamental facts about the normal density:

		0

These facts allow us say that the so-called “cross terms” (like
) integrate to zero and the square terms read off the variance. One of the reasons to assume a common distribution (such as the normal) is that almost any complicated calculation involving such distributions (differentiating, integrating) can usually be reduced to looking up a few well know facts about the so-called “moments” of the distribution, as we have done here. Of, course picking a distribution that accurately models reality take precedent over picking one that eases calculation.

The other entries of the Fisher Information matrix can be read off as easily and we derive:

Substituting our “typical” values of
from Section 3.2 we have

At first things look good. The
entries are growing with so we might expect the entries of
to shrink as increases. However, the entries they are all nearly identical so the matrix is ill-conditioned and we see larger than expected entries in the inverse. In fact in this case we have:

and these entries are not tending to zero- establishing (by the Cramer-Rao inequality) the difficulty of estimation.

Cramer-Rao Inequality Holds in General

By inspecting our last series of arguments, we can actually say a bit more. The difficulty in estimation was not due to our specific assumed values of
, but rather to the fact that the coin-flipping process we described earlier will nearly always land us in about as bad a situation for large . We can see that the larger the differences
and
the better things are for estimation. The “strong law of large numbers” states that as increases we expect (with probability 1) to have
and
. This means that it would be very rare (for large ) to see differences in
larger than we saw in our “typical case.” This lets us conclude that if there is no constructive bias then for large estimation is almost always as difficult as the example we worked out.

Now if there were any constructive bias in the experiment (such as apples were a bit more likely in the first basket and oranges were a bit more likely in the second basket) then the entries of would be forced to zero and the explicit estimate we gave earlier would in fact have shrinking error as grew large. However only the fraction of the data we can attribute to the bias is really helping us (so if it was say a th bias only about th of the data is useful to us) and we would need a lot of data to experience lowered error (but at least the error would be falling). The point is that the evenly distributed portion of the data is essentially not useful for inference, and that is why it is so important to be inferring things that the experiment was designed to measure (and why the limit on channel identifiers is bad since it limits the number of things we can simultaneously design for).

Footnotes

… Mount¹

http://www.mzlabs.com/

… Zumel²

http://www.quimba.com/

… apples)³

“Unbiased” simply means that
which can also be written as
. This means our estimate of variance doesn’t tend to be more over than under (or more under than over).

… estimates⁴

As an aside, some of the value in proposing a specific estimate (because the theory says there is no good one) is that it allows one to investigate the failure of the estimate without resorting to the larger theory. For example in this day of friendly computer languages and ubiquitous computers one can easily empirically confirm (by setting up a simulation experiment as suggested by Metropolis and Ulam[3]). One can check that our estimate is unbiased (by averaging many applications of it) and that it is not good (by observing the substantial error on each individual application even when is enormous). There is no rule that one should not get an empirical feel (or even an empirical confirmation) of a mathematical statement (presentation of math is subject to errors) and in this day there are likely many more readers who could quickly confirm or disprove the claims of this section by simulation than there are readers who would be inclined to check many lines of tedious algebra for a subtle error.

….⁵

“Normal” is a statistical term for the distribution associated with the Bell curve. Many quantities in nature have a nearly normal distribution.

… allowed⁶

And perhaps surprisingly not a function of the sample size.

… phrase.⁷

These examples are deliberately trivial.

… it.⁸

This is one of the nasty differences between prospective studies where the experimental layout is tailored to expose the quantities of interest and retrospective studies where we hope to infer new quantities from experiments that have relevant (but not specifically organized) data.

… unprofitable.⁹

By “quickly determine” we mean determine from past data we already have. What we have shown is we often can not determine what we need to know from past data, but must return to the market with new experiments that cost both time and money.

… difficult.¹⁰

This initial optimism of ours is perhaps a side-effect of a “can do” attitude.

… independent).¹¹

It is funny in statistics that we spend so much time reminding ourselves that is not always equal to that we actually sometimes find it surprising that
is generally true.

Related posts:

1. Paper on stock trading
2. Betting Best-Of Series
3. A Demonstration of Data Mining