Win-Vector Blog » significance http://www.win-vector.com/blog The Applied Theorist's Point of View Thu, 29 Jul 2010 17:09:49 +0000 en hourly 1 http://wordpress.org/?v=3.0.1 Statistics to English Translation, Part 2b: Calculating Significance http://www.win-vector.com/blog/2009/12/statistics-to-english-translation-part-2b-calculating-significance/?utm_source=rss&utm_medium=rss&utm_campaign=statistics-to-english-translation-part-2b-calculating-significance http://www.win-vector.com/blog/2009/12/statistics-to-english-translation-part-2b-calculating-significance/#comments Mon, 14 Dec 2009 07:02:40 +0000 Nina Zumel http://www.win-vector.com/blog/?p=1281
  • Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
  • “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
  • Living in A Lognormal World
    • ]]>     In the previous installment of the Statistics to English Translation, we discussed the technical meaning of the term ”significant”. In this installment, we look at how significance is calculated. This article will be a little more technically detailed than the last one, but our primary goal is still to help you decipher statements about significance in research papers: statements like “
    $ (F(2, 864) = 6.6, p = 0.0014)$ ”.

    As in the last article, we will concentrate on situations where we want to test the difference of means. You should read that previous article first, so you are familiar with the terminology that we use in this one.

    A pdf version of this current article can be found here.

    How is Significance Determined?

    Generally speaking, we calculate significance by computing a test statistic from the data. If we assume a specific null hypothesis, then we know that this test statistic will be distributed in a certain way. We can then compute how likely it is to observe our value of the test statistic, if we assume that the null hypothesis is true.

    We’ll explain the use of a test statistic with our Sneetch example from the last installment.

    The t-test for Difference of Means

    Suppose that the test scores for both Star-Bellies and Plain-Bellies are normally distributed, with the means and standard deviations as given in the table below.

      $ n$ (number of subjects) $ m$ (mean score) $ s$ (standard error)
    Star-Bellies 50 78 7
    Plain-Bellies 40 74 8

    Remember from the previous installment that we can estimate the true population means $ \mu_1$ and $ \mu_2$ as normally distributed around the empirical population means $ m_1$ and $ m_2$ respectively, with variances
    $ \sigma^2/{n_1}$ and
    $ \sigma^2/{n_2}$ . This is shown in Figure 1. Informally speaking, there is no significant difference in the two populations if the shaded overlap area in Figure 1 is large.

    Figure 1: The estimates of the means for two populations
    Image overlap

    Calculating this area is somewhat involved. Instead, we calculate the t-statistic:

    $\displaystyle t = \frac{(m_2 - m_1)}{s_D}$ (1)



    where $ s_D$ is called the pooled variance of the two populations.

    $\displaystyle {s_D}^2 = \frac{n_1\cdot {s_1}^2 + n_2\cdot {s_2}^2}{n_1 + n_2 - 2} \cdot (1/n_1 + 1/n_2)$ (2)


    For our Sneetch example, $ s_D = 1.6$ , and $ t=2.499$ , or the negative of that, depending on which group is Group 1. There are
    $ 50 + 40 - 2 = 88$ degrees of freedom.

    If the null hypothesis is true, and the two populations are identical, then $ t$ is distributed according to Student’s distribution with
    $ N_1 + N_2 - 2$ degrees of freedom    . Student’s distribution is sort of a “stretched out” bell curve; as the degrees of freedom increase (
    $ N_1 + N_2 \rightarrow \infty$ ), Student’s distribution approaches the standard normal distribution, $ N(0, 1)$ 1.

    In other words, if the null hypothesis is true, $ t$ should be near zero. The probability of seeing a $ t$ of a certain magnitude or greater under the null hypothesis is given by the area under the tails of Student’s distribution:

    Figure 2: The area under the tails for a given $ t$
    Image twotailedtest

    This area is $ p$ . For the Sneetch example, $ p = 0.014$ .

    The further out on the tails $ t$ is, the stronger the evidence that you should reject the null hypothesis. If you know for some reason that the mean of one population will be greater than or equal to the other, than you can use the one-tailed test:

    Figure 3: The one-tailed test for a given $ t$
    Image onetailedtest

    This test halves the p-value as compared to the two-tailed test, making a given $ t$ value twice as significant. When in doubt about which to use, the two-tailed test is more conservative against false positives2.

    In discussions of t-tests, you will often see statements of the form:

    The t-test meets the hypothesis that two means are equal if

    $\displaystyle \vert t\vert > t_{\alpha/2, \nu}$    


    for a two-tailed test, or

    $\displaystyle t > t_{\alpha, \nu}$    


    for a (right-sided) one-tailed test.

    The quantities on the right hand side of the two equations above are called the critical values for a given significance level $ \alpha$ (usually,
    $ \alpha = 0.05$ ) and $ \nu$ degrees of freedom. The critical values are the values for which the area of the right hand tail is equal to $ \alpha$ .

    Figure 4: Critical value for a one-tailed test. Reject the null hypothesis if
    $ t > t_{crit}$
    Image onetailedcritval

    For a two-tailed test, you must halve the area under a single tail.

    Figure 5: Critical value for a two-tailed test. Reject the null hypothesis if
    $ \vert t\vert > t_{crit}$
    Image twotailedcritval

    This convention dates back to the time when computational resources were scarce, and researchers had to use pre-computed tables of critical values, rather than calculating $ p$ directly. Today, general statistical packages such as R or Matlab can compute the CDFs of any number of standard distributions; once you can compute the CDF, directly computing $ p$ (the area under the tails) is straightforward. Despite this, many tutorials of the t-test (and of the F-test, and other significance tests) still adhere to the convention of comparing test statistics to critical values. This tends to needlessly ritualize the whole process, and make it seem more complicated and mysterious than it actually is, at least in my opinion.

    David Freedman was very much against the continued practice of using critical values, rather than reporting the actual p-value. The last chapter of Freedman, Pisani and Purves [FPP07] is worth reading for its discussion of this, and other potential pitfalls of significance tests.

    Some standard packages for evaluating t-tests, F-tests, or the ANOVA also present analysis results in terms of critical values. Most of them do usually print the actual p value as well, along with the value of the test statistic and the degrees of freedom. Most researchers rightfully report the test statistics along with the actual significance levels: “we conclude that there is a significant difference in mathematical performance (t(88) = 2.499, p = 0.014)… .” Here, 88 gives the degrees of freedom, $ t(88)$ is the value of the t-statistic, and $ p$ is of course the p-value.

    Similar comments apply to the F-test, discussed in more detail below.

    Assumptions

    Strictly speaking, the t-test is only valid for normally distributed data where both populations have equal variance. However, the test is fairly robust to non-normal data [Box53]. You can verify that the sample variances are “equal enough” – that is, they could plausibly both be sampled observations from populations with the same variance, by using the F-test. The F-statistic

    $\displaystyle F = {s_1}^2/{s_2}^2 $

    is distributed according to the F distribution with
    $ (n_1 - 1,n_2 - 1)$ degrees of freedom

    Figure 6: The F distribution
    Image Ftest

    In practice, the larger variance is usually put in the numerator, so $ F > 1$ . The test should still be two-tailed, so you should double the area under the right-hand tail3. In this situation, you want to check if you ƒshould accept the null hypothesis (that
    $ F \approx 1$ ) at a given significance level. If so, then you can go ahead and apply the t-test.

    There is a variation of the t-tests for distributions of unequal variance, called Welch’s t-test [Wikc]. In this case, you are only checking if the means are equal, not that the distributions are the same.

    The F-test for Analysis of Variance (ANOVA)

    ANOVA is an extension of the difference of means test above to the casae of more than two populations. The null hypothesis in this case is that all the sample means are equal – or more strictly, that all the treatment groups are drawn from the same population.

    The simplest version of the ANOVA is the one-way ANOVA, where there are $ k$ treatment groups (populations) with $ n_i$ subjects (or repetitions, or replications) each, for a total of $ N$ subjects. Each population corresponds to a different single factor (a treatment or a condition: for example, a type of medicine, or a Star-Bellied Sneetch vs. a Plain-Bellied Sneetch vs. a Grinch). Two- or three- way ANOVAs correspond to varying two or three different factors combinatorially. For example, we could do a two-way ANOVA of Sneetch math performance by considering both the belly type and the gender of the Sneetchs.

    Figure 7: Table for a Two-way ANOVA of Sneetch math performance
    Image twowayANOVA

    We will only discuss one-way ANOVA in this article, since that covers all the relevant ideas about calculating significance.

    For a one-way ANOVA, we have the population means $ m_i$ and variances $ {s_i}^2$ . We can also calculate the overall mean $ m_0$ , over the entire aggregate population.

    The between-groups mean sum of squares, which is an estimate of the between-groups variance, is given by

    $\displaystyle {s_B}^2 = \frac{1}{k-1} \sum_i {n_i \cdot (m_i - m_0)^2}$ (3)


    $ {s_B}^2$ is sometimes designated $ MS_B$ It is a measure of how the population means vary with respect to the grand mean.

    The within-group mean sum of squares is an estimate of the within-group variance:

    $\displaystyle {s_W}^2 = \frac{1}{N-k} \sum_i^k \sum_j^{n_i} {x_{ij} - m_i}^2$ (4)


    $ {s_W}^2$ is sometimes designated $ MS_W$ . It is a measure of the “average population variance”.

    Figure 8: Within-group and between-group variance
    Image sigmas

    If the null hypothesis is true, then

    $\displaystyle F = {s_B}^2/{s_W}^2 $

    is distributed according to the F distribution wiht
    $ (k-1, n-k)$ degrees of freedom.

    Figure 9: p-value for the one-tailed F-test
    Image Ftest

    That is, under the null hypothesis, the within-group and between-group variances should be about equal:
    $ F \approx 1$ . If $ F < 1$ , then some of the treatment groups overlap other groups substantially, so practically speaking, one might as well accept the null hypothesis. Hence, a one-sided F test is good enough. As with the t-test, research papers usually give the value of the F statistic, the degrees of freedom, and the p-value: “
    $ (F(2, 864) = 6.6, p = 0.0014)$ ”. In this example, the test statistic value is 6.6, and it was evaluated against the F distribution with (2, 864) degrees of freedom, which means that
    $ k = 3, n = 866$ . The p-value is 0.0014.

    Assumptions

    Like the t-test, ANOVA assumes that the data is normally distributed with equal variances. According to Box [Box53], ANOVA is fairly robust to unequal variances when the population sizes are about the same, but you might want to check anyway. If all the populations are the same size (all the $ n_i$ are the same), the easiest way to check for equality of variances is an F-test of the statistic
    $ F = {s_{max}}^2/{s_{min}}^2$ with $ n-1$ degrees of freedom[Sac84]. In other cases, you can use Bartlett’s Test [Wika] or Levene’s Test [Wikb]. Bartlett’s test uses a test statistic that is distributed as the $ \chi^2$ distribution, and Levene’s test uses one that is distributed as the F distribution. Levene’s test does not assume normally distributed data.

    If the data are not normally distributed, or have unequal variance, often they can be transformed to a form that is closer to obeying the assumptions of ANOVA. The following table of transformations is based on [Sac84, p. 517], and other sources [Hor].

    Figure 10: Table of Transformations
    \begin{figure}\begin{center} \begin{tabular}{\vert p{2.5in}\vert p{3.5in}\vert} ... ...} \ $\sigma \approx k\mu$\ & \ \hline \end{tabular} \end{center}\end{figure}

    Jim Deacon from the University of Edinburgh lists some suggestions as well [Dea]. He also reminds us that running ANOVA on the transformed data will identify significant differences in the transformed data. This is not the same as saying there are significant differences in the original data!

    Once the Null Hypothesis is Rejected

    If you are able to reject the ANOVA null hypothesis, you will usually want to know which population means are significantly different from the rest. Often, in fact, you are primarily interested in which population had the highest mean. For example, if you are comparing the efficacy of a new medicine A against existing medicines B and C, you are probably not too concerned about whether B and C perform significantly differently from each other, only about whether A is significantly better than both.

    If all you care about is whether the highest mean is significantly higher than the others, you can simply test where the statistic

    $\displaystyle (m_1 - m_2)/({s_W}^2 \frac{n_1 + n_2}{n_1\cdot n_2}) $

    falls on the Student-t distribution with $ n-k$ degrees of freedom. Here, $ {s_W}^2$ is the within-group variance, as calculated in Equation 4, $ m_1$ and $ m_2$ are the highest and second highest population means, $ n$ is the total number of samples (
    $ n = \sum{n_i}$ ), and $ k$ is the number of treatment groups.

    This test is usually written

    $\displaystyle m_1 - m_2 > t_{(n-k, \alpha/2)} \cdot \sqrt{{s_W}^2 \cdot \frac{n_1 + n_2}{n_1\cdot n_2}} = LSD_{(1,2)} $

    where
    $ t_{(n-k, \alpha/2)}$ is the (two-sided) critical value for significance level $ \alpha$ and $ n-k$ is the number of degrees of freedom to use. This quantity is called the least significant difference (LSD) between the highest and second highest means, and the test is usually called the LSD test.

    If you want to test all the population differences $ m_i - m_j$ for significance, (or test the highest value against all of the others explicitly) then you need to take some care with the LSD test. Remember that a significance level of $ \alpha$ means that with probability $ \alpha$ you will make a false positive error. To test all possible population differences is $ K$ = ($ k$ choose $ 2$ ) comparisons, or $ K = k-1$ comparisons, if you sort all the means in descending order and compare adjacent ones. Testing the highest mean against all the lower values is also $ K = k-1$ comparisons. This means you have a
    $ K \cdot \alpha$ probability of making a false positive error. So if you want the overall significance level to be $ \alpha$ , each individual comparison should use a stricter significance threshold
    $ p \leq \alpha/K$ .

    A preferred way to compare multiple means for significance (once the ANOVA null hypothesis has been rejected) is to use a multiple range test [Dea] or Tukey’s method [oST06], rather than the LSD test. Tukey’s method tests all pairwise comparison simultaneously, and the multiple range test starts with the broadest range (the highest and the lowest means), and works its way in until significance is lost.

    Conclusion

    We’ve skimmed over many complications in this discussion. Hopefully, though, what we have gone over is enough to demystify much of the statistical discussion in research papers. Perhaps, it will demystify the output of standard ANOVA and t-test packages for you, as well.

    Chong-ho Yu’s site [hY] gives a brief discussion of some of the issues that I’ve skimmed over. It also lists a few common non-parametric tests. These are tests that do not make assumptions about how the data is distributed, and so they may be more appropriate for data that is very non-normal, or for discrete data. They tend to have less power than parametric tests (that is, they have a lower true positive rate); so if the data is at all normal-like, parametric tests are preferred.

    Significance tests are used in other applications beyond testing the difference in means or variances. They are used for testing whether events follow an expected distribution, for testing if there is a correlation between two variables, and for evaluating the coefficients of a regression analysis. We hope to cover some of these applications in future installments of this series.

    Bibliography

    Box53
    G.E.P. Box, Non-normality and tests on variances, Biometrika 40 (1953), no. 3/4, 318-335.
    Dea
    Jim Deacon, A multiple range test for comparing means in an analysis of variance, http://www.biology.ed.ac.uk/research/groups/jdeacon/statistics/tress7.html.
    FPP07
    David Freedman, Robert Pisani, and Roger Purves, Statistics, 4th ed., W. W. Norton & Company, New York, 2007.
    Hor
    Rich Horsley, Transformations, http://www.ndsu.nodak.edu/ndsu/horsley/Transfrm.pdf, Class notes, Plant Sciences 724, North Dakota State University.
    hY
    Chong ho Yu, Parametric tests, http://www.creative-wisdom.com/teaching/WBI/parametric_test.shtml.
    oST06
    National Institute of Standards and Technology, Tukey’s method, NIST/SEMATECH e-Handbook of Statistical Methods, 2006, http://itl.nist.gov/div898/handbook/prc/section4/prc471.htm.
    Sac84
    Lothar Sachs, Applied statistics: A handbook of techniques, 2nd ed., Springer-Verlag, New York, 1984.
    Wika
    Wikipedia, Bartlett’s test, http://en.wikipedia.org/wiki/Bartlett's_test.
    Wikb
    —–, Levene’s test, http://en.wikipedia.org/wiki/Levene's_test.
    Wikc
    —–, Welch’s t test, http://en.wikipedia.org/wiki/Welch's_t_test.

    Footnotes

    …1
    Remember from the last installment that when you are estimating the mean of a distribution with unknown mean $ \mu$ and unknown variance $ \sigma^2$ , the 95% confidence interval around your estimate is
    $ m \pm 2\cdot \sigma/\sqrt{n}$ . Intuitively speaking, Student’s distribution is what you get if you calculate confidence intervals using the estimated variance $ s$ instead of the true but unknown variance $ \sigma$ . The distribution is stretched out compared to the normal distribution to reflect this increased uncertainty.
    … positives2
    In his textbook Statistics, Freedman tells an anecdote about a study that was published in the Journal of the AMA, claiming to demonstrate that cholesterol causes heart attacks. The treatment group that took a cholesterol reducing drug had “significantly fewer” heart attacks than the control group (
    $ p \approx 0.035$ ). A closer reading revealed that the researchers used a one-tailed test, which is equivalent to assuming that the treatment group was going to have fewer heart attacks. What if the drug had increased the risk of heart attack? The proper two-tailed significance of their results would have been
    $ p \approx 0.07$ , which is higher than JAMA‘s strict significance threshold of 0.05. [FPP07, p. 550]
    … tail3
    The area to the right of $ F$ with $ (a,b)$ degrees of freedom is equal to the area to the left of $ 1/F$ , with $ (b,a)$ degrees of freedom.

    Related posts:

    1. Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’
    2. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
    3. Living in A Lognormal World

    ]]>     http://www.win-vector.com/blog/2009/12/statistics-to-english-translation-part-2b-calculating-significance/feed/ 0     Statistics to English Translation, Part 2a: ’Significant’ Doesn’t Always Mean ’Important’ http://www.win-vector.com/blog/2009/12/statistics-to-english-translation-part-2a-%e2%80%99significant%e2%80%99-doesn%e2%80%99t-always-mean-%e2%80%99important%e2%80%99/?utm_source=rss&utm_medium=rss&utm_campaign=statistics-to-english-translation-part-2a-%25e2%2580%2599significant%25e2%2580%2599-doesn%25e2%2580%2599t-always-mean-%25e2%2580%2599important%25e2%2580%2599 http://www.win-vector.com/blog/2009/12/statistics-to-english-translation-part-2a-%e2%80%99significant%e2%80%99-doesn%e2%80%99t-always-mean-%e2%80%99important%e2%80%99/#comments Fri, 04 Dec 2009 20:39:20 +0000 Nina Zumel http://www.win-vector.com/blog/?p=1186
  • Statistics to English Translation, Part 2b: Calculating Significance
  • “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
  • R annoyances
    • ]]>     In this installment of our ongoing Statistics to English Translation series1, we will look at the technical meaning of the term ”significant”. As you might expect, what it means in statistics is not exactly what it means in everyday language.

    As always, a pdf version of this article is available as well.

    Does too much salt cause high blood pressure, or doesn’t it? That debate has raged for decades, with a slew of studies finding “yes” and a slew of others finding “no.” Two new studies out today in the journal Hypertension tip the scales in favor of reducing sodium – particularly for those 1 in 4 Americans who have high blood pressure. One study found that reducing salt intake from 9,700 milligrams a day to 6,500 milligrams decreased blood pressure significantly in blacks, Asians, and whites who had untreated mild hypertension. Another study found that switching to a lower-salt diet helped lower blood pressure in folks with treatment-resistant hypertension.
    - “10 salt shockers that could make hypertension worse,” U.S. News & World Report [Kot09]

    “Great!” you think. “Who needs to spend money on high-blood pressure meds? I can just cut down my salt!” Well, maybe so, maybe not. To come to that conclusion, you need more information than you were given in that paragraph. What was the “significant” decrease in blood pressure? What was the “before” and the “after”? Does “significant” mean important, or useful? And why has there been so much controversy over this?

    Let’s discuss the important points with an example.

    Image sneetches

    Suppose that we wanted to test for a difference in intelligence between two groups, say Star-Bellied Sneetches and Plain-Bellied Sneetches2. We take a group of 50 Star-Bellies and a group of 40 Plain-Bellies, and give them both a series of tests designed to measure their mathematical, linguistic, and problem-solving abilities. After evaluating the data, we conclude that there is “a significant difference in mathematical performance (t(88) = 2.499, p = 0.014) between the two groups”. The mean mathematics score of the Star-Bellies is 78, with a standard deviation of 7, and the mean mathematics score of the Plain-Bellies is 74, with a standard deviation of 8, for a difference of 4 points3.

    Should we interpret this result to mean that Star-Bellied Sneetches are better than Plain-Bellied ones at math? It depends.

    How Hypothesis Tests Work

    The Sneetch example above and the blood-pressure study cited earlier are both examples of hypothesis tests. In hypothesis testing, researchers set their proposed hypothesis (that there is an effect or a relationship) against the null hypothesis that there is no effect or relationship. In this article, we consider proposed relationships of the form

    The mean value of X measured for group A is different from the mean value of X measured for group B.

    In this case, the null hypothesis is

    The mean value of X is the same for groups A and B, and any difference observed in the data is only by observational chance.

    In fact, we are actually testing the stricter null hypothesis:

    The distribution of X is the same for groups A and B, and any difference observed is only by observational chance.

    A and B are sometimes called treatment groups; this terminology comes from the original applications of hypothesis testing procedures, in agriculture and medicine. In the blood pressure study above, the treatment is daily salt intake. One group ingests about 9,700 milligrams of sodium a day, the other group about 6,500 milligrams a day. The question of interest is: does the difference in sodium intake make a difference in the average blood pressure of the two groups? The null hypothesis is “No.”

    Significance

    We call an observed difference significant – meaning that a difference as large as we observed is probably not by chance – if the the value $ 1-p$ is “high enough.” In the Sneetch example, $ p = 0.014$ is the significance level of the result. To interpret the p-value, suppose the null hypothesis is true: there is truly no difference between Star-Bellied math scores and Plain-Bellied math scores. If this is so, then there is only a 0.014 (1.4%) chance that the difference in the average scores of the two groups will be 4 points or larger. In other words, if the null hypothesis is true, and we administer this same test to different groups of 50 Star-Bellies and 40 Plain-Bellies a hundred times, then the difference in scores will be 4 points or more only about once or twice.

    We interpret the fact that we have seen a difference that should be rare to be evidence that the null hypothesis isn’t true. So we reject the null hypothesis and say that there is a “significant difference” in the performance of the two groups. Alternatively, we could say that Star-Bellied Sneetches performed “significantly better” than Plain-Bellied Sneetches on the math test.

    Effect Size

    Four points (or about a 5% difference) is the effect size of the comparison. The effect size represents what might be called the “practical significance” of the result. In general, the larger the effect size, the better. In this example, Star-Bellies might truly outperform Plain-Bellies by about four points on average, but if we were to examine the relationship between math scores and real-life math performance (say, how well college-attending Sneetches do in their math and science courses), we might discover that it takes a test score difference of ten points or more to reliably predict which Sneetches will do better. In that case, a four point average difference would not be a practical difference.

    Evaluating a Result

    When evaluating a result, you should look both for its significance and its effect size. In practice, researchers usually consider a finding to be significant if
    $ p \leq 0.05$ . This is actually a pretty large $ p$ ; it means even if the null hypothesis is true, you would still observe a difference as large as the one that you observed about five times out of every one hundred trials. In fact, Sachs noted that
    $ p < 0.0027$ used to be the commonly used threshold for significance ([Sac84, p. 114]).

    Sometimes results are reported using an asterisk convention: (*) means
    $ p \leq 0.05$ , (**) means
    $ p \leq 0.01$ , and (***) means
    $ p \leq 0.001$ . Hopefully, the actual significance level is reported (it isn’t always), as well as the actual effect size (it isn’t always).

    Image cup_of_coffee

    The effect size in medical studies is often reported in the popular press with statements like “those who abstained from coffee had triple the risk of contracting colon cancer compared to those who drank three or more cups a day.” Does that mean that all confirmed Lapsang Souchong drinkers and the uncaffeinated should run out and learn to embrace Starbucks? Well, no. First of all, ask yourself: what is the baseline risk of colon cancer? If abstaining from coffee triples the risk from 0.01% to 0.03%, well, it probably isn’t worth worrying about. On the other hand, if the risk triples from 5% to 15%, perhaps that is a reason to take up espressos. You should also see who were the subjects of the study, and how similar they are to you. Suppose the study was done on Caucasian males in the U.S., ages 55-65, with no family history of colon cancer. If you are a young white American male, it’s possible that this study says something about your future health. If you are female or non-Caucasian or not living in the U.S., the finding may or may not be relevant to you. It depends on the mechanism that drives the relationship, and whether or not it applies to you as well as to the subjects of the study.

    “Significant” is not the same as “Important”

    With a large sample, even a small difference can be “statistically significant”… . This doesn’t necessarily make it important. Conversely, an important difference may not be statistically significant if the sample size is too small.
    - Freedman, Pisani and Purves, Statistics [FPP07, p. 550]

    The ability of a study to detect a significant difference depends almost entirely on its size. When a researcher designs a study, she has to decide how much risk of error – and what type of error – she is willing to tolerate.

    How big a risk [of inventing a difference] between two indistinguishable treatments are we willing to put up with? This risk is known as the significance level $ \alpha$ . [Sac84, p. 214]

    $ \alpha$ is the probability of rejecting a null hypothesis that should be accepted. This is a Type I error (a false positive). $ \alpha$ enters the design of the study as the threshold for p-values that the researcher will accept as significant.

    How big a risk do we allow of missing a substantial difference between two treatments? … This risk is called $ \beta$ . [Sac84, p. 214]

    $ \beta$ is the probability of accepting a null hypothesis that should have been rejected. This is a Type II error (a false negative). The quantity $ 1-\beta$ is known as the power of the test: the probability that the test will correctly reject the null hypothesis when the alternative hypothesis is true.

    How small a difference should still be recognized as significant? This difference is called $ \delta$ . [Sac84, p. 214]

    $ \delta$ is the minimum effect size that we are willing to consider “practically significant.”

    It is important to consider all three of $ \alpha$ , $ \beta$ , and $ \delta$ when determining an appropriate sample size for a trial. The power of a test and the significance of a result both increase as the sample size $ n$ increases. So if $ \delta$ is not specified, any difference can appear significant, with a large enough $ n$ , even if the difference is really by chance.

    The Central Limit Theorem

    To see why the above statement is true, we need a few more facts about estimating the mean. Suppose we have a random variable $ X$ that is normally (or nearly normally) distributed, with a true mean $ \mu $ and (unknown) variance $ \sigma^2$ . You want to estimate $ \mu $ by drawing $ n$ samples; the sample mean $ \bar{x}$ gives you an estimate of $ \mu $ . According to the Central Limit Theorem, if you were to repeat this experiment over and over again, you would see that the estimated $ \bar{x}$ has a normal distribution, with mean $ \mu $ and variance
    $ \sigma^2/n$ . So $ \bar{x}$ is a good estimate of $ \mu $ , one that improves with a larger sample size $ n$ .

    Another fact about normal distributions is that a little over 95% of the probability mass is within $ \pm 2$ standard deviations of the mean. So, for a single experiment, we can reason that the true mean $ \mu $ is in the interval
    $ \bar{x} \pm 2 \sigma/\sqrt{n}$ with 95% probability4.

    Figure 1: Confidence bounds on the estimate of $ \mu $ for different values of $ n$
    Image fig1

    So, as $ n$ gets larger, we zoom in on $ \mu $ 5.

    Now, back to the problem of checking for the difference of means. We’ll take $ n$ samples from population $ A$ and $ n$ from population $ B$ . Let’s assume for now that the variances are equal.

    Figure 2: Confidence bounds overlap; means may not be truly different
    Image fig2

    With 95% probability,
    $ \mu_A \in \bar{x}_A \pm 2\sigma/\sqrt{n}$ , and
    $ \mu_B \in \bar{x}_B \pm 2\sigma/\sqrt{n}$ . If
    $ \vert\bar{x}_A - \bar{x}_B\vert$ is small compared to
    $ 4 \sigma/\sqrt{n}$ , then the two confidence intervals overlap substantially, and we cannot reject the null hypothesis that
    $ \mu_A = \mu_B$ .

    If, on the other hand,
    $ \vert\bar{x}_A - \bar{x}_B\vert$ is wide compared to
    $ 4 \sigma/\sqrt{n}$ :

    Figure 3: Confidence bounds don’t overlap; means are significantly different
    Image fig3

    then the confidence intervals are well separated, and we can reject the null hypothesis.

    So $ \delta$ , the minimum significant distance – the “resolution” of the experiment – is about the distance when the two confidence intervals touch:
    $ 4 \sigma/\sqrt{n}$ , if our desired significance level is 0.05.

    Figure 4: Minimum significant distance for a given sample size $ n$
    Image fig4

    If $ \delta$ is too large, the experiment may be unable to detect important differences because the confidence intervals overlap too soon. This means that the sample size was too small (the test didn’t have enough power), and the experiment should be repeated with a larger test population.

    If $ \delta$ is too small, then the experiment will potentially detect statistically significant differences that are, for all practical intents and purposes, meaningless. To go back to the Sneetch example, if the math exam has one hundred questions, then an effect size of two points would correspond to one group answering two additional questions correctly, on average. Practically speaking, that’s probably not a very big difference. But if we made the experiment big enough, about 250 Sneetches in each group, it would be a statistically significant difference, to the 0.05 level. In theory, we could even make a difference of less than one point statistically significant! That is why knowing the effect size of a significant result is important.

    “Significant” is not the same as “True”

    The power and significance level of a test play similar roles to the sensitivity and specificity of a diagnostic test. You’ll remember from Part 1 of this series6that sensitivity and specificity are properties of the test, not how the test performs in a given population. To know the practical accuracy of a screening test, you must know the underlying prevalence of the condition that it is screening for. If it is crucial that the screening not miss any positive cases, then the test will be designed to be highly sensitive, possibly at the cost of specificity. In that case, the test will tend to have a high false positive rate if the condition is relatively rare. And yet, this same screening test will have a lower overall false positive rate when used in a population where the condition is more prevalent.

    The same is true for hypothesis tests. The probability that a statistically significant result is actually true depends on the underlying probability that results “of that type” tend to be true in the domain of study. It also depends on whether the researcher was trying to minimize the chance of a false positive error, or a false negative error.

    You should also be careful interpreting the results of exploratory work, where the researchers have run a series of several different studies, but only highlight the “significant” ones. Running twenty experiments and having one of them return a significant result to the $ p=0.05$ level is actually not significant at all.

    John Ioannides discusses these points (and a few others) in his 2005 essay “Why Most Published Research Findings are False”[Ioa05]. The essay made a few waves at the time of its publication, and it is still available online. We recommend that you read it, along with the 2007 followup article by Moonesinghe, et.al [MKJ07]. Now that you’ve read the first two installments of the Statistics to English translation, both essays should be a breeze!

    Some Points to Remember

    • “Significant” is a statistical statement that an observed relationship is unlikely to be by chance. It is not an necessarily a statement about the magnitude or the importance (or the truth!) of the relationship.
    • Knowing the effect size of a significant result will help you decide if the relationship is “practically significant.”
    • With a large enough sample size, any difference in means can appear significant, even when it is by chance.

    You now have a general idea what a “statistically significant result” is. The next installment will go into a little more technical detail of how significance is calculated. You should read that installment if you want to decipher statements in research papers like “
    $ (F(2, 864) = 6.6, p = 0.0014)$ ” — or if you are simply curious.

    Bibliography

    FPP07
    David Freedman, Robert Pisani, and Roger Purves, Statistics, 4th ed., W. W. Norton & Company, New York, 2007.
    Ioa05
    John P. A. Ioannidis, Why most published research findings are false, PLoS Med 2 (2005), no. 8, e124, Available as http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124.
    Kot09
    Deborah Kotz, 10 salt shockers that could make hypertension worse, U.S. News & World Report (2009), Online as http://health.usnews.com/articles/health/heart/2009/07/20/10-salt-shockers-that-could-make-hypertension-worse.html.
    MKJ07
    Ramal Moonesinghe, Muin J Khoury, and A. Cecile J. W Janssens, Most published research findings are false — but a little replication goes a long way, PLoS Med 4 (2007), no. 2, e28, Available as http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0040028.
    Sac84
    Lothar Sachs, Applied statistics: A handbook of techniques, 2nd ed., Springer-Verlag, New York, 1984.
    SS99
    Murray R. Spiegel and Larry J. Stephens, Schaum’s outline of statistics, 4th ed., McGraw-Hill, New York, 1999.

    Footnotes

    … series1
    http://www.win-vector.com/blog/category/statistics-to-english-translation/
    … Sneetches2
    “The Sneetchs,” from The Sneetches and Other Stories by Dr. Seuss.
    http://www.youtube.com/watch?v=Ln3V0HgW4eM
    and http://www.youtube.com/watch?v=s0LgMpfLD1Y
    … points3
    This example is based on Exercise 10.17 in [SS99]; the original exercise did not, unfortunately, involve Sneetches.
    … probability4
    The correct way to state this is that for a given (unknown) $ \mu $ , the estimate $ \bar{x}$ falls in the interval
    $ \mu \pm 2 \sigma/\sqrt{n}$ just over 95% of the time. This gets awkward to reason about. Luckily, symmetry arguments let us center the appropriate confidence interval around $ \bar{x}$ instead.
    …5
    Of course, we don’t actually know $ \sigma$ , so we don’t know exactly how fast we zoom in. That doesn’t affect our argument, though, since only $ n$ changes
    … series6
    http://www.win-vector.com/blog/2009/11/i-dont-think-that-means-what-you-think-it-means-statistics-to-english-translation-part-1-accuracy-measures/

    Nina Zumel 2009-12-04

    Related posts:

    1. Statistics to English Translation, Part 2b: Calculating Significance
    2. “I don’t think that means what you think it means;” Statistics to English Translation, Part 1: Accuracy Measures
    3. R annoyances

    ]]>     http://www.win-vector.com/blog/2009/12/statistics-to-english-translation-part-2a-%e2%80%99significant%e2%80%99-doesn%e2%80%99t-always-mean-%e2%80%99important%e2%80%99/feed/ 4