In statistics, testing multiple hypotheses simultaneously using methods appropriate for testing single hypotheses tends to yield many false positives: the so-called multiple comparisons problem. For example, assume that one were to test 1,000 null hypotheses, all of which are true, and to reject null hypotheses with a significance level of 0.05; due to random chance, one would expect 5% of the results to appear significant, yielding 50 false positives. Since the 1950s, statisticians had been developing methods for multiple comparisons that reduced the number of false positives, such as controlling the family-wise error rate using the Bonferroni correction, but these methods also increased the number of false negatives. In 1995, Yoav Benjamini and Yosef Hochberg proposed controlling the false discovery rate as a more statistically powerful alternative to controlling the FWER in multiple hypothesis testing. The pFDR and the q-value were introduced by John D. Storey in 2002 in order to improve upon a limitation of the FDR, namely that the FDR is not defined when there are no positive results.
Definition
Let there be a null hypothesis and an alternative hypothesis. Perform hypothesis tests; let the test statistics be i.i.d. random variables such that. That is, if is true for test , then follows the null distribution ; while if is true, then follows the alternative distribution. Let, that is, for each test, is true with probability and is true with probability. Denote the critical region at significance level by. Let an experiment yield a value for the test statistic. The q-value of is formally defined as That is, the q-value is the infimum of the pFDR if is rejected for test statistics with values. Equivalently, the q-value equals which is the infimum of the probability that is true given that is rejected.
The p-value is defined as the infimum of the probability that is rejected given that is true. Comparing the definitions of the p- and q-values, it can be seen that the q-value is the minimum posterior probability that is true.
Interpretation
The q-value can be interpreted as the false discovery rate : the proportion of false positives among all positive results. Given a set of test statistics and their associated q-values, rejecting the null hypothesis for all tests whose q-value is less than or equal to some threshold ensures that the expected value of the false discovery rate is.
involve simultaneously testing the expression of thousands of genes. Controlling the FWER avoids excessive false positives but imposes a strict threshold for the p-value that results in many false negatives. However, controlling the pFDR by selecting genes with significant q-values lowers the number of false negatives while ensuring that the expected value of the proportion of false positives among all positive results is low. For example, suppose that among 10,000 genes tested, 1,000 are actually differentially expressed and 9,000 are not:
If we consider every gene with a p-value of less than 0.05 to be differentially expressed, we expect that 450 of the 9,000 genes that are not differentially expressed will appear to be differentially expressed.
If we control the FWER to 0.05, there is only a 5% probability of obtaining at least one false positive. However, this very strict criterion will reduce the power such that few of the 1,000 genes that are actually differentially expressed will appear to be differentially expressed.
If we control the pFDR to 0.05 by considering all genes with a q-value of less than 0.05 to be differentially expressed, then we expect 5% of the positive results to be false positives. This strategy enables one to obtain relatively low numbers of both false positives and false negatives.
