Control of false positives

The collection of p-values for the F-tests at each timepoint is generally the only input for multiple testing procedures. Indeed, most methods consist in rejecting the null when the p-value is smaller than a preset threshold, chosen to guarantee that the corresponding number V of erroneous rejections of the null is controlled. In the last two decades, the questions raised by large-scale significance analysis of the relationships between components of complex systems and controlled experimental conditions, such as the search for key regulator genes using high-throughput gene expression data, have generated a plethora of simultaneous testing procedures and thresholding methods for high-dimensional data (see van der Laan and Dudoit, 2007 for a review of the leading procedures and Gropp et al. (2011a, 2011b), specifically, for ERP data analysis.

These multiple testing methods can be divided into two families according to the overall type-I error rate they control. The most popular reference methods, which are designed for a moderate number of simultaneous tests, such as for post-hoc comparisons in analysis of variance, aim at controlling the Family-Wise Error Rate (FWER) defined as FWER = P(V>1) . The Bonferroni correction is probably the simplest and best-known way to achieve this goal.

The function ‘erptest’ is designed for this kind of multiplicity correction:

tests = erptest(erpcz[,1:251],design=mod1,design0=mod0,method="bonferroni")

The following commands can be used to plot the results:

frames = seq(0,1001,4)
plot(frames,tests$signal,type="l",xlab="Time (ms)",
   ylab="Difference ERP curves")
points(frames[tests$significant],rep(0,length(tests$significant)),
   pch=16,col="blue")
title("Paired comparison at electrode CZ")

The following plot is generated:

erpbonftestcz

However, FWER-controlling procedures are usually far too conservative when the number of tests is large. A new family of methods aims to control, instead of the FWER, the False Discovery Rate (FDR), defined as the expected proportion of erroneous rejections of the null among the positive tests (see Benjamini and Hochberg, 1995). This change of objectives in large-scale multiple testing began with the introduction of the so-called Benjamini-Hochberg (BH) procedure. Under an assumption of independence among tests, Benjamini and Hochberg (1995) shows that their thresholding method guarantees the control of the FDR .

The function ‘erptest’ can also be used for a BH correction:

tests = erptest(erpcz[,1:251],design=mod1,design0=mod0,method="BH")

The following commands can be used to plot the results:

frames = seq(0,1001,4)
plot(frames,tests$signal,type="l",xlab="Time (ms)",
   ylab="Difference ERP curves")
points(frames[tests$significant],rep(0,length(tests$significant)),
   pch=16,col="blue")
title("Paired comparison at electrode CZ")

The following plot is generated:

erpbhtestcz

The possible procedures provided in ‘erptest’ are the collection of FDR or FWER controlling methods available in the generic function ‘p.adjust’.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s