Controlling the Maximum Familywise Type I Error Rate in Analyses of Multivariate Experiments

Multivariate experiments are often analyzed by multistage multiple-comparison procedures (MCPs) that prohibit univariate testing on individual dependent variables if an overall multivariate analysis of variance (MANOVA) test fails to reject the relevant overall null hypothesis. Although the sole function of the MANOVA test in such analyses is to control the overall Type I error rate, it is known that the most popular MANOVA-protected MCPs do not control the maximum familywise error rate (MFWER). In this article, we show that the MFWER associated with standard MANOVA-protected MCPs can be so large that the protection provided by the initial MANOVA test is illusory. We show that the MFWER can be controlled nonconservatively with modified protected MCPs and with single-stage MCPs that allow for the construction of simultaneous confidence intervals on effect sizes. We argue that, given the ease with which these MCPs can be implemented, there is no justification for continued use of the standard procedures.