Inferences about variance components and reliability-generalizability coefficients in the absence of random sampling

Generalizability theory (G theory) employs random-effects ANOVA to estimate the variance components included in generalizability coefficients, standard errors, and other indices of precision. The ANOVA models depend on random sampling assumptions, and the variance-component estimates are likely to be sensitive to violations of these assumptions. Yet, generalizability studies do not typically sample randomly. This kind of inconsistency between assumptions in statistical models and actual data collection procedures is not uncommon in science, but it does raise fundamental questions about the substantive inferences based on the statistical analyses. This article reviews criticisms of sampling assumptions in G theory (and in reliability theory) and examines the feasibility of using representative sampling, stratification, homogeneity assumptions, and replications to address these criticisms.