Generalizability theory in psychotherapy research: The impact of multiple sources of variance on the dependability of psychotherapy process ratings

A central task of psychotherapy process measurement development is the assessment of reliability and validity. The convention of reporting intraclass correlations (ICCs) for coding procedures assumes that variance in scores can be adequately explained by differences between individuals and error resulting from differences in coders. Given the complex coding procedures that are common in psychotherapy process studies ( multiple sessions may be rated by multiple coders on one or more multi-item scales), an ICC may fail to account for all of the relevant sources of variability in obtained scores. If process studies are to provide useful information about the mutative agents in psychotherapy, assessment procedures must be developed that dependably assess the constructs of interest. Generalizability theory provides a framework within which multiple sources of error can be simultaneously evaluated, thus improving the accuracy of reliability estimates and providing critical information for modification and improvement of coding procedures. To illustrate the applicability of generalizability theory to psychotherapy process research, the authors present the design and findings of a study investigating the generalizability of technique scales in the Psychotherapy Process Rating Scale for Borderline Personality Disorder. Implications for measurement development and procedural modifications are discussed.