Using automated clinical data for risk adjustment - Development and validation of six disease-specific mortality predictive models for pay-for-performance

Background: Clinically plausible risk-adjustment methods are needed to implement pay-for-performance protocols. Because billing data lacks clinical precision, may be gamed, and chart abstraction is costly, we sought to develop predictive models for mortality that maximally used automated laboratory data and intentionally minimized the use of administrative data (Laboratory Models). We also evaluated the additional value of vital signs and altered mental status (Full Models). Methods: Six models predicting in-hospital mortality for ischemic and hemorrhagic stroke, pneumonia, myocardial infarction, heart failure, and septicemia were derived from 194,903 admissions in 2000-2003 across 71 hospitals that imported laboratory data. Demographics, admission-based labs, International Classification of Diseases (ICD)-9 variables. vital signs, and altered mental status were sequentially entered as covariates. Models were validated using abstractions (629,490 admissions) from 195 hospitals. Finally, we constructed hierarchical models to compare hospital performance using the Laboratory Models and the Full Models. Results: Model c-statistics ranged from 0.81 to 0.89. As constructed. laboratory findings contributed more to the prediction of death compared with any other risk factor characteristic groups across most models except for stroke, where altered mental status was more important. Laboratory variables were between 2 and 67 times more important in predicting mortality than ICD-9 variables. The hospital-level risk-standardized mortality rates derived from the Laboratory Models were highly correlated with the results derived from the Full Models (average rho = 0.92). Conclusions: Mortality can be well predicted using models that maximize reliance on objective pathophysiologic variables whereas minimizing input from billing data. Such models should be less susceptible to the vagaries of billing information and inexpensive to implement.