Evaluation of three machine learning models for self-referral decision support on low back pain in primary care

Background: Most people experience low back pain (LBP) at least once in their life and for some patients this evolves into a chronic condition. One way to prevent acute LBP from transiting into chronic LBP, is to ensure that patients receive the right interventions at the right moment. We started research in the design of a clinical decision support system (CDSS) to support patients with LBP in their self-referral to primary care. For this, we explored the possibilities of using supervised machine learning. We compared the performances of the three classification models -i.e. 1. decision tree, 2. random forest, and 3. boosted tree - to get insight in which model performs best and whether it is already acceptable to use this model in real practice. Methods: The three models were generated by means of supervised machine learning with 70% of a training dataset (1288 cases with 65% GP, 33% physio, 2% self-care cases). The cases in the training dataset were fictive cases on low back pain collected during a vignette study with primary healthcare professionals. We also wanted to know the performance of the models on real-life low back pain cases that were not used to train the models. Therefore we also collected real-life cases on low back pain as test dataset. These cases were collected with the help of patients and healthcare professionals in primary care. For each model, the performance was measured during model validation - with 30% of the training dataset -as well as during model testing -with the test dataset containing real-life cases. The total observed accuracy as well as the kappa, and the sensitivity, specificity, and precision were used as performance measures to compare the models. Results: For the training dataset, the total observed accuracies of the decision tree, the random forest and boosted tree model were 70%, 69%, and 72% respectively. For the test dataset, the total observed accuracies were 71%, 53%, and 71% respectively. The boosted tree appeared to be the best for predicting a referral advice with a fair accuracy (Kappa between 0.2 and 0.4). Next to this, the measured evaluation measures show that all models provided a referral advice better than just a random guess. This means that all models learned some implicit knowledge of the provided referral advices in the training dataset. Conclusions: The study showed promising results on the possibility of using machine learning in the design of our CDSS. The boosted tree model performed best on the classification of low back pain cases, but still has to be improved. Therefore, new cases have to be collected, especially cases that are classified as self-care cases. This to be sure that also the self-care advice can be predicted well by the model.