Diagnostic Accuracy of a Mobile AI-Based Symptom Checker anda Web-Based Self-Referral Tool in Rheumatology: Multicenter Randomized Controlled Trial

Background: The diagnosis of inflammatory rheumatic diseases (IRDs) is often delayed due to unspecific symptoms and ashortage of rheumatologists. Digital diagnostic decision support systems (DDSSs) have the potential to expedite diagnosis andhelp patients navigate the health care system more efficiently. Objective: The aim of this study was to assess the diagnostic accuracy of a mobile artificial intelligence (AI)-based symptomchecker (Ada) and a web-based self-referral tool (Rheport) regarding IRDs. Methods: A prospective, multicenter, open-label, crossover randomized controlled trial was conducted with patients newlypresenting to 3 rheumatology centers. Participants were randomly assigned to complete a symptom assessment using either Adaor Rheport. The primary outcome was the correct identification of IRDs by the DDSSs, defined as the presence of any IRD inthe list of suggested diagnoses by Ada or achieving a prespecified threshold score with Rheport. The gold standard was thediagnosis made by rheumatologists. Results: A total of 600 patients were included, among whom 214 (35.7%) were diagnosed with an IRD. Most frequent IRDwas rheumatoid arthritis with 69 (11.5%) patients. Rheport's disease suggestion and Ada's top 1 (D1) and top 5 (D5) diseasesuggestions demonstrated overall diagnostic accuracies of 52%, 63%, and 58%, respectively, for IRDs. Rheport showed a sensitivityof 62% and a specificity of 47% for IRDs. Ada's D1 and D5 disease suggestions showed a sensitivity of 52% and 66%, respectively,and a specificity of 68% and 54%, respectively, concerning IRDs. Ada's diagnostic accuracy regarding individual diagnoses washeterogenous, and Ada performed considerably better in identifying rheumatoid arthritis in comparison to other diagnoses (D1:42%; D5: 64%). The Cohen kappa statistic of Rheport for agreement on any rheumatic disease diagnosis with Ada D1 was 0.15 (95%CI 0.08-0.18) and with Ada D5 was 0.08 (95% CI 0.00-0.16), indicating poor agreement for the presence of any rheumatic diseasebetween the 2 DDSSs. Conclusions: To our knowledge, this is the largest comparative DDSS trial with actual use of DDSSs by patients. The diagnosticaccuracies of both DDSSs for IRDs were not promising in this high-prevalence patient population. DDSSs may lead to a misuseof scarce health care resources. Our results underscore the need for stringent regulation and drastic improvements to ensure thesafety and efficacy of DDSSs. Trial Registration: German Register of Clinical Trials DRKS00017642; https://drks.de/search/en/trial/DRKS00017642