Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark

被引：29

作者：

Wagner, Martin ^{[1
,2
]}

Mueller-Stich, Beat-Peter ^{[1
,2
]}

Kisilenko, Anna ^{[1
,2
]}

Tran, Duc ^{[1
,2
]}

Heger, Patrick ^{[1
]}

Muendermann, Lars ^{[3
]}

Lubotsky, David M. ^{[1
,2
]}

Mueller, Benjamin ^{[1
,2
]}

Davitashvili, Tornike ^{[1
,2
]}

Capek, Manuela ^{[1
,2
]}

Reinke, Annika ^{[4
,5
,6
]}

Reid, Carissa ^{[7
]}

Yu, Tong ^{[8
,9
]}

Vardazaryan, Armine ^{[8
,9
]}

Nwoye, Chinedu Innocent ^{[8
,9
]}

Padoy, Nicolas ^{[8
,9
]}

Liu, Xinyang ^{[10
]}

Lee, Eung-Joo ^{[11
]}

Disch, Constantin ^{[12
]}

Meine, Hans ^{[12
,13
]}

Xia, Tong ^{[14
]}

Jia, Fucang ^{[14
]}

Kondo, Satoshi ^{[15
,27
]}

Reiter, Wolfgang ^{[16
]}

Jin, Yueming ^{[17
]}

Long, Yonghao ^{[17
]}

Jiang, Meirui ^{[17
]}

Dou, Qi ^{[17
]}

Heng, Pheng Ann ^{[17
]}

Twick, Isabell ^{[18
]}

Kirtac, Kadir ^{[18
]}

Hosgor, Enes ^{[18
]}

Bolmgren, Jon Lindstro ^{[18
]}

Stenzel, Michael ^{[18
]}

von Siemens, Bjorn ^{[18
]}

Zhao, Long ^{[19
]}

Ge, Zhenxiao ^{[19
]}

Sun, Haiming ^{[19
]}

Xie, Di ^{[19
]}

Guo, Mengqi ^{[20
]}

Liu, Daochang ^{[21
]}

Kenngott, Hannes G. ^{[1
]}

Nickel, Felix ^{[1
]}

von Frankenberg, Moritz ^{[22
]}

Mathis-Ullrich, Franziska ^{[23
]}

Kopp-Schneider, Annette ^{[7
]}

Maier-Hein, Lena ^{[4
,5
,6
,24
]}

Speidel, Stefanie ^{[25
]}

Bodenstedt, Sebastian ^{[26
]}

机构：

[1] Heidelberg Univ Hosp, Dept Gen Visceral & Transplantat Surg, Neuenheimer Feld 420, D-69120 Heidelberg, Germany

[2] Natl Ctr Tumor Dis NCT Heidelberg, Neuenheimer Feld 460, D-69120 Heidelberg, Germany

[3] KARL STORZ SE & Co KG, Corp Res & Technol, Data Assisted Solut, Dr Karl Storz Str 34, D-78332 Tuttlingen, Germany

[4] German Canc Res Ctr, Div Comp Assisted Med Intervent, Neuenheimer Feld 223, D-69120 Heidelberg, Germany

[5] German Canc Res Ctr, HIP Helmholtz Imaging Platform, Neuenheimer Feld 223, D-69120 Heidelberg, Germany

[6] Heidelberg Univ, Fac Math & Comp Sci, Neuenheimer Feld 205, D-69120 Heidelberg, Germany

[7] German Canc Res Ctr, Div Biostat, Neuenheimer Feld 280, Heidelberg, Germany

[8] Univ Strasbourg, ICube, CNRS, 300 Bd Sebastien Brant,CS 10413, F-67412 Illkirch Graffenstaden, France

[9] IHU Strasbourg, 1 Pl Hop, F-67000 Strasbourg, France

[10] Childrens Natl Hosp, Sheikh Zayed Inst Pediat Surg Innovat, 111 Michigan Ave NW, Washington, DC 20010 USA

[11] Univ Maryland, 2405 AV Williams Bldg, College Pk, MD 20742 USA

[12] Fraunhofer Inst Digital Med MEVIS, Max von Laue Str 2, D-28359 Bremen, Germany

[13] Univ Bremen, Med Image Comp Grp, Fraunhofer MEVIS, FB3, Fallturm 1, D-28359 Bremen, Germany

[14] Chinese Acad Sci, Shenzhen Inst Adv Technol, Lab Med Imaging & Digital Surg, Shenzhen 518055, Peoples R China

[15] Konika Minolta Inc, 1-2 Sakura Machi, Takatsuki, Osaka 5698503, Japan

[16] Wintegral GmbH, Ehrenbreitsteiner Str 36, D-80993 Munich, Germany

[17] Chinese Univ Hong Kong, Dept Comp Sci & Engn, Sha Tin, Ho Sin Hang Engn Bldg, Hong Kong, Peoples R China

[18] Caresyntax GmbH, Komturstr 18A, D-12099 Berlin, Germany

[19] Hikvis Res Inst, Hangzhou, Peoples R China

[20] Natl Univ Singapore, Sch Comp, Comp 1, 13 Comp Dr, Singapore 117417, Singapore

[21] Peking Univ, Natl Engn Res Ctr Visual Technol, Sch Comp Sci, Beijing, Peoples R China

[22] Salem Hosp Evangel Stadtmiss Heidelberg, Dept Surg, Zeppelinstr 11-33, D-69121 Heidelberg, Germany

[23] Karlsruhe Inst Technol, Inst Anthropomat & Robot, Hlth Robot & Automat Lab, Geb 40-28,KIT Campus Sud,Engler Bunte Ring 8, D-76131 Karlsruhe, Germany

[24] Heidelberg Univ, Med Fac, Neuenheimer Feld 672, D-69120 Heidelberg, Germany

[25] Natl Ctr Tumor Dis Dresden, Div Translat Surg Oncol, Fetscherstr 74, D-01307 Dresden, Germany

[26] Tech Univ Dresden, Cluster Excellence Ctr Tactile Internet Human In, D-01062 Dresden, Germany

[27] Muroran Inst Technol, Muroran, Hokkaido 0508585, Japan

来源：

MEDICAL IMAGE ANALYSIS | 2023年 / 86卷

关键词：

Surgical workflow analysis; Endoscopic vision; Surgical data science; Laparoscopic cholecystectomy; ARTIFICIAL-INTELLIGENCE; RECOGNITION; PERFORMANCE; CHALLENGE; TOOL;

D O I：

10.1016/j.media.2023.102770

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Purpose: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data singlecenter video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill. Methods: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. Results: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team). Conclusion: Surgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.

引用

页数：21

共 50 条

[1] Surgical skill assessment using machine learning algorithms
Lavanchy, J. L.
Zindel, J.
Kirtac, K.
Twick, I.
Hosgor, E.
Candinas, D.
Beldi, G.
BRITISH JOURNAL OF SURGERY, 2021, 108 : 30 - 30
[2] Comparative Analysis of Machine learning algorithms in OCR
Jain, Vanita
Dubey, Arun
Gupta, Amit
Sharma, Sanchit
PROCEEDINGS OF THE 10TH INDIACOM - 2016 3RD INTERNATIONAL CONFERENCE ON COMPUTING FOR SUSTAINABLE GLOBAL DEVELOPMENT, 2016, : 1089 - 1092
[3] Validation of machine learning algorithms
Burzykowski, Tomasz
Geubbelmans, Melvin
Rousseau, Axel-Jan
Valkenborg, Dirk
AMERICAN JOURNAL OF ORTHODONTICS AND DENTOFACIAL ORTHOPEDICS, 2023, 164 (02) : 295 - 297
[4] Machine learning algorithms in microbial classification: a comparative analysis
Wu, Yuandi
Gadsden, S. Andrew
FRONTIERS IN ARTIFICIAL INTELLIGENCE, 2023, 6
[5] Comparative Analysis of Machine Learning Algorithms for Rainfall Prediction
Patil, Rudragoud
Bedekar, Gayatri
INNOVATIVE DATA COMMUNICATION TECHNOLOGIES AND APPLICATION, ICIDCA 2021, 2022, 96 : 833 - 842
[6] Machine Learning Algorithms for Document Classification: Comparative Analysis
Rashid, Faizur
Gargaare, Suleiman M. A.
Aden, Abdulkadir H.
Abdi, Afendi
INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 2022, 13 (04) : 260 - 265
[7] Comparative Analysis of Different Machine Learning Algorithms in Classification
Wang, Lincong
Xu, Weiwen
Zhu, Zhenghao
2022 INTERNATIONAL CONFERENCE ON BIG DATA, INFORMATION AND COMPUTER NETWORK (BDICN 2022), 2022, : 257 - 263
[8] Development of Virtual Skill Trainers and Their Validation Study Analysis Using Machine Learning
Shedage, Seema
Farmer, Jake
Demirel, Doga
Halic, Tansel
Kockara, Sinan
Arikatla, Venkata
Sexton, Kevin
Ahmadi, Shahryar
5TH INTERNATIONAL CONFERENCE ON INFORMATION SYSTEM AND DATA MINING (ICISDM 2021), 2021, : 8 - 13
[9] CHURN PREDICTION - A COMPARATIVE ANALYSIS WITH SUPERVISED MACHINE LEARNING ALGORITHMS
Gangadharan, Chika K.
Alex, Roshni
Sabu, M. K.
ADVANCES AND APPLICATIONS IN MATHEMATICAL SCIENCES, 2021, 20 (12): : 3049 - 3060
[10] A comparative analysis of machine learning algorithms for predicting wave runup
Ahmet Durap
Anthropocene Coasts, 6

← 1 2 3 4 5 →