Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark

被引：29

作者：

Wagner, Martin ^{[1
,2
]}

Mueller-Stich, Beat-Peter ^{[1
,2
]}

Kisilenko, Anna ^{[1
,2
]}

Tran, Duc ^{[1
,2
]}

Heger, Patrick ^{[1
]}

Muendermann, Lars ^{[3
]}

Lubotsky, David M. ^{[1
,2
]}

Mueller, Benjamin ^{[1
,2
]}

Davitashvili, Tornike ^{[1
,2
]}

Capek, Manuela ^{[1
,2
]}

Reinke, Annika ^{[4
,5
,6
]}

Reid, Carissa ^{[7
]}

Yu, Tong ^{[8
,9
]}

Vardazaryan, Armine ^{[8
,9
]}

Nwoye, Chinedu Innocent ^{[8
,9
]}

Padoy, Nicolas ^{[8
,9
]}

Liu, Xinyang ^{[10
]}

Lee, Eung-Joo ^{[11
]}

Disch, Constantin ^{[12
]}

Meine, Hans ^{[12
,13
]}

Xia, Tong ^{[14
]}

Jia, Fucang ^{[14
]}

Kondo, Satoshi ^{[15
,27
]}

Reiter, Wolfgang ^{[16
]}

Jin, Yueming ^{[17
]}

Long, Yonghao ^{[17
]}

Jiang, Meirui ^{[17
]}

Dou, Qi ^{[17
]}

Heng, Pheng Ann ^{[17
]}

Twick, Isabell ^{[18
]}

Kirtac, Kadir ^{[18
]}

Hosgor, Enes ^{[18
]}

Bolmgren, Jon Lindstro ^{[18
]}

Stenzel, Michael ^{[18
]}

von Siemens, Bjorn ^{[18
]}

Zhao, Long ^{[19
]}

Ge, Zhenxiao ^{[19
]}

Sun, Haiming ^{[19
]}

Xie, Di ^{[19
]}

Guo, Mengqi ^{[20
]}

Liu, Daochang ^{[21
]}

Kenngott, Hannes G. ^{[1
]}

Nickel, Felix ^{[1
]}

von Frankenberg, Moritz ^{[22
]}

Mathis-Ullrich, Franziska ^{[23
]}

Kopp-Schneider, Annette ^{[7
]}

Maier-Hein, Lena ^{[4
,5
,6
,24
]}

Speidel, Stefanie ^{[25
]}

Bodenstedt, Sebastian ^{[26
]}

机构：

[1] Heidelberg Univ Hosp, Dept Gen Visceral & Transplantat Surg, Neuenheimer Feld 420, D-69120 Heidelberg, Germany

[2] Natl Ctr Tumor Dis NCT Heidelberg, Neuenheimer Feld 460, D-69120 Heidelberg, Germany

[3] KARL STORZ SE & Co KG, Corp Res & Technol, Data Assisted Solut, Dr Karl Storz Str 34, D-78332 Tuttlingen, Germany

[4] German Canc Res Ctr, Div Comp Assisted Med Intervent, Neuenheimer Feld 223, D-69120 Heidelberg, Germany

[5] German Canc Res Ctr, HIP Helmholtz Imaging Platform, Neuenheimer Feld 223, D-69120 Heidelberg, Germany

[6] Heidelberg Univ, Fac Math & Comp Sci, Neuenheimer Feld 205, D-69120 Heidelberg, Germany

[7] German Canc Res Ctr, Div Biostat, Neuenheimer Feld 280, Heidelberg, Germany

[8] Univ Strasbourg, ICube, CNRS, 300 Bd Sebastien Brant,CS 10413, F-67412 Illkirch Graffenstaden, France

[9] IHU Strasbourg, 1 Pl Hop, F-67000 Strasbourg, France

[10] Childrens Natl Hosp, Sheikh Zayed Inst Pediat Surg Innovat, 111 Michigan Ave NW, Washington, DC 20010 USA

[11] Univ Maryland, 2405 AV Williams Bldg, College Pk, MD 20742 USA

[12] Fraunhofer Inst Digital Med MEVIS, Max von Laue Str 2, D-28359 Bremen, Germany

[13] Univ Bremen, Med Image Comp Grp, Fraunhofer MEVIS, FB3, Fallturm 1, D-28359 Bremen, Germany

[14] Chinese Acad Sci, Shenzhen Inst Adv Technol, Lab Med Imaging & Digital Surg, Shenzhen 518055, Peoples R China

[15] Konika Minolta Inc, 1-2 Sakura Machi, Takatsuki, Osaka 5698503, Japan

[16] Wintegral GmbH, Ehrenbreitsteiner Str 36, D-80993 Munich, Germany

[17] Chinese Univ Hong Kong, Dept Comp Sci & Engn, Sha Tin, Ho Sin Hang Engn Bldg, Hong Kong, Peoples R China

[18] Caresyntax GmbH, Komturstr 18A, D-12099 Berlin, Germany

[19] Hikvis Res Inst, Hangzhou, Peoples R China

[20] Natl Univ Singapore, Sch Comp, Comp 1, 13 Comp Dr, Singapore 117417, Singapore

[21] Peking Univ, Natl Engn Res Ctr Visual Technol, Sch Comp Sci, Beijing, Peoples R China

[22] Salem Hosp Evangel Stadtmiss Heidelberg, Dept Surg, Zeppelinstr 11-33, D-69121 Heidelberg, Germany

[23] Karlsruhe Inst Technol, Inst Anthropomat & Robot, Hlth Robot & Automat Lab, Geb 40-28,KIT Campus Sud,Engler Bunte Ring 8, D-76131 Karlsruhe, Germany

[24] Heidelberg Univ, Med Fac, Neuenheimer Feld 672, D-69120 Heidelberg, Germany

[25] Natl Ctr Tumor Dis Dresden, Div Translat Surg Oncol, Fetscherstr 74, D-01307 Dresden, Germany

[26] Tech Univ Dresden, Cluster Excellence Ctr Tactile Internet Human In, D-01062 Dresden, Germany

[27] Muroran Inst Technol, Muroran, Hokkaido 0508585, Japan

来源：

MEDICAL IMAGE ANALYSIS | 2023年 / 86卷

关键词：

Surgical workflow analysis; Endoscopic vision; Surgical data science; Laparoscopic cholecystectomy; ARTIFICIAL-INTELLIGENCE; RECOGNITION; PERFORMANCE; CHALLENGE; TOOL;

D O I：

10.1016/j.media.2023.102770

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Purpose: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data singlecenter video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill. Methods: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. Results: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team). Conclusion: Surgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.

引用

页数：21

共 50 条

[31] Comparative study of machine learning algorithms in predicting asphaltene precipitation with a novel validation technique
Khalighi, Jafar
Cheremisin, Alexey
EARTH SCIENCE INFORMATICS, 2023, 16 (04) : 3097 - 3111
[32] Comparative study of machine learning algorithms in predicting asphaltene precipitation with a novel validation technique
Jafar Khalighi
Alexey Cheremisin
Earth Science Informatics, 2023, 16 : 3097 - 3111
[33] A Comparative Analysis Study of Machine Learning Algorithms in the Quantitative Analysis of LIBS Steel
Zheng, Ji-Shi
Hou, Wei-Yu
Zhao, Zhao-Lin
Yang, Jia-Cheng
Ye, Hong-Ji
Ren, Xiang-Xu
Kong, Ling-Hua
Journal of Network Intelligence, 2024, 9 (01): : 313 - 328
[34] Validation workflow for machine learning interatomic potentials for complex ceramics
Ghaffari, Kimia
Bavdekar, Salil
Spearot, Douglas E.
Subhash, Ghatu
COMPUTATIONAL MATERIALS SCIENCE, 2024, 239
[35] Comparative analysis of machine learning algorithms for steel plate defect classification
Kordatos I.D.
Benardos P.
International Journal of Mechatronics and Manufacturing Systems, 2022, 15 (04) : 246 - 263
[36] Comparative Analysis of Predictive Machine Learning Algorithms for Chronic Kidney Disease
Kangra, Kirti
Singh, Jaswinder
2021 INTERNATIONAL CONFERENCE ON COMPUTATIONAL PERFORMANCE EVALUATION (COMPE-2021), 2021, : 790 - 796
[37] Comparative analysis of machine learning algorithms for prediction of smart grid stability†
Bashir, Ali Kashif
Khan, Suleman
Prabadevi, B.
Deepa, N.
Alnumay, Waleed S.
Gadekallu, Thippa Reddy
Maddikunta, Praveen Kumar Reddy
INTERNATIONAL TRANSACTIONS ON ELECTRICAL ENERGY SYSTEMS, 2021, 31 (09):
[38] Machine Learning for Enhanced COPD Diagnosis: A Comparative Analysis of Classification Algorithms
Elashmawi, Walaa H.
Djellal, Adel
Sheta, Alaa
Surani, Salim
Aljahdali, Sultan
DIAGNOSTICS, 2024, 14 (24)
[39] A Comparative Classification Analysis of Abdominal Aortic Aneurysms by Machine Learning Algorithms
Balaji Rengarajan
Wei Wu
Crystal Wiedner
Daijin Ko
Satish C. Muluk
Mark K. Eskandari
Prahlad G. Menon
Ender A. Finol
Annals of Biomedical Engineering, 2020, 48 : 1419 - 1429
[40] Comparative analysis of machine learning algorithms for the classification of underwater marine debris
Jalil, B.
Maggiani, L.
Valcarenghi, L.
2023 IEEE INTERNATIONAL WORKSHOP ON METROLOGY FOR THE SEA; LEARNING TO MEASURE SEA HEALTH PARAMETERS, METROSEA, 2023, : 116 - 120

← 1 2 3 4 5 →