Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole benchmark

被引：29

作者：

Wagner, Martin ^{[1
,2
]}

Mueller-Stich, Beat-Peter ^{[1
,2
]}

Kisilenko, Anna ^{[1
,2
]}

Tran, Duc ^{[1
,2
]}

Heger, Patrick ^{[1
]}

Muendermann, Lars ^{[3
]}

Lubotsky, David M. ^{[1
,2
]}

Mueller, Benjamin ^{[1
,2
]}

Davitashvili, Tornike ^{[1
,2
]}

Capek, Manuela ^{[1
,2
]}

Reinke, Annika ^{[4
,5
,6
]}

Reid, Carissa ^{[7
]}

Yu, Tong ^{[8
,9
]}

Vardazaryan, Armine ^{[8
,9
]}

Nwoye, Chinedu Innocent ^{[8
,9
]}

Padoy, Nicolas ^{[8
,9
]}

Liu, Xinyang ^{[10
]}

Lee, Eung-Joo ^{[11
]}

Disch, Constantin ^{[12
]}

Meine, Hans ^{[12
,13
]}

Xia, Tong ^{[14
]}

Jia, Fucang ^{[14
]}

Kondo, Satoshi ^{[15
,27
]}

Reiter, Wolfgang ^{[16
]}

Jin, Yueming ^{[17
]}

Long, Yonghao ^{[17
]}

Jiang, Meirui ^{[17
]}

Dou, Qi ^{[17
]}

Heng, Pheng Ann ^{[17
]}

Twick, Isabell ^{[18
]}

Kirtac, Kadir ^{[18
]}

Hosgor, Enes ^{[18
]}

Bolmgren, Jon Lindstro ^{[18
]}

Stenzel, Michael ^{[18
]}

von Siemens, Bjorn ^{[18
]}

Zhao, Long ^{[19
]}

Ge, Zhenxiao ^{[19
]}

Sun, Haiming ^{[19
]}

Xie, Di ^{[19
]}

Guo, Mengqi ^{[20
]}

Liu, Daochang ^{[21
]}

Kenngott, Hannes G. ^{[1
]}

Nickel, Felix ^{[1
]}

von Frankenberg, Moritz ^{[22
]}

Mathis-Ullrich, Franziska ^{[23
]}

Kopp-Schneider, Annette ^{[7
]}

Maier-Hein, Lena ^{[4
,5
,6
,24
]}

Speidel, Stefanie ^{[25
]}

Bodenstedt, Sebastian ^{[26
]}

机构：

[1] Heidelberg Univ Hosp, Dept Gen Visceral & Transplantat Surg, Neuenheimer Feld 420, D-69120 Heidelberg, Germany

[2] Natl Ctr Tumor Dis NCT Heidelberg, Neuenheimer Feld 460, D-69120 Heidelberg, Germany

[3] KARL STORZ SE & Co KG, Corp Res & Technol, Data Assisted Solut, Dr Karl Storz Str 34, D-78332 Tuttlingen, Germany

[4] German Canc Res Ctr, Div Comp Assisted Med Intervent, Neuenheimer Feld 223, D-69120 Heidelberg, Germany

[5] German Canc Res Ctr, HIP Helmholtz Imaging Platform, Neuenheimer Feld 223, D-69120 Heidelberg, Germany

[6] Heidelberg Univ, Fac Math & Comp Sci, Neuenheimer Feld 205, D-69120 Heidelberg, Germany

[7] German Canc Res Ctr, Div Biostat, Neuenheimer Feld 280, Heidelberg, Germany

[8] Univ Strasbourg, ICube, CNRS, 300 Bd Sebastien Brant,CS 10413, F-67412 Illkirch Graffenstaden, France

[9] IHU Strasbourg, 1 Pl Hop, F-67000 Strasbourg, France

[10] Childrens Natl Hosp, Sheikh Zayed Inst Pediat Surg Innovat, 111 Michigan Ave NW, Washington, DC 20010 USA

[11] Univ Maryland, 2405 AV Williams Bldg, College Pk, MD 20742 USA

[12] Fraunhofer Inst Digital Med MEVIS, Max von Laue Str 2, D-28359 Bremen, Germany

[13] Univ Bremen, Med Image Comp Grp, Fraunhofer MEVIS, FB3, Fallturm 1, D-28359 Bremen, Germany

[14] Chinese Acad Sci, Shenzhen Inst Adv Technol, Lab Med Imaging & Digital Surg, Shenzhen 518055, Peoples R China

[15] Konika Minolta Inc, 1-2 Sakura Machi, Takatsuki, Osaka 5698503, Japan

[16] Wintegral GmbH, Ehrenbreitsteiner Str 36, D-80993 Munich, Germany

[17] Chinese Univ Hong Kong, Dept Comp Sci & Engn, Sha Tin, Ho Sin Hang Engn Bldg, Hong Kong, Peoples R China

[18] Caresyntax GmbH, Komturstr 18A, D-12099 Berlin, Germany

[19] Hikvis Res Inst, Hangzhou, Peoples R China

[20] Natl Univ Singapore, Sch Comp, Comp 1, 13 Comp Dr, Singapore 117417, Singapore

[21] Peking Univ, Natl Engn Res Ctr Visual Technol, Sch Comp Sci, Beijing, Peoples R China

[22] Salem Hosp Evangel Stadtmiss Heidelberg, Dept Surg, Zeppelinstr 11-33, D-69121 Heidelberg, Germany

[23] Karlsruhe Inst Technol, Inst Anthropomat & Robot, Hlth Robot & Automat Lab, Geb 40-28,KIT Campus Sud,Engler Bunte Ring 8, D-76131 Karlsruhe, Germany

[24] Heidelberg Univ, Med Fac, Neuenheimer Feld 672, D-69120 Heidelberg, Germany

[25] Natl Ctr Tumor Dis Dresden, Div Translat Surg Oncol, Fetscherstr 74, D-01307 Dresden, Germany

[26] Tech Univ Dresden, Cluster Excellence Ctr Tactile Internet Human In, D-01062 Dresden, Germany

[27] Muroran Inst Technol, Muroran, Hokkaido 0508585, Japan

来源：

MEDICAL IMAGE ANALYSIS | 2023年 / 86卷

关键词：

Surgical workflow analysis; Endoscopic vision; Surgical data science; Laparoscopic cholecystectomy; ARTIFICIAL-INTELLIGENCE; RECOGNITION; PERFORMANCE; CHALLENGE; TOOL;

D O I：

10.1016/j.media.2023.102770

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

Purpose: Surgical workflow and skill analysis are key technologies for the next generation of cognitive surgical assistance systems. These systems could increase the safety of the operation through context-sensitive warnings and semi-autonomous robotic assistance or improve training of surgeons via data-driven feedback. In surgical workflow analysis up to 91% average precision has been reported for phase recognition on an open data singlecenter video dataset. In this work we investigated the generalizability of phase recognition algorithms in a multicenter setting including more difficult recognition tasks such as surgical action and surgical skill. Methods: To achieve this goal, a dataset with 33 laparoscopic cholecystectomy videos from three surgical centers with a total operation time of 22 h was created. Labels included framewise annotation of seven surgical phases with 250 phase transitions, 5514 occurences of four surgical actions, 6980 occurences of 21 surgical instruments from seven instrument categories and 495 skill classifications in five skill dimensions. The dataset was used in the 2019 international Endoscopic Vision challenge, sub-challenge for surgical workflow and skill analysis. Here, 12 research teams trained and submitted their machine learning algorithms for recognition of phase, action, instrument and/or skill assessment. Results: F1-scores were achieved for phase recognition between 23.9% and 67.7% (n = 9 teams), for instrument presence detection between 38.5% and 63.8% (n = 8 teams), but for action recognition only between 21.8% and 23.3% (n = 5 teams). The average absolute error for skill assessment was 0.78 (n = 1 team). Conclusion: Surgical workflow and skill analysis are promising technologies to support the surgical team, but there is still room for improvement, as shown by our comparison of machine learning algorithms. This novel HeiChole benchmark can be used for comparable evaluation and validation of future work. In future studies, it is of utmost importance to create more open, high-quality datasets in order to allow the development of artificial intelligence and cognitive robotics in surgery.

引用

页数：21

共 50 条

[21] Comparative Analysis of Machine Learning Algorithms in Breast Cancer Classification
Chaurasiya, Satish
Rajak, Ranjit
WIRELESS PERSONAL COMMUNICATIONS, 2023, 131 (02) : 763 - 772
[22] A COMPARATIVE ANALYSIS OF MACHINE LEARNING ALGORITHMS FOR IPO UNDERPERFORMANCE PREDICTION
Sonsare, Pravinkumar M.
Pande, Ashtavinayak
Kumar, Sudhanshu
Kurve, Akshay
Shanbhag, Chinmay
JOURNAL OF ADVANCED APPLIED SCIENTIFIC RESEARCH, 2023, 5 (06):
[23] Comparative Study of Machine Learning Algorithms for Twitter Sentiment Analysis
Indulkar, Yash
Patil, Abhijit
2021 INTERNATIONAL CONFERENCE ON EMERGING SMART COMPUTING AND INFORMATICS (ESCI), 2021, : 295 - 299
[24] A Comparative Analysis of Machine Learning Algorithms in Energy Poverty Prediction
Kalfountzou, Elpida
Papada, Lefkothea
Tourkolias, Christos
Mirasgedis, Sevastianos
Kaliampakos, Dimitris
Damigos, Dimitris
ENERGIES, 2025, 18 (05)
[25] Enhancing GPS Accuracy with Machine Learning: A Comparative Analysis of Algorithms
Zontul, Metin
Ersan, Ziya Gokalp
Yelmen, Ilkay
Cevik, Taner
Anka, Ferzat
Gesoglu, Kevser
TRAITEMENT DU SIGNAL, 2024, 41 (03) : 1441 - 1450
[26] Analysis of three intrusion detection system benchmark datasets using machine learning algorithms
Kayacik, HG
Zincir-Heywood, N
INTELLIGENCE AND SECURITY INFORMATICS, PROCEEDINGS, 2005, 3495 : 362 - 367
[27] Rehabilitation monitoring and assessment: a comparative analysis of feature engineering and machine learning algorithms on the UI-PRMD and KIMORE benchmark datasets
Zaher, Moamen
Ghoneim, Amr S.
Abdelhamid, Laila
Atia, Ayman
JOURNAL OF INFORMATION AND TELECOMMUNICATION, 2025,
[28] Development and validation of 'Patient Optimizer' (POP) algorithms for predicting surgical risk with machine learning
Kowadlo, Gideon
Mittelberg, Yoel
Ghomlaghi, Milad
Stiglitz, Daniel K.
Kishore, Kartik
Guha, Ranjan
Nazareth, Justin
Weinberg, Laurence
BMC MEDICAL INFORMATICS AND DECISION MAKING, 2024, 24 (01)
[29] Development and validation of ‘Patient Optimizer’ (POP) algorithms for predicting surgical risk with machine learning
Gideon Kowadlo
Yoel Mittelberg
Milad Ghomlaghi
Daniel K. Stiglitz
Kartik Kishore
Ranjan Guha
Justin Nazareth
Laurence Weinberg
BMC Medical Informatics and Decision Making, 24
[30] Can deep learning algorithms outperform benchmark machine learning algorithms in flood susceptibility modeling?
Binh Thai Pham
Chinh Luu
Tran Van Phong
Phan Trong Trinh
Shirzadi, Ataollah
Renoud, Somayeh
Asadi, Shahrokh
Hiep Van Le
von Meding, Jason
Clague, John J.
JOURNAL OF HYDROLOGY, 2021, 592

← 1 2 3 4 5 →