Research at the Division of Intelligent Medical Systems
The mission of our division is to improve the quality of interventional healthcare in a data-driven manner using Artificial Intelligence (AI) as a central concept. Our research can be categorized in the three main pillars: Intelligent Medical Imaging Systems, Methods for Intelligent Systems, and Validation of Intelligent Systems with a strong focus on Applications in Interventional Healthcare and Open Science.
Intelligent Medical Imaging Systems
A core component of all interventional data science lies in the real-time perception of the environment. Current clinical decision-making is largely based on human perception relying mainly on visual and tactile feedback (palpation). This holds particularly true for the field of surgery. Most currently available medical imaging modalities commonly rely on the use of ionizing radiation, suffer from poor resolution or contrast and/or lack the capacity to operate in real time, thus not fulfilling the needs of an interventional environment. We challenge the current state of the art by proposing novel interventional imaging concepts based on biophotonics techniques. In this context we pioneered machine learning as an approach to solving the inverse problem of reconstructing clinically relevant tissue properties from optical or photoacoustic spectral measurements obtained with multispectral optical or photoacoustic measurements.
Our novel interventional imaging concepts serve as the basis for a wide range of tasks including (a) tumor detection and localization, (b) perfusion monitoring, and (c) full semantic scene segmentation for context-aware assistance.
© dkfz.de
Representative publications - Intelligent Medical Imaging Systems
Ayala, L., Adler, T. et al. Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery. Science Advances 2023. https://doi.org/10.1126/sciadv.add6778
Gröhl, J. et al. Learned spectral decoloring enables photoacoustic oximetry. Scientific Reports 2021. https://doi.org/10.1038/s41598-021-83405-8
Seidlitz, S., Sellner, J. et al. Robust deep learning-based semantic organ segmentation in hyperspectral images. Medical Image Analysis 2022. https://doi.org/10.1016/j.media.2022.102488
Methods for Intelligent Systems
To provide assistance to caregivers, the imaging data must be interpreted in the context of the relevant, available knowledge and according to key clinical questions. In this regard, the surgical domain faces several (often) unique challenges pertaining to data science methodology. Specifically, surgical data science often suffers from extremely high data variability and heterogeneity and a lack of representative, annotated data. To overcome this hurdle, we have pioneered different concepts to address data sparsity in the surgical domain, with a specific focus on synthetic data. As building trust in AI is key for bringing AI-based solutions into clinical practice, a particular second focus of our research has further been dedicated to explainable, uncertainty-aware machine learning, with a novel class of invertible neural networks serving as the core underlying technique.
Reliable machine learning in cancer care. Uncertainty-aware machine learning is achieved with a novel class of invertible neural networks serving as core underlying technique, here for representing ambiguity in registration problems.
© dkfz.de
Further information
Representative publications - Methods for Intelligent Systems
Eisenmann, M. et al. Why Is the Winner the Best? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023. https://openaccess.thecvf.com/content/CVPR2023/html/Eisenmann_Why_Is_the_Winner_the_Best_CVPR_2023_paper.html
Godau, P., Kalinowski, P.et al. Deployment of Image Analysis Algorithms Under Prevalence Shifts. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. https://doi.org/10.1007/978-3-031-43898-1_38
Adler, T. J. et al. Out of Distribution Detection for Intra-operative Functional Imaging. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging and Clinical Image-Based Procedures (MICCAI UNSURE) 2019. https://doi.org/10.1007/978-3-030-32689-0_8
Ardizzone, L. et al. Analyzing Inverse Problems with Invertible Neural Networks. International Conference on Learning Representations (ICLR) 2019. http://arxiv.org/abs/1808.04730
Maier-Hein, L. et al. Surgical data science for next-generation interventions. Nature Biomedical Engineering 2017. https://doi.org/10.1038/s41551-017-0132-7
Validation of Intelligent Systems
Increasing evidence points to poor validation being one of the major reasons for the failure of AI-based solutions in clinical practice. Our division is working on new methodology and corresponding open source tools for the reliable and robust validation of algorithms.
Problem-aware metric recommendations. Popular performance metrics often fail to capture clinical interest and adversely impact validation. To overcome this issue, we formed the international ‘Metrics Reloaded’ consortium comprising more than 70 experts worldwide under DKFZ and Helmholtz Imaging lead, with the goal of addressing these pitfalls by enabling the selection of metrics that match the domain interest in a problem-aware approach (Maier-Hein et al. 2024).
Representative publications - Validation of Intelligent Systems
Maier-Hein, L. et al. Metrics reloaded: Recommendations for image analysis validation. Nature Methods 2024. https://doi.org/10.1038/s41592-023-02151-z
Reinke, A. et al. Understanding metric-related pitfalls in image analysis validation. Nature Methods 2024. https://doi.org/10.1038/s41592-023-02150-0
Wiesenfarth, M. et al. Methods and open-source toolkit for analyzing and visualizing challenge results. Scientific Reports 2021. https://doi.org/10.1038/s41598-021-82017-6
Reinke, A. How to Exploit Weaknesses in Biomedical Challenge Design and Organization. Medical Image Computing and Computer Assisted Intervention (MICCAI) 2018. https://doi.org/10.1007/978-3-030-00937-3_45
Maier-Hein, L. et al. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications 2018. https://doi.org/10.1038/s41467-018-07619-7
Applications in Interventional Healthcare
The ultimate and most challenging goal regarding surgical data is their usage to provide real-time, context-aware assistance to the physician in and beyond the operating room. Clinical applications include:
- Surgical action recognition (1st prize MICCAI CholecTriplet Challenge 2022)
- Instrument detection (1st prize MICCAI PitVis Workflow Recognition in Endoscopic Pituitary Surgery Challenge2023)
- Polyp detection (1st prize IEEE EndoCV Challenge 2022)
- Tissue classification (1st prize MICCAI GIANA Challenge 2021)
- Perfusion monitoring in laparoscopic kidney surgery (Ayala et al. 2023)
- AI-assisted surgical training (Müller et al. 2022)
- Ultrasound-navigated percutaneous needle insertions (Franz et al. 2017)
Examples for applications in interventional healthcare. (a): First real-time in-human perfusion monitoring in laparoscopic cancer therapy with spectral imaging. Novel machine learning methodology enables the real-time detection of tissue ischemia based on spectral imaging data (Ayala et al. 2023). (b): AI-assisted surgical training. Robust tracking of gestures enables intuitive teaching of surgical trainees. In collaboration with University Hospital Heidelberg (Müller et al. 2022).
Representative publications - Applications in Interventional Healthcare
Ayala, L. et al. Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery. Science Advances 2023. https://doi.org/10.1126/sciadv.add6778
Franz, A. M. et al. First clinical use of the EchoTrack guidance approach for radiofrequency ablation of thyroid gland nodules. International Journal of Computer Assisted Radiology and Surgery 2017. https://doi.org/10.1007/s11548-017-1560-2
Maier-Hein, L. et al. Surgical data science – from concepts toward clinical translation. Medical Image Analysis 2022. https://doi.org/10.1016/j.media.2021.102306
Müller, L.-R. et al. Robust hand tracking for surgical telestration. International Journal of Computer Assisted Radiology and Surgery 2022. https://doi.org/10.1007/s11548-022-02637-9
Open Science
Our division is strongly committed to the principle of open science.