Research

A core component of all interventional data science lies in the real-time perception of the environment. Current clinical decision-making is largely based on human perception relying mainly on visual and tactile feedback (palpation). This holds particularly true for the field of surgery. Most currently available medical imaging modalities commonly rely on the use of ionizing radiation, suffer from poor resolution or contrast and/or lack the capacity to operate in real time, thus not fulfilling the needs of an interventional environment. We challenge the current state of the art by proposing novel interventional imaging concepts based on biophotonics techniques. In this context we pioneered machine learning as an approach to solving the inverse problem of reconstructing clinically relevant tissue properties from optical or photoacoustic spectral measurements obtained with multispectral optical or photoacoustic measurements.

Further information

• European Research Council (ERC) Consolidator Grant NEURAL SPICING
• Publication highlights - Intelligent Systems in Spectral Imaging
• Publication highlights - Intelligent Systems in Photoacoustic Imaging

Representative publications

Baumann, A. et al. CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis. International Conference on Learning Representations (ICLR) 2026. https://openreview.net/forum?id=TpbhS1yfz0

Sellner, J. et al. Xeno-learning: knowledge transfer across species in deep learning-based spectral image analysis. Nature Biomedical Engineering 2026. https://doi.org/10.1038/s41551-025-01585-4 

Seidlitz, S. et al. AI-powered skin spectral imaging enables instant sepsis diagnosis and outcome prediction in critically ill patients. Science Advances 2025. https://doi.org/10.1126/sciadv.adw1968

Ayala, L., Adler, T. et al. Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery. Science Advances 2023. https://doi.org/10.1126/sciadv.add6778

Nölke, J-H. et al. Photoacoustic quantification of tissue oxygenation using conditional invertible neural networks. IEEE Transactions on Medical Imaging 2024. https://doi.org/10.1109/TMI.2024.3403417

 

To provide assistance to caregivers, the imaging data must be interpreted in the context of the relevant, available knowledge and according to key clinical questions. In this regard, the surgical domain faces several (often) unique challenges pertaining to data science methodology. Specifically, surgical data science often suffers from extremely high data variability and heterogeneity and a lack of representative, annotated data. To overcome this hurdle, we have pioneered different concepts to address data sparsity in the surgical domain, with a specific focus on synthetic data. As building trust in AI is key for bringing AI-based solutions into clinical practice, a particular second focus of our research has further been dedicated to explainable, uncertainty-aware machine learning, with a novel class of invertible neural networks serving as the core underlying technique.

Further information

• Publication highlights

Representative publications

Godau, P., Kalinowski, P.et al. Navigating prevalence shifts in image analysis algorithm deployment. Medical Image Analysis 2025. https://doi.org/10.1016/j.media.2025.103504

Yamlahi, A. et al. Self-distillation for surgical action recognition. Medical Image Computing and Computer Assisted Interventions (MICCAI) 2023. https://link.springer.com/chapter/10.1007/978-3-031-43996-4_61

Eisenmann, M. et al. Why Is the Winner the Best? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023. https://openaccess.thecvf.com/content/CVPR2023/html/Eisenmann_Why_Is_the_Winner_the_Best_CVPR_2023_paper.html 

Adler, T. J. et al. Out of Distribution Detection for Intra-operative Functional Imaging. Uncertainty for Safe Utilization of Machine Learning in Medical Imaging and Clinical Image-Based Procedures (MICCAI UNSURE) 2019. https://doi.org/10.1007/978-3-030-32689-0_8

Ardizzone, L. et al. Analyzing Inverse Problems with Invertible Neural Networks. International Conference on Learning Representations (ICLR) 2019. http://arxiv.org/abs/1808.04730

Maier-Hein, L. et al. Surgical data science for next-generation interventions. Nature Biomedical Engineering 2017. https://doi.org/10.1038/s41551-017-0132-7

Increasing evidence points to poor validation being one of the major reasons for the failure of AI-based solutions in clinical practice. Our division is working on new methodology and corresponding open source tools for the reliable and robust validation of algorithms.

Further information

• Publication highlights
• BIAS Initiative
• Metrics Reloaded
• Endoscopic Vision Challenge (EndoVis)
• ORena SAVE FOCUS challenge

Representative publications

Maier-Hein, L. et al. Metrics reloaded: Recommendations for image analysis validation. Nature Methods 2024. https://doi.org/10.1038/s41592-023-02151-z 

Reinke, A. et al. Understanding metric-related pitfalls in image analysis validation. Nature Methods 2024. https://doi.org/10.1038/s41592-023-02150-0 

Wiesenfarth, M. et al. Methods and open-source toolkit for analyzing and visualizing challenge results. Scientific Reports 2021. https://doi.org/10.1038/s41598-021-82017-6

Reinke, A. How to Exploit Weaknesses in Biomedical Challenge Design and Organization. Medical Image Computing and Computer Assisted Intervention (MICCAI) 2018. https://doi.org/10.1007/978-3-030-00937-3_45

Maier-Hein, L. et al. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications 2018. https://doi.org/10.1038/s41467-018-07619-7

The ultimate and most challenging goal regarding surgical data is their usage to provide real-time, context-aware assistance to the physician in and beyond the operating room. Clinical applications include:

• Instrument detection (1st prize MICCAI PitVis Workflow Recognition in Endoscopic Pituitary Surgery Challenge 2023)
• Surgical action recognition (1st prize MICCAI CholecTriplet Challenge 2022)
• Polyp detection (1st prize IEEE EndoCV Challenge 2022)
• Tissue classification (1st prize MICCAI GIANA Challenge 2021)
• Perfusion monitoring in laparoscopic kidney surgery (Ayala et al. 2023)
• AI-assisted surgical training (Müller et al. 2022)
• Ultrasound-navigated percutaneous needle insertions (Franz et al. 2017)

Further information

• Publication highlights
• NCT Surgical Oncology Program

Representative publications

Ayala, L. et al. Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery. Science Advances 2023. https://doi.org/10.1126/sciadv.add6778

Franz, A. M. et al. First clinical use of the EchoTrack guidance approach for radiofrequency ablation of thyroid gland nodules. International Journal of Computer Assisted Radiology and Surgery 2017. https://doi.org/10.1007/s11548-017-1560-2

Maier-Hein, L. et al. Surgical data science – from concepts toward clinical translation. Medical Image Analysis 2022. https://doi.org/10.1016/j.media.2021.102306

Müller, L.-R. et al. Robust hand tracking for surgical telestration. International Journal of Computer Assisted Radiology and Surgery 2022. https://doi.org/10.1007/s11548-022-02637-9

Our division is committed to making its research more environmentally sustainable, with a particular focus on reducing the environmental impact of our computing activities. In recognition of these efforts, IMSY has been awarded the Green DiSC Bronze certification, an externally assessed certification for sustainable computational research.

… more information on the Green DiSC certification