Computer Assisted Medical Interventions

COMBIOSCOPY: Computational biophotonics in endoscopic cancer diagnosis and therapy

Key challenges in endoscopic tumor diagnosis and therapy consist of the detection and discrimination of malignant tissue as well as the precise navigation of medical instruments. Currently, a low level of sensitivity and specificity in tumor detection and lack of global orientation lead to both over- and undertreatment, tumor recurrence, intra-operative complications, and high costs. The goal of this multidisciplinary project is to revolutionize clinical endoscopic imaging based on the systematic integration of two new but independant fields of research up until this point: Biophotonics and computer-assisted interventions (COMputational BIOphotonics in endoSCOPY).

For the first time, quantitative multi-modal imaging biomarkers based on structural and functional data are being developed to enhance the physician’s view by providing information that cannot be seen with the naked eye. To this extent, white light images co-registered with multispectral optical and photoacoustic images will be processed in a combined manner to dynamically reconstruct not only the visible surface in 3D but also subsurface anatomical and functional detail such as 3D vessel topology, blood volume and oxygenation. Spatio-temporal registration of multi-modal data acquired before and during the procedure will enable (1) the highly specific local tissue classification and discrimination based on tissue shape, texture, function and radiological contrast imagery as well as (2) global context-aware instrument guidance.

This innovative approach to radiation-free real-time imaging will be implemented and evaluated by means of computer-assisted colonoscopy and laparoscopy. The potential socioeconomic impact of providing high precision minimally-invasive tumor diagnosis and therapy at low cost is extremely high.

Figure 1. Machine learning - based real-time quantification of tissue oxygenation in laparoscopic surgery.


  • Dr. Anant Vemuri (Scientist)
  • Dr. Thomas Kirchner (Doctoral Student)
  • Franz Sattler (Student Assistant)
  • Dr. Sebastian Wirkert (Doctoral Student)
  • Dominik Waibel (Master's student)
  • Angelika Laha (Master's student)
  • Dr. Sara Moccia (PhD intern)
  • Yan Zhang (Master's student)
  • Justin Iszatt (Bachelor's student)

Key collaborators

  • Dr. Daniel S. Elson and Dr. Neil T. Clancy
    Imperial College London, Hamlyn Centre for Robotic Surgery

  • Dr. Peter Sauer
    University of Heidelberg, Interdisciplinary Endoscopy Centre

  • Dr. Hannes Kenngott
    University of Heidelberg, Division of Minimally-invasive Surgery of the Department of General Surgery

  • PD Dr. Dogu Teber and Dr. Tobias Simpfendörfer
    University of Heidelberg, Department of Urology

  • Prof. Dr. Carsten Rother
    Head of Visual Learning Lab Heidelberg

  • M.D. Dr. Med. Edgar Santos
    Universität Heidelberg, University Clinic for Neurosurgery

Selected publications

  1. Ardizzone L., Kruse J., Wirkert S., Rahner D., Pellegrini E. W., Klessen R. S., Maier-Hein L., Rother C., Köthe U. (2018), arXiv:1808.04730v1, Analyzing Inverse Problems with Invertible Neural Networks. (ArXiv:

  2. Gröhl, J, Kirchner, T., and Maier-Hein, L. (2018) Confidence Estimation for Quantitative Photoacoustic Imaging. In Photons Plus Ultrasound: Imaging and Sensing 2018, 10494:104941C. International Society for Optics and Photonics, 2018. (DOI:

  3. Jaeger, A.H., Franz, A.M., O’Donoghue, K., Seitel, A., Trauzettel, F., Maier-Hein, L., Cantillon-Murphy, P. (2017) Anser-EMT – The first open-source electromagnetic tracking platform for image-guided interventions. In: Int J CARS. (DOI:

  4. Kirchner, T., Gröhl, J., Sattler, F., Bischoff, M. S., Laha, A., Nolden, M., & Maier-Hein, L. (2018). Real-time in vivo blood oxygenation measurements with an open-source software platform for translational photoacoustic research (Conference Presentation). In Photons Plus Ultrasound: Imaging and Sensing 2018 (Vol. 10494, p. 1049407). International Society for Optics and Photonics. (DOI:

  5. Kirchner, T., Gröhl, J., and Maier-Hein, L. Context Encoding Enables Machine Learning-Based Quantitative Photoacoustics. Journal of Biomedical Optics 23, no. 5 (2018): 056008. (DOI:

  6. Kirchner, T., Wild E., Maier-Hein, K.H., Maier-Hein, L. (2016) Freehand photoacoustic tomography for 3D angiography using local gradient information. Proc. SPIE 9708, Photons Plus Ultrasound: Imaging and Sensing 2016, 97083G. (DOI:

  7. Klemm, M., Kirchner, T., Gröhl, J., Cheray D., Nolden M., Seitel A., Hoppe, H., Maier-Hein, L., Franz, A. M. (2016) MITK-OpenIGTLink for combining open-source toolkits in real-time computer-assisted interventions, Int J CARS. (DOI:

  8. Lin, J., Clancy, N.T., Qi J., Hu, Y., Tatla, T., Stoyanov, D., Hein L.M., Elson, D.S. (2018). Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks. Medical Image Analysis. (DOI:

  9. Lin J, Clancy NT, Hu Y, Qi J, Tatla T, Stoyanov D, Maier-Hein L, Elson DS. (2017) Endoscopic depth measurement and super-spectral-resolution imaging. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention 2017 Sep 10 (pp. 39-47). Springer, Cham.(DOI:

  10. Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A.P., Carass, A., Feldmann, C., Frangi, A. F., Full, P.M., Ginneken, B., Hanbury, A., Honauer, K., Kozubek, M., Landman, B.A., März, K., MaierO., Maier-Hein K., Menze, B.H., Müller, H., Neher, P.F., Niessen, W., Rajpoot, N., Sharp, G.C., Sirinukunwattana, K., Speidel, S., Stock, C., Stoyanov, D., Taha, A.A., Sommen, F., Wang, C., Weber, M., Zheng, G., Jannin, P., Kopp-Schneider, A. Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions. arXiv preprint. (ArXiv:

  11. Moccia, S., Wirkert, S.J., Kenngott, H., Vemuri, A.S., Apitz, M., Mayer, B., De Momi, E., Mattos, L.S., Maier-Hein, L. (2018). Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy. In IEEE Transactions on Biomedical engineering, (pre-print) (DOI:

  12. Sattler, F., Kirchner, T., Gröhl, J., & Maier-Hein, L. (2018). Real-time delay multiply and sum beamforming for multispectral photoacoustics (Conference Presentation). In Photons Plus Ultrasound: Imaging and Sensing 2018 (Vol. 10494, p. 104942Q). International Society for Optics and Photonics. (DOI:

  13. Vemuri, A.S., Wirkert S.J., Maier-Hein, L. (2018) Hyerspectral camera selection for health care applications. Journal of Biomedical Optics. (under review).

  14. Waibel, D., Gröhl, J., Isensee, F., Kirchner, T., Maier-Hein, K., and Maier-Hein, L. (2018) Reconstruction of Initial Pressure from Limited View Photoacoustic Images Using Deep Learning. In Photons Plus Ultrasound: Imaging and Sensing 2018, 10494:104942S. International Society for Optics and Photonics, 2018. (DOI:

  15. Wirkert, S.J., Isensee, F., Vemuri, A.S., Maier-Hein, K., Fei, B., Maier-Hein, L., (2018) Domain and task specific multispectral band selection, Proc. SPIE 10486, Design and Quality for Biomedical Technologies XI, 104860H (14 March 2018). (DOI:

  16. Wirkert, S.J., Vemuri, A.S., Kenngott, H.G., Moccia, S., Götz, M., Mayer, B.F.B., Maier-Hein, K.H., Elson, D.S., Maier-Hein, L. (2017). Physiological Parameter Estimation from Multispectral Images Unleashed. In: Medical Image Computing and Computer-Assisted Intervention ? MICCAI 2017, Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L. Duchesne, S., eds. (Cham: Springer International Publishing) pp. 134-141. (DOI:

  17. Wirkert, S.J., Kenngott, H., Mayer, B., Mietkowski, P., Wagner, M., Sauer, P., Clancy, N.T., Elson, D.S., Maier-Hein, L. (2016). Robust near real-time estimation of physiological parameters from megapixel multispectral images with inverse Monte Carlo and random forest regression. In: Int J. CARS (Special Issue: IPCAI), pp. 909-917. (DOI:

  18. Wirkert, S.J., Clancy, N.T., Stoyanov, D., Arya, S., Hanna, G.B., Schlemmer, H.-P., Sauer, P., Elson, D.S., and Maier-Hein, L. (2014). Endoscopic Sheffield Index for Unsupervised In Vivo Spectral Band Selection. In: Computer-Assisted and Robotic Endoscopy, X. Luo, T. Reichl, D. Mirota, and T. Soper, eds. (Cham: Springer International Publishing), pp. 110-120. (DOI:

  19. Zhang, Y., Wirkert, S.J., Iszatt, J., Kenngott H., Wagner, M., Mayer, B., Stock, C., Clancy, N.T., Elson, D.S., Maier-Hein, L. (2017). Tissue classification for laparoscopic image understanding based on multispectral texture analysis. In: J. Medical Imaging. (DOI:


  1. Maier-Hein, L., Kirchner, T., Groehl, J., MACHINE LEARNING-BASED QUANTITATIVE PHOTOACOUSTIC TOMOGRAPHY (PAT) european patent pending under no. 16177204.1


  • Best Poster Award Annual PhD Retreat of the German Cancer Research Center (2018)
    Janek Gröhl, for work on photoacoustic imaging.
  • Berlin-Brandenburg Academy Prize (2017)
    Lena Maier-Hein, for new imaging techniques to differentiate tumors more reliably from healthy tissue and perform safer surgical anticancer therapies, sponsored by the Monika Kutzner Foundation.

  • Best pitch at Science Sparks Start-ups (2017)
    Sebastian Wirkert, Anant Vemuri and Lena Maier-Hein, for "Rainbow Surgery", sponsored by Phenex Pharmaceuticals AG.

  • conhIT-Nachwuchspreis (2017)
    Janek Gröhl, for the best Masters thesis "Machine learning based quantitative photoacoustic tomography", awarded by conhIT

  • Emil Salzer Prize (2016)
    Prof. Dr. Lena Maier-Hein for "Using sound and light for navigating inside the body", awarded by DKFZ on behalf of Baden-Wuerttemberg’s Ministry of Science, Research and the Arts.

  • Thomas-Gessmann-Förderpreis (2015) 
    Justin Iszatt, for Master Thesis "Multispektrale Bildgebung in der Medizin - Entwicklung eines multispektralen Laparoskops zur Schätzung des Sauerstoffgehalts in Geweben", awarded by the Thomas Gessmann-Stiftung
  • KUKA 2nd place Award for Best Paper (2014)
    Sebastian Wirkert et al., for the paper "Endoscopic Sheffield Index for Unsuspervised In Vivo Spectral Band Selection", awarded at the MICCAI CARE workshop.

Keynotes and Invited Talks on COMBIOSCOPY

07 / 2018                The International Workshop of Medical Imaging (Moscow, Russia)
07 / 2018                Beyond Gynecological Surgery Congress (Clermont-Ferrand, France)
06 / 2018                GNB 2018 - Sixth National Congress of Bioengineering (Milano, Italy)
02 / 2018                Medical Information, Information Retrieval, and Data Sciences (Toulouse, France)
11 / 2017                AIS Challenge: Live Surgery & The Operating Theatre of the Future (online talk)
03 / 2017                134th Annual Congress of the German Society for Surgery (Munich, Germany)
11 / 2016                First European Workshop of MedTech Alsace (Strasbourg, France)
09 / 2016                Meet and Match on Optical Imaging (Mannheim, Germany)
09 / 2016                European Health Science Match (Heidelberg, Germany)
02 / 2016                3rd EMBO Conference on Visualizing Biological Data (Heidelberg, Germany)
02 / 2016                BioPro Baden-Württemberg Meet and Match: Optical Imaging: Future Trends in Medical Applications (Mannheim, Germany)
02 / 2016                Dutch society of Pattern Recognition and Image Processing - Spring 2016 Meeting (Rotterdam, The Netherlands)

to top
powered by webEdition CMS