Computer Assisted Medical Interventions

COMBIOSCOPY: Computational biophotonics in endoscopic cancer diagnosis and therapy

< Back to Research overview


The goal of the COMBIOSCOPY project is to develop new concepts for interventional imaging that (1) provide fast discrimination of local tissue with a high contrast-to-noise-ratio  (2) are radiation-free to prevent the patient and staff from being exposed to harmful ionizing radiation and (3) feature a compact design at a low cost for a wide range of applicability and acceptance.  This shall be achieved by the systematic integration and mutual enhancement of biophotonics and computer assisted intervention techniques. Core of the project are novel machine learning - based algorithms that convert high-dimensional multispectral data acquired with multispectral optical and photoacoustic imaging techniques into intuitive information that can be used by physicians for real-time clinical decision making.

Multispectral optical imaging

Replacing traditional open surgery with minimally-invasive techniques for complicated interventions such as partial tumor resection or anastomosis one of the most important challenges in modern healthcare. In these and many other procedures, characterization of the tissue perfusion and oxygenation remains challenging by means of visual inspection. Conventional laparoscopes are limited by “imitating” the human eye; multispectral cameras remove this arbitrary restriction of recording only red, green and blue colors. Instead, they capture many specific bands of light. Although these could help characterize important indications such as ischemia the lack of powerful digital image processing and bulky, nonstandard equipment prevents realizing the full potential of multispectral imaging (MSI) to date.

In the scope of the COMBIOSCOPY project, we developed the first multispectral laparoscopic imaging setup featuring  (1) a compact and lightweight laparoscope built from commercially available parts which is straightforward to assemble and  (2) the possibility to complement the conventional surgical view on the patient (RGB video images) with relevant morphological and functional information at an imaging rate of 30Hz (Fig. 1). By combining the first video-rate capable multispectral sensor with advanced image processing techniques, our approach pioneers fluent perfusion monitoring and tissue discrimination with multispectral imaging.

One of the main challenges that we addressed in the project is the absence of a quantitative reference for the functional parameters - traditionally a key requirement for applying machine learning techniques. To overcome this hurdle and enable functional multispectral biophotonics, for the first time in this field physical models are combined with powerful machine learning techniques [1,13]. The physical model is employed to create highly accurate simulations, which in turn teach the algorithm to relate multispectral pixels to underlying functional changes.

Further scientific contributions are related to automatic band selection [14], (2) machine learning-based tissue classification [3,10] and  (3) the systematic analysis of solutions to inverse problems [16].

Figure 1. Machine learning - based real-time quantification of tissue oxygenation in laparoscopic surgery.

Multispectral photoacoustic imaging

While multispectral optical imaging is a passive technique that yields 2D reflection images but requires no contact with the tissue and no additional illumination unit, multispectral photoacoustic imaging has the advantage of providing tomographic images at a depth range of several centimeters. One of the primary obstacles towards clinical translation of PAT in these scenarios is that optical properties cannot be quantified reliably. The measured initial pressure distribution, even though proportional to the absorption of chromophores, is also dependent on the light fluence which is in turn highly dependent on the distribution of chromophores and scatterers within the tissue. As a consequence, quantification of chromophore concentrations from the underlying initial pressure distribution is an ill-posed inverse problem. Current model based approaches are far from satisfactory when dealing with this problem as they are either too slow, inaccurate or make assumptions that do not hold when the method is applied to clinical settings. To address this major challenge, we presented the first machine learning based method for quantitative PAT (Kirchner et al. 2018). The method is based on encoding the relevant context for each voxel in a so-called context image. A machine learning algorithm uses the context images to estimate the fluence in each voxel, which – in turn – can be used to correct the optical absorption measured. The patented [A] concept has yielded extremely promising results in silico and thus has high potential for solving one of the primary issues related to PAT.

Further scientific contributions in the context of PAT are related to 3D photoacoustic vessel angiography with optical tracking [4], uncertainty estimation in quantitative photoacoustic imaging [7], and photoacoustic image reconstruction from time series data [9].


  • Dominik Waibel (Masters)
  • Angelika Laha (Masters)
  • Sara Moccia (PhD intern)
  • Yan Zhang (Masters)
  • Justin Iszatt (Bachelors)

Key collaborators

  • Dr. Daniel S. Elson and Dr. Neil T. Clancy
    Imperial College London, Hamlyn Centre for Robotic Surgery

  • Dr. Peter Sauer
    University of Heidelberg, Interdisciplinary Endoscopy Centre

  • Dr. Hannes Kenngott
    University of Heidelberg, Division of Minimally-invasive Surgery of the Department of General Surgery

  • PD Dr. Dogu Teber and Dr. Tobias Simpfendörfer
    University of Heidelberg, Department of Urology

  • Prof. Dr. Carsten Rother
    Head of Visual Learning Lab Heidelberg

  • M.D. Dr. Med. Edgar Santos
    Universität Heidelberg, University Clinic for Neurosurgery

Selected publications

  1. Ardizzone L., Kruse J., Wirkert S., Rahner D., Pellegrini E. W., Klessen R. S., Maier-Hein L., Rother C., Köthe U. (2018), arXiv:1808.04730v1, Analyzing Inverse Problems with Invertible Neural Networks. (ArXiv:

  2. Gröhl, J, Kirchner, T., and Maier-Hein, L. (2018) Confidence Estimation for Quantitative Photoacoustic Imaging. In Photons Plus Ultrasound: Imaging and Sensing 2018, 10494:104941C. International Society for Optics and Photonics, 2018. (DOI:

  3. Jaeger, A.H., Franz, A.M., O’Donoghue, K., Seitel, A., Trauzettel, F., Maier-Hein, L., Cantillon-Murphy, P. (2017) Anser-EMT – The first open-source electromagnetic tracking platform for image-guided interventions. In: Int J CARS. (DOI:

  4. Kirchner, T., Gröhl, J., Sattler, F., Bischoff, M. S., Laha, A., Nolden, M., & Maier-Hein, L. (2018). Real-time in vivo blood oxygenation measurements with an open-source software platform for translational photoacoustic research (Conference Presentation). In Photons Plus Ultrasound: Imaging and Sensing 2018 (Vol. 10494, p. 1049407). International Society for Optics and Photonics. (DOI:

  5. Kirchner, T., Gröhl, J., and Maier-Hein, L. Context Encoding Enables Machine Learning-Based Quantitative Photoacoustics. Journal of Biomedical Optics 23, no. 5 (2018): 056008. (DOI:

  6. Kirchner, T., Wild E., Maier-Hein, K.H., Maier-Hein, L. (2016) Freehand photoacoustic tomography for 3D angiography using local gradient information. Proc. SPIE 9708, Photons Plus Ultrasound: Imaging and Sensing 2016, 97083G. (DOI:

  7. Klemm, M., Kirchner, T., Gröhl, J., Cheray D., Nolden M., Seitel A., Hoppe, H., Maier-Hein, L., Franz, A. M. (2016) MITK-OpenIGTLink for combining open-source toolkits in real-time computer-assisted interventions, Int J CARS. (DOI:

  8. Lin, J., Clancy, N.T., Qi J., Hu, Y., Tatla, T., Stoyanov, D., Hein L.M., Elson, D.S. (2018). Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks. Medical Image Analysis. (DOI:

  9. Lin J, Clancy NT, Hu Y, Qi J, Tatla T, Stoyanov D, Maier-Hein L, Elson DS. (2017) Endoscopic depth measurement and super-spectral-resolution imaging. InInternational Conference on Medical Image Computing and Computer-Assisted Intervention 2017 Sep 10 (pp. 39-47). Springer, Cham.(DOI:

  10. Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A.P., Carass, A., Feldmann, C., Frangi, A. F., Full, P.M., Ginneken, B., Hanbury, A., Honauer, K., Kozubek, M., Landman, B.A., März, K., MaierO., Maier-Hein K., Menze, B.H., Müller, H., Neher, P.F., Niessen, W., Rajpoot, N., Sharp, G.C., Sirinukunwattana, K., Speidel, S., Stock, C., Stoyanov, D., Taha, A.A., Sommen, F., Wang, C., Weber, M., Zheng, G., Jannin, P., Kopp-Schneider, A. Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions. arXiv preprint. (ArXiv:

  11. Moccia, S., Wirkert, S.J., Kenngott, H., Vemuri, A.S., Apitz, M., Mayer, B., De Momi, E., Mattos, L.S., Maier-Hein, L. (2018). Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy. In IEEE Transactions on Biomedical engineering, (pre-print) (DOI:

  12. Sattler, F., Kirchner, T., Gröhl, J., & Maier-Hein, L. (2018). Real-time delay multiply and sum beamforming for multispectral photoacoustics (Conference Presentation). In Photons Plus Ultrasound: Imaging and Sensing 2018 (Vol. 10494, p. 104942Q). International Society for Optics and Photonics. (DOI:

  13. Vemuri, A.S., Wirkert S.J., Maier-Hein, L. (2018) Hyerspectral camera selection for health care applications. Journal of Biomedical Optics. (under review).

  14. Waibel, D., Gröhl, J., Isensee, F., Kirchner, T., Maier-Hein, K., and Maier-Hein, L. (2018) Reconstruction of Initial Pressure from Limited View Photoacoustic Images Using Deep Learning. In Photons Plus Ultrasound: Imaging and Sensing 2018, 10494:104942S. International Society for Optics and Photonics, 2018. (DOI:

  15. Wirkert, S.J., Isensee, F., Vemuri, A.S., Maier-Hein, K., Fei, B., Maier-Hein, L., (2018) Domain and task specific multispectral band selection, Proc. SPIE 10486, Design and Quality for Biomedical Technologies XI, 104860H (14 March 2018). (DOI:

  16. Wirkert, S.J., Vemuri, A.S., Kenngott, H.G., Moccia, S., Götz, M., Mayer, B.F.B., Maier-Hein, K.H., Elson, D.S., Maier-Hein, L. (2017). Physiological Parameter Estimation from Multispectral Images Unleashed. In: Medical Image Computing and Computer-Assisted Intervention ? MICCAI 2017, Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L. Duchesne, S., eds. (Cham: Springer International Publishing) pp. 134-141. (DOI:

  17. Wirkert, S.J., Kenngott, H., Mayer, B., Mietkowski, P., Wagner, M., Sauer, P., Clancy, N.T., Elson, D.S., Maier-Hein, L. (2016). Robust near real-time estimation of physiological parameters from megapixel multispectral images with inverse Monte Carlo and random forest regression. In: Int J. CARS (Special Issue: IPCAI), pp. 909-917. (DOI:

  18. Wirkert, S.J., Clancy, N.T., Stoyanov, D., Arya, S., Hanna, G.B., Schlemmer, H.-P., Sauer, P., Elson, D.S., and Maier-Hein, L. (2014). Endoscopic Sheffield Index for Unsupervised In Vivo Spectral Band Selection. In: Computer-Assisted and Robotic Endoscopy, X. Luo, T. Reichl, D. Mirota, and T. Soper, eds. (Cham: Springer International Publishing), pp. 110-120. (DOI:

  19. Zhang, Y., Wirkert, S.J., Iszatt, J., Kenngott H., Wagner, M., Mayer, B., Stock, C., Clancy, N.T., Elson, D.S., Maier-Hein, L. (2017). Tissue classification for laparoscopic image understanding based on multispectral texture analysis. In: J. Medical Imaging. (DOI:


  1. Maier-Hein, L., Kirchner, T., Groehl, J., MACHINE LEARNING-BASED QUANTITATIVE PHOTOACOUSTIC TOMOGRAPHY (PAT) european patent pending under no. 16177204.1


  • Best Poster Award Annual PhD Retreat of the German Cancer Research Center (2018)
    Janek Gröhl, for work on photoacoustic imaging.
  • Berlin-Brandenburg Academy Prize (2017)
    Lena Maier-Hein, for new imaging techniques to differentiate tumors more reliably from healthy tissue and perform safer surgical anticancer therapies, sponsored by the Monika Kutzner Foundation.

  • Best pitch at Science Sparks Start-ups (2017)
    Sebastian Wirkert, Anant Vemuri and Lena Maier-Hein, for "Rainbow Surgery", sponsored by Phenex Pharmaceuticals AG.

  • conhIT-Nachwuchspreis (2017)
    Janek Gröhl, for the best Masters thesis "Machine learning based quantitative photoacoustic tomography", awarded by conhIT

  • Emil Salzer Prize (2016)
    Prof. Dr. Lena Maier-Hein for "Using sound and light for navigating inside the body", awarded by DKFZ on behalf of Baden-Wuerttemberg’s Ministry of Science, Research and the Arts.

  • Thomas-Gessmann-Förderpreis (2015) 
    Justin Iszatt, for Master Thesis "Multispektrale Bildgebung in der Medizin - Entwicklung eines multispektralen Laparoskops zur Schätzung des Sauerstoffgehalts in Geweben", awarded by the Thomas Gessmann-Stiftung
  • KUKA 2nd place Award for Best Paper (2014)
    Sebastian Wirkert et al., for the paper "Endoscopic Sheffield Index for Unsuspervised In Vivo Spectral Band Selection", awarded at the MICCAI CARE workshop.

Keynotes and Invited Talks on COMBIOSCOPY

07 / 2018                The International Workshop of Medical Imaging (Moscow, Russia)
06 / 2018                GNB 2018 - Sixth National Congress of Bioengineering (Milano, Italy)
02 / 2018                Medical Information, Information Retrieval, and Data Sciences (Toulouse, France)
03 / 2017                134th Annual Congress of the German Society for Surgery (Munich, Germany)
02 / 2016                Dutch society of Pattern Recognition and Image Processing - Spring 2016 Meeting (Rotterdam, The Netherlands)

Invited talks on COMBIOSCOPY

07 / 2018                Beyond Gynecological Surgery Congress (Clermont-Ferrand, France)

11 / 2017                AIS Challenge: Live Surgery & The Operating Theatre of the Future (online talk)

11 / 2016                First European Workshop of MedTech Alsace (Strasbourg, France)

09 / 2016                Meet and Match on Optical Imaging (Mannheim, Germany)

09 / 2016                European Health Science Match (Heidelberg, Germany)

02 / 2016                3rd EMBO Conference on Visualizing Biological Data (Heidelberg, Germany)

02 / 2016                BioPro Baden-Württemberg Meet and Match: Optical Imaging: Future Trends in Medical Applications (Mannheim, Germany)


to top