Computer Assisted Medical Interventions

Surgical Multispectral Imaging

Replacing traditional open surgery with minimally-invasive techniques for complicated interventions such as partial tumor resection is one of the most important challenges in modern healthcare. In these and many other procedures, characterization of the tissue perfusion and oxygenation remains challenging by means of visual inspection. Conventional laparoscopes are limited by "imitating" the human eye. Multispectral cameras remove this arbitrary restriction of recording only red, green and blue colors. Instead, they capture many specific bands of light. Although multispectral imaging (MSI) could help characterize important indications such as ischemia, the lack of powerful digital image processing and convenient, standardized equipment prevents realizing the full potential of MSI to date.
In the scope of the COMBIOSCOPY project, we developed the first video-rate, multispectral, laparoscopic imaging device by combining a standard laparoscope with a multispectral sensor, which is capable to capture the relevant morphological and functional information at an imaging rate of 30 Hz. (Fig. 1). By combining this device with advanced image processing techniques (Wirkert et al., 2016; Wirkert et al. 2017; Wirkert et al., 2019), our approach pioneers fluent perfusion monitoring and tissue discrimination with multispectral imaging.

Fig. 1: Machine learning-based real-time quantification of tissue oxygenation in laparoscopic surgery.

One of the main challenges that we addressed in the project is the absence of a quantitative reference for the functional parameters. The availability of such labeled data is traditionally a key requirement for applying machine learning techniques. To overcome this hurdle and to enable functional multispectral imaging, for the first time in this field, physical models are combined with powerful machine learning techniques. The physical model is employed to create multispectral reflectance spectra, which in turn teach the algorithm to relate multispectral reflectance measurements to underlying functional tissue parameters. The benefit of the methodology is currently being investigated for various clinical applications including perfusion monitoring in minimally-invasive surgery and visualization of hemodynamic changes in the brain, as shown in Fig. 2. It shows a phenomenon in the brain called Spreading Depolarization (SD), which is characterized by the depolarization of the neurons that causes a wave of low oxygenation that travel in the cortex of the brain.

Fig. 2: Tissue oxygenation map (%SO2) on the brain. Spreading depolarizations can be seen traveling in the surface of the brain.

Further scientific contributions are related to automatic band selection (Wirkert et al., 2014; Wirkert, Isensee, et al. 2018; Wirkert et al., 2019) machine learning-based tissue classification (Moccia et al., 2018), automatic light source calibration (Ayala et al., 2020) and the systematic analysis of solutions to inverse problems (Ardizzone et al., 2019; Adler2019 et al., 2019).


  • Sebastian Pirmann (Master's Student)
  • Dr. Anant Vemuri (Scientist)
  • Dr. Sebastian Wirkert (Doctoral Student)
  • Dr. Sara Moccia (PhD intern)
  • Yan Zhang (Master's Student)
  • Justin Iszatt (Bachelor's Student)

Key Collaborators

  • Prof. Dr. med. Dogu Teber, Karlsruhe Municipal Hospital
  • Prof. Dr. med. Beat Müller, Department for General, Visceral and Transplant Surgery, Section for Minimally Invasive Surgery, Heidelberg University Hospital
  • PD. Dr. med. Felix Nickel, Department for General, Visceral and Transplant Surgery, Section for Minimally Invasive Surgery, Heidelberg University Hospital
  • Dr. med Hannes Kenngott, Department for General Visceral and Transplant Surgery, Heidelberg University Hospital
  • MD, Dr. med. Edgar Santos, Department for Neurosurgery, Heidelberg University Hospital
  • Dr. med. Peter Sauer, Interdisciplinary Endoscopy Center, Heidelberg University Hospital
  • Dr. Daniel S. Elson, Hamlyn Centre for Robotic Surgery, Institute of Global Health Innovation, Imperial College London, UK; Department of Surgery and Cancer, Imperial College London, UK
  • Dr. Neil T. Clancy, Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, UK; Centre for Medical Image Computing (CMIC), Department of Medical Physics and Biomedical Engineering, University College London, UK
  • Prof. Dr. Danail Stoyanov, Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, UK; Centre for Medical Image Computing (CMIC), Department of Computer Science, University College London, UK
  • Prof. Dr. Carsten Rother, Head of Visual Learning Lab Heidelberg
  • Diaspective Vision GmbH


Adler, T. J., Ardizzone, L., Vemuri, A., Ayala, L., Gröhl, J., Kirchner, T., Wirkert, S., Kruse, J., Rother, C., Köthe, U., & Maier-Hein, L. (2019). Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks. International Journal of Computer Assisted Radiology and Surgery, 14(6), 997–1007.

Ardizzone, L., Kruse, J., Rother, C., Köthe, U., & the. (2019). Analyzing Inverse Problems with Invertible Neural Networks. International Conference on Learning Representations.

Ayala, L. A., Wirkert, S. J., Gröhl, J., Herrera, M. A., Hernandez-Aguilera, A., Vemuri, A., Santos, E., & Maier-Hein, L. (2019). Live Monitoring of Haemodynamic Changes with Multispectral Image Analysis. In OR 2.0 Context-Aware Operating Theaters and Machine Learning in Clinical Neuroimaging (pp. 38–46).

Ayala, L., Seidlitz, S., Vemuri, A., Wirkert, S. J., Kirchner, T., Adler, T. J., Engels, C., Teber, D., & Maier-Hein, L. (2020). Light source calibration for multispectral imaging in surgery. International Journal of Computer Assisted Radiology and Surgery, 15(7), 1117–1125.

Ayala, L., Wirkert, S., Herrera, M., Hernández-Aguilera, A., Vermuri, A., Santos, E., & Maier-Hein, L. (2019). Abstract: Multispectral Imaging Enables Visualization of Spreading Depolarizations in Gyrencephalic Brain. In Bildverarbeitung für die Medizin 2019 (pp. 244–244).

Clancy, N. T., Jones, G., Maier-Hein, L., Elson, D. S., & Stoyanov, D. (2020). Surgical spectral imaging. Medical Image Analysis, 63, 101699.

Lin, J., Clancy, N. T., Qi, J., Hu, Y., Tatla, T., Stoyanov, D., Maier-Hein, L., & Elson, D. S. (2018). Dual-modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks. Medical Image Analysis, 48, 162–176.

Moccia, S., Wirkert, S. J., Kenngott, H., Vemuri, A. S., Apitz, M., Mayer, B., De Momi, E., Mattos, L. S., & Maier-Hein, L. (2018). Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy. IEEE Transactions on Biomedical Engineering, 65(11), 2649–2659.

Wirkert, Sebastian J., Clancy, N. T., Stoyanov, D., Arya, S., Hanna, G. B., Schlemmer, H.-P., Sauer, P., Elson, D. S., & Maier-Hein, L. (2014). Endoscopic Sheffield Index for Unsupervised In Vivo Spectral Band Selection. In Computer-Assisted and Robotic Endoscopy (pp. 110–120).

Wirkert, Sebastian J., Isensee, F., Vemuri, A. S., Ayala, L. A., Maier-Hein, K. H., Fei, B., & Maier-Hein, L. (2019). Task-specific multispectral band selection. ArXiv:1905.11297 [Physics].

Wirkert, Sebastian J., Isensee, F., Vemuri, A. S., Maier-Hein, K., Fei, B., & Maier-Hein, L. (2018). Domain and task specific multispectral band selection (Conference Presentation). Design and Quality for Biomedical Technologies XI, 10486, 104860H.

Wirkert, Sebastian J., Kenngott, H., Mayer, B., Mietkowski, P., Wagner, M., Sauer, P., Clancy, N. T., Elson, D. S., & Maier-Hein, L. (2016). Robust near real-time estimation of physiological parameters from megapixel multispectral images with inverse Monte Carlo and random forest regression. International Journal of Computer Assisted Radiology and Surgery, 11(6), 909–917.

Wirkert, Sebastian J., Vemuri, A. S., Kenngott, H. G., Moccia, S., Götz, M., Mayer, B. F. B., Maier-Hein, K. H., Elson, D. S., & Maier-Hein, L. (2017). Physiological Parameter Estimation from Multispectral Images Unleashed. In Medical Image Computing and Computer Assisted Intervention − MICCAI 2017 (pp. 134–141).

Wirkert, Sebastian Josef. (2018b). Multispectral image analysis in laparoscopy – A machine learning approach to live perfusion monitoring [PhD Thesis, Karlsruher Institut für Technologie (KIT)].

Zhang, Y., Wirkert, S., Iszatt, J., Kenngott, H., Wagner, M., Mayer, B., Stock, C., Clancy, N. T., Elson, D. S., & Maier-Hein, L. (2017). Tissue classification for laparoscopic image understanding based on multispectral texture analysis. Journal of Medical Imaging, 4(1), 015001.


Maier-Hein, L., Wirkert, S., Vemuri, A., Ayala, L. Seidlitz, S., Kirchner, T., Adler, T., Method and system for augmented imaging in open treatment using multispectral information, WO2020025684A1 (Pending)


Bench to Bedside Award at MICCAI workshop: OR2.0 Context-Aware Operating Theaters (2019)
L. Ayala et al. for his paper "Life Monitoring of Hemodynamic Changes with Multispectral Image Analysis"

SMIT Young Investigator Award (2019)
L. Ayala et al. for his paper "Deep Learning Approach to live Monitoring of Hemodynamic Changes with Multispectral Image Analysis"

BVM Award (2019)
S. Wirkert for his doctoral thesis "Multispectral Image Analysis in Laparoscopy - A Machine Learning Approach"

Berlin-Brandenburg Academy Prize (2017)
L. Maier-Hein for outstanding achievements in cancer research

Best Pitch at Science Sparks Startups (2017)
S. Wirkert, A. Vemuri for their contribution "Rainbow Surgery"

Emil Salzer Prize (2016)
Lena Maier-Hein for her contributions to the fields of computer science, physics and medicine

Thomas-Gessmann Prize (2015)
Justin Iszatt for the Bachelor's thesis "Multispektrale Bildgebung in der Medizin - Entwicklung eines multispektralen Laparoskops zur Schätzung des Sauerstoffgehalts in Geweben"

KUKA Best Paper Award at the MICCAI CARE workshop (2014)
S. Wirkert et al. for his paper "Endoscopic Sheffield Index for Unsuspervised In Vivo Spectral Band Selection"

to top
powered by webEdition CMS