Publication highlights - Intelligent Systems in Spectral Imaging

Main contribution: We propose a new deep-learning method to automatically recalibrate hyperspectral images during surgery, thus eliminating the tedious process of white reference re-calibration.

Hyperspectral imaging (HSI) is emerging as a promising novel imaging modality with various potential surgical applications. Currently available cameras, however, suffer from poor integration into the clinical workflow because they require the lights to be switched off, or the camera to be manually recalibrated as soon as lighting conditions change. Given this critical bottleneck, the contribution of this paper is threefold: (1) We demonstrate that dynamically changing lighting conditions in the operating room dramatically affect the performance of HSI applications, namely physiological parameter estimation, and surgical scene segmentation. (2) We propose a novel learning-based approach to automatically recalibrating hyperspectral images during surgery and show that it is sufficiently accurate to replace the tedious process of white reference-based recalibration.
(3) Based on a total of 742 HSI cubes from a phantom, porcine models, and rats we show that our recalibration method not only outperforms previously proposed methods, but also generalizes across species, lighting conditions, and image processing tasks.
Due to its simple workflow integration as well as high accuracy, speed, and generalization capabilities, our method could evolve as a central component in clinical surgical HSI.

Baumann A., Ayala L., Studier-Fischer A., Sellner J., Özdemir B., Kowalewski K., Ilic S., Seidlitz A., and Lena Maier-Hein. (2024) Deep intra-operative illumination calibration of hyperspectral cameras. Medical Image Computing and Computer Assisted Intervention - MICCAI 2024. [MICCAI 2024 Straight accept and oral presentation] [pdf].

Main contribution: We propose a strategy to mitigate perfusion distribution shifts in spectral imaging by leveraging synthetic data for test-time augmentation.

Surgical scene segmentation is crucial for providing context-aware surgical assistance. Recent studies highlight the significant advantages of hyperspectral imaging (HSI) over traditional RGB data in enhancing segmentation performance. Nevertheless, the current hyperspectral imaging (HSI) datasets remain limited and do not capture the full range of tissue variations encountered clinically. Based on a total of 615 hyperspectral images from a total of 16 pigs, featuring porcine organs in different perfusion states, we carry out an exploration of distribution shifts in spectral imaging caused by perfusion alterations. We further introduce a novel strategy to mitigate such distribution shifts, utilizing synthetic data for test-time augmentation. The effect of perfusion changes on state-of-the-art (SOA) segmentation networks depended on the organ and the specific perfusion alteration induced. In the case of the kidney, we observed a performance decline of up to 93% when applying a state-of-the-art (SOA) network under ischemic conditions. Our method improved on the state-of-the-art (SOA) by up to 4.6 times. Given its potential wide-ranging relevance to diverse pathologies, our approach may serve as a pivotal tool to enhance neural network generalization within the realm of spectral imaging.

Bin Qasim A., Motta A., Studier-Fischer A., Sellner J., Ayala L., Hübner M., Bressan M., Özdemir B., Kowalewski K., Nickel F., Seidlitz S. & Lena Maier-Hein. (2024) Test-time augmentation with synthetic data addresses distribution shifts in spectral imaging. International Journal of Computer Assisted Radiology and Surgery 19. [IPCAI CASCINATION & Zeiss Machine Learning in CAI award] [pdf]

Main contribution: We make openly accessible the first video-rate spectral imaging database comprising over 200,000 images from ten different subjects and different levels of perfusion.

Spectral imaging has the potential to become a key technique in interventional medicine as it unveils much richer optical information compared to conventional RBG (red, green, and blue)-based imaging. Thus allowing for high-resolution functional tissue analysis in real time. Its higher information density particularly shows promise for the development of powerful perfusion monitoring methods for clinical use. However, even though in vivo validation of such methods is crucial for their clinical translation, the biomedical field suffers from a lack of publicly available datasets for this purpose. Closing this gap, we generated the SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA). It comprises ten spectral videos (∼20 Hz, approx. 20,000 frames each) systematically recorded of the hands of ten healthy human participants in different functional states. We paired each spectral video with concisely tracked regions of interest, and corresponding diffuse reflectance measurements recorded with a spectrometer. Providing the first openly accessible in human spectral video dataset for perfusion monitoring, our work facilitates the development and validation of new functional imaging methods.

Ayala L., Mindroc-Filimon D., Rees M., Hübner M., Sellner J., Seidlitz S., Tizabi M., Wirkert S., Seitel A. & Maier-Hein L. (2024) The SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA) for video-rate functional imaging of the skin. Nature Scientific Data 11. [pdf]

Main contribution: We show that state-of-the-art surgical scene segmentation networks fail under geometric domain shifts particularly in the context of hyperspectral imaging and propose so far unexplored topology-altering data augmentation schemes to our target application.

Robust semantic segmentation of intraoperative image data could pave the way for automatic surgical scene understanding and autonomous robotic surgery. Geometric domain shifts, however – although common in real-world open surgeries due to variations in surgical procedures or situs occlusions – remain a topic largely unaddressed in the field. To address this gap in the literature, we (1) present the first analysis of state-of-the-art (SOA) semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data, and (2) address generalizability with a dedicated augmentation technique termed 'Organ Transplantation' that we adapted from the general computer vision community. According to a comprehensive validation on six different OOD data sets comprising 600 RGB and yperspectral imaging (HSI) cubes from 33 pigs semantically annotated with 19 classes, we demonstrate a large performance drop of SOA organ segmentation networks applied to geometric OOD data. Surprisingly, this holds true not only for conventional RGB data (drop of Dice similarity coefficient (DSC) by 46 %) but also for HSI data (drop by 45 %), despite the latter's rich information content per pixel. Using our augmentation scheme improves on the SOA DSC by up to 67% (RGB) and 90% (HSI)) and renders performance on par with in-distribution performance on real OOD test data. The simplicity and effectiveness of our augmentation scheme makes it a valuable network-independent tool for addressing geometric domain shifts in semantic scene segmentation of intraoperative data. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Sellner, J., Seidlitz, S., Studier-Fischer, A., Motta, A., Özdemir, B., Müller-Stich, B. P., Nickel, F., & Maier-Hein, L. (2023). Semantic Segmentation of Surgical Hyperspectral Images Under Geometric Domain Shifts. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 (pp. 618–627). [MICCAI 2023 Young Scientist Award & straight accept] [pdf] [video]

Main contribution: Based on 506 surgical hyperspectral images from 20 pigs semantically annotated with 19 organ classes, we show that hyperspectral imaging outperforms RGB imaging during deep learning-based scene segmentation.

Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases — consistently across modalities — with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean DSC of 0.90 ((standard deviation (SD)) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 ((standard deviation (SD)) 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Seidlitz, S., Sellner, J., Odenthal, J., Özdemir, B., Studier-Fischer, A., Knödler, S., Ayala, L., Adler, T. J., Kenngott, H. G., Tizabi, M., Wagner, M., Nickel, F., Müller-Stich, B. P., Maier-Hein, L. (2022). Robust deep learning-based semantic organ segmentation in hyperspectral images. Medical Image Analysis, 80. [pdf]

Main contribution: We present the first real-time application and analysis of multispectral imaging in laparoscopic surgery on human subjects and in a clinical setting.

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent–free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning–based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Ayala, L., Adler, T. J., Seidlitz, S., Wirkert, S., Engels, C., Seitel, A., Sellner, J., Aksenov, A., Bodenbach, M., Bader, P., Baron, S., Vemuri, A., Wiesenfarth, M., Schreck, N., Mindroc, D., Tizabi, M., Pirmann, S., Everitt, B., Kopp-Schneider, A., ... Maier-Hein, L. (2023). Spectral imaging enables contrast agent–free real-time ischemia monitoring in laparoscopic surgery. Science Advances, 9(10). [pdf]

Main contribution: Based on a hyperspectral imaging dataset comprising 9059 images annotated with 20 different porcine organ classes, we show that the organ class is the greatest source of spectral variability and demonstrate that deep learning-based tissue discrimination is feasible with high accuracy of above 95%.

Visual discrimination of tissue during surgery can be challenging since different tissues appear similar to the human eye. Hyperspectral imaging (HSI) removes this limitation by associating each pixel with high-dimensional spectral information. While previous work has shown its general potential to discriminate tissue, clinical translation has been limited due to the method's current lack of robustness and generalizability. Specifically, the scientific community is lacking a comprehensive spectral tissue atlas, and it is unknown whether variability in spectral reflectance is primarily explained by tissue type rather than the recorded individual or specific acquisition conditions. The contribution of this work is threefold: (1) Based on an annotated medical HSI data set (9059 images from 46 pigs), we present a tissue atlas featuring spectral fingerprints of 20 different porcine organs and tissue types. (2) Using the principle of mixed model analysis, we show that the greatest source of variability related to HSI images is the organ under observation. (3) We show that HSI-based fully-automatic tissue differentiation of 20 organ classes with deep neural networks is possible with high accuracy (> 95%). We conclude from our study that automatic tissue discrimination based on HSI data is feasible and could thus aid in intraoperative decision making and pave the way for context-aware computer-assisted surgery systems and autonomous robotics.

Studier-Fischer, A., Seidlitz, S., Sellner, J., Özdemir, B., Wiesenfarth, M., Ayala, L., Odenthal, J., Knödler, S., Kowalewski, K. F., Haney, C. M., Camplisson, I., Dietrich, M., Schmidt, K., Salg, G. A., Kenngott, H. G., Adler, T. J., Schreck, N., Kopp-Schneider, A., Maier-Hein, K., ... Nickel, F. (2022). Spectral organ fingerprints for machine learning-based intraoperative tissue classification with hyperspectral imaging in a porcine model. Scientific Reports, 12(1). [pdf]

Main contribution: We release the first hyperspectral visceral surgery dataset including standardized measurements for 20 physiological organ classes in a total of 11 pigs.

Hyperspectral Imaging (HSI) is a relatively new medical imaging modality that exploits an area of diagnostic potential formerly untouched. Although exploratory translational and clinical studies exist, no surgical HSI datasets are openly accessible to the general scientific community. To address this bottleneck, this publication releases HeiPorSPECTRAL (https://www.heiporspectral.org; https://doi.org/10.5281/zenodo.7737674), the first annotated high-quality standardized surgical HSI dataset. It comprises 5,758 spectral images acquired with the TIVITA® Tissue and annotated with 20 physiological porcine organs from 8 pigs per organ distributed over a total number of 11 pigs. Each HSI image features a resolution of 480 × 640 pixels acquired over the 500–1000 nm wavelength range. The acquisition protocol has been designed such that the variability of organ spectra as a function of several parameters including the camera angle and the individual can be assessed. A comprehensive technical validation confirmed both the quality of the raw data and the annotations. We envision potential reuse within this dataset, but also its reuse as baseline data for future research questions outside this dataset.

Studier-Fischer, A., Seidlitz, S., Sellner, J., Bressan, M., Özdemir, B., Ayala, L., Odenthal, J., Knoedler, S., Kowalewski, K.-F., Haney, C. M., Salg, G., Dietrich, M., Kenngott, H., Gockel, I., Hackert, T., Müller-Stich, B. P., Maier-Hein, L., & Nickel, F. (2023). HeiPorSPECTRAL - the Heidelberg Porcine HyperSPECTRAL Imaging Dataset of 20 Physiological Organs. Scientific Data, 10(1). [pdf]

Esophagectomy is the mainstay of esophageal cancer treatment, but anastomotic insufficiency related morbidity and mortality remain challenging for patient outcome. Therefore, the objective of this work was to optimize anastomotic technique and gastric conduit perfusion with hyperspectral imaging (HSI) for total minimally invasive esophagectomy (MIE) with linear stapled anastomosis. A live porcine model (n = 58) for MIE was used with gastric conduit formation and simulation of linear stapled side-to-side esophagogastrostomy. Four main experimental groups differed in stapling length (3 vs. 6 cm) and simulation of anastomotic position on the conduit (cranial vs. caudal). Tissue oxygenation around the anastomotic simulation site was evaluated using HSI and was validated with histopathology. The tissue oxygenation (ΔStO2) after the anastomotic simulation remained constant only for the short stapler in caudal position (−0.4 ± 4.4%, n.s.) while it was impaired markedly in the other groups (short-cranial: −15.6 ± 11.5%, p = 0.0002; long-cranial: −20.4 ± 7.6%, p = 0.0126; long-caudal: −16.1 ± 9.4%, p < 0.0001). Tissue samples from a vascular stomach as measured by HSI showed correspondent eosinophilic pre-necrotic changes in 35.7 ± 9.7% of the surface area. Tissue oxygenation at the site of anastomotic simulation of the gastric conduit during MIE is influenced by stapling technique. Optimal oxygenation was achieved with a short stapler (3 cm) and sufficient distance of the simulated anastomosis to the cranial end of the gastric conduit. HSI tissue deoxygenation corresponded to histopathologic necrotic tissue changes. The experimental model with HSI and ML allow for systematic optimization of gastric conduit perfusion and anastomotic technique while clinical translation will have to be proven.

Nickel, F., Studier-Fischer, A., Özdemir, B., Odenthal, J., Müller, L. R., Knoedler, S., Kowalewski, K. F., Camplisson, I., Allers, M. M., Dietrich, M., Schmidt, K., Salg, G. A., Kenngott, H. G., Billeter, A. T., Gockel, I., Sagiv, C., Hadar, O. E., Gildenblat, J., Ayala, L., Seidlitz, S., Maier-Hein, L., & Müller-Stich, B. P. (2023). Optimization of anastomotic technique and gastric conduit perfusion with hyperspectral imaging and machine learning in an experimental model for minimally invasive esophagectomy. European Journal of Surgical Oncology. [pdf]

Main contribution: We propose an approach to band selection that can be tailored to a variety of different domains.

Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset of bands, the band selection methods proposed to date are mainly restricted by the availability of often hard to obtain reference measurements. We address this bottleneck with a new approach to band selection that leverages highly accurate Monte Carlo (MC) simulations. We hypothesize that a so chosen small subset of bands can reproduce or even improve upon the results of a quasi continuous spectral measurement. We further investigate whether novel domain adaptation techniques can address the inevitable domain shift stemming from the use of simulations. Initial results based on in silico and in vivo experiments suggest that 10-20 bands are sufficient to closely reproduce results from spectral measurements with 101 bands in the 500-700 nm range. The investigated domain adaptation technique, which only requires unlabeled in vivo measurements, yielded better results than the pure in silico band selection method. Overall, our method could guide development of fast multispectral imaging systems suited for interventional use without relying on complex hardware setups or manually labeled data.

Ayala, L., Isensee, F., Wirkert, S. J., Vemuri, A. S., Maier-Hein, K. H., Fei, B., & Maier-Hein, L. (2022). Band selection for oxygenation estimation with multispectral/hyperspectral imaging. Biomedical Optics Express, 13(3). [pdf]