Intelligent Systems in Spectral Imaging - German Cancer Research Center

CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis

Main contribution: Addressing the critical bottleneck of hardware variability in spectral imaging, we introduce CARL, a camera-agnostic framework that unifies spatio-spectral encoding through a novel spectral encoder and self-supervision strategy, enabling robust knowledge transfer across spectrally heterogeneous datasets as demonstrated in a large-scale study spanning medical imaging, autonomous driving, and remote sensing.

Spectral imaging offers promising applications across diverse domains, including medicine and urban scene understanding, and is already established as a critical modality in remote sensing. However, variability in channel dimensionality and captured wavelengths among spectral cameras impede the development of AI-driven methodologies, leading to camera-specific models with limited generalizability and inadequate cross-camera applicability. To address this bottleneck, we introduce CARL, a model for Camera-Agnostic Representation Learning across RGB, multispectral, and hyperspectral imaging modalities. To enable the conversion of a spectral image with any channel dimensionality to a camera-agnostic representation, we introduce a novel spectral encoder, featuring a self-attention-cross-attention mechanism, to distill salient spectral information into learned spectral representations. Spatio-spectral pre-training is achieved with a novel feature-based self-supervision strategy tailored to CARL. Large-scale experiments across the domains of medical imaging, autonomous driving, and satellite imaging demonstrate our model's unique robustness to spectral heterogeneity, outperforming on datasets with simulated and real-world cross-camera spectral variations. The scalability and versatility of the proposed approach position our model as a backbone for future spectral foundation models.

Baumann, A., Ayala, L., Seidlitz, S., Sellner, J., Studier-Fischer, A., Özdemir, B., Maier-Hein, L., & Ilic, S. (2026). CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis. International Conference on Learning Representations (ICLR). [pdf]

Xeno-learning: knowledge transfer across species in deep learning-based spectral image analysis

Main contribution: Limited human data, needed to train artificial intelligence (AI) models, has constrained the development of hyperspectral imaging (HSI) for surgical applications. We developed xeno-learning, an AI method that harnesses shared biological responses found across species, allowing experimental data from animals to be used in training AI models for HSI applications in humans.

Optical imaging techniques, such as hyperspectral imaging combined with machine learning-based analysis, have the potential to revolutionize clinical surgical imaging. However, these modalities face a shortage of large-scale, representative clinical data for training machine learning-based algorithms. While preclinical animal data are abundantly available through standardized experiments and allow for controlled induction of pathological tissue states, it is not ethically possible to obtain similar data from patients. To leverage this situation, we propose ‘xeno-learning’, a cross-species knowledge-transfer concept inspired by xeno-transplantation. Here, using a total of 14,013 hyperspectral images from humans as well as porcine and rat models, we show that, although spectral signatures of organs differ substantially across species, relative changes resulting from pathologies or surgical manipulation such as malperfusion or injection of contrast agent are comparable. Such changes learnt in one species can be transferred to a new species through a ‘physiology-based data augmentation’ method, enabling the large-scale secondary use of preclinical animal data for human application. The resulting benefits promise a high impact of the proposed knowledge-transfer concept on future developments in the field.

Sellner, J., Studier-Fischer, A., Qasim, A. B., Seidlitz, S., Schreck, N., Tizabi, M., Wiesenfarth, M., Kopp-Schneider, A., Heinecke, J., Brandt, J., Knoedler, S., Max Haney, C., Salg, G., Özdemir, B., Dietrich, M., Michel, M. S., Nickel, F., Kowalewski, K.-F., & Maier-Hein, L. (2026). Xeno-learning: Knowledge transfer across species in deep learning-based spectral image analysis. Nature Biomedical Engineering, 1–15. [pdf]

AI-powered skin spectral imaging enables instant sepsis diagnosis and outcome prediction in critically ill patients

Main contribution: In a prospective observational study involving more than 480 intensive care unit patients, we show that deep learning-based hyperspectral imaging of the skin enables rapid and noninvasive prediction of sepsis and mortality and even outperforms widely used clinical biomarkers and scores when combined with readily available clinical data.

With sepsis remaining a leading cause of mortality, early identification of patients with sepsis and those at high risk of death is a challenge of high socioeconomic importance. Given the potential of hyperspectral imaging (HSI) to monitor microcirculatory alterations, we propose a deep learning approach to automated sepsis diagnosis and mortality prediction using a single HSI cube acquired within seconds. In a prospective observational study, we collected HSI data from the palms and fingers of more than 480 intensive care unit patients. Neural networks applied to HSI measurements predicted sepsis and mortality with areas under the receiver operating characteristic curve (AUROCs) of 0.80 and 0.72, respectively. Performance improved substantially with additional clinical data, reaching AUROCs of 0.94 for sepsis and 0.83 for mortality. We conclude that deep learning–based HSI analysis enables rapid and noninvasive prediction of sepsis and mortality, with a potential clinical value for enhancing diagnosis and treatment.

Seidlitz, S., Hölzl, K., von Garrel, A., Sellner, J., Katzenschlager, S., Hölle, T., Fischer, D., von der Forst, M., Schmitt, F. C. F., Studier-Fischer, A., Weigand, M. A., Maier-Hein, L., & Dietrich, M. (2025). AI-powered skin spectral imaging enables instant sepsis diagnosis and outcome prediction in critically ill patients. Science Advances, 11(29). [pdf]

Deep intra-operative illumination calibration of hyperspectral cameras

The image illustrates a workflow for intraoperative imaging using deep learning. It compares traditional calibration before surgery with a new approach involving deep learning for recalibration. The right side shows organ segmentation and perfusion estimation results, highlighting improvements in accuracy and analysis during surgical procedures.

Main contribution: We propose a new deep-learning method to automatically recalibrate hyperspectral images during surgery, thus eliminating the tedious process of white reference re-calibration.

Hyperspectral imaging (HSI) is emerging as a promising novel imaging modality with various potential surgical applications. Currently available cameras, however, suffer from poor integration into the clinical workflow because they require the lights to be switched off, or the camera to be manually recalibrated as soon as lighting conditions change. Given this critical bottleneck, the contribution of this paper is threefold: (1) We demonstrate that dynamically changing lighting conditions in the operating room dramatically affect the performance of HSI applications, namely physiological parameter estimation, and surgical scene segmentation. (2) We propose a novel learning-based approach to automatically recalibrating hyperspectral images during surgery and show that it is sufficiently accurate to replace the tedious process of white reference-based recalibration.
(3) Based on a total of 742 HSI cubes from a phantom, porcine models, and rats we show that our recalibration method not only outperforms previously proposed methods, but also generalizes across species, lighting conditions, and image processing tasks.
Due to its simple workflow integration as well as high accuracy, speed, and generalization capabilities, our method could evolve as a central component in clinical surgical HSI.

Baumann A., Ayala L., Studier-Fischer A., Sellner J., Özdemir B., Kowalewski K., Ilic S., Seidlitz A., and Lena Maier-Hein. (2024) Deep intra-operative illumination calibration of hyperspectral cameras. Medical Image Computing and Computer Assisted Intervention - MICCAI 2024. [MICCAI 2024 Straight accept and oral presentation] [pdf].

Test-time augmentation with synthetic data addresses distribution shifts in spectral imaging

Graph showing L1-normalized reflectance values plotted against wavelength (500-1000 nm). Two inset diagrams represent "Physiological" and "Avascular" kidney states. The main graph displays varying reflectance data for both states, indicating differences in spectral imaging analysis relevant to medical research.

Main contribution: We propose a strategy to mitigate perfusion distribution shifts in spectral imaging by leveraging synthetic data for test-time augmentation.

Surgical scene segmentation is crucial for providing context-aware surgical assistance. Recent studies highlight the significant advantages of hyperspectral imaging (HSI) over traditional RGB data in enhancing segmentation performance. Nevertheless, the current hyperspectral imaging (HSI) datasets remain limited and do not capture the full range of tissue variations encountered clinically. Based on a total of 615 hyperspectral images from a total of 16 pigs, featuring porcine organs in different perfusion states, we carry out an exploration of distribution shifts in spectral imaging caused by perfusion alterations. We further introduce a novel strategy to mitigate such distribution shifts, utilizing synthetic data for test-time augmentation. The effect of perfusion changes on state-of-the-art (SOA) segmentation networks depended on the organ and the specific perfusion alteration induced. In the case of the kidney, we observed a performance decline of up to 93% when applying a state-of-the-art (SOA) network under ischemic conditions. Our method improved on the state-of-the-art (SOA) by up to 4.6 times. Given its potential wide-ranging relevance to diverse pathologies, our approach may serve as a pivotal tool to enhance neural network generalization within the realm of spectral imaging.

Bin Qasim A., Motta A., Studier-Fischer A., Sellner J., Ayala L., Hübner M., Bressan M., Özdemir B., Kowalewski K., Nickel F., Seidlitz S. & Lena Maier-Hein. (2024) Test-time augmentation with synthetic data addresses distribution shifts in spectral imaging. International Journal of Computer Assisted Radiology and Surgery 19. [IPCAI CASCINATION & Zeiss Machine Learning in CAI award] [pdf]

The SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA) for video-rate functional imaging of the skin

The image illustrates a measurement setup involving a spectral camera to monitor pressure over time (from 0 to 1100 seconds) in a study with ten subjects. It shows a graph of pressure levels and images captured during the experiment, highlighting the data collection process.

Main contribution: We make openly accessible the first video-rate spectral imaging database comprising over 200,000 images from ten different subjects and different levels of perfusion.

Spectral imaging has the potential to become a key technique in interventional medicine as it unveils much richer optical information compared to conventional RBG (red, green, and blue)-based imaging. Thus allowing for high-resolution functional tissue analysis in real time. Its higher information density particularly shows promise for the development of powerful perfusion monitoring methods for clinical use. However, even though in vivo validation of such methods is crucial for their clinical translation, the biomedical field suffers from a lack of publicly available datasets for this purpose. Closing this gap, we generated the SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA). It comprises ten spectral videos (∼20 Hz, approx. 20,000 frames each) systematically recorded of the hands of ten healthy human participants in different functional states. We paired each spectral video with concisely tracked regions of interest, and corresponding diffuse reflectance measurements recorded with a spectrometer. Providing the first openly accessible in human spectral video dataset for perfusion monitoring, our work facilitates the development and validation of new functional imaging methods.

Ayala L., Mindroc-Filimon D., Rees M., Hübner M., Sellner J., Seidlitz S., Tizabi M., Wirkert S., Seitel A. & Maier-Hein L. (2024) The SPECTRAL Perfusion Arm Clamping dAtaset (SPECTRALPACA) for video-rate functional imaging of the skin. Nature Scientific Data 11. [pdf]

Semantic Segmentation of Surgical Hyperspectral Images Under Geometric Domain Shifts

The image compares idealized surgical scene inputs and outputs with actual surgical scenes. On the left, the ideal input shows labeled organs like the liver and colon. The right illustrates a real scene with hand interaction and labeled organ segments, questioning performance in real-world applications.

Main contribution: We show that state-of-the-art surgical scene segmentation networks fail under geometric domain shifts particularly in the context of hyperspectral imaging and propose so far unexplored topology-altering data augmentation schemes to our target application.

Robust semantic segmentation of intraoperative image data could pave the way for automatic surgical scene understanding and autonomous robotic surgery. Geometric domain shifts, however – although common in real-world open surgeries due to variations in surgical procedures or situs occlusions – remain a topic largely unaddressed in the field. To address this gap in the literature, we (1) present the first analysis of state-of-the-art (SOA) semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data, and (2) address generalizability with a dedicated augmentation technique termed 'Organ Transplantation' that we adapted from the general computer vision community. According to a comprehensive validation on six different OOD data sets comprising 600 RGB and yperspectral imaging (HSI) cubes from 33 pigs semantically annotated with 19 classes, we demonstrate a large performance drop of SOA organ segmentation networks applied to geometric OOD data. Surprisingly, this holds true not only for conventional RGB data (drop of Dice similarity coefficient (DSC) by 46 %) but also for HSI data (drop by 45 %), despite the latter's rich information content per pixel. Using our augmentation scheme improves on the SOA DSC by up to 67% (RGB) and 90% (HSI)) and renders performance on par with in-distribution performance on real OOD test data. The simplicity and effectiveness of our augmentation scheme makes it a valuable network-independent tool for addressing geometric domain shifts in semantic scene segmentation of intraoperative data. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Sellner, J., Seidlitz, S., Studier-Fischer, A., Motta, A., Özdemir, B., Müller-Stich, B. P., Nickel, F., & Maier-Hein, L. (2023). Semantic Segmentation of Surgical Hyperspectral Images Under Geometric Domain Shifts. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 (pp. 618–627). [MICCAI 2023 Young Scientist Award & straight accept] [pdf] [video]

Robust deep learning-based semantic organ segmentation in hyperspectral images

Diagram illustrating a robust deep learning model for semantic organ segmentation in hyperspectral images. It compares pixel, patch-based, and image-level segmentation approaches, highlighting the benefits of hyperspectral images over traditional methods, and presents validation results and performance metrics.

Main contribution: Based on 506 surgical hyperspectral images from 20 pigs semantically annotated with 19 organ classes, we show that hyperspectral imaging outperforms RGB imaging during deep learning-based scene segmentation.

Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases — consistently across modalities — with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean DSC of 0.90 ((standard deviation (SD)) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 ((standard deviation (SD)) 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Seidlitz, S., Sellner, J., Odenthal, J., Özdemir, B., Studier-Fischer, A., Knödler, S., Ayala, L., Adler, T. J., Kenngott, H. G., Tizabi, M., Wagner, M., Nickel, F., Müller-Stich, B. P., Maier-Hein, L. (2022). Robust deep learning-based semantic organ segmentation in hyperspectral images. Medical Image Analysis, 80. [pdf]

Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery

A flowchart illustrating a methodology for real-time tissue assessment in clinical settings. It shows the steps of clamping and monitoring using ICG fluorescence and MSI video, noting outcomes like “clamping successful” or “clamping failed,” with a focus on tissue imaging technology.

Main contribution: We present the first real-time application and analysis of multispectral imaging in laparoscopic surgery on human subjects and in a clinical setting.

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent–free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning–based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Ayala, L., Adler, T. J., Seidlitz, S., Wirkert, S., Engels, C., Seitel, A., Sellner, J., Aksenov, A., Bodenbach, M., Bader, P., Baron, S., Vemuri, A., Wiesenfarth, M., Schreck, N., Mindroc, D., Tizabi, M., Pirmann, S., Everitt, B., Kopp-Schneider, A., ... Maier-Hein, L. (2023). Spectral imaging enables contrast agent–free real-time ischemia monitoring in laparoscopic surgery. Science Advances, 9(10). [pdf]

Spectral organ fingerprints for intraoperative tissue classification with hyperspectral imaging

A graphic featuring a pig silhouette, labeled with various organs such as the stomach, liver, and kidney. Below, spectral imaging data graphs represent different organs, showing variations in spectral properties. The overall theme illustrates research in intelligent systems in spectral imaging for biomedical applications.

Main contribution: Based on a hyperspectral imaging dataset comprising 9059 images annotated with 20 different porcine organ classes, we show that the organ class is the greatest source of spectral variability and demonstrate that deep learning-based tissue discrimination is feasible with high accuracy of above 95%.

Visual discrimination of tissue during surgery can be challenging since different tissues appear similar to the human eye. Hyperspectral imaging (HSI) removes this limitation by associating each pixel with high-dimensional spectral information. While previous work has shown its general potential to discriminate tissue, clinical translation has been limited due to the method's current lack of robustness and generalizability. Specifically, the scientific community is lacking a comprehensive spectral tissue atlas, and it is unknown whether variability in spectral reflectance is primarily explained by tissue type rather than the recorded individual or specific acquisition conditions. The contribution of this work is threefold: (1) Based on an annotated medical HSI data set (9059 images from 46 pigs), we present a tissue atlas featuring spectral fingerprints of 20 different porcine organs and tissue types. (2) Using the principle of mixed model analysis, we show that the greatest source of variability related to HSI images is the organ under observation. (3) We show that HSI-based fully-automatic tissue differentiation of 20 organ classes with deep neural networks is possible with high accuracy (> 95%). We conclude from our study that automatic tissue discrimination based on HSI data is feasible and could thus aid in intraoperative decision making and pave the way for context-aware computer-assisted surgery systems and autonomous robotics.

Studier-Fischer, A., Seidlitz, S., Sellner, J., Özdemir, B., Wiesenfarth, M., Ayala, L., Odenthal, J., Knödler, S., Kowalewski, K. F., Haney, C. M., Camplisson, I., Dietrich, M., Schmidt, K., Salg, G. A., Kenngott, H. G., Adler, T. J., Schreck, N., Kopp-Schneider, A., Maier-Hein, K., ... Nickel, F. (2022). Spectral organ fingerprints for machine learning-based intraoperative tissue classification with hyperspectral imaging in a porcine model. Scientific Reports, 12(1). [pdf]

HeiPorSPECTRAL - the Heidelberg Porcine HyperSPECTRAL Imaging Dataset of 20 Physiological Organs

The image displays a diagram outlining a study involving eight pigs and twenty organs. It categorizes data based on various situses and angles, showing multiple recordings per organ with annotations indicating the organ's surface and polygon outlines for anatomical reference.

Main contribution: We release the first hyperspectral visceral surgery dataset including standardized measurements for 20 physiological organ classes in a total of 11 pigs.

Hyperspectral Imaging (HSI) is a relatively new medical imaging modality that exploits an area of diagnostic potential formerly untouched. Although exploratory translational and clinical studies exist, no surgical HSI datasets are openly accessible to the general scientific community. To address this bottleneck, this publication releases HeiPorSPECTRAL (https://www.heiporspectral.org; https://doi.org/10.5281/zenodo.7737674), the first annotated high-quality standardized surgical HSI dataset. It comprises 5,758 spectral images acquired with the TIVITA® Tissue and annotated with 20 physiological porcine organs from 8 pigs per organ distributed over a total number of 11 pigs. Each HSI image features a resolution of 480 × 640 pixels acquired over the 500–1000 nm wavelength range. The acquisition protocol has been designed such that the variability of organ spectra as a function of several parameters including the camera angle and the individual can be assessed. A comprehensive technical validation confirmed both the quality of the raw data and the annotations. We envision potential reuse within this dataset, but also its reuse as baseline data for future research questions outside this dataset.

Studier-Fischer, A., Seidlitz, S., Sellner, J., Bressan, M., Özdemir, B., Ayala, L., Odenthal, J., Knoedler, S., Kowalewski, K.-F., Haney, C. M., Salg, G., Dietrich, M., Kenngott, H., Gockel, I., Hackert, T., Müller-Stich, B. P., Maier-Hein, L., & Nickel, F. (2023). HeiPorSPECTRAL - the Heidelberg Porcine HyperSPECTRAL Imaging Dataset of 20 Physiological Organs. Scientific Data, 10(1). [pdf]

Optimization of anastomotic technique and gastric conduit perfusion with hyperspectral imaging and machine learning in an experimental model for minimally invasive esophagectomy

The image displays two diagrams illustrating tissue capillary perfusion in different gastric areas. The first focuses on the gastropileplica dextra, and the second shows the gastropileplica with transverse perfusion. Below are graphs comparing standard deviation indices (SDI) for control and intervention groups, highlighting significant findings.

Esophagectomy is the mainstay of esophageal cancer treatment, but anastomotic insufficiency related morbidity and mortality remain challenging for patient outcome. Therefore, the objective of this work was to optimize anastomotic technique and gastric conduit perfusion with hyperspectral imaging (HSI) for total minimally invasive esophagectomy (MIE) with linear stapled anastomosis. A live porcine model (n = 58) for MIE was used with gastric conduit formation and simulation of linear stapled side-to-side esophagogastrostomy. Four main experimental groups differed in stapling length (3 vs. 6 cm) and simulation of anastomotic position on the conduit (cranial vs. caudal). Tissue oxygenation around the anastomotic simulation site was evaluated using HSI and was validated with histopathology. The tissue oxygenation (ΔStO2) after the anastomotic simulation remained constant only for the short stapler in caudal position (−0.4 ± 4.4%, n.s.) while it was impaired markedly in the other groups (short-cranial: −15.6 ± 11.5%, p = 0.0002; long-cranial: −20.4 ± 7.6%, p = 0.0126; long-caudal: −16.1 ± 9.4%, p < 0.0001). Tissue samples from a vascular stomach as measured by HSI showed correspondent eosinophilic pre-necrotic changes in 35.7 ± 9.7% of the surface area. Tissue oxygenation at the site of anastomotic simulation of the gastric conduit during MIE is influenced by stapling technique. Optimal oxygenation was achieved with a short stapler (3 cm) and sufficient distance of the simulated anastomosis to the cranial end of the gastric conduit. HSI tissue deoxygenation corresponded to histopathologic necrotic tissue changes. The experimental model with HSI and ML allow for systematic optimization of gastric conduit perfusion and anastomotic technique while clinical translation will have to be proven.

Nickel, F., Studier-Fischer, A., Özdemir, B., Odenthal, J., Müller, L. R., Knoedler, S., Kowalewski, K. F., Camplisson, I., Allers, M. M., Dietrich, M., Schmidt, K., Salg, G. A., Kenngott, H. G., Billeter, A. T., Gockel, I., Sagiv, C., Hadar, O. E., Gildenblat, J., Ayala, L., Seidlitz, S., Maier-Hein, L., & Müller-Stich, B. P. (2023). Optimization of anastomotic technique and gastric conduit perfusion with hyperspectral imaging and machine learning in an experimental model for minimally invasive esophagectomy. European Journal of Surgical Oncology. [pdf]

Band selection for oxygenation estimation with multispectral/hyperspectral imaging

Three images show a spectral imaging technique highlighting different wavelengths of light, with selected regions marked in each. A graph below displays reflectance values across various wavelengths, indicating how the image data varies with different spectral bands. This illustrates the concept of band selection in imaging.

Main contribution: We propose an approach to band selection that can be tailored to a variety of different domains.

Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset of bands, the band selection methods proposed to date are mainly restricted by the availability of often hard to obtain reference measurements. We address this bottleneck with a new approach to band selection that leverages highly accurate Monte Carlo (MC) simulations. We hypothesize that a so chosen small subset of bands can reproduce or even improve upon the results of a quasi continuous spectral measurement. We further investigate whether novel domain adaptation techniques can address the inevitable domain shift stemming from the use of simulations. Initial results based on in silico and in vivo experiments suggest that 10-20 bands are sufficient to closely reproduce results from spectral measurements with 101 bands in the 500-700 nm range. The investigated domain adaptation technique, which only requires unlabeled in vivo measurements, yielded better results than the pure in silico band selection method. Overall, our method could guide development of fast multispectral imaging systems suited for interventional use without relying on complex hardware setups or manually labeled data.

Ayala, L., Isensee, F., Wirkert, S. J., Vemuri, A. S., Maier-Hein, K. H., Fei, B., & Maier-Hein, L. (2022). Band selection for oxygenation estimation with multispectral/hyperspectral imaging. Biomedical Optics Express, 13(3). [pdf]