ERC Consolidator Grant NEURAL SPICING: Neural Spectral Image Decoding
Had we the capacity to image the molecular tissue composition in real time, without exposing the patient or clinical staff to radiation, then this would revolutionize healthcare. Spectral imaging techniques, such as multispectral diffuse reflectance imaging and photoacoustics, have the potential to recover important tissue properties including oxygenation, temperature and the concentration of water. However, decades of research invested in solving the inverse problem of reconstructing clinically relevant tissue properties from spectral measurements have failed to produce solutions that can quantify tissue parameters robustly in a clinical setting. Initial attempts to address the limitations of model-based approaches with machine learning were hampered by the absence of labeled reference data needed for supervised algorithm training. While training data simulation can potentially address this bottleneck, the domain gap between real and simulated images remains a huge unsolved challenge.
This project at the intersection of physics, biology, medicine, and data science bridges the domain gap with a "learning-to-simulate" approach and reframes the quantification problem as a decoding problem that can be tackled with physics-constrained neural networks. It is based on the hypothesis that unlabeled and weakly labeled measurement data can be leveraged to improve the realism of simulated spectral images and ultimately the quantification accuracy. In building upon modern machine learning techniques, the concept naturally allows different sources of uncertainty to be handled and even exploited, requires no labeled data for the target task, and enables imaging systems to learn from their experience.
The proposed second generation of spectral imaging is inherently safe and low-cost and could thus pave the way for a new era in interventional healthcare, with clinical applications ranging from cancer diagnosis to the staging and therapy of cardiovascular and inflammatory diseases.
Neural spectral image decodingfor quantitative optical imaging in interventional healthcare. A physics-based simulation framework for virtual spectral imagingcontinuously adapts itself to real data in order to minimize the domain gapbetween real and simulated data. A neural network-based decoding framework leverages both simulated data and real data to learn the reconstruction of relevant tissue properties from measurement data. As the project relies on the explicit and fast decomposition of all of the different parameters which are relevant for the image formation process (e.g. tissue temperature, oxygenation, concentration of water/lipids), it offers spectacular opportunities for a wide range of clinical applications in interventional healthcare.
© dkfz.de
Key Collaborators
Throughout this project we closely collaborate with the following technical, clinical and industrial partners:
Technical collaborators
Prof. Dr. Sarah E. Bohndiek and Dr. Janek Gröhl
University of Cambridge, Department of Physics
Prof. Dr. Carsten Rother and Prof. Dr. Ullrich Köthe
Visual Learning Lab Heidelberg
Clinical collaborators
Prof. Dr. Beat Müller, PD Dr. Felix Nickel
Department of General, Visceral and Transplantation Surgery, Heidelberg University Hospital
Prof. Dr. Jürgen Debus
Department of Radiotherapy, Heidelberg University Hospital
Prof. Dr. Dogu Teber
Urology Clinic, Städtisches Klinikum Karlsruhe
Dr. Edgar Santos
Department of Neurosurgery, Heidelberg University Hospital
Industrial collaborators
iThera Medical GmbH (Munich, Germany)
Diaspective Vision GmbH (Am Salzhaff, Germany)
Karl Storz SE & Co. KG (Tuttlingen, Germany)
Funding
This project receives funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 101002198)
Awards
Best PhD Talk Award for Kris Dreher at the DKFZ PhD Retreat held at Evangelische Tagungsstätte Löwenstein from 17th-19th July 2023 for the talk entitled "Unsupervised sim2real transfer of spectral images with conditional invertible neural networks". July 2023
3rd Prize BiOS Student 3-Minute Poster Presentation for Niklas Holzwarth at SPIE Photonics West. January 2023