Computer Assisted Medical Interventions

Range imaging for markerless navigation

Real-time markerless patient localization

One of the main difficulties revolving around computer-assisted intervention (CAI) is acquiring the 3D structure of a patient’s anatomy in an accurate, fast and robust manner. Existing systems for navigation along rigid structures typically compute the location of subsurface structures in relation to the position of markers that have been rigidly attached to the skull or the skin of the patient. Due to the continuously changing morphology of organs in soft-tissue interventions, however, the hardware applied for this purpose must be able to capture the patient’s anatomy locally in a fast and robust manner. To address this issue, we propose using the novel Time-of-Flight (ToF) camera technique for locating the patient in real-time during surgery [SeitA12]. Contrary to all conventional imaging modalities available, this technique enables the acquisition of morphological information in real time, without radiation, requiring no tactile contact and at very low cost. To improve image quality and in turn make the technique sufficiently accurate for clinical application, we proposed a new camera calibration concept for dealing with systematic distance errors [MersS13a] and are now able to reduce reconstruction errors by up to 74%. Our studies have shown that the ToF technique carries great potential as a low-cost intra-operative imaging modality, although existing hardware should be further optimized to yield submillimeter reconstruction accuracy.

Endoscopic markerless patient localization

In collaboration with the company Richard Wolf GmbH and the Friedrich-Alexander-University Erlangen-Nuremberg, we further assessed the value of the first industrial prototypical endoscope (Richard Wolf GmbH) with the integrated ToF technique [MaieL13]. In order to compare the quality of surfaces acquired with the new ToF endoscope with surfaces that have been reconstructed by other state-of-the-art 3D surface reconstruction methods, we have established an international consortium with leading institutes in the field, including Imperial College London, University College London, Karlsruhe Institute of Technology, University of Erlangen-Nuremberg, and Université d’Auvergne. Together with these organizations, we conducted a comprehensive multi-center study for which we investigated the accuracy and robustness of the most recent techniques and methods in the field (article under revision). The study revealed that submillimeter accuracy can be achieved, but robustness to clinically relevant factors like smoke or bleeding is a severe issue that has to be addressed.

Markerless intra-operative registration

Correspondences established automatically between surfaces extracted from a CT scan and acquired with a Time-of-Flight (ToF) camera (reprinted from [dosST14])
© dkfz.de

In CAI, the term registration typically refers to the alignment of preoperative patient-specific models to intra-operatively acquired data. It may be used to augment the surgeon’s view by visualizing structures beneath the visible tissue surface. In order to use the range imaging data for augmented reality applications, the intra-interventionally acquired data must be continuously registered to surface data extracted from tomographic images. Shape-matching methods can be classified into two categories: These include global matching methods that establish correspondences between the input shapes without any prior knowledge of their respective poses in relation to each other and fine registration methods that assume a rough alignment of the input data and. The latter methods are often used to compute an initial

For global surface registration, we have developed a new approach to automatic correspondence search in the presence of deformation and high-level noise [dosST4]. Shape-based registration is formulated as an optimization problem, where the metric to be optimized quantifies the goodness of fit between two sets of surface nodes extracted from the input data. The metric takes the descriptor similarity, the spatial distribution of the points, and compatibility of pairwise assignments into account. Optimization is achieved by means of a tree-based search approach. Several in-silico and in-vitro experiments have shown that this approach can establish high quality surface correspondences, even in the presence of severe noise (cf. Fig. 1). These correspondences established are used to initialize a fine surface matching algorithm.

The iterative closest point (ICP) algorithm, which had been introduced in the early 1990s, is the most cited algorithm for the fine geometric alignment of 3D models. As an important contribution of the junior group, we proposed a generalization of this popular method that keeps the main benefits of the original algorithm, namely the guaranteed convergence, the general applicability in various fields, and the straightforward implementation while targeting one of the key issues: The assumption of isotropic localization errors in the input data [MaieL12a]. In our extension, which was developed in close collaboration with the Vanderbilt University, anisotropic localization errors associated with the surface points and that are very typical for all range imaging techniques, are integrated into the algorithm via covariance matrices assigned to all of the surface nodes. In all of our experiments, registration accuracy was improved by more than 50%. Because of its general applicability, its flexibility in incorporating prior knowledge, its convergence properties, and the dramatic improvement in accuracy compared to the original ICP, its future potential is high.


Application: Markerless mobile augmented reality

Visualization of anatomical data for diagnosis, surgical planning, or orientation during interventional therapy is an integral part of modern health care. As only few medical imaging modalities are capable of providing real-time images of the patient’s anatomy, a common procedure requires the physician to mentally establish a correspondence between the real object and the 3D virtual image generated on an external display. To address this, we have presented a new approach to on-patient visualization of 3D medical images which combines the concept of augmented reality with an intuitive interaction scheme. Our method requires mounting a range imaging device on a portable display (e.g., a tablet PC), as shown in Fig. 2. During the visualization process, the position of the camera and thus the user’s direction of view is continuously ascertained by means of a surface registration algorithm. By moving the device along the body of the patient, the physician gains the impression of having a direct view into the human body. The patented concept, which had yielded promising results in initial feasibility studies [KilgT12a] and has recently been ported to the graphics processing unit (GPU) [HeimE14], can be used for intervention planning or as a teaching aid and for various other applications that require intuitive visualization of 3D data. A marker-based variant of the mobile AR concept [MüllM13a] is currently being validated in vivo.

Patented concept for on-patient visualization: the pose of a portable device (e.g. iPad) relative to the patient is continuously estimated by registering ToF range images to static volume data. This allows the physician to look directly into the human body (reprinted from [BaueS13])
© dkfz.de

Application: Markerless needle navigation

Percutaneous needle insertions are increasingly being used for diagnosis and treatment of abdominal lesions. The challenging part of CT-guided punctures is transferring the planned insertion trajectory to the patient, which conventionally requires needle to be repositioned several times in addition to control CT scans. Previously presented navigation systems, however, have thus far not become widely accepted in clinical routine. This is because their benefit to the patient are not capable of outweighing the additional higher costs and the increased complexity in terms of bulky tracking systems and specialized markers for registration and tracking. To address this issue, we have presented a markerless navigation system that uses one single modality for patient localization and instrument guidance. The main idea consists of utilizing a range imaging device that enables both distance and intensity information of the patient to be recorded during an intervention. Registration can then be performed by matching the range data to the acquired CT planning data. Needle guidance is achieved by augmenting the live intensity image of the scene with a projection of a virtual 3D model of the instrument, as illustrated in Fig. 3. A first feasibility study yielded a median targeting accuracy ranging within several millimeters [SeitA12a]. Although system performance remains to be improved, the concept has high potential with regard to finding widespread application for clinical use due to its simple integration into clinical workflow.

Overview of the markerless navigation approach [SeitA12a/b]. Before the intervention, a CT volume is acquired and used for both trajectory planning and for creating a surface representation of the abdomen. During the intervention, a range imaging device is used to acquire a partial surface of the abdomen. The insertion trajectory planned before the intervention is then transferred to the patient during the intervention by means of surface registration. Guidance of the needle toward the target is finally accomplished by using the intensity image of the range imaging device.
© dkfz.de

Selected awards

2013: Heinz Maier-Leibnitz Prize, awarded by the German Research Foundation (DFG)

2011: Best scientific talk, 2nd prize, at the 23rd Meeting of the Society for Medical Innovation and Technology (SMIT) 2011, Tel Aviv, Israel

Selected publications

Bauer S, Seitel A, Hofmann H, Blum T, Wasza J, Balda M, Meinzer HP, Navab N, Hornegger J, Maier-Hein L. Real-Time Range Imaging in Health Care: A Survey. Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications LNCS 8200, 2013, pp 228-254

Dos Santos TR, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer HP, Meinzer HP, Heimann T, Maier-Hein L. Pose-Independent Surface Matching for Intra-Operative Soft-Tissue Marker-Less Registration. Med Imag Anal (accepted), 2014.

Dos Santos T, Seitel A, Meinzer HP, Maier-Hein L. Correspondences Search for Surface-Based Intra-Operative Registration. 13th International Conference on Medical Image Computing and Computer-Assisted Intervention- MICCAI 2010, LNCS 6362: 660-667, 2010.

Heim E, Kilgus T, Haase S, Iszatt J, Franz AM, Seitel A, Müller M, Fangerau M, Hornegger J, Meinzer HP, Maier-Hein L. GPGPU-beschleunigter anisotroper ICP zur Registrierung von Tiefendaten. Bildverarbeitung für die Medizin, 24-29, 2014.

Kilgus T, Franz AM, Seitel A, März K, Bartha L, Fangerau M, Mersmann S, Groch A, Meinzer HP, Maier-Hein L. Registration of partially overlapping surfaces for range image based augmented reality on mobile devices. Medical Imaging: Image-Guided Procedures, Robotic Interventions, and Modeling. DR. Holmes III, KH. Wong, SPIE, 2012.

Maier-Hein L, Mountney P, Bartoli A, Elhawary H, Elson D, Groch A, Kolb A, Rodrigues M, Sorger J, Speidel S, and Stoyanov D. Optical Techniques for 3D Surface Reconstruction in Computer-assisted Laparoscopic Surgery. Med Imag Anal, 17(8): 974-996, 2013.

Maier-Hein L, Franz AM, dos Santos TR, Schmidt M, Fangerau M, Meinzer HP, Fitzpatrick JM. Convergent Iterative Closest-Point Algorithm to Accomodate Anisotropic and Inhomogenous Localization Error. IEEE T Patter Anal 34 (8), 1520-1532, 2012.

Maier-Hein L, Franz AM, Fangerau M, Schmidt M, Seitel A, Mersmann S, Kilgus T, Groch A, Yung K, dos Santos TR, Meinzer HP. Towards Mobile Augmented Reality for On-Patient Visualization of Medical Images. Bildverarbeitung für die Medizin. Handels H, Ehrhardt J, Deserno TM, Meinzer HP, Tolxdorff T, 389-393, 2011.

Maier-Hein L, Schmidt M, Franz AM, Dos Santos TR, Seitel A, Jähne B, Fitzpatrick JM, Meinzer HP. Accounting for Anisotropic Noise in Fine Registration of Time-of-Flight Range Data with High-Resolution Surface Data. 13th International Conference on Medical Image Computing and Computer-Assisted Intervention- MICCAI 2010. LNCS 6361: 251-258, 2010.

Mersmann S, Seitel A, Erz M, Jähne B, Nickel F, Mieth M, Mehrabi A, Maier-Hein L. Calibration of time-of-flight cameras for accurate intraoperative surface reconstruction. Med Phys 40 (8), 082701, 2013.

Müller M, Rassweiler MC, Klein J, Seitel A, Gondan M, Baumhauer M, Teber D, Rassweiler J, Meinzer HP, Maier-Hein L. Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J Cars, 8:663-674, 2013.

Seitel A, Yung K, Mersmann S, Kilgus T, Groch A, dos Santos T, Franz AM, Nolden M, Meinzer HP, Maier-Hein L. MITK-ToF - range data within MITK. Int J Cars 7 (1):87-96, 2012.

Seitel A. Markerless Navigation for Percutaneous Needle Insertions. Heidelberg: Ruprecht-Karls-Universität Heidelberg, Dissertation, 2012

to top
powered by webEdition CMS