Efficient data annotation
With the breakthrough success of deep learning-based algorithms for automatic image annotation, the availability of reference annotations for their training is one of the major bottlenecks in many fields. In computer assisted minimally-invasive surgery, for example, the amount of publically available annotated data is steadily increasing. However, it is extremely challenging to capture all the variations (e.g. all medical devices applied in an intervention) that may occur in clinical practice. Furthermore, algorithms trained on data of one specific domain (e.g. the specific hardware in a specific hospital for a specific intervention) typically do not generalize well to new domains. This is an important limitation as curation of (sufficient) training data is extremely labor-intensive and is currently hindering progress in the field. Several different approaches have been introduced to address this issue:
Crowdsourcing:
In crowdsourcing-based approaches, annotation tasks are outsourced to masses of anonymous workers in an online community. We have explored this concept in the context of 3D diagnostic image annotation (Heim et al., 2015; Heim et al., 2018) and endoscopic video annotation (Maier-Hein et al., 2016) . In order to deal with the high variation in annotation quality, we proposed a novel approach to estimate the skills of a user based on the interaction with the annotation software (Heim et al., 2018). Furthermore, we investigated first concepts to crowd-algorithm collaboration for large-scale image annotation (Maier-Hein et al., 2016).
Figure 1: Automated quality control in crowdsourcing. A machine learning algorithm analyzes the interaction behaviour of the users and detects spammers accordingly (Heim et al., 2018).
© dkfz.de
Self-supervised learning
Another approach to achieve efficient data annotation is to create methods that exploit natural structure in data to "self-supervise'' their learning. This concept holds great potential for the field of Surgical Data Science as the bottleneck related to training data acquisition is often due to the small amount of annotated medical image data rather than the amount of raw medical data that. One of our contributions (Ross et al., 2018), for example, was based on the hypothesis that unlabeled video data can be used to learn a representation of the target domain (here: the specific hardware setup and surgical procedure that an instrument segmentation algorithm is applied in) to boost the performance of state-of-the-art machine learning algorithms. The core component of this approach is a so-called auxiliary task (here: recolorization) that is used to pre-train a convolutional neural network (CNN) for the target task.
Domain adaptation
The concept of domain adaptation has successfully been applied in various fields to compensate for the fact that algorithms trained on a specific training domain typically do not generalize well on the target domain. In (Wirkert et al., 2017), for example, we used domain adaptation in the context of oxygenation estimation from multispectral imaging data in order to compensate that no reference labels for oxygenation could be obtained. The approach involved using unlabeled real measurements of the target domain to give different weights to training samples that were generated with a Monte-Carlo approach, in order to train the algorithm.
Alumni
- Dr. Eric Heim (Doctoral Student)
- Martina Zündel (Student Assistant)
Key Collaborators
- Dr. Daniel Kondermann, QUALITY MATCH, http://www.quality-match.com
- Dr. Hannes Kenngott, Division of Visceral Surgery of the Department of General Surgery, University of Heidelberg
- Prof. Dr. Stefanie Speidel, Department for Translational Surgical Oncology, National Center for Tumor Diseases Dresden
Publications
Bittel, S., Roethlingshoefer, V., Kenngott, H., Wagner, M., Bodenstedt, S., Ross, T., Speidel, S., & Meier-Hein, L. (2017). How to create the largest in-vivo endoscopic dataset. Intravascular Imaging and Computer Assisted Stenting, and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis.
Cardoso, M. J., Arbel, T., Lee, S., Cheplygina, V., Balocco, S., Mateus, D., Zahnd, G., Maier-Hein, L., Demirci, S., Granger, E., & others. (2017). Intravascular Imaging and Computer Assisted Stenting, and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. CVII-STENT and Second International Workshop, LABELS.
Ganz, M., Kondermann, D., Andrulis, J., Knudsen, G. M., & Maier-Hein, L. (2017). Crowdsourcing for error detection in cortical surface delineations. International Journal of Computer Assisted Radiology and Surgery, 12(1), 161–166.
Heim, E., Roß, T., Seitel, A., März, K., Stieltjes, B., Eisenmann, M., Lebert, J., Metzger, J., Sommer, G., Sauter, A. W., & others. (2018). Large-scale medical image annotation with crowd-powered algorithms. Journal of Medical Imaging, 5(3), 034002.
Heim, E., Seitel, A., Andrulis, J., Isensee, F., Stock, C., Ross, T., & Maier-Hein, L. (2017). Clickstream analysis for crowd-based object segmentation with confidence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12), 2814–2826.
Heim, E., Seitel, A., Stock, C., Ross, T., & Maier-Hein, L. (2017). Clickstreamanalyse zur Qualitätssicherung in der crowdbasierten Bildsegmentierung. In Bildverarbeitung für die Medizin 2017 (pp. 17–17). Springer.
Maier-Hein, L., Kondermann, D., Roß, T., Mersmann, S., Heim, E., Bodenstedt, S., Kenngott, H. G., Sanchez, A., Wagner, M., Preukschas, A., & others. (2015). Crowdtruth validation: A new paradigm for validating algorithms that rely on image correspondences. International Journal of Computer Assisted Radiology and Surgery, 10(8), 1201–1212.
Maier-Hein, L., Mersmann, S., Kondermann, D., Bodenstedt, S., Sanchez, A., Stock, C., Kenngott, H. G., Eisenmann, M., & Speidel, S. (2014). Can masses of non-experts train highly accurate image classifiers? International Conference on Medical Image Computing and Computer-Assisted Intervention, 438–445.
Maier-Hein, L., Ross, T., Gröhl, J., Glocker, B., Bodenstedt, S., Stock, C., Heim, E., Götz, M., Wirkert, S., Kenngott, H., & others. (2016). Crowd-algorithm collaboration for large-scale endoscopic image annotation with confidence. International Conference on Medical Image Computing and Computer-Assisted Intervention, 616–623.
Maier-Hein, L., Mersmann, S., Kondermann, D., Stock, C., Kenngott, H. G., Sanchez, A., Wagner, M., Preukschas, A., Wekerle, A.-L., Helfert, S., & others. (2014). Crowdsourcing for reference correspondence generation in endoscopic images. International Conference on Medical Image Computing and Computer-Assisted Intervention, 349–356.
Maier-Hein, L., Wagner, M., Ross, T., Reinke, A., Bodenstedt, S., Full, P. M., Hempe, H., Mindroc-Filimon, D., Scholz, P., Tran, T. N., & others. (2020). Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room. ArXiv Preprint ArXiv:2005.03501.
Ross, T., Reinke, A., Full, P. M., Wagner, M., Kenngott, H., Apitz, M., Hempe, H., Filimon, D. M., Scholz, P., Tran, T. N., Bruno, P., Arbeláez, P., Bian, G.-B., Bodenstedt, S., Bolmgren, J. L., Bravo-Sánchez, L., Chen, H.-B., González, C., Guo, D., ... Maier-Hein, L. (2020). Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge. Medical Image Analysis, 101920. https://doi.org/10.1016/j.media.2020.101920
Ross, T., Zimmerer, D., Vemuri, A., Isensee, F., Wiesenfarth, M., Bodenstedt, S., Both, F., Kessler, P., Wagner, M., Müller, B., & others. (2018). Exploiting the potential of unlabeled endoscopic video data with self-supervised learning. International Journal of Computer Assisted Radiology and Surgery, 13(6), 925–933.
Awards
IPCAI Best Presentation Award: Runner Up (2018)
Tobias Roß for the presentation "Exploiting the potential of unlabeled endoscopic video data with self-supervised learning"
KUKA IJCARS Best Paper Award, 3rd Prize (2016)
M. Ganz, D. Kondermann, J. Andrula, G. M. Knubben and L. Maier-Hein for the paper "Crowdsourcing for error detection in cortical surface delineations."
CURAC Best Paper Award (2015)
E. Heim, T. Ross, T. Norajitra, M. Nolden, K. März, D. Kondermann, S. Speidel, K. Maier-Hein and L. Maier-Hein for the paper "Crowdgestützte Organsegmentierung: Möglichkeiten und Grenzen."