Computer Assisted Medical Interventions

Crowd-powered Medical Image Annotation

< Back to Research overview

© dkfz.de

With the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training has become one of the major bottlenecks in the field of surgical data science. To address this challenge, our group is investigating novel large-scale data annotation methods [1-9] based on crowdsourcing - a new concept that involves outsourcing tasks to anonymous workers in an online community. Our scientific innovations are mainly related to dealing with uncertainties of both the algorithms developed and the crowd. Main contributions include a new method for estimating an anonymous user's annotation performance based on clickstream analysis [5,7]. Furthermore, we were the first to investigate the concept of crowd-algorithm collaboration in the field of large-scale medical image annotation [1].

Key collaborators

  • Dr. Hannes Kenngott
    University of Heidelberg, Division of Visceral Surgery of the Department of General Surgery

  • Dr. Christian Stock 
    Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ)
  • Prof. Dr. Stefanie Speidel Translationale Chirurgische Onkologie, NCT Dresden

Funding

Selected publications

  1. Maier-Hein L, Ross T, Gröhl J, Glocker B, Bodenstedt S, Stock S, Heim E, et al. Crowd-Algorithm Collaboration for Large-Scale Endoscopic Image Annotation with Confidence. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 2016, pp 616–23, https://doi.org/10.1007/978-3-319-46723-8_71.

     

  2. Maier-Hein L, Kondermann D, Roß T, Mersmann S, Heim E, Bodenstedt S, Kenngott HG, et al. Crowdtruth Validation: A New Paradigm for Validating Algorithms That Rely on Image Correspondences. International Journal of Computer Assisted Radiology and Surgery. 2015; 10(8): pp 1201–12. https://doi.org/10.1007/s11548-015-1168-3.

 

  1. Maier-Hein L, Mersmann S, Kondermann D, Bodenstedt S, Sanchez A, Stock C, Kenngott HG, Eisenmann M, and Speidel S. Can Masses of Non-Experts Train Highly Accurate Image Classifiers?  In International Conference on Medical Image Computing and Computer-Assisted Intervention. 2014, pp 438–45, https://doi.org/10.1007/978-3-319-10470-6_55.

 

  1. Maier-Hein L, Mersmann S, Kondermann D, Stock C, Kenngott HG, Sanchez A, Wagner M, et al. Crowdsourcing for Reference Correspondence Generation in Endoscopic Images. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 2014, pp 349-56, https://doi.org/10.1007/978-3-319-10470-6_44.

 

  1. Heim E, Seitel A, Stock C, Ross T, and Maier-Hein L. Clickstreamanalyse zur Qualitätssicherung in der crowdbasierten Bildsegmentierung. In Bildverarbeitung für die Medizin 2017, 17–17.  https://link.springer.com/chapter/10.1007/978-3-662-54345-0_10.

 

  1. Heim E, Roß T, Norajitra T, Nolden M, März K, Kondermann D, Speidel S, Maier-Hein KH, and Maier-Hein L,  Crowdgestützte Organsegmentierung: Möglichkeiten und Grenzen, In 14. Jahrestagung der Deutschen Gesellschaft für Computer und Roboterassistierte Chirurgie 2015, pp. 37-42

 

  1. Heim E, Seitel A, Andrulis J, Isensee F, Stock C, Ross T, and Maier-Hein L. Clickstream Analysis for Crowd-Based Object Segmentation with Confidence. arXiv:1611.08527 [Cs], November 25, 2016. http://arxiv.org/abs/1611.08527.

 

  1. Ganz M, Kondermann D, Andrulis J, Knudsen GM, Maier-Hein, L, Crowdsourcing for error detection in cortical surface delineations. International Journal of Computer Assisted Radiology and Surgery. 2017;12(1):161-6, 10.1007/s11548-016-1445-9

 

  1. Roethlingshoefer V, Bittel S, Kenngott H, Wagner M, Bodenstedt S, Ross T, Speidel S, Maier-Hein L, How to Create the Largest In-Vivo Endoscopic Dataset, LABELS Workshop, MICCAI 2017

Awards

  • CURAC KUKA Best Paper Award (2015)
    Eric Heim et al. for the best CURAC paper 2015 “Crowdgestützte Organsegmentierung: Möglichkeiten und Grenzen,”  awarded at 14. Jahrestagung der Deutschen Gesellschaft für Computer und Roboterassistierte Chirurgie

  • IJCARS KUKA Best Paper Award 3rd place (2016)
    Ganz Melanie et al. for the IJCARS paper “Crowdsourcing for error detection in cortical surface delineations.” awarded at International Congress and Exhibition on Computer Assisted Radiology and Surgery (CARS)

< Back to Research overview

to top