Cookie Settings

We use cookies to optimize our website. These include cookies that are necessary for the operation of the site, as well as those that are only used for anonymous statistic. You can decide for yourself which categories you want to allow. Further information can be found in our data privacy protection .

Essential

These cookies are necessary to run the core functionalities of this website and cannot be disabled.

Name Webedition CMS
Purpose This cookie is required by the CMS (Content Management System) Webedition for the system to function correctly. Typically, this cookie is deleted when the browser is closed.
Name econda
Purpose Session cookie emos_jcsid for the web analysis software econda. This runs in the “anonymized measurement” mode. There is no personal reference. As soon as the user leaves the site, tracking is ended and all data in the browser are automatically deleted.
Statistics

These cookies help us understand how visitors interact with our website by collecting and analyzing information anonymously. Depending on the tool, one or more cookies are set by the provider.

Name econda
Purpose Statistics
External media

Content from external media platforms is blocked by default. If cookies from external media are accepted, access to this content no longer requires manual consent.

Name YouTube
Purpose Show YouTube content
Name Twitter
Purpose activate Twitter Feeds

Crowd-powered Medical Image Annotation

© dkfz.de

With the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training has become one of the major bottlenecks in the field of surgical data science. To address this challenge, our group is investigating novel large-scale data annotation methods [1-9] based on crowdsourcing - a new concept that involves outsourcing tasks to anonymous workers in an online community. Our scientific innovations are mainly related to dealing with uncertainties of both the algorithms developed and the crowd. Main contributions include a new method for estimating an anonymous user's annotation performance based on clickstream analysis [5,7]. Furthermore, we were the first to investigate the concept of crowd-algorithm collaboration in the field of large-scale medical image annotation [1].

Key collaborators

  • Dr. Hannes Kenngott
    University of Heidelberg, Division of Visceral Surgery of the Department of General Surgery

  • Dr. Christian Stock 
    Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ)
  • Prof. Dr. Stefanie Speidel Translationale Chirurgische Onkologie, NCT Dresden

Funding

Selected publications

  1. Maier-Hein L, Ross T, Gröhl J, Glocker B, Bodenstedt S, Stock S, Heim E, et al. Crowd-Algorithm Collaboration for Large-Scale Endoscopic Image Annotation with Confidence. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 2016, pp 616–23, https://doi.org/10.1007/978-3-319-46723-8_71.

     

  2. Maier-Hein L, Kondermann D, Roß T, Mersmann S, Heim E, Bodenstedt S, Kenngott HG, et al. Crowdtruth Validation: A New Paradigm for Validating Algorithms That Rely on Image Correspondences. International Journal of Computer Assisted Radiology and Surgery. 2015; 10(8): pp 1201–12. https://doi.org/10.1007/s11548-015-1168-3.

 

  1. Maier-Hein L, Mersmann S, Kondermann D, Bodenstedt S, Sanchez A, Stock C, Kenngott HG, Eisenmann M, and Speidel S. Can Masses of Non-Experts Train Highly Accurate Image Classifiers?  In International Conference on Medical Image Computing and Computer-Assisted Intervention. 2014, pp 438–45, https://doi.org/10.1007/978-3-319-10470-6_55.

 

  1. Maier-Hein L, Mersmann S, Kondermann D, Stock C, Kenngott HG, Sanchez A, Wagner M, et al. Crowdsourcing for Reference Correspondence Generation in Endoscopic Images. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 2014, pp 349-56, https://doi.org/10.1007/978-3-319-10470-6_44.

 

  1. Heim E, Seitel A, Stock C, Ross T, and Maier-Hein L. Clickstreamanalyse zur Qualitätssicherung in der crowdbasierten Bildsegmentierung. In Bildverarbeitung für die Medizin 2017, 17–17.  https://link.springer.com/chapter/10.1007/978-3-662-54345-0_10.

 

  1. Heim E, Roß T, Norajitra T, Nolden M, März K, Kondermann D, Speidel S, Maier-Hein KH, and Maier-Hein L,  Crowdgestützte Organsegmentierung: Möglichkeiten und Grenzen, In 14. Jahrestagung der Deutschen Gesellschaft für Computer und Roboterassistierte Chirurgie 2015, pp. 37-42

 

  1. Heim E, Seitel A, Andrulis J, Isensee F, Stock C, Ross T, and Maier-Hein L. Clickstream Analysis for Crowd-Based Object Segmentation with Confidence. arXiv:1611.08527 [Cs], November 25, 2016. http://arxiv.org/abs/1611.08527.

 

  1. Ganz M, Kondermann D, Andrulis J, Knudsen GM, Maier-Hein, L, Crowdsourcing for error detection in cortical surface delineations. International Journal of Computer Assisted Radiology and Surgery. 2017;12(1):161-6, 10.1007/s11548-016-1445-9

 

  1. Roethlingshoefer V, Bittel S, Kenngott H, Wagner M, Bodenstedt S, Ross T, Speidel S, Maier-Hein L, How to Create the Largest In-Vivo Endoscopic Dataset, LABELS Workshop, MICCAI 2017

Awards

  • CURAC KUKA Best Paper Award (2015)
    Eric Heim et al. for the best CURAC paper 2015 “Crowdgestützte Organsegmentierung: Möglichkeiten und Grenzen,”  awarded at 14. Jahrestagung der Deutschen Gesellschaft für Computer und Roboterassistierte Chirurgie

  • IJCARS KUKA Best Paper Award 3rd place (2016)
    Ganz Melanie et al. for the IJCARS paper “Crowdsourcing for error detection in cortical surface delineations.” awarded at International Congress and Exhibition on Computer Assisted Radiology and Surgery (CARS)

to top
powered by webEdition CMS