Research
- Research Topics
- Cell Biology and Tumor Biology
- Stem Cells and Cancer
- Inflammatory Stress in Stem Cells
- Experimental Hematology
- Molecular Embryology
- Signal Transduction and Growth Control
- Epigenetics
- Redox Regulation
- Vascular Oncology and Metastasis
- Clinical Neurobiology
- Molecular Neurogenetics
- Chaperones and Proteases
- Vascular Signaling and Cancer
- Molecular Neurobiology
- Mechanisms Regulating Gene Expression
- Molecular Biology of Centrosomes and Cilia
- Dermato-Oncology
- Pediatric Leukemia
- Molecular Hematology - Oncology
- Cancer Progression and Metastasis
- Metastatic Niches
- Molecular Leukemogenesis
- DNA Repair Mechanisms and Cancer
- Neuronal Signaling and Morphogenesis
- Cell Signaling and Metabolism
- Cell Fate Engineering and Disease Modeling
- Cancer Drug Development
- Cell Morphogenesis and Signal Transduction
- Tumour Metabolism and Microenvironment
- Helmholtz Professorship Cell Biology
- Functional and Structural Genomics
- Molecular Genome Analysis
- Molecular Genetics
- Pediatric Neurooncology
- Cancer Genome Research
- Chromatin Networks
- Functional Genome Analysis
- Theoretical Systems Biology
- Neuroblastoma Genomics
- Signaling and Functional Genomics
- Signal Transduction in Cancer and Metabolism
- RNA Biology and Cancer
- Systems Biology of Signal Transduction
- Molecular thoracic Oncology
- Proteomics of Stem Cells and Cancer
- Computational Genomics and System Genetics
- Regulatory Genomics and Cancer Evolution
- Applied Functional Genomics
- Applied Bioinformatics
- Translational Medical Oncology
- Cancer Epigenomics
- Translational Pediatric Sarcoma Research
- Neuropathology
- Pediatric Oncology
- Neurooncology
- Somatic Evolution and Early Detection
- Translational Control and Metabolism
- Pediatric Glioma Research
- Soft-Tissue Sarcoma
- Precision Sarcoma Research
- Neural Stem Cell Genome Integrity in Brain Development and Diseases
- Mechanisms of Leukemogenesis
- Genome Instability in Tumors
- Developmental Origins of Pediatric Cancer
- Brain Tumor Translational Targets
- Brain Cancer Metabolism
- Translational Functional Cancer Genomics
- SPRINT
- Cancer Risk Factors and Prevention
- Cancer Epidemiology
- Biostatistics
- Clinical Epidemiology and Aging Research
- Health Economics
- Physical Activity, Prevention and Cancer
- Preventive Oncology
- Digital Biomarkers for Oncology
- Genomic Epidemiology
- Cancer Survivorship
- Molecular Epidemiology
- Immunology and Cancer
- Cellular Immunology
- Molecular Oncology of Gastrointestinal Tumors
- Translational Immunotherapy
- B Cell Immunology
- Immune Diversity
- Structural Biology of Infection and Immunity
- Applied Tumor-Immunity
- Neuroimmunology and Brain Tumor Immunology
- T Cell Metabolism
- Adaptive Immunity and Lymphoma
- Immune Regulation in Cancer
- GMP & T Cell Therapy
- News
- Imaging and Radiooncology
- Radiology
- Research
- Computational Radiology Research Group
- Contrast Agents In Radiology Research Group
- Neuro-Oncologic Imaging Research Group
- Radiological Early Response Assessment Of Modern Cancer Therapies
- Imaging In Monoclonal Plasma Cell Disorders
- 7 Tesla MRI - Novel Imaging Biomarkers
- Functional Imaging
- Visualization And Forensic Imaging
- PET/MRI
- Dual- and Multienergy CT
- Radiomics Research Group
- Prostate Research Group
- Breast Imaging Research Group
- Bone marrow
- Musculoskeletal Imaging
- Staff
- Patients
- Research
- Medical Physics in Radiology
- X-Ray Imaging and Computed Tomography
- Medical Informatics for Translational Oncology
- Federated Information Systems
- Medical Physics in Radiation Oncology
- Biomedical Physics in Radiation Oncology
- Computer Assisted Medical Interventions
- Medical Image Computing
- Radiooncology - Radiobiology
- Radiation Oncology
- Molecular Radiooncology
- Nuclear Medicine
- Medical Imaging and Radiology - Cancer Prevention
- Molecular Biology of Systemic Radiotherapy
- Translational Radiation Oncology
- Molecular Mechanisms of Head and Neck Tumors
- Radiology
- Infection, Inflammation and Cancer
- Tumor Virology
- Viral Transformation Mechanisms
- Pathogenesis of Virus-Associated Tumors
- Applied Tumor Biology
- Virotherapy
- Virus-associated Carcinogenesis
- Chronic Inflammation and Cancer
- Microbiome and Cancer
- Cell Plasticity and Epigenetic Remodeling
- Cellular Polarity and Viral Infection
- Noroviruses
- Laboratory of Oncolytic Virus Immuno-Therapeutics
- Infections and Cancer Epidemiology
- Tumorvirus-specific Vaccination Strategies
- Mammalian Cell Cycle Control Mechanisms
- Molecular Therapy of Virus-Associated Cancers
- Immunotherapy and Immunoprevention - Molecular Vaccine Design
- DNA Vectors
- Episomal-Persistent DNA in Cancer- and Chronic Diseases
- Other Units
- Cell Biology and Tumor Biology
- Research Groups A-Z
- Junior Research Groups
- Core Facilities
- Center for Preclinical Research
- Chemical Biology Core Facility
- Electron Microscopy
- Flow Cytometry
- Genomics and Proteomics
- Information Technology
- Library
- Kataloge - Catalogues
- Zeitschriften - Journals
- E-Books - Ebooks
- Datenbanken - Databases
- Dokument-Lieferung - Document Delivery
- Publikationsdatenbank - Publication database
- DKFZ Archiv - DKFZ Archive
- Open Access
- Science 2.0
- Ansprechpartner - Contact
- More Information - Service
- Anschrift - Address
- Antiquariat - Second Hand
- Aufstellungssystematik - Shelf Classification
- Ausleihe - Circulation
- Benutzerhinweise - Library Use
- Beschaffungsvorschläge - Desiderata
- Fakten und Zahlen - Facts and Numbers
- Kooperationen, Konsortien - Cooperations, Consortia
- Kopieren, Scannen - Copying, Scans
- Kurse, Führungen - Courses, Introductions
- Öffnungszeiten - Opening Hours
- DEAL-Info
- DKFZ-Intern - internal only
- Light Microscopy
- Omics IT and Data Management Core Facility
- Small Animal Imaging
- Metabolomics Core Technology Platform
- Data Science @ DKFZ
- INFORM
- Databox
- Baden-Württemberg Cancer Registry
- Cooperations & Networks
- National Cooperations
- International Cooperations
- Cooperational Research Program with Israel: DKFZ - MOST in Cancer Research
- Program
- Members of the Program Committee
- Call 2019
- Publication Database
- German-Israeli Cancer Research Schools
- Archive
- Heidelberg - Israel, Science and Culture
- Symposium 40 Years of German-Israeli Cooperation
- 35th Anniversary Symposium
- 34th Meeting of the DKFZ-MOST Program
- 40th Anniversary Publication
- 30th Anniversary Publication
- 20th Anniversary Publication
- Flyer - The Cancer Cooperation Program
- List Publications 1976-2004
- Highlight-Projects
- Cooperational Research Program with Israel: DKFZ - MOST in Cancer Research
- Cooperations with industrial companies
- Helmholtz Alliances
- DKFZ PostDoc Network
- Cross Program Topic RNA@DKFZ
- Cross Program Topic Epigenetics@dkfz
- Cross Program Topic Single Cell Sequencing
- WHO Collaborating Centers
- DKFZ Site Dresden
Unraveling uncertainty in benchmarking: Methods and open-source toolkit for analyzing and visualizing challenge results
Although the scientific impact of biomedical image analysis challenges is steadily increasing, there is surprisingly a huge discrepancy between the challenges' impact and their quality control. In particular, challenge rankings are sensitive to a range of challenge design parameters. For example, rankings and thus the identified challenge winner may strongly depend on the chosen ranking method or on a couple of test cases. Thus, the validity and transferability of challenge results may be questioned due to possibly considerable instabilities in rankings. Yet, most publications of challenges ignore the uncertainty associated with rankings and result presentations are often limited to the ranking list and simple visualizations of the metric values for each algorithm.
Thus, the purpose of this work is to propose methodology along with an open-source framework for systematically analyzing and visualizing results of challenges. It intends to help challenge organizers and participants to gain further insights into both the algorithms' performance and the assessment data set itself in an intuitive manner.
Visualization approaches for both challenges designed around a single task and for challenges comprising multiple tasks are presented. The proposed tools involve bootstrapping, significance testing and unsupervised learning. They allow to investigate questions such as, e.g., whether there are influential test cases, whether the winner is consistently superior to other algorithms across test cases, whether the winner is significantly superior, what range of ranks for a specific algorithm is supported by the data, which task yields clear separation of algorithms and a stable ranking and which tasks are similar with respect to their rankings.
All techniques are illustrated by synthetic and real-world assessment data.
Dr. Manuel Wiesenfarth

Division of Biostatistics, DKFZ