Computer Assisted Medical Interventions

Biomedical Image Analysis ChallengeS (BIAS) Initiative


The importance of data science techniques in almost all fields of biomedicine is increasing at an enormous pace. This holds particularly true for the field of biomedical image analysis, which plays a crucial role in many areas including tumor detection, classification, staging and progression modeling as well as automated analysis of cancer cell images acquired using microscopy.

While clinical trials are the state-of-the-art methods to assess the effect of new medication in a comparative manner, benchmarking in the field of image analysis is performed by so-called challenges. Challenges are international competitions, typically hosted by individual researchers, institutes, or societies, that aim to assess the performance of multiple algorithms on identical data sets and encourage benchmarking. They are often published in prestigious journals, are associated with significant amounts of prize money (up to €1 million on platforms like Kaggle) and receive a huge amount of attention, indicated by the number of downloads, citations and views. Our recent comprehensive analysis of biomedical image analysis challenges (Maier-Hein et al., 2018), which involved 38 researchers from 30 institutes worldwide, however, revealed a huge discrepancy between the impact of a challenge and the quality (control) of the design and reporting standard. We showed that (1) "common practice related to challenge reporting is poor and does not allow for adequate interpretation and reproducibility of results"', (2) "challenge design is very heterogeneous and lacks common standards, although these are requested by the community"' and (3) "challenge rankings are sensitive to a range of challenge design parameters, such as the metric variant applied, the type of test case aggregation performed and the observer annotating the data" (Maier-Hein et al., 2018). We also showed that security holes in challenge design can potentially be exploited by both challenge organizers and participants to tune rankings (e.g. by selective test case submission (participants) or retrospective tuning of the ranking scheme (organizers)) (Reinke et al., 2018). The conclusion from our studies was that "journal editors and reviewers should provide motivation to raise challenge quality by establishing a rigorous review process."

The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network is a global initiative with the aim of improving the quality of research publications and research itself. A key mission in this context is to achieve accurate, complete and transparent reporting of health research studies to support reproducibility and usefulness. A core activity of the network is to assist in the development, dissemination and implementation of robust reporting guidelines, where a guideline is defined as "a checklist, flow diagram or structured text to guide authors in reporting a specific type of research" (The EQUATOR network, 2008). Between 2006 and 2019, more than 400 reporting guidelines have been published under the umbrella of the Equator network. A well-known guideline is the CONSORT statement developed for reporting of randomized controlled trials. Prominent journals, such as Lancet, Jama or the British Medical Journal require the CONSORT checklist to be submitted along with the actual paper when reporting results of a randomized controlled trial.

Inspired by this success story, the Biomedical Image Analysis ChallengeS (BIAS) initiative was founded by the MICCAI board challenge working group (, lead by Prof. Dr. Lena Maier-Hein. Our goal is to bring biomedical image analysis challenges to the next level of quality.

As a first step towards better scientific practice, we presented a guideline to standardize and facilitate the writing and reviewing process of biomedical image analysis challenges and help readers of challenges interpret and reproduce results by making relevant information explicit (Maier-Hein et al., 2020). The guideline was used to enhance the quality of challenge designs and to simplify the challenge submission process for prestigious conferences (MICCAI 2018, 2019 and 2020, the IEEE International Symposium on Biomedical Imaging (ISBI) 2020 and the Conference on Medical Imaging with Deep Learning (MIDL) 2020). In our most recent contribution (Wiesenfarth et al., 2019), we presented methodology along with an open-source framework to facilitate the visualization of challenge results.

As stated above, common practice in biomedical challenge organization can be exploited to tune challenge rankings. To prevent such incidents and to improve the quality of challenges, the MICCAI board challenge working group and the MICCAI 2020 Satellite Event team decided to introduce the concept of challenge registration. Similar to how clinical trials have to be registered before starting, the complete design of accepted MICCAI challenges will be put online before the challenges take place. Changes to the design (e.g. to the metrics or ranking schemes applied) must be well-justified and officially be registered online (as a new version of the challenge design). Registered challenges are listed here:


  • Lena Maier-Hein (Chair), Annika Reinke, Matthias Eisenmann, Sinan Onogur, Minu D. Tizabi, Tim Rädsch, Doreen Heckmann-Nötzel, Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ)
  • Annette Kopp-Schneider, Division of Biostatistics, DKFZ
  • Spyridon (Spyros) Bakas, Center for Biomedical Image Computing & Analytics (CBICA), Perelman School of Medicine, University of Pennsylvania
  • Michal Kozubek, Centre for Biomedical Image Analysis, Masaryk University
  • Bennett A. Landman, Electrical Engineering, Vanderbilt University
  • Anne L. Martel, Physical Sciences, Sunnybrook Research Institute; Department Medical Biophysics, University of Toronto
  • Tal Arbel, Centre for Intelligent Machines, McGill University
  • Allan Hanbury, Institute of Information Systems Engineering, Technische Universität (TU) Wien; Complexity Science Hub Vienna
  • Pierre Jannin, Laboratoire Traitement du Signal et de l’Image (LTSI) - UMR_S 1099, Université de Rennes 1, Inserm
  • Henning Müller, University of Applied Sciences Western Switzerland (HES-SO); Medical Faculty, University of Geneva
  • Julio Saez-Rodriguez, Institute of Computational Biomedicine, Heidelberg University; Faculty of Medicine, Heidelberg, University Hospital; Joint Research Centre for Computational Biomedicine, Westfälische Technische Hochschule (RWTH) Aachen
  • Bram van Ginneken, Radboud University Medical Center; Fraunhofer MEVIS
  • Paul Jäger, Fabian Isensee, Jens Petersen, Klaus Maier-Hein, Division of Medical Image Computing, DKFZ
  • Carole H. Sudre, School of Biomedical Engineering and Imaging Science, King's College London (KCL); Centre for Medical Image Computing, University College London (UCL); MRC Unit for Lifelong Health and Ageing at UCL
  • Michela Antonelli, Centre for Medical Image Computing, UCL; School of Biomedical Engineering and Imaging Science, KCL
  • M. Jorge Cardoso, Department of Biomedical Engineering, School of Biomedical Engineering & Imaging Sciences, KCL; Department of Medical Physics and Biomedical Engineering, UCL
  • Veronika Cheplygina, IT University of Copenhagen
  • Keyvan Farahani, Center for Biomedical Informatics and Information Technology, National Cancer Institute (NIH)
  • Ben Glocker, Biomedical Image Analysis Group, Department of Computing, Imperial College London
  • Charles E. Kahn, Perelman School of Medicine, University of Pennsylvania
  • Jens Kleesiek, Translational Image-guided Oncology (TIO), Institute for AI in Medicine (IKIM), University Medicine Essen
  • Tashin Kurc, Stony Brook Cancer Center, Stony Brook University
  • Geert Litjens, Department of Pathology, Radboud University Medical Center; Radboud University Medical Center, Radboud Institute for Health Sciences
  • Bjoern Menze, Department of Quantitative Biomedicine, University of Zurich
  • Mauricio Reyes, Healthcare Imaging A.I., Insel Data Science Center, Bern University Hospital
  • Nicola Rieke, NVIDIA; Technical University of Munich (TUM)
  • Bram Stieltjes, Department of Radiology, University Hospital of Basel
  • Ronald M. Summers, Radiology and Imaging Sciences, Clinical Center, NIH
  • Sotirios A. Tsaftaris, School of Engineering, The University of Edinburgh


Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A. P., Carass, A., Feldmann, C., Frangi, A. F., Full, P. M., van Ginneken, B., Hanbury, A., Honauer, K., Kozubek, M., Landman, B. A., März, K., ... Kopp-Schneider, A. (2018). Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications, 9(1), 5217.

Maier-Hein, L., Reinke, A., Kozubek, M., Martel, A. L., Arbel, T., Eisenmann, M., Hanbuary, A., Jannin, P., Müller, H., Onogur, S., Saez-Rodriguez, J., van Ginneken, B., Kopp-Schneider, A., & Landman, B. (2020). BIAS: Transparent reporting of biomedical image analysis challenges. Medical Image Analysis, 101796.

Reinke, A., Eisenmann, M., Onogur, S., Stankovic, M., Scholz, P., Full, P. M., Bogunovic, H., Landman, B. A., Maier, O., Menze, B., Sharp, G. C., Sirinukunwattana, K., Speidel, S., van der Sommen, F., Zheng, G., Müller, H., Kozubek, M., Arbel, T., Bradley, A. P., ... Maier-Hein, L. (2018). How to Exploit Weaknesses in Biomedical Challenge Design and Organization. In A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-López, & G. Fichtinger (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (pp. 388–395). Springer International Publishing.

Wiesenfarth, M., Reinke, A., Landman, B. A., Eisenmann, M., Saiz, L. A., Cardoso, M. J., Maier-Hein, L., & Kopp-Schneider, A. (2021). Methods and open-source toolkit for analyzing and visualizing challenge results. Scientific Reports, 11(1), 2369.

The EQUATOR network – Enhancing the QUAlity and Transparency Of health Research., 2008. Accessed: 2019-09-12.


to top
powered by webEdition CMS