Computer Assisted Medical Interventions

Biomedical Image Analysis ChallengeS (BIAS) Initiative

© dkfz.de

The importance of data science techniques in almost all fields of biomedicine is increasing at an enormous pace. This holds particularly true for the field of biomedical image analysis, which plays a crucial role in many areas including tumor detection, classification, staging and progression modeling as well as automated analysis of cancer cell images acquired using microscopy.

While clinical trials are the state-of-the-art methods to assess the effect of new medication in a comparative manner, benchmarking in the field of image analysis is performed by so-called challenges. Challenges are international competitions, typically hosted by individual researchers, institutes, or societies, that aim to assess the performance of multiple algorithms on identical data sets and encourage benchmarking. They are often published in prestigious journals, are associated with significant amounts of prize money (up to €1 million on platforms like Kaggle) and receive a huge amount of attention, indicated by the number of downloads, citations and views. Our recent comprehensive analysis of biomedical image analysis challenges (Maier-Hein et al., 2018), which involved 38 researchers from 30 institutes worldwide, however, revealed a huge discrepancy between the impact of a challenge and the quality (control) of the design and reporting standard. We showed that (1) "common practice related to challenge reporting is poor and does not allow for adequate interpretation and reproducibility of results"', (2) "challenge design is very heterogeneous and lacks common standards, although these are requested by the community"' and (3) "challenge rankings are sensitive to a range of challenge design parameters, such as the metric variant applied, the type of test case aggregation performed and the observer annotating the data" (Maier-Hein et al., 2018). We also showed that security holes in challenge design can potentially be exploited by both challenge organizers and participants to tune rankings (e.g. by selective test case submission (participants) or retrospective tuning of the ranking scheme (organizers)) (Reinke et al., 2018). The conclusion from our studies was that "journal editors and reviewers should provide motivation to raise challenge quality by establishing a rigorous review process."

The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network is a global initiative with the aim of improving the quality of research publications and research itself. A key mission in this context is to achieve accurate, complete and transparent reporting of health research studies to support reproducibility and usefulness. A core activity of the network is to assist in the development, dissemination and implementation of robust reporting guidelines, where a guideline is defined as "a checklist, flow diagram or structured text to guide authors in reporting a specific type of research" (The EQUATOR network, 2008). Between 2006 and 2019, more than 400 reporting guidelines have been published under the umbrella of the Equator network. A well-known guideline is the CONSORT statement developed for reporting of randomized controlled trials. Prominent journals, such as Lancet, Jama or the British Medical Journal require the CONSORT checklist to be submitted along with the actual paper when reporting results of a randomized controlled trial.

Inspired by this success story, the Biomedical Image Analysis ChallengeS (BIAS) initiative was founded by the MICCAI board challenge working group (http://www.miccai.org/events/challenges/), lead by Prof. Dr. Lena Maier-Hein. Our goal is to bring biomedical image analysis challenges to the next level of quality.

As a first step towards better scientific practice, we presented a guideline to standardize and facilitate the writing and reviewing process of biomedical image analysis challenges and help readers of challenges interpret and reproduce results by making relevant information explicit (Maier-Hein et al., 2020). The guideline was used to enhance the quality of challenge designs and to simplify the challenge submission process for prestigious conferences (MICCAI 2018, 2019 and 2020, the IEEE International Symposium on Biomedical Imaging (ISBI) 2020 and the Conference on Medical Imaging with Deep Learning (MIDL) 2020). In our most recent contribution (Wiesenfarth et al., 2019), we presented methodology along with an open-source framework to facilitate the visualization of challenge results.

As stated above, common practice in biomedical challenge organization can be exploited to tune challenge rankings. To prevent such incidents and to improve the quality of challenges, the MICCAI board challenge working group and the MICCAI 2020 Satellite Event team decided to introduce the concept of challenge registration. Similar to how clinical trials have to be registered before starting, the complete design of accepted MICCAI challenges will be put online before the challenges take place. Changes to the design (e.g. to the metrics or ranking schemes applied) must be well-justified and officially be registered online (as a new version of the challenge design). Registered challenges are listed here: http://www.miccai.org/events/challenges/.

Members:

  • Lena Maier-Hein (Chair), Annika Reinke, Matthias Eisenmann, Sinan Onogur, Division of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ)
  • Spyridon (Spyros) Bakas, Center for Biomedical Image Computing & Analytics (CBICA), Perelman School of Medicine, University of Pennsylvania
  • Annette Kopp-Schneider, Division of Biostatistics, DKFZ
  • Michal Kozubek, Centre for Biomedical Image Analysis, Masaryk University
  • Bennett A. Landman, Electrical Engineering, Vanderbilt University
  • Anne L. Martel, Physical Sciences, Sunnybrook Research Institute; Department Medical Biophysics, University of Toronto
  • Tal Arbel, Centre for Intelligent Machines, McGill University
  • Allan Hanbury, Institute of Information Systems Engineering, Technische Universität (TU) Wien; Complexity Science Hub Vienna
  • Pierre Jannin, Laboratoire Traitement du Signal et de l’Image (LTSI) - UMR_S 1099, Université de Rennes 1, Inserm
  • Henning Müller, University of Applied Sciences Western Switzerland (HES-SO); Medical Faculty, University of Geneva
  • Julio Saez-Rodriguez, Institute of Computational Biomedicine, Heidelberg University; Faculty of Medicine, Heidelberg, University Hospital; Joint Research Centre for Computational Biomedicine, Westfälische Technische Hochschule (RWTH) Aachen
  • Bram van Ginneken, Radboud University Medical Center, Nijmegen

References

Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A. P., Carass, A., Feldmann, C., Frangi, A. F., Full, P. M., van Ginneken, B., Hanbury, A., Honauer, K., Kozubek, M., Landman, B. A., März, K., ... Kopp-Schneider, A. (2018). Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications, 9(1), 5217. https://doi.org/10.1038/s41467-018-07619-7

Maier-Hein, L., Reinke, A., Kozubek, M., Martel, A. L., Arbel, T., Eisenmann, M., Hanbuary, A., Jannin, P., Müller, H., Onogur, S., Saez-Rodriguez, J., van Ginneken, B., Kopp-Schneider, A., & Landman, B. (2020). BIAS: Transparent reporting of biomedical image analysis challenges. Medical Image Analysis, 101796. https://doi.org/10.1016/j.media.2020.101796

Reinke, A., Eisenmann, M., Onogur, S., Stankovic, M., Scholz, P., Full, P. M., Bogunovic, H., Landman, B. A., Maier, O., Menze, B., Sharp, G. C., Sirinukunwattana, K., Speidel, S., van der Sommen, F., Zheng, G., Müller, H., Kozubek, M., Arbel, T., Bradley, A. P., ... Maier-Hein, L. (2018). How to Exploit Weaknesses in Biomedical Challenge Design and Organization. In A. F. Frangi, J. A. Schnabel, C. Davatzikos, C. Alberola-López, & G. Fichtinger (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (pp. 388–395). Springer International Publishing. https://doi.org/10.1007/978-3-030-00937-3_45

Wiesenfarth, M., Reinke, A., Landman, B. A., Cardoso, M. J., Maier-Hein, L., & Kopp-Schneider, A. (2019). Methods and open-source toolkit for analyzing and visualizing challenge results. ArXiv:1910.05121 [Cs, Stat]. http://arxiv.org/abs/1910.05121

The EQUATOR network – Enhancing the QUAlity and Transparency Of health Research. http://www.equator-network.org, 2008. Accessed: 2019-09-12.

 

to top
powered by webEdition CMS