The importance of data science techniques in almost all fields of biomedicine is increasing at an enormous pace. This holds particularly true for the field of biomedical image analysis, which plays a crucial role in many areas including tumor detection, classification, staging and progression modeling as well as automated analysis of cancer cell images acquired using microscopy.
While clinical trials are the state-of-the-art methods to assess the effect of new medication in a comparative manner, benchmarking in the field of image analysis is performed by so-called challenges. Challenges are international competitions, typically hosted by individual researchers, institutes, or societies, that aim to assess the performance of multiple algorithms on identical data sets and encourage benchmarking. They are often published in prestigious journals, are associated with significant amounts of prize money (up to €1 million on platforms like Kaggle) and receive a huge amount of attention, indicated by the number of downloads, citations and views. Our recent comprehensive analysis of biomedical image analysis challenges (Maier-Hein et al., 2018), which involved 38 researchers from 30 institutes worldwide, however, revealed a huge discrepancy between the impact of a challenge and the quality (control) of the design and reporting standard. We showed that (1) "common practice related to challenge reporting is poor and does not allow for adequate interpretation and reproducibility of results"', (2) "challenge design is very heterogeneous and lacks common standards, although these are requested by the community"' and (3) "challenge rankings are sensitive to a range of challenge design parameters, such as the metric variant applied, the type of test case aggregation performed and the observer annotating the data" (Maier-Hein et al., 2018). We also showed that security holes in challenge design can potentially be exploited by both challenge organizers and participants to tune rankings (e.g. by selective test case submission (participants) or retrospective tuning of the ranking scheme (organizers)) (Reinke et al., 2018). The conclusion from our studies was that "journal editors and reviewers should provide motivation to raise challenge quality by establishing a rigorous review process."
The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network is a global initiative with the aim of improving the quality of research publications and research itself. A key mission in this context is to achieve accurate, complete and transparent reporting of health research studies to support reproducibility and usefulness. A core activity of the network is to assist in the development, dissemination and implementation of robust reporting guidelines, where a guideline is defined as "a checklist, flow diagram or structured text to guide authors in reporting a specific type of research" (The EQUATOR network, 2008). Between 2006 and 2019, more than 400 reporting guidelines have been published under the umbrella of the Equator network. A well-known guideline is the CONSORT statement developed for reporting of randomized controlled trials. Prominent journals, such as Lancet, Jama or the British Medical Journal require the CONSORT checklist to be submitted along with the actual paper when reporting results of a randomized controlled trial.
Inspired by this success story, the Biomedical Image Analysis ChallengeS (BIAS) initiative was founded by the MICCAI board challenge working group, since 2021 known as MICCAI Special Interest Group on Biomedical Image Analysis Challenges (https://miccai.org/index.php/special-interest-groups/challenges/), lead by Prof. Dr. Lena Maier-Hein. Our goal is to bring biomedical image analysis challenges to the next level of quality.
As a first step towards better scientific practice, we presented a guideline (Maier-Hein et al. 2020) to standardize and facilitate the writing and reviewing process of biomedical image analysis challenges and help readers of challenges interpret and reproduce results by making relevant information explicit (Maier-Hein et al., 2020). The guideline was used to enhance the quality of challenge designs and to simplify the challenge submission process for prestigious conferences (MICCAI 2018, 2019 and 2020, the IEEE International Symposium on Biomedical Imaging (ISBI) 2020 and the Conference on Medical Imaging with Deep Learning (MIDL) 2020), see https://www.biomedical-challenges.org/. In our most recent contribution (Wiesenfarth et al., 2019), we presented methodology along with an open-source framework to facilitate the visualization of challenge results.
A light version fo the checklist which can be used for journal submission and reviews of challenge reports, can be downloaded here.
As stated above, common practice in biomedical challenge organization can be exploited to tune challenge rankings. To prevent such incidents and to improve the quality of challenges, the MICCAI board challenge working group and the MICCAI 2020 Satellite Event team decided to introduce the concept of challenge registration. Similar to how clinical trials have to be registered before starting, the complete design of accepted MICCAI challenges will be put online before the challenges take place. Changes to the design (e.g. to the metrics or ranking schemes applied) must be well-justified and officially be registered online (as a new version of the challenge design). Registered challenges are listed here: http://www.miccai.org/special-interest-groups/challenges/miccai-registered-challenges/.
The BIAS initiative furthermore aims to provide best practice recommendations for choosing the correct performance metrics for a specific biomedical research problem. Metrics are the key to objective, transparent and comparative performance assessment in the field of image analysis. Unfortunately, the field suffers from a lack of guidelines/standards for choosing appropriate performance metrics that reflect the task-specific needs, and practical pitfalls in metric usage are often overseen (Reinke et al. 2021). To address this issue, the BIAS initiative founded a large consortium with more than 60 international experts (including researchers from Google Health, Harvard University, Imperial College, Radiological Society of North America (RSNA) and National Institutes of Health (NIH)) aiming to provide guidelines and tools for choosing performance metrics in a problem-aware manner. The resulting recommendation framework involves capturing the characteristics of the given research problem in a problem fingerprint, which comprises domain interest-related, target structure-related and test set-related properties. Based on this fingerprint, an image processing task-specific mapping is applied to match the task to a set of metrics that reflect the needs of the target domain.