Catálogo de publicaciones - libros

Systematic evaluation has had a strong impact on many data analysis domains, for example, TREC and CLEF in information retrieval, ImageCLEF in image retrieval, and many challenges in conferences such as MICCAI for medical imaging and ICPR for pattern recognition. With Kaggle, a platform for machine learning challenges has also had a significant success in crowdsourcing solutions. This shows the importance to systematically evaluate algorithms and that the impact is far larger than simply evaluating a single system. Many of these challenges also showed the limits of the commonly used paradigm to prepare a data collection and tasks, distribute these and then evaluate the participants’ submissions. Extremely large datasets are cumbersome to download, while shipping hard disks containing the data becomes impractical. Confidential data can often not be shared, for example medical data, and also data from company repositories. Real-time data will never be available via static data collections as the data change over time and data preparation often takes much time. The Evaluation-as-a-Service (EaaS) paradigm tries to find solutions for many of these problems and has been applied in the VISCERAL project. In EaaS, the data are not moved but remain on a central infrastructure. In the case of VISCERAL, all data were made available in a cloud environment. Participants were provided with virtual machines on which to install their algorithms. Only a small part of the data, the training data, was visible to participants. The major part of the data, the test data, was only accessible to the organizers who ran the algorithms in the participants’ virtual machines on the test data to obtain impartial performance measures.

This chapter gives a brief overview of the VISCERAL Registration System that is used for all the VISCERAL Benchmarks and is released as open source on GitHub. The system can be accessed by both participants and administrators, reducing the direct participant–organizer interaction and handling the documentation available for each of the benchmarks organized by VISCERAL. Also, the upload of the VISCERAL usage and participation agreements is integrated, as well as the attribution of virtual machines that allow participation in the VISCERAL Benchmarks. In the second part, a summary of the various steps in the continuous evaluation chain mainly consisting of the submission, algorithm execution and storage as well as the evaluation of results is given. The final part consists of the cloud infrastructure detail, describing the process of defining requirements, selecting a cloud solution provider, setting up the infrastructure and running the benchmarks. This chapter concludes with a short experience report outlining the encountered challenges and lessons learned.

This chapter describes the ethical and privacy aspects of using medical data in the context of the VISCERAL project. The project had as main goals the creation of a benchmark for organ segmentation, landmark detection, lesion detection and similar case retrieval. The availability of a large amount of imaging data was extremely important for the project goals, and thus, we present an analysis of the procedures that were followed for getting access to the data from IRB (internal review board) approval to data extraction and usage. This chapter details the requirements stated by medical ethics committees in three partner countries that supplied data. The exact procedure from request to data distribution is explained. The specific requirements of each data provider (each from a different country) are described in detail. The final data collection was made available in anonymized form in the Microsoft Azure cloud with the restriction of having it on servers that are located inside the European Union.

This chapter describes the annotation of the medical image data that were used in the VISCERAL project. Annotation of regions in the 3D images is non-trivial, and tools need to be chosen to limit the manual work and have semi-automated annotation available. For this, several tools that were available free of charge or with limited costs were tested and compared. The GeoS tool was finally chosen for the annotation based on the detailed analysis, allowing for efficient and effective annotations. 3D slice was chosen for smaller structures with low contrast to complement the annotations. A detailed quality control was also installed, including an automatic tool that attributes organs to annotate and volumes to specific annotators, and then compares results. This allowed to judge the confidence in specific annotators and also to iteratively refine the annotation instructions to limit the subjectivity of the task as much as possible. For several structures, some subjectivity remains and this was measured via double annotations of the structure. This allows the judgement of the quality of automatic segmentations.

In the VISCERAL project, several datasets containing medical imaging data and corresponding manual expert annotations have been created. These datasets were used for training and evaluation of participant algorithms in the VISCERAL Benchmarks. In addition to Gold Corpus datasets, the architecture of VISCERAL enables the creation of annotations of far larger datasets, which are generated by the collective ensemble of submitted algorithms. In this chapter, three Gold Corpus datasets created for the VISCERAL Anatomy, Detection and Retrieval Benchmarks are described. Additionally, we present two datasets that have been created as a result of the anatomy and retrieval challenge.

This chapter provides an overview of the metrics used in the VISCERAL segmentation benchmarks, namely Anatomy 1, 2 and 3. In particular, it provides an overview of 20 evaluation metrics for segmentation, from which four metrics were selected to be used in VISCERAL benchmarks. It also provides an analysis of these metrics in three ways: first by analysing fuzzy implementations of these metrics using fuzzy segmentations produced either synthetically or by fusing participant segmentations and second by comparing segmentation rankings produced by these metrics with rankings performed manually by radiologists. Finally, a metric selection is performed using an automatic selection framework, and the selection result is validated using the manual rankings. Furthermore, this chapter provides an overview of metrics used for the Lesion Detection Benchmark.

While a growing number of benchmark studies compare the performance of algorithms for automated organ segmentation or lesion detection in images with restricted fields of view, few efforts have been made so far towards benchmarking these and related routines for the automated identification of bones, inner organs and relevant substructures visible in an image volume of the abdomen, the trunk or the whole body. The VISCERAL project has organized a series of benchmark editions designed for segmentation and landmark localization in medical images of multiple modalities, resolutions and fields of view acquired during daily clinical routine work. Participating groups are provided with data and computing resources on a cloud-based framework, where they can develop and test their algorithms, the submitted executables of which are then run and evaluated on unseen test data by the VISCERAL organizers.

Health providers currently construct their differential diagnosis for a given medical case most often based on textbook knowledge and clinical experience. Data mining of the large amount of medical records generated daily in hospitals is only very rarely done, limiting the reusability of these cases. As part of the VISCERAL project, the Retrieval benchmark was organized to evaluate available approaches for medical case-based retrieval. Participant algorithms were required to find and rank relevant medical cases from a large multimodal dataset (including semantic RadLex terms extracted from text and visual 3D data) for common query topics. The relevance assessment of the cases was done by medical experts who selected cases that are useful for a differential diagnosis for the given query case. The approaches that integrated information from both the RadLex terms and the 3D volumes (mixed techniques) obtained the best results based on five standard evaluation metrics. The benchmark set up, dataset description and result analysis are presented.

Automatic segmentation of anatomical structures in CT scans is an essential step in the analysis of radiological patient data and is a prerequisite for large-scale content-based image retrieval (CBIR). Many existing segmentation methods are tailored to a single structure and/or require an atlas, which entails multistructure deformable registration and is time-consuming. We present a fully automatic atlas-free segmentation of multiple organs of the ventral cavity in contrast-enhanced CT scans of the whole trunk (CECT). Our method uses a pipeline approach based on the rules that determine the order in which the organs are isolated and how they are segmented. Each organ is individually segmented with a generic four-step procedure. Our method is unique in that it does not require any predefined atlas or a costly registration step and in that it uses the same generic segmentation approach for all organs. Experimental results on the segmentation of seven organs—liver, left and right kidneys, left and right lungs, trachea, and spleen—on 20 CECT scans of the VISCERAL Anatomy training dataset and 10 CECT scans of the test dataset yield an average DICE volume overlap similarity score of 90.95 and 88.50%, respectively.

In this chapter, we introduce an automatic multiorgan segmentation method using a hierarchical-shape-prior-guided level set method. The hierarchical shape priors are organized according to the anatomical hierarchy of the human body, so that the children structures are always contained by the parent structure. This hierarchical approach solves two challenges of multiorgan segmentation. First, it gradually refines the prediction of the organs’ position by locating and segmenting the larger parent structure. Second, it solves the ambiguity of boundary between two attaching organs by looking at a large scale and imposing the additional shape constraint of the higher-level structures. To improve the segmentation accuracy, a model-guided local phase term is introduced and integrated with the conventional region-based energy function to guide the level set propagation. Finally, a novel coherent propagation method is implemented to speed up the model-based level set segmentation. In the VISCERAL Anatomy challenge, the proposed method delivered promising results on a number of abdominal organs.