Dissertation

Assessing Methods and Tools to Improve Reporting, Increase Transparency, and Reduce Failures in Machine Learning Applications in Healthcare

Garbin, Christian

2020

Overview

Artificial intelligence (AI) had a few false starts – the AI winters of the 1970s and 1980s. We are now in what looks like an AI summer. There are many useful applications of AI in the field. But there are still unfulfilled promises and outright failures. From self-driving cars that work only in constrained cases, to medical image analysis products that would replace radiologists but never did, we still struggle to translate successful research into successful real-world applications.The software engineering community has accumulated a large body of knowledge over the decades on how to develop, release, and maintain products. AI products, being software products, benefit from some of that accumulated knowledge, but not all of it. AI products diverge from traditional software products in fundamental ways: their main component is not a specific piece of code, written for a specific purpose, but a generic piece of code, a model, customized by a training process driven by hyperparameters and a dataset. Datasets are usually large and models are opaque. We cannot directly inspect them as we can inspect the code of traditional software products. We need other methods to detect failures in AI products.This thesis investigates recent advancements that promote auditing, supported by transparency, as a mechanism to detect potential failures in AI products, focusing on healthcare applications. It reviews practices that apply to the early stages of the machine learning (ML) lifecycle, when datasets and models are created. These stages are unique to AI products.The proposed solution uses checklists to increase the transparency of these early stages. The thesis investigates checklists and guidelines that have been recently proposed in the AI industry and research communities, focusing on medical applications. Out of those checklists, it selected datasheets for dataset and model cards to increase the transparency of the dataset and model creation, respectively.As a demonstration of the increased transparency afforded by these methods, this thesis applies the selected checklists to a well-known medical imaging dataset, ChestX-ray8, and to a well-known model, CheXNet. The dataset datasheet created for ChestX-ray8 and the model card created for CheXNet show how a well-structured format increases transparency. The increased transparency, in turn, allows the ML community to interact with other stakeholders, e.g. domain experts in medical imaging, early on to find potential problems and opportunities for improvement.

• Publisher: ProQuest Dissertations & Theses
• Subject: Artificial intelligence / Computer Engineering / Computer science / Health care management / Information Technology / Medical imaging
• ISBN: 9798557001830