Journal Article

2 The factive assumption: reframing epistemic justification for opaque AI in clinical medicine

Wells, Sarah Scriven

2025

Overview

IntroductionThe deployment of opaque machine learning (ML) models in clinical settings raises a critical problem of epistemic justification. Standard approaches like explainability often fail, as they rely on a ”factive assumption” requiring access to a model’s internal mechanisms. This paper challenges this assumption, investigating an alternative foundation for justification.Materials and MethodsThis paper employs conceptual analysis, drawing on epistemology and explainable AI (xAI). We critique access-based justification via Gettier-style problems and synthesise Catherine Elgin’s non-factive theory of understanding with recent work on clinical AI. This involves analysing critiques of intelligibility (Fleisher, 2022) and the proposal for interactive Toy Surrogate Models (TSMs) (Páez, 2024).ResultsThe factive assumption sets an unattainable standard, leading to fragile, artefactual understanding. Treating post-hoc explainability tools as analogous to scientific idealisations is shown to be flawed. A non-factive account of understanding—the ability to ”grasp” and reason with model outputs via tools like TSMs—provides a more robust epistemic warrant. Justification is thus relocated from model fidelity to the clinician’s structured, counterfactual reasoning within norm-governed practices.ConclusionFactive standards of explainability are neither attainable nor necessary for justifying opaque clinical AI. A more philosophically sustainable approach grounds justification in non-factive, practice-based understanding. This framework reorients responsibility from the model to the clinician’s cognitive engagement, aligning the use of AI with existing professional norms for managing uncertainty.

• Publisher: BMJ Publishing Group LTD
• Subject: Epistemology