Journal Article

How to Conduct Rigorous Supervised Machine Learning in Information Systems Research: The Supervised Machine Learning Report Card

Kühl, Niklas, Satzger, Gerhard, Hirt, Robin, Baier, Lucas, Schmitz, Björn

2021

Overview

In the last decade, applying supervised machine learning (SML) has become increasingly popular in the information systems (IS) field. However, SML results rely on many different data-preprocessing techniques, algorithms, and ways to implement them, which has contributed to an inconsistency in the way researchers have documented their SML efforts and, thus, the degree to which others can reproduce their results. In one sense, we can understand this inconsistency given the goals and motivations for SML applications vary and the research area’s rapid evolution. However, for the IS research community, the inconsistency poses a big challenge because, even with full access to the data, researchers can neither completely evaluate the SML approaches that previous research has adopted or replicate previous research results. Therefore, in this paper, we provide the IS community with guidelines for comprehensively and rigorously conducting and documenting SML research. First, we review the literature concerning steps and SML process frameworks to extract relevant problem characteristics that researchers should report and relevant choices that they should make in applying SML. Second, we integrate these characteristics and choices into a comprehensive “Supervised Machine Learning Report Card (SMLR)” that researchers can use in future SML endeavors. Third, we apply this report card to a set of 121 relevant papers published in renowned IS outlets between 2010 and 2018 and demonstrate how and where these papers’ authors could have improved their documentation and, thus, how and where researchers can better document their SML approaches in the future. Thus, with this work, we help researchers more completely and rigorously apply and document SML approaches and, thereby, enable researchers to more deeply evaluate and reproduce/replicate results in the IS field.

• Publisher: Association for Information Systems
• Subject: Algorithms / Information systems / Literature reviews / Machine learning / Researchers