Paper

SalienTrack: providing salient information for semi-automated self-tracking feedback with model explanations

Liu, Jiaying, Wang, Yunlong, Schultz-McArdle, Jordan, Park, Homin, Lim, Brian Y, Kay, Judy, Rosenthal, Stephanie

2022

Overview

Self-tracking can improve people's awareness of their unhealthy behaviors and support reflection to inform behavior change. Increasingly, new technologies make tracking easier, leading to large amounts of tracked data. However, much of that information is not salient for reflection and self-awareness. To tackle this burden for reflection, we created the SalienTrack framework, which aims to 1) identify salient tracking events, 2) select the salient details of those events, 3) explain why they are informative, and 4) present the details as manually elicited or automatically shown feedback. We implemented SalienTrack in the context of nutrition tracking. To do this, we first conducted a field study to collect photo-based mobile food tracking over 1-5 weeks. We then report how we used this data to train an explainable-AI model of salience. Finally, we created interfaces to present salient information and conducted a formative user study to gain insights about how SalienTrack could be integrated into an interface for reflection. Our key contributions are the SalienTrack framework, a demonstration of its implementation for semi-automated feedback in an important and challenging self-tracking context and a discussion of the broader uses of the framework.

• Publisher: Cornell University Library, arXiv.org
• Subject: Annotations / Automation / Feedback / Food / Machine learning / Nutrition / Tracking