Journal Article

The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation

Cadena Cesar, Blum, Hermann, Siegwart, Roland, Nieto, Juan, Paul-Edouard, Sarlin

2021

Overview

Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for anomaly detection in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects. We adapt state-of-the-art methods to recent semantic segmentation models and compare uncertainty estimation approaches based on softmax confidence, Bayesian learning, density estimation, image resynthesis, as well as supervised anomaly detection methods. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art. Results, data and submission information can be found at https://fishyscapes.com/.

• Publisher: Springer Nature B.V
• Subject: Anomalies / Benchmarks / Blind spot area / Deep learning / Estimates / Evaluation / Image segmentation / Machine learning / Object recognition / Safety critical / Semantic segmentation / Semantics / Task complexity / Uncertainty