Catalogue Search | MBRL
Are you sure you want to remove the book from the shelf?
{{itemTitle}}
277,173
result(s) for
"performance evaluation"
Sort by:
HOTA: A Higher Order Metric for Evaluating Multi-object Tracking
Multi-object tracking (MOT) has been notoriously difficult to evaluate. Previous metrics overemphasize the importance of either detection or association. To address this, we present a novel MOT evaluation metric, higher order tracking accuracy (HOTA), which explicitly balances the effect of performing accurate detection, association and localization into a single unified metric for comparing trackers. HOTA decomposes into a family of sub-metrics which are able to evaluate each of five basic error types separately, which enables clear analysis of tracking performance. We evaluate the effectiveness of HOTA on the MOTChallenge benchmark, and show that it is able to capture important aspects of MOT performance not previously taken into account by established metrics. Furthermore, we show HOTA scores better align with human visual evaluation of tracking performance.
Journal Article
MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking
Standardized benchmarks have been crucial in pushing the performance of computer vision algorithms, especially since the advent of deep learning. Although leaderboards should not be over-claimed, they often provide the most objective measure of performance and are therefore important guides for research. We present MOTChallenge, a benchmark for single-camera Multiple Object Tracking (MOT) launched in late 2014, to collect existing and new data and create a framework for the standardized evaluation of multiple object tracking methods. The benchmark is focused on multiple people tracking, since pedestrians are by far the most studied object in the tracking community, with applications ranging from robot navigation to self-driving cars. This paper collects the first three releases of the benchmark: (i) MOT15, along with numerous state-of-the-art results that were submitted in the last years, (ii) MOT16, which contains new challenging videos, and (iii) MOT17, that extends MOT16 sequences with more precise labels and evaluates tracking performance on three different object detectors. The second and third release not only offers a significant increase in the number of labeled boxes, but also provide labels for multiple object classes beside pedestrians, as well as the level of visibility for every single object of interest. We finally provide a categorization of state-of-the-art trackers and a broad error analysis. This will help newcomers understand the related work and research trends in the MOT community, and hopefully shed some light into potential future research directions.
Journal Article
Comparison of Full-Reference Image Quality Models for Optimization of Image Processing Systems
The performance of objective image quality assessment (IQA) models has been evaluated primarily by comparing model predictions to human quality judgments. Perceptual datasets gathered for this purpose have provided useful benchmarks for improving IQA methods, but their heavy use creates a risk of overfitting. Here, we perform a large-scale comparison of IQA models in terms of their use as objectives for the optimization of image processing algorithms. Specifically, we use eleven full-reference IQA models to train deep neural networks for four low-level vision tasks: denoising, deblurring, super-resolution, and compression. Subjective testing on the optimized images allows us to rank the competing models in terms of their perceptual performance, elucidate their relative advantages and disadvantages in these tasks, and propose a set of desirable properties for incorporation into future IQA models.
Journal Article
VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change
Visual place recognition (VPR) is the process of recognising a previously visited place using visual information, often under varying appearance conditions and viewpoint changes and with computational constraints. VPR is related to the concepts of localisation, loop closure, image retrieval and is a critical component of many autonomous navigation systems ranging from autonomous vehicles to drones and computer vision systems. While the concept of place recognition has been around for many years, VPR research has grown rapidly as a field over the past decade due to improving camera hardware and its potential for deep learning-based techniques, and has become a widely studied topic in both the computer vision and robotics communities. This growth however has led to fragmentation and a lack of standardisation in the field, especially concerning performance evaluation. Moreover, the notion of viewpoint and illumination invariance of VPR techniques has largely been assessed qualitatively and hence ambiguously in the past. In this paper, we address these gaps through a new comprehensive open-source framework for assessing the performance of VPR techniques, dubbed “VPR-Bench”. VPR-Bench (Open-sourced at: https://github.com/MubarizZaffar/VPR-Bench) introduces two much-needed capabilities for VPR researchers: firstly, it contains a benchmark of 12 fully-integrated datasets and 10 VPR techniques, and secondly, it integrates a comprehensive variation-quantified dataset for quantifying viewpoint and illumination invariance. We apply and analyse popular evaluation metrics for VPR from both the computer vision and robotics communities, and discuss how these different metrics complement and/or replace each other, depending upon the underlying applications and system requirements. Our analysis reveals that no universal SOTA VPR technique exists, since: (a) state-of-the-art (SOTA) performance is achieved by 8 out of the 10 techniques on at least one dataset, (b) SOTA technique in one community does not necessarily yield SOTA performance in the other given the differences in datasets and metrics. Furthermore, we identify key open challenges since: (c) all 10 techniques suffer greatly in perceptually-aliased and less-structured environments, (d) all techniques suffer from viewpoint variance where lateral change has less effect than 3D change, and (e) directional illumination change has more adverse effects on matching confidence than uniform illumination change. We also present detailed meta-analyses regarding the roles of varying ground-truths, platforms, application requirements and technique parameters. Finally, VPR-Bench provides a unified implementation to deploy these VPR techniques, metrics and datasets, and is extensible through templates.
Journal Article
Performance measurement and leisure management
The chapters in this text each present a different case study of performance measurement. They cover a wide range of sectors in the leisure industry including public recreation centres, theme parks, play facilities, sport organisations, hospitality, and the Olympic Games. The evidence from these cases covers examples from three different continents and five different countries. All the chapters report empirical research and all the cases explore managerial implications.
Book
Evaluation of Six Atmospheric Reanalyses over Arctic Sea Ice from Winter to Early Summer
This study evaluates the performance of six atmospheric reanalyses (ERA-Interim, ERA5, JRA-55, CFSv2, MERRA-2, and ASRv2) over Arctic sea ice from winter to early summer. The reanalyses are evaluated using observations from the Norwegian Young Sea Ice campaign (N-ICE2015), a 5-month ice drift in pack ice north of Svalbard. N-ICE2015 observations include surface meteorology, vertical profiles from radiosondes, as well as radiative and turbulent heat fluxes. The reanalyses simulate surface analysis variables well throughout the campaign, but have difficulties with most forecast variables. Wintertime (January–March) correlation coefficients between the reanalyses and observations are above 0.90 for the surface pressure, 2-m temperature, total column water vapor, and downward longwave flux. However, all reanalyses have a positive wintertime 2-m temperature bias, ranging from 1° to 4°C, and negative (i.e., upward) net longwave bias of 3–19 W m−2. These biases are associated with poorly represented surface inversions and are largest during cold-stable periods. Notably, the recent ERA5 and ASRv2 datasets have some of the largest temperature and net longwave biases, respectively. During spring (April–May), reanalyses fail to simulate observed persistent cloud layers. Therefore they overestimate the net shortwave flux (5–79 W m−2) and underestimate the net longwave flux (8–38 W m−2). Promisingly, ERA5 provides the best estimates of downward radiative fluxes in spring and summer, suggesting improved forecasting of Arctic cloud cover. All reanalyses exhibit large negative (upward) residual heat flux biases during winter, and positive (downward) biases during summer. Turbulent heat fluxes over sea ice are simulated poorly in all seasons.
Journal Article