Catalogue Search | MBRL
Are you sure you want to remove the book from the shelf?
{{itemTitle}}
121,743
result(s) for
"Benchmark"
Sort by:
Benchmarking transaction and analytical processing systems : the creation of a mixed workload benchmark and its application
Systems for Online Transaction Processing (OLTP) and Online Analytical Processing (OLAP) are currently separate. The potential of the latest technologies and changes in operational and analytical applications over the last decade have given rise to the unification of these systems, which can be of benefit for both workloads. Research and industry have reacted and prototypes of hybrid database systems are now appearing. Benchmarks are the standard method for evaluating, comparing and supporting the development of new database systems. Because of the separation of OLTP and OLAP systems, existing benchmarks are only focused on one or the other. With the rise of hybrid database systems, benchmarks to assess these systems will be needed as well. Based on the examination of existing benchmarks, a new benchmark for hybrid database systems is introduced in this book. It is furthermore used to determine the effect of adding OLAP to an OLTP workload and is applied to analyze the impact of typically used optimizations in the historically separate OLTP and OLAP domains in mixed-workload scenarios.
Book
Updated benchmarking of variant effect predictors using deep mutational scanning
The assessment of variant effect predictor (VEP) performance is fraught with biases introduced by benchmarking against clinical observations. In this study, building on our previous work, we use independently generated measurements of protein function from deep mutational scanning (DMS) experiments for 26 human proteins to benchmark 55 different VEPs, while introducing minimal data circularity. Many top‐performing VEPs are unsupervised methods including EVE, DeepSequence and ESM‐1v, a protein language model that ranked first overall. However, the strong performance of recent supervised VEPs, in particular VARITY, shows that developers are taking data circularity and bias issues seriously. We also assess the performance of DMS and unsupervised VEPs for discriminating between known pathogenic and putatively benign missense variants. Our findings are mixed, demonstrating that some DMS datasets perform exceptionally at variant classification, while others are poor. Notably, we observe a striking correlation between VEP agreement with DMS data and performance in identifying clinically relevant variants, strongly supporting the validity of our rankings and the utility of DMS for independent benchmarking.
Synopsis
Common sources of bias in variant effect predictor benchmarking are assessed using data from deep mutational scanning experiments. ESM‐1v, EVE and DeepSequence are among the top performers on both functionally validated and clinically observed variants.
Deep mutational scanning datasets from 26 human proteins are used to benchmark 55 computational predictors of missense variant effect.
The top‐performing methods include several very recent predictors and are based mostly on unsupervised machine learning methodologies.
There is a strong correlation between predictor performance when benchmarked against deep mutational scanning data and clinical variants.
Graphical Abstract
Common sources of bias in variant effect predictor benchmarking are assessed using data from deep mutational scanning experiments. ESM‐1v, EVE and DeepSequence are among the top performers on both functionally validated and clinically observed variants.
Journal Article
LaSOT: A High-quality Large-scale Single Object Tracking Benchmark
Despite great recent advances in visual tracking, its further development, including both algorithm design and evaluation, is limited due to lack of dedicated large-scale benchmarks. To address this problem, we present LaSOT, a high-quality Large-scale Single Object Tracking benchmark. LaSOT contains a diverse selection of 85 object classes, and offers 1550 totaling more than 3.87 million frames. Each video frame is carefully and manually annotated with a bounding box. This makes LaSOT, to our knowledge, the largest densely annotated tracking benchmark. Our goal in releasing LaSOT is to provide a dedicated high quality platform for both training and evaluation of trackers. The average video length of LaSOT is around 2500 frames, where each video contains various challenge factors that exist in real world video footage,such as the targets disappearing and re-appearing. These longer video lengths allow for the assessment of long-term trackers. To take advantage of the close connection between visual appearance and natural language, we provide language specification for each video in LaSOT. We believe such additions will allow for future research to use linguistic features to improve tracking. Two protocols, full-overlap and one-shot, are designated for flexible assessment of trackers. We extensively evaluate 48 baseline trackers on LaSOT with in-depth analysis, and results reveal that there still exists significant room for improvement. The complete benchmark, tracking results as well as analysis are available at http://vision.cs.stonybrook.edu/~lasot/.
Journal Article
ImageNet Large Scale Visual Recognition Challenge
The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide a detailed analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the 5 years of the challenge, and propose future directions and improvements.
Journal Article
A new algorithm for normal and large-scale optimization problems: Nomadic People Optimizer
Metaheuristic algorithms have received much attention recently for solving different optimization and engineering problems. Most of these methods were inspired by nature or the behavior of certain swarms, such as birds, ants, bees, or even bats, while others were inspired by a specific social behavior such as colonies, or political ideologies. These algorithms faced an important issue, which is the balancing between the global search (exploration) and local search (exploitation) capabilities. In this research, a novel swarm-based metaheuristic algorithm which depends on the behavior of nomadic people was developed, it is called “Nomadic People Optimizer (NPO)”. The proposed algorithm simulates the nature of these people in their movement and searches for sources of life (such as water or grass for grazing), and how they have lived hundreds of years, continuously migrating to the most comfortable and suitable places to live. The algorithm was primarily designed based on the multi-swarm approach, consisting of several clans and each clan looking for the best place, in other words, for the best solution depending on the position of their leader. The algorithm is validated based on 36 unconstrained benchmark functions. For the comparison purpose, six well-established nature-inspired algorithms are performed for evaluating the robustness of NPO algorithm. The proposed and the benchmark algorithms are tested for large-scale optimization problems which are associated with high-dimensional variability. The attained results demonstrated a remarkable solution for the NPO algorithm. In addition, the achieved results evidenced the potential high convergence, lower iterations, and less time-consuming required for finding the current best solution.
Journal Article
Image Matching Across Wide Baselines: From Paper to Practice
We introduce a comprehensive benchmark for local features and robust estimation algorithms, focusing on the downstream task—the accuracy of the reconstructed camera pose—as our primary metric. Our pipeline’s modular structure allows easy integration, configuration, and combination of different methods and heuristics. This is demonstrated by embedding dozens of popular algorithms and evaluating them, from seminal works to the cutting edge of machine learning research. We show that with proper settings, classical solutions may still outperform the perceived state of the art. Besides establishing the actual state of the art, the conducted experiments reveal unexpected properties of structure from motion pipelines that can help improve their performance, for both algorithmic and learned methods. Data and code are online (https://github.com/ubc-vision/image-matching-benchmark), providing an easy-to-use and flexible framework for the benchmarking of local features and robust estimation methods, both alongside and against top-performing methods. This work provides a basis for the Image Matching Challenge (https://image-matching-challenge.github.io).
Journal Article
Benchmarking Low-Light Image Enhancement and Beyond
In this paper, we present a systematic review and evaluation of existing single-image low-light enhancement algorithms. Besides the commonly used low-level vision oriented evaluations, we additionally consider measuring machine vision performance in the low-light condition via face detection task to explore the potential of joint optimization of high-level and low-level vision enhancement. To this end, we first propose a large-scale low-light image dataset serving both low/high-level vision with diversified scenes and contents as well as complex degradation in real scenarios, called Vision Enhancement in the LOw-Light condition (VE-LOL). Beyond paired low/normal-light images without annotations, we additionally include the analysis resource related to human, i.e. face images in the low-light condition with annotated face bounding boxes. Then, efforts are made on benchmarking from the perspective of both human and machine visions. A rich variety of criteria is used for the low-level vision evaluation, including full-reference, no-reference, and semantic similarity metrics. We also measure the effects of the low-light enhancement on face detection in the low-light condition. State-of-the-art face detection methods are used in the evaluation. Furthermore, with the rich material of VE-LOL, we explore the novel problem of joint low-light enhancement and face detection. We develop an enhanced face detector to apply low-light enhancement and face detection jointly. The features extracted by the enhancement module are fed to the successive layer with the same resolution of the detection module. Thus, these features are intertwined together to unitedly learn useful information across two phases, i.e. enhancement and detection. Experiments on VE-LOL provide a comparison of state-of-the-art low-light enhancement algorithms, point out their limitations, and suggest promising future directions. Our dataset has supported the Track “Face Detection in Low Light Conditions” of CVPR UG2+ Challenge (2019–2020) (http://cvpr2020.ug2challenge.org/).
Journal Article
Artificial lemming algorithm: a novel bionic meta-heuristic technique for solving real-world engineering optimization problems
The advent of the intelligent information era has witnessed a proliferation of complex optimization problems across various disciplines. Although existing meta-heuristic algorithms have demonstrated efficacy in many scenarios, they still struggle with certain challenges such as premature convergence, insufficient exploration, and lack of robustness in high-dimensional, nonconvex search spaces. These limitations underscore the need for novel optimization techniques that can better balance exploration and exploitation while maintaining computational efficiency. In response to this need, we propose the Artificial Lemming Algorithm (ALA), a bio-inspired metaheuristic that mathematically models four distinct behaviors of lemmings in nature: long-distance migration, digging holes, foraging, and evading predators. Specifically, the long-distance migration and burrow digging behaviors are dedicated to highly exploring the search domain, whereas the foraging and evading predators behaviors provide exploitation during the optimization process. In addition, ALA incorporates an energy-decreasing mechanism that enables dynamic adjustments to the balance between exploration and exploitation, thereby enhancing its ability to evade local optima and converge to global solutions more robustly. To thoroughly verify the effectiveness of the proposed method, ALA is compared with 17 other state-of-the-art meta-heuristic algorithms on the IEEE CEC2017 benchmark test suite and the IEEE CEC2022 benchmark test suite. The experimental results indicate that ALA has reliable comprehensive optimization performance and can achieve superior solution accuracy, convergence speed, and stability in most test cases. For the 29 10-, 30-, 50-, and 100-dimensional CEC2017 functions, ALA obtains the lowest Friedman average ranking values among all competitor methods, which are 1.7241, 2.1034, 2.7241, and 2.9310, respectively, and for the 12 CEC2022 functions, ALA again wins the optimal Friedman average ranking of 2.1667. Finally, to further evaluate its applicability, ALA is implemented to address a series of optimization cases, including constrained engineering design, photovoltaic (PV) model parameter identification, and fractional-order proportional-differential-integral (FOPID) controller gain tuning. Our findings highlight the competitive edge and potential of ALA for real-world engineering applications. The source code of ALA is publicly available at https://github.com/StevenShaw98/Artificial-Lemming-Algorithm.
Journal Article
Update: use of the benchmark dose approach in risk assessment
The Scientific Committee (SC) reconfirms that the benchmark dose (BMD) approach is a scientifically more advanced method compared to the NOAEL approach for deriving a Reference Point (RP). Most of the modifications made to the SC guidance of 2009 concern the section providing guidance on how to apply the BMD approach. Model averaging is recommended as the preferred method for calculating the BMD confidence interval, while acknowledging that the respective tools are still under development and may not be easily accessible to all. Therefore, selecting or rejecting models is still considered as a suboptimal alternative. The set of default models to be used for BMD analysis has been reviewed, and the Akaike information criterion (AIC) has been introduced instead of the log‐likelihood to characterise the goodness of fit of different mathematical models to a dose–response data set. A flowchart has also been inserted in this update to guide the reader step‐by‐step when performing a BMD analysis, as well as a chapter on the distributional part of dose–response models and a template for reporting a BMD analysis in a complete and transparent manner. Finally, it is recommended to always report the BMD confidence interval rather than the value of the BMD. The lower bound (BMDL) is needed as a potential RP, and the upper bound (BMDU) is needed for establishing the BMDU/BMDL per ratio reflecting the uncertainty in the BMD estimate. This updated guidance does not call for a general re‐evaluation of previous assessments where the NOAEL approach or the BMD approach as described in the 2009 SC guidance was used, in particular when the exposure is clearly smaller (e.g. more than one order of magnitude) than the health‐based guidance value. Finally, the SC firmly reiterates to reconsider test guidelines given the expected wide application of the BMD approach. http://onlinelibrary.wiley.com/doi/10.2903/sp.efsa.2017.EN-1147/full
Journal Article
Butterfly optimization algorithm: a novel approach for global optimization
Real-world problems are complex as they are multidimensional and multimodal in nature that encourages computer scientists to develop better and efficient problem-solving methods. Nature-inspired metaheuristics have shown better performances than that of traditional approaches. Till date, researchers have presented and experimented with various nature-inspired metaheuristic algorithms to handle various search problems. This paper introduces a new nature-inspired algorithm, namely butterfly optimization algorithm (BOA) that mimics food search and mating behavior of butterflies, to solve global optimization problems. The framework is mainly based on the foraging strategy of butterflies, which utilize their sense of smell to determine the location of nectar or mating partner. In this paper, the proposed algorithm is tested and validated on a set of 30 benchmark test functions and its performance is compared with other metaheuristic algorithms. BOA is also employed to solve three classical engineering problems (spring design, welded beam design, and gear train design). Results indicate that the proposed BOA is more efficient than other metaheuristic algorithms.
Journal Article