Journal Article

Updated benchmarking of variant effect predictors using deep mutational scanning

Livesey, Benjamin J, Marsh, Joseph A

2023

Overview

The assessment of variant effect predictor (VEP) performance is fraught with biases introduced by benchmarking against clinical observations. In this study, building on our previous work, we use independently generated measurements of protein function from deep mutational scanning (DMS) experiments for 26 human proteins to benchmark 55 different VEPs, while introducing minimal data circularity. Many top‐performing VEPs are unsupervised methods including EVE, DeepSequence and ESM‐1v, a protein language model that ranked first overall. However, the strong performance of recent supervised VEPs, in particular VARITY, shows that developers are taking data circularity and bias issues seriously. We also assess the performance of DMS and unsupervised VEPs for discriminating between known pathogenic and putatively benign missense variants. Our findings are mixed, demonstrating that some DMS datasets perform exceptionally at variant classification, while others are poor. Notably, we observe a striking correlation between VEP agreement with DMS data and performance in identifying clinically relevant variants, strongly supporting the validity of our rankings and the utility of DMS for independent benchmarking. Synopsis Common sources of bias in variant effect predictor benchmarking are assessed using data from deep mutational scanning experiments. ESM‐1v, EVE and DeepSequence are among the top performers on both functionally validated and clinically observed variants. Deep mutational scanning datasets from 26 human proteins are used to benchmark 55 computational predictors of missense variant effect. The top‐performing methods include several very recent predictors and are based mostly on unsupervised machine learning methodologies. There is a strong correlation between predictor performance when benchmarked against deep mutational scanning data and clinical variants. Graphical Abstract Common sources of bias in variant effect predictor benchmarking are assessed using data from deep mutational scanning experiments. ESM‐1v, EVE and DeepSequence are among the top performers on both functionally validated and clinically observed variants.

• Publisher: Nature Publishing Group UK,EMBO Press,John Wiley and Sons Inc,Springer Nature
• Subject: Amino acids / Benchmark / Benchmarking / Benchmarks / Bias / Circularity / Datasets / DMS / EMBO10 / EMBO16 / Humans / Machine learning / MAVE / Mutation / Mutation, Missense / Performance assessment / Proteins / Proteins - genetics / Scanning / VEP