Journal Article

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

Goyal, Siddharth, Ma, Jerry, Guo, Demi, Zitnick, C. Lawrence, Rives, Alexander, Lin, Zeming, Fergus, Rob, Liu, Jason, Meier, Joshua, Sercu, Tom, Ott, Myle

2021

Overview

SignificanceLearning biological properties from sequence data is a logical step toward generative and predictive artificial intelligence for biology. Here, we propose scaling a deep contextual language model with unsupervised learning to sequences spanning evolutionary diversity. We find that without prior knowledge, information emerges in the learned representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity. We show the learned representations are useful across benchmarks for remote homology detection, prediction of secondary structure, long-range residue–residue contacts, and mutational effect. Unsupervised representation learning enables state-of-the-art supervised prediction of mutational effect and secondary structure and improves state-of-the-art features for long-range contact prediction. In the field of artificial intelligence, a combination of scale in data and model capacity enabled by unsupervised learning has led to major advances in representation learning and statistical generation. In the life sciences, the anticipated growth of sequencing promises unprecedented data on natural sequence diversity. Protein language modeling at the scale of evolution is a logical step toward predictive and generative artificial intelligence for biology. To this end, we use unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity. The resulting model contains information about biological properties in its representations. The representations are learned from sequence data alone. The learned representation space has a multiscale organization reflecting structure from the level of biochemical properties of amino acids to remote homology of proteins. Information about secondary and tertiary structure is encoded in the representations and can be identified by linear projections. Representation learning produces features that generalize across a range of applications, enabling state-of-the-art supervised prediction of mutational effect and secondary structure and improving state-of-the-art features for long-range contact prediction.

• Publisher: National Academy of Sciences
• Subject: Amino acid sequence / Amino acids / Artificial intelligence / Biological properties / Biological Sciences / Biophysics and Computational Biology / Computer Sciences / Generative artificial intelligence / Homology / Language / Learning / Model testing / Physical Sciences / Protein structure / Proteins / Representation learning / Secondary structure / Sequences / Structure-function relationships / Tertiary structure / Unsupervised learning