Explore the vast range of titles available.

We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

Monoallelic expression with random choice between the maternal and paternal alleles defines an unusual class of genes comprising X-inactivated genes and a few autosomal gene families. Using a genome-wide approach, we assessed allele-specific transcription of about 4000 human genes in clonal cell lines and found that more than 300 were subject to random monoallelic expression. For a majority of monoallelic genes, we also observed some clonal lines displaying biallelic expression. Clonal cell lines reflect an independent choice to express the maternal, the paternal, or both alleles for each of these genes. This can lead to differences in expressed protein sequence and to differences in levels of gene expression. Unexpectedly widespread monoallelic expression suggests a mechanism that generates diversity in individual cells and their clonal descendants.

The ability to image light elements in soft matter at atomic resolution enables unprecedented insight into the structure and properties of molecular heterostructures and beam-sensitive nanomaterials. In this study, we introduce a scanning transmission electron microscopy technique combining a pre-specimen phase plate designed to produce a probe with structured phase with a high-speed direct electron detector to generate nearly linear contrast images with high efficiency. We demonstrate this method by using both experiment and simulation to simultaneously image the atomic-scale structure of weakly scattering amorphous carbon and strongly scattering gold nanoparticles. Our method demonstrates strong contrast for both materials, making it a promising candidate for structural determination of heterogeneous soft/hard matter samples even at low electron doses comparable to traditional phase-contrast transmission electron microscopy. Simulated images demonstrate the extension of this technique to the challenging problem of structural determination of biological material at the surface of inorganic crystals. Scanning transmission electron microscopy is a powerful material probe, but constrained to large atomic number samples due to the issues of beam damage and weak scattering. Here, Ophus et al. propose a method that produces linear phase contrast in a focused electron beam to image dose-sensitive objects.

The development of mature B cells involves a series of molecular decisions which culminate in the expression of a single light-chain and heavy-chain antigen receptor on the cell surface 1 , 2 . There are two alleles for each receptor locus, so the ultimate choice of one receptor type must involve a process of allelic exclusion. One way to do this is with a feedback mechanism that downregulates rearrangement after the generation of a productive receptor molecule 3 , but recent work suggests that monoallelic epigenetic changes may also take place even before rearrangement 4 . To better understand the basis for distinguishing between alleles, we have analysed DNA replication timing. Here we show that all of the B-cell-receptor loci ( μ , κ and λ) and the TCRβ locus replicate asynchronously. This pattern, which is established randomly in each cell early in development and maintained by cloning, represents an epigenetic mark for allelic exclusion, because it is almost always the early-replicating allele which is initially selected to undergo rearrangement in B cells. These results indicate that allelic exclusion in the immune system may be very similar to the process of X chromosome inactivation.

BackgroundRespiratory illness is a leading cause of morbidity in adults and the number one cause of mortality in children, yet billions of people lack access to medical imaging to assist in its diagnosis. Although ultrasound is highly sensitive and specific for respiratory illness such as pneumonia, its deployment is limited by a lack of sonographers. As a solution, we tested a standardised lung ultrasound volume sweep imaging (VSI) protocol based solely on external body landmarks performed by individuals without prior ultrasound experience after brief training. Each step in the VSI protocol is saved as a video clip for later interpretation by a specialist.MethodsDyspneic hospitalised patients were scanned by ultrasound naive operators after 2 hours of training using the lung ultrasound VSI protocol. Separate blinded readers interpreted both lung ultrasound VSI examinations and standard of care chest radiographs to ascertain the diagnostic value of lung VSI considering chest X-ray as the reference standard. Comparison to clinical diagnosis as documented in the medical record and CT (when available) were also performed. Readers offered a final interpretation of normal, abnormal, or indeterminate/borderline for each VSI examination, chest X-ray, and CT.ResultsOperators scanned 102 subjects (0–89 years old) for analysis. Lung VSI showed a sensitivity of 93% and a specificity of 91% for an abnormal chest X-ray and a sensitivity of 100% and a specificity of 93% for a clinical diagnosis of pneumonia. When any cases with an indeterminate rating on chest X-ray or ultrasound were excluded (n=38), VSI lung ultrasound showed 92% agreement with chest X-ray (Cohen’s κ 0.83 (0.68 to 0.97, p<0.0001)). Among cases with CT (n=21), when any ultrasound with an indeterminate rating was excluded (n=3), there was 100% agreement with VSI.ConclusionLung VSI performed by previously inexperienced ultrasound operators after brief training showed excellent agreement with chest X-ray and high sensitivity and specificity for a clinical diagnosis of pneumonia. Blinded readers were able to identify other respiratory diseases including pulmonary oedema and pleural effusion. Deployment of lung VSI could benefit the health of the global community.

Societal Impact Statement Grapevine leaves are emblematic of the strong visual associations people make with plants. Leaf shape is immediately recognizable at a glance, and therefore, this is used to distinguish grape varieties. In an era of computationally enabled machine learning‐derived representations of reality, we can revisit how we view and use the shapes and forms that plants display to understand our relationship with them. Using computational approaches combined with time‐honored methods, we can predict theoretical leaves that are possible, enabling us to understand the genetics, development, and environmental responses of plants in new ways. Summary Grapevine leaves are a model morphometric system. Sampling over 10,000 leaves using dozens of landmarks, the genetic, developmental, and environmental basis of leaf shape has been studied and a morphospace for the genus Vitis predicted. Yet, these representations of leaf shape fail to capture the exquisite features of leaves at high resolution. We measure the shapes of 139 grapevine leaves using 1672 pseudo‐landmarks derived from 90 homologous landmarks with Procrustean approaches. From hand traces of the vasculature and blade, we have derived a method to automatically detect landmarks and place pseudo‐landmarks that results in a high‐resolution representation of grapevine leaf shape. Using polynomial models, we create continuous representations of leaf development in 10 Vitis spp. We visualize a high‐resolution morphospace in which genetic and developmental sources of leaf shape variance are orthogonal to each other. Using classifiers, Vitis vinifera, Vitis spp., rootstock and dissected leaf varieties as well as developmental stages are accurately predicted. Theoretical eigenleaf representations sampled from across the morphospace that we call synthetic leaves can be classified using models. By predicting a high‐resolution morphospace and delimiting the boundaries of leaf shapes that can plausibly be produced within the genus Vitis, we can sample synthetic leaves with realistic qualities. From an ampelographic perspective, larger numbers of leaves sampled at lower resolution can be projected onto this high‐resolution space, or, synthetic leaves can be used to increase the robustness and accuracy of machine learning classifiers. Las hojas de la vid son emblemáticas de las fuertes asociaciones visuales que la gente hace con las plantas. A primera vista, la forma de la hoja es inmediatamente reconocible, y por eso se utiliza para distinguir las variedades de uva. En la era de las representaciones computacionales de la realidad derivadas del aprendizaje automático, podemos revisar cómo vemos y utilizamos las formas de las plantas para entender nuestra relación con ellas. Utilizando enfoques computacionales combinados con métodos consagrados, podemos predecir las hojas teóricas que son posibles para entender de nuevas maneras la genética, el desarrollo y las respuestas medioambientales de las plantas. Grapevine leaves are emblematic of the strong visual associations people make with plants. Leaf shape is immediately recognizable at a glance, and therefore, this is used to distinguish grape varieties. In an era of computationally enabled machine learning–derived representations of reality, we can revisit how we view and use the shapes and forms that plants display to understand our relationship with them. Using computational approaches combined with time‐honored methods, we can predict theoretical leaves that are possible, enabling us to understand the genetics, development, and environmental responses of plants in new ways.