Explore the vast range of titles available.

Deep learning algorithms trained to predict masked words from large amount of text have recently been shown to generate activations similar to those of the human brain. However, what drives this similarity remains currently unknown. Here, we systematically compare a variety of deep language models to identify the computational principles that lead them to generate brain-like representations of sentences. Specifically, we analyze the brain responses to 400 isolated sentences in a large cohort of 102 subjects, each recorded for two hours with functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). We then test where and when each of these algorithms maps onto the brain responses. Finally, we estimate how the architecture, training, and performance of these models independently account for the generation of brain-like representations. Our analyses reveal two main findings. First, the similarity between the algorithms and the brain primarily depends on their ability to predict words from context. Second, this similarity reveals the rise and maintenance of perceptual, lexical, and compositional representations within each cortical region. Overall, this study shows that modern language algorithms partially converge towards brain-like solutions, and thus delineates a promising path to unravel the foundations of natural language processing. Charlotte Caucheteux and Jean-Rémi King examine the ability of transformer neural networks trained on word prediction tasks to fit representations in the human brain measured with fMRI and MEG. Their results provide further insight into the workings of transformer language models and their relevance to brain responses.

Considerable progress has recently been made in natural language processing: deep learning algorithms are increasingly able to generate, summarize, translate and classify texts. Yet, these language models still fail to match the language abilities of humans. Predictive coding theory offers a tentative explanation to this discrepancy: while language models are optimized to predict nearby words, the human brain would continuously predict a hierarchy of representations that spans multiple timescales. To test this hypothesis, we analysed the functional magnetic resonance imaging brain signals of 304 participants listening to short stories. First, we confirmed that the activations of modern language models linearly map onto the brain responses to speech. Second, we showed that enhancing these algorithms with predictions that span multiple timescales improves this brain mapping. Finally, we showed that these predictions are organized hierarchically: frontoparietal cortices predict higher-level, longer-range and more contextual representations than temporal cortices. Overall, these results strengthen the role of hierarchical predictive coding in language processing and illustrate how the synergy between neuroscience and artificial intelligence can unravel the computational bases of human cognition. Current machine learning language algorithms make adjacent word-level predictions. In this work, Caucheteux et al. show that the human brain probably uses long-range and hierarchical predictions, taking into account up to eight possible words into the future.

Deep language algorithms, like GPT-2, have demonstrated remarkable abilities to process text, and now constitute the backbone of automatic translation, summarization and dialogue. However, whether these models encode information that relates to human comprehension still remains controversial. Here, we show that the representations of GPT-2 not only map onto the brain responses to spoken stories, but they also predict the extent to which subjects understand the corresponding narratives. To this end, we analyze 101 subjects recorded with functional Magnetic Resonance Imaging while listening to 70 min of short stories. We then fit a linear mapping model to predict brain activity from GPT-2’s activations. Finally, we show that this mapping reliably correlates ( R = 0.50 , p < 10 - 15 ) with subjects’ comprehension scores as assessed for each story. This effect peaks in the angular, medial temporal and supra-marginal gyri, and is best accounted for by the long-distance dependencies generated in the deep layers of GPT-2. Overall, this study shows how deep language models help clarify the brain computations underlying language comprehension.

An outstanding challenge for consciousness research is to characterize the neural signature of conscious access independently of any decisional processes. Here we present a model-based approach that uses inter-trial variability to identify the brain dynamics associated with stimulus processing. We demonstrate that, even in the absence of any task or behavior, the electroencephalographic response to auditory stimuli shows bifurcation dynamics around 250–300 milliseconds post-stimulus. Namely, the same stimulus gives rise to late sustained activity on some trials, and not on others. This late neural activity is predictive of task-related reports, and also of reports of conscious contents that are randomly sampled during task-free listening. Source localization further suggests that task-free conscious access recruits the same neural networks as those associated with explicit report, except for frontal executive components. Studying brain dynamics through variability could thus play a key role for identifying the core signatures of conscious access, independent of report. Current knowledge on the neural basis of consciousness mostly relies on situations where people report their perception. Here, the authors provide evidence for the idea that bifurcation in brain dynamics reflects conscious perception independent of report.

Brain activation in response to spoken motor commands can be detected by electroencephalography (EEG) in clinically unresponsive patients. The prevalence and prognostic importance of a dissociation between commanded motor behavior and brain activation in the first few days after brain injury are not well understood. We studied a prospective, consecutive series of patients in a single intensive care unit who had acute brain injury from a variety of causes and who were unresponsive to spoken commands, including some patients with the ability to localize painful stimuli or to fixate on or track visual stimuli. Machine learning was applied to EEG recordings to detect brain activation in response to commands that patients move their hands. The functional outcome at 12 months was determined with the Glasgow Outcome Scale-Extended (GOS-E; levels range from 1 to 8, with higher levels indicating better outcomes). A total of 16 of 104 unresponsive patients (15%) had brain activation detected by EEG at a median of 4 days after injury. The condition in 8 of these 16 patients (50%) and in 23 of 88 patients (26%) without brain activation improved such that they were able to follow commands before discharge. At 12 months, 7 of 16 patients (44%) with brain activation and 12 of 84 patients (14%) without brain activation had a GOS-E level of 4 or higher, denoting the ability to function independently for 8 hours (odds ratio, 4.6; 95% confidence interval, 1.2 to 17.1). A dissociation between the absence of behavioral responses to motor commands and the evidence of brain activation in response to these commands in EEG recordings was found in 15% of patients in a consecutive series of patients with acute brain injury. (Supported by the Dana Foundation and the James S. McDonnell Foundation.).

Humans can spontaneously detect complex algebraic structures. Historically, two opposing views explain this ability, at the root of language and music acquisition. Some argue for the existence of an innate and specific mechanism. Others argue that this ability emerges from experience: i.e. when generic learning principles continuously process sensory inputs. These two views, however, remain difficult to test experimentally. Here, we use deep learning models to evaluate the factors that lead to the spontaneous detection of algebraic structures in the auditory modality. Specifically, we use self-supervised learning to train multiple deep-learning models with a variable amount of either natural (environmental sounds) and/or cultural sounds (speech or music) to evaluate the impact of these stimuli. We then expose these models to the experimental paradigms classically used to evaluate the processing of algebraic structures. Like humans, these models spontaneously detect repeated sequences, probabilistic chunks, and complex algebraic structures. Also like humans, this ability diminishes with structure complexity. Importantly, this ability can emerge from experience alone: the more the models are exposed to natural sounds, the more they spontaneously detect increasingly complex structures. Finally, this ability does not emerge in models pretrained only on speech, and emerges more rapidly in models pretrained with music than environmental sounds. Overall, our study provides an operational framework to clarify sufficient built-in and acquired principles that model human’s advanced capacity to detect algebraic structures in sounds.

Decoding speech from brain activity is a long-awaited goal in both healthcare and neuroscience. Invasive devices have recently led to major milestones in this regard: deep-learning algorithms trained on intracranial recordings can now start to decode elementary linguistic features such as letters, words and audio-spectrograms. However, extending this approach to natural speech and non-invasive brain recordings remains a major challenge. Here we introduce a model trained with contrastive learning to decode self-supervised representations of perceived speech from the non-invasive recordings of a large cohort of healthy individuals. To evaluate this approach, we curate and integrate four public datasets, encompassing 175 volunteers recorded with magneto-encephalography or electro-encephalography while they listened to short stories and isolated sentences. The results show that our model can identify, from 3 seconds of magneto-encephalography signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities on average across participants, and with up to 80% in the best participants—a performance that allows the decoding of words and phrases absent from the training set. The comparison of our model with a variety of baselines highlights the importance of a contrastive objective, pretrained representations of speech and a common convolutional architecture simultaneously trained across multiple participants. Finally, the analysis of the decoder’s predictions suggests that they primarily depend on lexical and contextual semantic representations. Overall, this effective decoding of perceived speech from non-invasive recordings delineates a promising path to decode language from brain activity, without putting patients at risk of brain surgery. Deep learning can help develop non-invasive technology for decoding speech from brain activity, which could improve the lives of patients with brain injuries. Défossez et al. report a contrastive-learning approach to decode speech listening from human participants, using public databases of recordings based on non-invasive magnetic and electrical measurements.

Long-range cortico-cortical functional connectivity has long been theorized to be necessary for conscious states. In the present work, we estimate long-range cortical connectivity in a series of intracranial and scalp EEG recordings experiments. In the two first experiments intracranial-EEG (iEEG) was recorded during four distinct states within the same individuals: conscious wakefulness (CW), rapid-eye-movement sleep (REM), stable periods of slow-wave sleep (SWS) and deep propofol anaesthesia (PA). We estimated functional connectivity using the following two methods: weighted Symbolic-Mutual-Information (wSMI) and phase-locked value (PLV). Our results showed that long-range functional connectivity in the delta-theta frequency band specifically discriminated CW and REM from SWS and PA. In the third experiment, we generalized this original finding on a large cohort of brain-injured patients. FC in the delta-theta band was significantly higher in patients being in a minimally conscious state (MCS) than in those being in a vegetative state (or unresponsive wakefulness syndrome). Taken together the present results suggest that FC of cortical activity in this slow frequency band is a new and robust signature of conscious states.

Working memory and conscious perception are thought to share similar brain mechanisms, yet recent reports of non-conscious working memory challenge this view. Combining visual masking with magnetoencephalography, we investigate the reality of non-conscious working memory and dissect its neural mechanisms. In a spatial delayed-response task, participants reported the location of a subjectively unseen target above chance-level after several seconds. Conscious perception and conscious working memory were characterized by similar signatures: a sustained desynchronization in the alpha/beta band over frontal cortex, and a decodable representation of target location in posterior sensors. During non-conscious working memory, such activity vanished. Our findings contradict models that identify working memory with sustained neural firing, but are compatible with recent proposals of ‘activity-silent’ working memory. We present a theoretical framework and simulations showing how slowly decaying synaptic changes allow cell assemblies to go dormant during the delay, yet be retrieved above chance-level after several seconds. Many everyday activities require you to store information in your brain for immediate use. For example, imagine that you are cooking a meal: You have to remember the ingredients, add them in the correct order, and operate the stove. This ability is called working memory. Researchers have long believed that, whenever we store information in our working memory, we are conscious of that information. That is, if someone asks you, you can report the information. Scientists usually also think that working memory comes with constant brain activity. This means that for as long as you have to remember something, the cells in your brain that code for that information will be active. Trübutschek et al. now show that we can sometimes store information in working memory without being conscious of it and without the need for constant brain activity. As part of the experiment, a barely visible square-shaped target was briefly flashed in 1 of 20 different locations on a computer screen. Human volunteers had to locate the square and indicate whether they had seen it or not. Importantly, they had to guess the location of the target whenever they had not seen it. While the volunteers performed this task, their brain activity was monitored using magnetoencephalography, a noninvasive technique that captures the magnetic fields created by electrical signals in the brain. Even when the volunteers had not seen the target, they could often correctly guess where it had been up to four seconds later, more often than would be predicted by chance alone. The experiment ruled out the possibility that this so-called “blindsight” was simply due to the volunteers accidentally reporting not having seen a target, when they had actually seen it. It also excluded the possibility that the volunteers guessed the location long before they had to report it and simply consciously stored that guess. Instead, without the participant knowing, the brain appears to have stored the target location in working memory using parts of the brain near the back of the head that process visual information. Importantly, this non-conscious storage did not come with constant brain activity, but seemed to rely on other, “activity-silent” mechanisms that are hidden to standard recording techniques. Although Trübutschek et al. show that the brain can unknowingly store information, they did not test other aspects of working memory. Future studies are needed to examine whether the brain can also non-consciously manipulate or use information in its working memory. In addition, future research also needs to investigate the exact mechanism that stores information without constant brain activity.

Combinatoric linguistic operations underpin human language processes, but how meaning is composed and refined in the mind of the reader is not well understood. We address this puzzle by exploiting the ubiquitous function of negation. We track the online effects of negation (“not”) and intensifiers (“really”) on the representation of scalar adjectives (e.g., “good”) in parametrically designed behavioral and neurophysiological (MEG) experiments. The behavioral data show that participants first interpret negated adjectives as affirmative and later modify their interpretation towards, but never exactly as, the opposite meaning. Decoding analyses of neural activity further reveal significant above chance decoding accuracy for negated adjectives within 600 ms from adjective onset, suggesting that negation does not invert the representation of adjectives (i.e., “not bad” represented as “good”); furthermore, decoding accuracy for negated adjectives is found to be significantly lower than that for affirmative adjectives. Overall, these results suggest that negation mitigates rather than inverts the neural representations of adjectives. This putative suppression mechanism of negation is supported by increased synchronization of beta-band neural activity in sensorimotor areas. The analysis of negation provides a steppingstone to understand how the human brain represents changes of meaning over time.