Explore the vast range of titles available.

Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval. Machine learning algorithms can decode objects that people see or imagine from their brain activity. Here the authors present a predictive decoder combined with deep neural network representations that generalizes beyond the training set and correctly identifies novel objects that it has never been trained on.

Extensive sampling of neural activity during rich cognitive phenomena is critical for robust understanding of brain function. Here we present the Natural Scenes Dataset (NSD), in which high-resolution functional magnetic resonance imaging responses to tens of thousands of richly annotated natural scenes were measured while participants performed a continuous recognition task. To optimize data quality, we developed and applied novel estimation and denoising techniques. Simple visual inspections of the NSD data reveal clear representational transformations along the ventral visual pathway. Further exemplifying the inferential power of the dataset, we used NSD to build and train deep neural network models that predict brain activity more accurately than state-of-the-art models from computer vision. NSD also includes substantial resting-state and diffusion data, enabling network neuroscience perspectives to constrain and enhance models of perception and memory. Given its unprecedented scale, quality and breadth, NSD opens new avenues of inquiry in cognitive neuroscience and artificial intelligence. The authors measured high-resolution fMRI activity from eight individuals who saw and memorized thousands of annotated natural images over 1 year. This massive dataset enables new paths of inquiry in cognitive neuroscience and artificial intelligence.

Recent computational neuroscience developments have used deep neural networks to model neural responses in higher visual areas. This Perspective describes key algorithmic underpinnings in computer vision and artificial intelligence that have contributed to this progress and outlines how deep networks could drive future improvements in understanding sensory cortical processing. Fueled by innovation in the computer vision and artificial intelligence communities, recent developments in computational neuroscience have used goal-driven hierarchical convolutional neural networks (HCNNs) to make strides in modeling neural single-unit and population responses in higher visual cortical areas. In this Perspective, we review the recent progress in a broader modeling context and describe some of the key technical innovations that have supported it. We then outline how the goal-driven HCNN approach can be used to delve even more deeply into understanding the development and organization of sensory cortical processing.

Mental imagery can be advantageous, unnecessary and even clinically disruptive. With methodological constraints now overcome, research has shown that visual imagery involves a network of brain areas from the frontal cortex to sensory areas, overlapping with the default mode network, and can function much like a weak version of afferent perception. Imagery vividness and strength range from completely absent (aphantasia) to photo-like (hyperphantasia). Both the anatomy and function of the primary visual cortex are related to visual imagery. The use of imagery as a tool has been linked to many compound cognitive processes and imagery plays both symptomatic and mechanistic roles in neurological and mental disorders and treatments.

The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity – a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations – suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems. Through controlled model-to-brain comparisons across a large-scale survey of deep neural networks, the authors show the data models are trained on matters far more for downstream brain prediction than design factors such as architecture and training task.

Non-recurrent deep convolutional neural networks (CNNs) are currently the best at modeling core object recognition, a behavior that is supported by the densely recurrent primate ventral stream, culminating in the inferior temporal (IT) cortex. If recurrence is critical to this behavior, then primates should outperform feedforward-only deep CNNs for images that require additional recurrent processing beyond the feedforward IT response. Here we first used behavioral methods to discover hundreds of these ‘challenge’ images. Second, using large-scale electrophysiology, we observed that behaviorally sufficient object identity solutions emerged ~30 ms later in the IT cortex for challenge images compared with primate performance-matched ‘control’ images. Third, these behaviorally critical late-phase IT response patterns were poorly predicted by feedforward deep CNN activations. Notably, very-deep CNNs and shallower recurrent CNNs better predicted these late IT responses, suggesting that there is a functional equivalence between additional nonlinear transformations and recurrence. Beyond arguing that recurrent circuits are critical for rapid object identification, our results provide strong constraints for future recurrent model development.Using model- and primate behavior-driven image selection with large-scale electrophysiology in monkeys performing core recognition tasks, Kar et al. provide evidence that automatically engaged recurrent circuits are critical for rapid object identification.

The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans. However, the stage-wise computations therein remain poorly understood. Here, we compared temporal (magnetoencephalography) and spatial (functional MRI) visual brain representations with representations in an artificial deep neural network (DNN) tuned to the statistics of real-world visual recognition. We showed that the DNN captured the stages of human visual processing in both time and space from early visual areas towards the dorsal and ventral streams. Further investigation of crucial DNN parameters revealed that while model architecture was important, training on real-world categorization was necessary to enforce spatio-temporal hierarchical relationships with the brain. Together our results provide an algorithmically informed view on the spatio-temporal dynamics of visual object recognition in the human visual brain.

Cognitive maps are internal representations of large-scale navigable spaces. While they have been long studied in rodents, recent work in humans reveals new insights into how cognitive maps are encoded, anchored to environmental landmarks and used to plan routes. Similar neural mechanisms might be used to form ‘maps’ of nonphysical spaces. The 'cognitive map' hypothesis proposes that brain builds a unified representation of the spatial environment to support memory and guide future action. Forty years of electrophysiological research in rodents suggest that cognitive maps are neurally instantiated by place, grid, border and head direction cells in the hippocampal formation and related structures. Here we review recent work that suggests a similar functional organization in the human brain and yields insights into how cognitive maps are used during spatial navigation. Specifically, these studies indicate that (i) the human hippocampus and entorhinal cortex support map-like spatial codes, (ii) posterior brain regions such as parahippocampal and retrosplenial cortices provide critical inputs that allow cognitive maps to be anchored to fixed environmental landmarks, and (iii) hippocampal and entorhinal spatial codes are used in conjunction with frontal lobe mechanisms to plan routes during navigation. We also discuss how these three basic elements of cognitive map based navigation—spatial coding, landmark anchoring and route planning—might be applied to nonspatial domains to provide the building blocks for many core elements of human thought.

The inferotemporal (IT) cortex is responsible for object recognition, but it is unclear how the representation of visual objects is organized in this part of the brain. Areas that are selective for categories such as faces, bodies, and scenes have been found 1 – 5 , but large parts of IT cortex lack any known specialization, raising the question of what general principle governs IT organization. Here we used functional MRI, microstimulation, electrophysiology, and deep networks to investigate the organization of macaque IT cortex. We built a low-dimensional object space to describe general objects using a feedforward deep neural network trained on object classification 6 . Responses of IT cells to a large set of objects revealed that single IT cells project incoming objects onto specific axes of this space. Anatomically, cells were clustered into four networks according to the first two components of their preferred axes, forming a map of object space. This map was repeated across three hierarchical stages of increasing view invariance, and cells that comprised these maps collectively harboured sufficient coding capacity to approximately reconstruct objects. These results provide a unified picture of IT organization in which category-selective regions are part of a coarse map of object space whose dimensions can be extracted from a deep network. Primate inferotemporal cortex contains a coarse map of object space consisting of four networks, identified using functional imaging, electrophysiology and deep networks.

In order to better understand how the brain perceives faces, it is important to know what objective drives learning in the ventral visual stream. To answer this question, we model neural responses to faces in the macaque inferotemporal (IT) cortex with a deep self-supervised generative model, β -VAE, which disentangles sensory data into interpretable latent factors, such as gender or age. Our results demonstrate a strong correspondence between the generative factors discovered by β -VAE and those coded by single IT neurons, beyond that found for the baselines, including the handcrafted state-of-the-art model of face perception, the Active Appearance Model, and deep classifiers. Moreover, β -VAE is able to reconstruct novel face images using signals from just a handful of cells. Together our results imply that optimising the disentangling objective leads to representations that closely resemble those in the IT at the single unit level. This points at disentangling as a plausible learning objective for the visual brain. Little is known about the brain’s computations that enable the recognition of faces. Here, the authors use unsupervised deep learning to show that the brain disentangles faces into semantically meaningful factors, like age or the presence of a smile, at the single neuron level.