Explore the vast range of titles available.

The phasic dopamine response is traditionally thought to signal reward prediction errors. Redgrave and Gurney evaluate evidence from studies of basal ganglia circuitry and signal timing, and propose instead that the short-latency dopamine signal is important for discovering novel actions. An influential concept in contemporary computational neuroscience is the reward prediction error hypothesis of phasic dopaminergic function. It maintains that midbrain dopaminergic neurons signal the occurrence of unpredicted reward, which is used in appetitive learning to reinforce existing actions that most often lead to reward. However, the availability of limited afferent sensory processing and the precise timing of dopaminergic signals suggest that they might instead have a central role in identifying which aspects of context and behavioural output are crucial in causing unpredicted events.

Key Points The basal ganglia are one of the fundamental processing units of the mammalian brain. Progressive degeneration of one of their major components, the ascending dopamine projection to the striatum, is a central pathological feature of Parkinson's disease. Imaging and post-mortem investigations reveal that degeneration of the dopamine projection is uneven in most cases, with input to caudolateral sectors of the putamen most severely affected. In the animal learning literature an important distinction has been forged between goal-directed and habitual control of behaviour. When behaviour is goal-directed, action selection is determined by the relative utility of predicted outcomes, whereas habits are under stimulus control and largely independent of outcome value. A seminal series of investigations in rodents by Balleine and colleagues established that the dorsomedial associative territories of the striatum are crucial for goal-directed control, whereas laterally located sensorimotor territories are essential for habits. Formal behavioural tests (for example, outcome devaluation) were used to determine whether an observed behaviour (for example, pressing a lever) was under goal-directed or habitual control. Recent neuroimaging studies using the same formal tests suggest that a similar spatial segregation of goal-directed and habitual control is present within the human striatum. As the loss of dopamine in Parkinson's disease is predominantly from the caudolateral sensorimotor territories, we would expect patients to experience major deficits in their production of habits. Because the same behavioural output can be directed by processing in spatially segregated regions of the basal ganglia, it must be assumed that the efferent projections of goal-directed and habitual control circuits must at some point converge on the 'final common motor path'. Given that the loss of dopamine in the basal ganglia is associated with enhanced oscillatory and inhibitory outputs, we suggest that for goal-directed control to be expressed, the distorting inhibitory signals from the habit system must be overcome at the point where the goal-directed and habitual control circuits converge. We conclude by reviewing evidence suggesting that many of the behavioural difficulties experienced by patients with Parkinson's disease can be interpreted in terms of an impaired automatic control of normal habits, coupled with distorting inhibitory influences imposed on the expression of residual goal-directed behaviours. In the light of this analysis, future work will need to establish how far the reported cognitive deficits in Parkinson's disease are due to the primary disease state (additional loss of dopamine from goal-directed circuits) or are a result of goal-directed control being overwhelmed by the absence of automatic control routines that are normally provided by the stimulus–response habit systems. In this Review, Redgrave et al . provide an updated model of basal ganglia architecture in which spatially segregated functional territories contribute to goal-directed and habitual control circuits. They propose that in Parkinson's disease, selective dopamine loss impairs habitual control and distorts goal-directed behaviours. Progressive loss of the ascending dopaminergic projection in the basal ganglia is a fundamental pathological feature of Parkinson's disease. Studies in animals and humans have identified spatially segregated functional territories in the basal ganglia for the control of goal-directed and habitual actions. In patients with Parkinson's disease the loss of dopamine is predominantly in the posterior putamen, a region of the basal ganglia associated with the control of habitual behaviour. These patients may therefore be forced into a progressive reliance on the goal-directed mode of action control that is mediated by comparatively preserved processing in the rostromedial striatum. Thus, many of their behavioural difficulties may reflect a loss of normal automatic control owing to distorting output signals from habitual control circuits, which impede the expression of goal-directed action.

Operant learning requires that reinforcement signals interact with action representations at a suitable neural interface. Much evidence suggests that this occurs when phasic dopamine, acting as a reinforcement prediction error, gates plasticity at cortico-striatal synapses, and thereby changes the future likelihood of selecting the action(s) coded by striatal neurons. But this hypothesis faces serious challenges. First, cortico-striatal plasticity is inexplicably complex, depending on spike timing, dopamine level, and dopamine receptor type. Second, there is a credit assignment problem-action selection signals occur long before the consequent dopamine reinforcement signal. Third, the two types of striatal output neuron have apparently opposite effects on action selection. Whether these factors rule out the interface hypothesis and how they interact to produce reinforcement learning is unknown. We present a computational framework that addresses these challenges. We first predict the expected activity changes over an operant task for both types of action-coding striatal neuron, and show they co-operate to promote action selection in learning and compete to promote action suppression in extinction. Separately, we derive a complete model of dopamine and spike-timing dependent cortico-striatal plasticity from in vitro data. We then show this model produces the predicted activity changes necessary for learning and extinction in an operant task, a remarkable convergence of a bottom-up data-driven plasticity model with the top-down behavioural requirements of learning theory. Moreover, we show the complex dependencies of cortico-striatal plasticity are not only sufficient but necessary for learning and extinction. Validating the model, we show it can account for behavioural data describing extinction, renewal, and reacquisition, and replicate in vitro experimental data on cortico-striatal plasticity. By bridging the levels between the single synapse and behaviour, our model shows how striatum acts as the action-reinforcement interface.

Plasticity at synapses between the cortex and striatum is considered critical for learning novel actions. However, investigations of spike-timing-dependent plasticity (STDP) at these synapses have been performed largely in brain slice preparations, without consideration of physiological reinforcement signals. This has led to conflicting findings, and hampered the ability to relate neural plasticity to behavior. Using intracellular striatal recordings in intact rats, we show here that pairing presynaptic and postsynaptic activity induces robust Hebbian bidirectional plasticity, dependent on dopamine and adenosine signaling. Such plasticity, however, requires the arrival of a reward-conditioned sensory reinforcement signal within 2 s of the STDP pairing, thus revealing a timing-dependent eligibility trace on which reinforcement operates. These observations are validated with both computational modeling and behavioral testing. Our results indicate that Hebbian corticostriatal plasticity can be induced by classical reinforcement learning mechanisms, and might be central to the acquisition of novel actions. Spike timing dependent plasticity (STDP) has been studied extensively in slices but whether such pairings can induce plasticity in vivo is not known. Here the authors report an experimental paradigm that achieves bidirectional corticostriatal STDP in vivo through modulation by behaviourally relevant reinforcement signals, mediated by dopamine and adenosine signaling.

Blindsight is the residual visuo-motor ability without subjective awareness observed after lesions of the primary visual cortex (V1). Various visual functions are retained, however, instrumental visual associative learning remains to be investigated. Here we examined the secondary reinforcing properties of visual cues presented to the hemianopic field of macaque monkeys with unilateral V1 lesions. Our aim was to test the potential role of visual pathways bypassing V1 in reinforcing visual instrumental learning. When learning the location of a hidden area in an oculomotor search task, conditioned visual cues presented to the lesion-affected hemifield operated as an effective secondary reinforcer. We noted that not only the hidden area location, but also the vector of the saccade entering the target area was reinforced. Importantly, when the visual reinforcement signal was presented in the lesion-affected field, the monkeys continued searching, as opposed to stopping when the cue was presented in the intact field. This suggests the monkeys were less confident that the target location had been discovered when the reinforcement cue was presented in the affected field. These results indicate that the visual signals mediated by the residual visual pathways after V1 lesions can access fundamental reinforcement mechanisms but with impaired visual awareness.

Responses of midbrain dopamine (DA) neurons reflecting expected reward from sensory cues are critical for reward-based associative learning. However, critical pathways by which reward-related visual information is relayed to DA neurons remain unclear. To address this question, we investigated Pavlovian conditioning in macaque monkeys with unilateral primary visual cortex (V1) lesions (an animal model of ‘blindsight’). Anticipatory licking responses to obtain juice drops were elicited in response to visual conditioned stimuli (CS) in the affected visual field. Subsequent pharmacological inactivation of the superior colliculus (SC) suppressed the anticipatory licking. Concurrent single unit recordings indicated that DA responses reflecting the reward expectation could be recorded in the absence of V1, and that these responses were also suppressed by SC inactivation. These results indicate that the subcortical visual circuit can relay reward-predicting visual information to DA neurons and integrity of the SC is necessary for visually-elicited classically conditioned responses after V1 lesion. To survive and thrive, animals must learn to approach cues in their environment that are likely to lead to a desirable outcome and avoid those that might lead them to harm. A group of brain regions known as the midbrain dopamine system helps many animals to achieve this. Dopamine is the brain’s reward signal. Cues that predict rewards, such as the sight or smell of food, activate midbrain dopamine neurons. However, the details of this process remained unclear. Takakuwa et al. have now examined how visual information that signals reward reaches the midbrain dopamine neurons. The anatomy of the visual system suggests two main possibilities. Information may travel directly from the eyes to an area of the midbrain called the superior colliculus, and then onto the dopamine neurons. Alternatively, information may travel to the midbrain indirectly via a pathway that includes additional processing in the brain’s outer layer, the visual cortex. To distinguish between these routes, Takakuwa et al. studied monkeys in which the indirect pathway via the visual cortex had been damaged. Some people with damage to this pathway have a disorder called blindsight. They are able to detect the movement or location of stimuli, but they cannot consciously see those stimuli. The monkeys with damage to visual cortex were able to learn that an image on a screen predicted the delivery of fruit juice. After repeated trials, the monkeys began to lick the spout dispensing the juice whenever the image appeared, even if no juice was delivered. The monkeys’ midbrain dopamine neurons also sent more signals in response to the images, and showed greater activity when the images predicted large rewards than small ones. Takakuwa et al. next inactivated the superior colliculus with a drug and showed that this prevented both the licking behavior and the increased signaling. Together the findings show that visual information about potential rewards can reach midbrain dopamine neurons via a direct route through the superior colliculus, without needing to pass via the visual cortex. The next step is to determine how and when the visual cortex may get involved in this process to help animals maximize rewards.

Unexpected, biologically salient stimuli elicit a short-latency, phasic response in midbrain dopaminergic (DA) neurons. Although this signal is important for reinforcement learning, the information it conveys to forebrain target structures remains uncertain. One way to decode the phasic DA signal would be to determine the perceptual properties of sensory inputs to DA neurons. After local disinhibition of the superior colliculus in anesthetized rats, DA neurons became visually responsive, whereas disinhibition of the visual cortex was ineffective. As the primary source of visual afferents, the limited processing capacities of the colliculus may constrain the visual information content of phasic DA responses.

Intrinsic motivations drive the acquisition of knowledge and skills on the basis of novel or surprising stimuli or the pleasure to learn new skills. In so doing, they are different from extrinsic motivations that are mainly linked to drives that promote survival and reproduction. Intrinsic motivations have been implicitly exploited in several psychological experiments but, due to the lack of proper paradigms, they are rarely a direct subject of investigation. This article investigates how different intrinsic motivation mechanisms can support the learning of visual skills, such as \"foveate a particular object in space\", using a gaze contingency paradigm. In the experiment participants could freely foveate objects shown in a computer screen. Foveating each of two \"button\" pictures caused different effects: one caused the appearance of a simple image (blue rectangle) in unexpected positions, while the other evoked the appearance of an always-novel picture (objects or animals). The experiment studied how two possible intrinsic motivation mechanisms might guide learning to foveate one or the other button picture. One mechanism is based on the sudden, surprising appearance of a familiar image at unpredicted locations, and a second one is based on the content novelty of the images. The results show the comparative effectiveness of the mechanism based on image novelty, whereas they do not support the operation of the mechanism based on the surprising location of the image appearance. Interestingly, these results were also obtained with participants that, according to a post experiment questionnaire, had not understood the functions of the different buttons suggesting that novelty-based intrinsic motivation mechanisms might operate even at an unconscious level.

The vertebrate basal ganglia play an important role in action selection—the resolution of conflicts between alternative motor programs. The effective operation of basal ganglia circuitry is also known to rely on appropriate levels of the neurotransmitter dopamine. We investigated reducing or increasing the tonic level of simulated dopamine in a prior model of the basal ganglia integrated into a robot control architecture engaged in a foraging task inspired by animal behaviour. The main findings were that progressive reductions in the levels of simulated dopamine caused slowed behaviour and, at low levels, an inability to initiate movement. These states were partially relieved by increased salience levels (stronger sensory/motivational input). Conversely, increased simulated dopamine caused distortion of the robot’s motor acts through partially expressed motor activity relating to losing actions. This could also lead to an increased frequency of behaviour switching. Levels of simulated dopamine that were either significantly lower or higher than baseline could cause a loss of behavioural integration, sometimes leaving the robot in a ‘behavioral trap’. That some analogous traits are observed in animals and humans affected by dopamine dysregulation suggests that robotic models could prove useful in understanding the role of dopamine neurotransmission in basal ganglia function and dysfunction.

Midbrain dopaminergic neurons respond to unexpected and biologically salient events, but little is known about the sensory systems underlying this response. Here we describe, in the rat, a direct projection from a primary visual structure, the midbrain superior colliculus (SC), to the substantia nigra pars compacta (SNc) where direct synaptic contacts are made with both dopaminergic and non-dopaminergic neurons. Complementary electrophysiological data reveal that short-latency visual responses in the SNc are abolished by ipsilateral lesions of the SC and increased by local collicular stimulation. These results show that the tectonigral projection is ideally located to relay short-latency visual information to dopamine-containing regions of the ventral midbrain. We conclude that it is within this afferent sensory circuitry that the critical perceptual discriminations that identify stimuli as both unpredicted and biologically salient are made.