Spatio-Temporal Credit Assignment in Neuronal Population Learning

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Cuidados de enfermagem preventivos em aves, em contexto zoológico:

Tanto o enriquecimento ambiental, como o treino pelo método do condicionamento operante por reforço positivo, são bastante relevantes no desempenho bem-sucedido de medidas de saúde preventivas em animais de cativeiro. A prevenção apresenta-se como de extrema importância no bem-estar das aves em contexto zoológico. Um maneio e enriquecimento apropriados, bem como programas de treino que incluam comportamentos médicos, podem não só diminuir a ocorrência de patologias, muitas vezes antropogénicas, mas também permitir o correto diagnóstico precoce das mesmas e controlo da saúde animal. Desta forma, contribui-se para o melhoramento do dia-a-dia dos animais e consequentemente da sua vida.

Two -way direction of influence in mother-infant interaction as denoted by concurrent reciprocal learning processes

This series of 5 single-subject studies used the operant conditioning paradigm to investigate, within the two-way influence process, how (a) contingent infant attention can reinforce maternal verbal behaviors during a period of mother-infant interaction and under subsequent experimental manipulation. Differential reinforcement was used to determine if it is possible that an infant attending to the mother (denoted by head-turns towards the image of the mother plus eye contact) increases (reinforces) the mother's verbal response (to a cue from the infant) upon which the infant behavior is contingent. There was also (b) an evaluation during the contrived parent-infant interaction for concurrent operant learning of infant vocal behavior via contingent verbal responding (reinforcement) implemented by the mother. Further, it was noted (c) whether or not the mother reported being aware that her responses were influenced by the infant's behavior. Findings showed: the operant conditioning of the maternal verbal behaviors were reinforced by contingent infant attention; and the operant conditioning of infant vocalizations was reinforced by contingent maternal verbal behaviors. No parent reported (1) being aware of the increase in their verbal response reinforced during operant conditioning of parental behavior nor a decrease in those responses during the DRA reversal phase, or (2) noticing a contingency between infant's and mother's response. By binomial 1-tail tests, the verbal-behavior patterns of the 5 mothers were conditioned by infant reinforcement (p < 0.02) and, concurrently, the vocal-response patterns of the 5 infants were conditioned by maternal reinforcement (p < 0.02). A program of systematic empirical research on the determinants of concurrent conditioning within mother-child interaction may provide a way to evaluate the differential effectiveness of interventions aimed at improving parent-child interactions. The work conducted in the present study is one step in this direction. ^

Impaired GABA Plasticity at ovBNST Synapses Predicts Compulsive Drinking in Schedule-Induced Polydipsia

Schedule-Induced Polydipsia (SIP) is an animal model of adjunctive drinking induced when a hungry rat receives food on a fixed interval of time. This model has been implemented as a model of compulsive behaviour and may represent a powerful tool to understand the neural mechanisms of compulsion. The bed nucleus of the stria terminalis (BNST) is thought to translate challenges to energy homeostasis into consummatory behaviours, and is therefore likely to contribute to drinking behaviours displayed by food restricted rats in the SIP paradigm. Furthermore, the BNST seems implicated in various compulsive behaviors, including compulsive water drinking in rats. Therefore, the goal of this project was to determine whether compulsive drinking in the SIP paradigm was associated with alterations in transmission at oval BNST (ovBNST) synapses. Rats undergoing the SIP procedure had restricted food access (1-hours/day) for a total of 29 days. After 7 days of food restriction and for the next 21 consecutive days, the rats had daily 2-hour access to operant conditioning chambers where they were presented with a 45-mg food pellet every minute. Water consumed during these 2-hour sessions was measured and the rats that drank 15 ml or more water for a minimum of 3 consecutive days were considered High Drinkers (HD; n=17) or otherwise, Low Drinkers (LD; n=13). Brain slices whole-cell patch clamp recordings conducted 18-hours after the last SIP training showed that chronic food restriction changed low frequency stimulation (LFS) - induced long-term potentiation of ovBNST inhibitory synaptic transmission (iLTP) into LFS - induced long-term depression (iLTD) in a majority of neurons, regardless of drinking behaviours. However, ad libitum access to food between the last day of SIP training and brain slice recordings (18-hour refeed) rescued LFS-induced iLTP in LD but not in HD, suggesting that impaired bi-directional plasticity of ovBNST synapses may contribute to compulsive drinking in the SIP paradigm.

Restoring function after neuromuscular trauma: novel paradigms for enhancing volitional control of neural activity in untethered environments and vHAB, a gamified therapy platform

Thesis (Ph.D.)--University of Washington, 2016-08

Operant serial feature-positive conditional discrimination with composite features and different topography responses

In a serial feature-positive conditional discrimination procedure the properties of a target stimulus A are defined by the presence or not of a feature stimulus X preceding it. In the present experiment, composite features preceded targets associated with two different topography operant responses (right and left bar pressing); matching and non-matching-to-sample arrangements were also used. Five water-deprived Wistar rats were trained in 6 different trials: X-R®Ar and X-L®Al, in which X and A were same modality visual stimuli and the reinforcement was contingent to pressing either the right (r) or left (l) bar that had the light on during the feature (matching-to-sample); Y-R®Bl and Y-L®Br, in which Y and B were same modality auditory stimuli and the reinforcement was contingent to pressing the bar that had the light off during the feature (non-matching-to-sample); A- and B- alone. After 100 training sessions, the animals were submitted to transfer tests with the targets used plus a new one (auditory click). Average percentages of stimuli with a response were measured. Acquisition occurred completely only for Y-L®Br+; however, complex associations were established along training. Transfer was not complete during the tests since concurrent effects of extinction and response generalization also occurred. Results suggest the use of both simple conditioning and configurational strategies, favoring the most recent theories of conditional discrimination learning. The implications of the use of complex arrangements for discussing these theories are considered.

Subtype-selective GABAergic drugs facilitate extinction of mouse operant behaviour

Several recent studies have shown that reducing gamma-aminobutyric acid (GABA)-mediated neurotransmission retards extinction of aversive conditioning. However, relatively little is known about the effect of GABA on extinction of appetitively motivated tasks. We examined the effect of chlordiazepoxide (CDP), a classical benzodiazepine (BZ) and two novel subtype-selective BZs when administered to male C57Bl/6 mice during extinction following training on a discrete-trial fixed-ratio 5 (FR5) food reinforced lever-press procedure. Initially CDP had no effect, but after several extinction sessions CDP significantly facilitated extinction, i.e. slowed responding, compared with vehicle-treated mice. This effect was not due to drug accumulation because mice switched from vehicle treatment to CDP late in extinction showed facilitation immediately. Likewise, this effect could not be attributed to sedation because the dose of CDP used (15 mg/kg i.p.) did not suppress locomotor activity. The two novel subtype-selective BZ partial agonists, L-838417 and TP13, selectively facilitated extinction in similar fashion to CDP. The non-GABAergic anxiolytic buspirone was also tested and found to have similar effects when administered at a non-sedating dose. These studies demonstrate that GABA-mediated processes are important during extinction of an appetitively motivated task, but only after the animals have experienced several extinction sessions.

Operant measurement of auditory threshold in prelingually deaf users of cochlear implants: II

Two experiments evaluated an operant procedure for establishing stimulus control using auditory and electrical stimuli as a baseline for measuring the electrical current threshold of electrodes implanted in the cochlea. Twenty-one prelingually deaf children, users of cochlear implants, learned a Go/No Go auditory discrimination task (i.e., pressing a button in the presence of the stimulus but not in its absence). When the simple discrimination baseline became stable, the electrical current was manipulated in descending and ascending series according to an adapted staircase method. Thresholds were determined for three electrodes, one in each location in the cochlea (basal, medial, and apical). Stimulus control was maintained within a certain range of decreasing electrical current but was eventually disrupted. Increasing the current recovered stimulus control, thus allowing the determination of a range of electrical currents that could be defined as the threshold. The present study demonstrated the feasibility of the operant procedure combined with a psychophysical method for threshold assessment, thus contributing to the routine fitting and maintenance of cochlear implants within the limitations of a hospital setting.

Operant and respondent procedures to establish social stimuli as reinforcers in children with autism

According to the DSM-IV- TR (American Psychiatric Association, 2000), one of the core deficits in autism is in the impairment of social interaction. Some have suggested that underlying these deficits is the reality that individuals with autism do not find social stimuli to be as reinforcing as other types of stimuli (Dawson, 2008). An interesting and growing body of literature supports the notion that symptoms in autism may be caused by a general reduction in social motivation (Chevallier et al., 2012). A review of the literature suggests that social orienting and social motivation are low in individuals with autism, and including social motivation as a target for therapeutic intervention should be pursued (Helt et al., 2008). Through our understanding of learning processes, researchers in behavior analysis and related fields have been able to use conditioning procedures to change the function of neutral or ineffective stimuli, including tokens (Ayllon & Azrin, 1968), facial expressions (Gewirtz & Pelaez-Nogueras, 1992) and praise (Dozier et al., 2012). The current study aimed to use operant and respondent procedures to condition social stimuli that were empirically shown to not be reinforcing prior to conditioning. Further, this study aimed to compare the two procedures in their effectiveness to condition social stimuli to function as reinforcers, and in their maintenance of effects over time. Using a multiple-baseline, multi-element design, one social stimulus was conditioned under each procedure to compare the different response rates following conditioning. Finally, the study sought to determine if conditioning social stimuli to function as reinforcers had any effect on the social functioning of young children with autism. Six children diagnosed with autism between the ages of 18 months and 3 years participated. Results show that the respondent procedure (pairing) resulted in more robust and enduring effects than the operant procedure (Sd procedure). Results of a social communication assessment (ESCS, Mundy et al., 2003) before and after conditioning demonstrate gains in all areas of social communication, particularly in the areas of initiating and responding to joint attention.^

Operant and Respondent Procedures to Establish Social Stimuli as Reinforcers in Children with Autism

According to the DSM-IV- TR (American Psychiatric Association, 2000), one of the core deficits in autism is in the impairment of social interaction. Some have suggested that underlying these deficits is the reality that individuals with autism do not find social stimuli to be as reinforcing as other types of stimuli (Dawson, 2008). An interesting and growing body of literature supports the notion that symptoms in autism may be caused by a general reduction in social motivation (Chevallier et al., 2012). A review of the literature suggests that social orienting and social motivation are low in individuals with autism, and including social motivation as a target for therapeutic intervention should be pursued (Helt et al., 2008). Through our understanding of learning processes, researchers in behavior analysis and related fields have been able to use conditioning procedures to change the function of neutral or ineffective stimuli, including tokens (Ayllon & Azrin, 1968), facial expressions (Gewirtz & Pelaez-Nogueras, 1992) and praise (Dozier et al., 2012). The current study aimed to use operant and respondent procedures to condition social stimuli that were empirically shown to not be reinforcing prior to conditioning. Further, this study aimed to compare the two procedures in their effectiveness to condition social stimuli to function as reinforcers, and in their maintenance of effects over time. Using a multiple-baseline, multi-element design, one social stimulus was conditioned under each procedure to compare the different response rates following conditioning. Finally, the study sought to determine if conditioning social stimuli to function as reinforcers had any effect on the social functioning of young children with autism. Six children diagnosed with autism between the ages of 18 months and 3 years participated. Results show that the respondent procedure (pairing) resulted in more robust and enduring effects than the operant procedure (Sd procedure). Results of a social communication assessment (ESCS, Mundy et al., 2003) before and after conditioning demonstrate gains in all areas of social communication, particularly in the areas of initiating and responding to joint attention.

Enhanced D1 And D2 Inhibitions Induced By Low-frequency Trains Of Conditioning Stimuli: Differential Effects On H- And T-reflexes And Possible Mechanisms.

Mechanically evoked reflexes have been postulated to be less sensitive to presynaptic inhibition (PSI) than the H-reflex. This has implications on investigations of spinal cord neurophysiology that are based on the T-reflex. Preceding studies have shown an enhanced effect of PSI on the H-reflex when a train of ~10 conditioning stimuli at 1 Hz was applied to the nerve of the antagonist muscle. The main questions to be addressed in the present study are if indeed T-reflexes are less sensitive to PSI and whether (and to what extent and by what possible mechanisms) the effect of low frequency conditioning, found previously for the H-reflex, can be reproduced on T-reflexes from the soleus muscle. We explored two different conditioning-to-test (C-T) intervals: 15 and 100 ms (corresponding to D1 and D2 inhibitions, respectively). Test stimuli consisted of either electrical pulses applied to the posterior tibial nerve to elicit H-reflexes or mechanical percussion to the Achilles tendon to elicit T-reflexes. The 1 Hz train of conditioning electrical stimuli delivered to the common peroneal nerve induced a stronger effect of PSI as compared to a single conditioning pulse, for both reflexes (T and H), regardless of C-T-intervals. Moreover, the conditioning train of pulses (with respect to a single conditioning pulse) was proportionally more effective for T-reflexes as compared to H-reflexes (irrespective of the C-T interval), which might be associated with the differential contingent of Ia afferents activated by mechanical and electrical test stimuli. A conceivable explanation for the enhanced PSI effect in response to a train of stimuli is the occurrence of homosynaptic depression at synapses on inhibitory interneurons interposed within the PSI pathway. The present results add to the discussion of the sensitivity of the stretch reflex pathway to PSI and its functional role.

Comparison among four commonly used demineralizing agents for root conditioning: a scanning electron microscopy

Dental roots that have been exposed to the oral cavity and periodontal pocket environment present superficial changes, which can prevent connective tissue reattachment. Demineralizing agents have been used as an adjunct to the periodontal treatment aiming at restoring the biocompatibility of roots. OBJECTIVE: This study compared four commonly used demineralizing agents for their capacity of removing smear layer and opening dentin tubules. METHODS: Fifty fragments of human dental roots previously exposed to periodontal disease were scaled and randomly divided into the following groups of treatment: 1) CA: demineralization with citric acid for 3 min; 2) TC-HCl: demineralization with tetracycline-HCl for 3 min; 3) EDTA: demineralization with EDTA for 3 min; 4) PA: demineralization with 37% phosphoric acid for 3 min; 5) Control: rubbing of saline solution for 3 min. Scanning electron microscopy was used to check for the presence of residual smear layer and for measuring the number and area of exposed dentin tubules. RESULTS: Smear layer was present in 100% of the specimens from the groups PA and control; in 80% from EDTA group; in 33.3% from TC-HCl group and 0% from CA group. The mean numbers of exposed dentin tubules in a standardized area were: TC-HCl=43.8±25.2; CA=39.3±37; PA=12.1±16.3; EDTA=4.4±7.5 and Control=2.3±5.7. The comparison showed significant differences between the following pairs of groups: TC-HCl and Control; TC-HCl and EDTA; CA and Control; and CA and EDTA. The mean percentages of area occupied by exposed dentin tubules were: CA=0.12±0.17%; TC-HCl=0.08±0.06%; PA=0.03±0.05%; EDTA=0.01±0.01% and Control=0±0%. The CA group differed significantly from the others except for the TC-HCl group. CONCLUSION: There was a decreasing ability for smear layer removal and dentin tubule widening as follows: AC>TC-HCl>PA>EDTA. This information can be of value as an extra parameter for choosing one of them for root conditioning.

Operant measurement of auditory threshold in prelingually deaf users of cochlear implants: II

Two experiments evaluated an operant procedure for establishing stimulus control using auditory and electrical stimuli as a baseline for measuring the electrical current threshold of electrodes implanted in the cochlea. Twenty-one prelingually deaf children, users of cochlear implants, learned a Go/No Go auditory discrimination task (i.e., pressing a button in the presence of the stimulus but not in its absence). When the simple discrimination baseline became stable, the electrical current was manipulated in descending and ascending series according to an adapted staircase method. Thresholds were determined for three electrodes, one in each location in the cochlea (basal, medial, and apical). Stimulus control was maintained within a certain range of decreasing electrical current but was eventually disrupted. Increasing the current recovered stimulus control, thus allowing the determination of a range of electrical currents that could be defined as the threshold. The present study demonstrated the feasibility of the operant procedure combined with a psychophysical method for threshold assessment, thus contributing to the routine fitting and maintenance of cochlear implants within the limitations of a hospital setting.

RWMODEL II: Computer simulation of the Rescorla-Wagner model of Pavlovian conditioning

RWMODEL II simulates the Rescorla-Wagner model of Pavlovian conditioning. It is written in Delphi and runs under Windows 3.1 and Windows 95. The program was designed for novice and expert users and can be employed in teaching, as well as in research. It is user friendly and requires a minimal level of computer literacy but is sufficiently flexible to permit a wide range of simulations. It allows the display of empirical data, against which predictions from the model can be validated.