Spatio-temporal credit assignment in population learning

We present a model for plasticity induction in reinforcement learning which is based on a cascade of synaptic memory traces. In the cascade of these so called eligibility traces presynaptic input is first corre lated with postsynaptic events, next with the behavioral decisions and finally with the external reinforcement. A population of leaky integrate and fire neurons endowed with this plasticity scheme is studied by simulation on different tasks. For operant co nditioning with delayed reinforcement, learning succeeds even when the delay is so large that the delivered reward reflects the appropriateness, not of the immediately preceeding response, but of a decision made earlier on in the stimulus - decision sequence . So the proposed model does not rely on the temporal contiguity between decision and pertinent reward and thus provides a viable means of addressing the temporal credit assignment problem. In the same task, learning speeds up with increasing population si ze, showing that the plasticity cascade simultaneously addresses the spatial problem of assigning credit to the different population neurons. Simulations on other task such as sequential decision making serve to highlight the robustness of the proposed sch eme and, further, contrast its performance to that of temporal difference based approaches to reinforcement learning.

Spatio-Temporal Credit Assignment in Neuronal Population Learning

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Spike-based decision learning of nash equilibria in two-player games

Humans and animals face decision tasks in an uncertain multi-agent environment where an agent's strategy may change in time due to the co-adaptation of others strategies. The neuronal substrate and the computational algorithms underlying such adaptive decision making, however, is largely unknown. We propose a population coding model of spiking neurons with a policy gradient procedure that successfully acquires optimal strategies for classical game-theoretical tasks. The suggested population reinforcement learning reproduces data from human behavioral experiments for the blackjack and the inspector game. It performs optimally according to a pure (deterministic) and mixed (stochastic) Nash equilibrium, respectively. In contrast, temporal-difference(TD)-learning, covariance-learning, and basic reinforcement learning fail to perform optimally for the stochastic strategy. Spike-based population reinforcement learning, shown to follow the stochastic reward gradient, is therefore a viable candidate to explain automated decision learning of a Nash equilibrium in two-player games.

Incidental task sequence learning: perceptual rather than conceptual?

In four experiments we investigated whether incidental task sequence learning occurs when no instructional task cues are available (i.e. with univalent stimuli). We manipulated task sequence by presenting three simple binary-choice tasks (colour, form or letter case decisions) in regular repeated or random order. Participants were required to use the same two response keys for each of the tasks. We manipulated response sequence by ordering the stimuli so as to produce either a regular or a random order of left versus right-hand key presses. When sequencing in both, or either, separate stream (i.e. task sequence and/or response sequence) was changed to random, only those participants who had processed both sequences together showed evidence of sequence learning in terms of significant response time disruption (Experiments 1-3). This effect disappeared when the sequences were uncorrelated (Experiment 4). The results indicate that only the correlated integration of task sequence and response sequence produced a reliable incidental learning effect. As this effect depends on the predictable ordering of stimulus categories, it suggests that task sequence learning is perceptual rather than conceptual in nature.

Kostenorientiertes Controlling von E-Learning-Plattformen mit dem TCO-Konzept

Zur administrativen Unterstützung von Lehr- und Lernprozessen werden E-Learning-Plattformen eingesetzt, die auf der Grundlage des Internet Funktionen zur Distribution von Lehr- und Lernmaterialien und zur Kommunikation zwischen Lehrenden und Lernenden anbieten. Zahlreiche wissenschaftliche Beiträge und Marktstudien beschäftigen sich mit der multikriteriellen Evaluation dieser Softwareprodukte zur informatorischen Fundierung strategischer Investitionsentscheidungen. Demgegenüber werden Instrumente zum kostenorientierten Controlling von E-Learning-Plattformen allenfalls marginal thematisiert. Dieser Beitrag greift daher das Konzept der Total Cost of Ownership (TCO) auf, das einen methodischen Ansatzpunkt zur Schaffung von Kostentransparenz von E-Learning-Plattformen bildet. Aufbauend auf den konzeptionellen Grundlagen werden Problembereiche und Anwendungspotenziale für das kostenorientierte Controlling von LMS identifiziert. Zur softwaregestützten Konstruktion und Analyse von TCO-Modellen wird das Open Source-Werkzeug TCO-Tool eingeführt und seine Anwendung anhand eines synthetischen Fallbeispiels erörtert. Abschließend erfolgt die Identifikation weiterführender Entwicklungsperspektiven des TCO-Konzepts im Kontext des E-Learning. Die dargestellte Thematik ist nicht nur von theoretischem Interesse, sondern adressiert auch den steigenden Bedarf von Akteuren aus der Bildungspraxis nach Instrumenten zur informatorischen Fundierung von Investitions- und Desinvestitionsentscheidungen im Umfeld des E-Learning.

E-Learning 2.0 in hydraulic engineering education

In this contribution the experiences with e-Learning 2.0 applications by using a Wiki for the education in hydraulic engineering are shown. Up to now important information for the students has been prepared by the instructor. For this project the students were asked to collaborate and search on their own for the information they needed. Therefore a Wiki-system was used. For the engineering practice a self dependent realisation of tasks is an important requirement which students should be prepared for. With the help of online communication there should be shown the possibilities for students for working together in an interdisciplinary team. The positive experiences as well as the results of the evaluation of this project plead for a continuation of the application of e-Learning 2.0 for education. The comparison of results of tests without using Wiki and with using Wiki shows a qualitative tendency of better marks. In this contribution we present the application of Wiki in hydraulic engineering but the results can also be used for other engineering disciplines.

ProFormA: An XML-based exchange format for programming tasks

Unterstützungssysteme für die Programmierausbildung sind weit verbreitet, doch gängige Standards für den Austausch von allgemeinen (Lern-) Inhalten und Tests erfüllen nicht die speziellen Anforderungen von Programmieraufgaben wie z. B. den Umgang mit komplexen Einreichungen aus mehreren Dateien oder die Kombination verschiedener (automatischer) Bewertungsverfahren. Dadurch können Aufgaben nicht zwischen Systemen ausgetauscht werden, was aufgrund des hohen Aufwands für die Entwicklung guter Aufgaben jedoch wünschenswert wäre. In diesem Beitrag wird ein erweiterbares XML-basiertes Format zum Austausch von Programmieraufgaben vorgestellt, das bereits von mehreren Systemen prototypisch genutzt wird. Die Spezifikation des Austauschformats ist online verfügbar [PFMA].

Learning in Virtual Worlds: Research and Applications

Three-dimensional (3D) immersive virtual worlds have been touted as being capable of facilitating highly interactive, engaging, multimodal learning experiences. Much of the evidence gathered to support these claims has been anecdotal but the potential that these environments hold to solve traditional problems in online and technology-mediated education—primarily learner isolation and student disengagement—has resulted in considerable investments in virtual world platforms like Second Life, OpenSimulator, and Open Wonderland by both professors and institutions. To justify this ongoing and sustained investment, institutions and proponents of simulated learning environments must assemble a robust body of evidence that illustrates the most effective use of this powerful learning tool. In this authoritative collection, a team of international experts outline the emerging trends and developments in the use of 3D virtual worlds for teaching and learning. They explore aspects of learner interaction with virtual worlds, such as user wayfinding in Second Life, communication modes and perceived presence, and accessibility issues for elderly or disabled learners. They also examine advanced technologies that hold potential for the enhancement of learner immersion and discuss best practices in the design and implementation of virtual world-based learning interventions and tasks. By evaluating and documenting different methods, approaches, and strategies, the contributors to Learning in Virtual Worlds offer important information and insight to both scholars and practitioners in the field. AU Press is an open access publisher and the book is available for free in PDF format as well as for purchase on our website: http://bit.ly/1W4yTRA

Deleterious effects of roving on learned tasks

In typical perceptual learning experiments, one stimulus type (e.g., a bisection stimulus offset either to the left or right) is presented per trial. In roving, two different stimulus types (e.g., a 30′ and a 20′ wide bisection stimulus) are randomly interleaved from trial to trial. Roving can impair both perceptual learning and task sensitivity. Here, we investigate the relationship between the two. Using a bisection task, we found no effect of roving before training. We next trained subjects and they improved. A roving condition applied after training impaired sensitivity.

Code-specific learning rules improve action selection by populations of spiking neurons

Population coding is widely regarded as a key mechanism for achieving reliable behavioral decisions. We previously introduced reinforcement learning for population-based decision making by spiking neurons. Here we generalize population reinforcement learning to spike-based plasticity rules that take account of the postsynaptic neural code. We consider spike/no-spike, spike count and spike latency codes. The multi-valued and continuous-valued features in the postsynaptic code allow for a generalization of binary decision making to multi-valued decision making and continuous-valued action selection. We show that code-specific learning rules speed up learning both for the discrete classification and the continuous regression tasks. The suggested learning rules also speed up with increasing population size as opposed to standard reinforcement learning rules. Continuous action selection is further shown to explain realistic learning speeds in the Morris water maze. Finally, we introduce the concept of action perturbation as opposed to the classical weight- or node-perturbation as an exploration mechanism underlying reinforcement learning. Exploration in the action space greatly increases the speed of learning as compared to exploration in the neuron or weight space.

Implicit task sequence learning in patients with Parkinson's disease, frontal lesions and amnesia: The critical role of fronto–striatal loops

The purpose of this study was to investigate the role of the fronto–striatal system for implicit task sequence learning. We tested performance of patients with compromised functioning of the fronto–striatal loops, that is, patients with Parkinson's disease and patients with lesions in the ventromedial or dorsolateral prefrontal cortex. We also tested amnesic patients with lesions either to the basal forebrain/orbitofrontal cortex or to thalamic/medio-temporal regions. We used a task sequence learning paradigm involving the presentation of a sequence of categorical binary-choice decision tasks. After several blocks of training, the sequence, hidden in the order of tasks, was replaced by a pseudo-random sequence. Learning (i.e., sensitivity to the ordering) was assessed by measuring whether this change disrupted performance. Although all the patients were able to perform the decision tasks quite easily, those with lesions to the fronto–striatal loops (i.e., patients with Parkinson's disease, with lesions in the ventromedial or dorsolateral prefrontal cortex and those amnesic patients with lesions to the basal forebrain/orbitofrontal cortex) did not show any evidence of implicit task sequence learning. In contrast, those amnesic patients with lesions to thalamic/medio-temporal regions showed intact sequence learning. Together, these results indicate that the integrity of the fronto–striatal system is a prerequisite for implicit task sequence learning.

Implicit task sequence learning

Implicit task sequence learning (TSL) can be considered as an extension of implicit sequence learning which is typically tested with the classical serial reaction time task (SRTT). By design, in the SRTT there is a correlation between the sequence of stimuli to which participants must attend and the sequence of motor movements/key presses with which participants must respond. The TSL paradigm allows to disentangle this correlation and to separately manipulate the presences/absence of a sequence of tasks, a sequence of responses, and even other streams of information such as stimulus locations or stimulus-response mappings. Here I review the state of TSL research which seems to point at the critical role of the presence of correlated streams of information in implicit sequence learning. On a more general level, I propose that beyond correlated streams of information, a simple statistical learning mechanism may also be involved in implicit sequence learning, and that the relative contribution of these two explanations differ according to task requirements. With this differentiation, conflicting results can be integrated into a coherent framework.

Improving memory in elderly adults by instructing them in professional actors' learning strategies

This study examined a new type of cognitive intervention. For four weeks, participants (ages 65 to 82) were instructed in professional acting techniques, followed by rehearsal and performance of theatrical scenes. Although the training was not targeted in any way to the tasks used in pre- and post-testing, participants produced significantly higher recall and recognition scores after the intervention. It is suggested that the cognitive effort involved in analyzing and adopting theatrical characters' motivations (and then experiencing those characters' mental/emotional states during performance) is responsible for the observed improvement. A secondary strand of this study showed that participants who were given annotated scripts in which the implied goals of the characters were made explicit demonstrated significantly faster access to the stored material, as measured by a computer latency task.

Statistical Learning, Syllable Processing, and Speech Production in Healthy Hearing and Hearing-Impaired Preschool Children: A Mismatch Negativity Study

OBJECTIVES The objectives of the present study were to investigate temporal/spectral sound-feature processing in preschool children (4 to 7 years old) with peripheral hearing loss compared with age-matched controls. The results verified the presence of statistical learning, which was diminished in children with hearing impairments (HIs), and elucidated possible perceptual mediators of speech production. DESIGN Perception and production of the syllables /ba/, /da/, /ta/, and /na/ were recorded in 13 children with normal hearing and 13 children with HI. Perception was assessed physiologically through event-related potentials (ERPs) recorded by EEG in a multifeature mismatch negativity paradigm and behaviorally through a discrimination task. Temporal and spectral features of the ERPs during speech perception were analyzed, and speech production was quantitatively evaluated using speech motor maximum performance tasks. RESULTS Proximal to stimulus onset, children with HI displayed a difference in map topography, indicating diminished statistical learning. In later ERP components, children with HI exhibited reduced amplitudes in the N2 and early parts of the late disciminative negativity components specifically, which are associated with temporal and spectral control mechanisms. Abnormalities of speech perception were only subtly reflected in speech production, as the lone difference found in speech production studies was a mild delay in regulating speech intensity. CONCLUSIONS In addition to previously reported deficits of sound-feature discriminations, the present study results reflect diminished statistical learning in children with HI, which plays an early and important, but so far neglected, role in phonological processing. Furthermore, the lack of corresponding behavioral abnormalities in speech production implies that impaired perceptual capacities do not necessarily translate into productive deficits.

Cerebellar Pathology Does Not Impair Performance on Identification or Categorization Tasks

In comparison to the basal ganglia, prefrontal cortex, and medial temporal lobes, the cerebellum has been absent from recent research on the neural substrates of categorization and identification, two prominent tasks in the learning and memory literature. To investigate the contribution of the cerebellum to these tasks, we tested patients with cerebellar pathology (seven with bilateral degeneration, six with unilateral lesions, and two with midline damage) on rule-based and information-integration categorization tasks and an identification task. In rule-based tasks, it is assumed that participants learn the categories through an explicit reasoning process. In information-integration tasks, optimal performance requires the integration of information from multiple stimulus dimensions, and participants are typically unaware of the decision strategy. The identification task, in contrast, required participants to learn arbitrary, color-word associations. The cerebellar patients performed similar to matched controls on all three tasks and performance did not vary with the extent of cerebellar pathology. Although the interpretation of these null results requires caution, these data contribute to the current debate on cerebellar contributions to cognition by providing boundary conditions on understanding the neural substrates of categorization and identification, and help define the functional domain of the cerebellum in learning and memory.