Spike-based decision learning of nash equilibria in two-player games

Humans and animals face decision tasks in an uncertain multi-agent environment where an agent's strategy may change in time due to the co-adaptation of others strategies. The neuronal substrate and the computational algorithms underlying such adaptive decision making, however, is largely unknown. We propose a population coding model of spiking neurons with a policy gradient procedure that successfully acquires optimal strategies for classical game-theoretical tasks. The suggested population reinforcement learning reproduces data from human behavioral experiments for the blackjack and the inspector game. It performs optimally according to a pure (deterministic) and mixed (stochastic) Nash equilibrium, respectively. In contrast, temporal-difference(TD)-learning, covariance-learning, and basic reinforcement learning fail to perform optimally for the stochastic strategy. Spike-based population reinforcement learning, shown to follow the stochastic reward gradient, is therefore a viable candidate to explain automated decision learning of a Nash equilibrium in two-player games.

Predicting spike timing of neocortical pyramidal neurons by simple threshold models

Neurons generate spikes reliably with millisecond precision if driven by a fluctuating current--is it then possible to predict the spike timing knowing the input? We determined parameters of an adapting threshold model using data recorded in vitro from 24 layer 5 pyramidal neurons from rat somatosensory cortex, stimulated intracellularly by a fluctuating current simulating synaptic bombardment in vivo. The model generates output spikes whenever the membrane voltage (a filtered version of the input current) reaches a dynamic threshold. We find that for input currents with large fluctuation amplitude, up to 75% of the spike times can be predicted with a precision of +/-2 ms. Some of the intrinsic neuronal unreliability can be accounted for by a noisy threshold mechanism. Our results suggest that, under random current injection into the soma, (i) neuronal behavior in the subthreshold regime can be well approximated by a simple linear filter; and (ii) most of the nonlinearities are captured by a simple threshold process.

Spine Ca2+ signaling in spike-timing-dependent plasticity

Calcium is a second messenger, which can trigger the modification of synaptic efficacy. We investigated the question of whether a differential rise in postsynaptic Ca2+ ([Ca2+]i) alone is sufficient to account for the induction of long-term potentiation (LTP) and long-term depression (LTD) of EPSPs in the basal dendrites of layer 2/3 pyramidal neurons of the somatosensory cortex. Volume-averaged [Ca2+]i transients were measured in spines of the basal dendritic arbor for spike-timing-dependent plasticity induction protocols. The rise in [Ca2+]i was uncorrelated to the direction of the change in synaptic efficacy, because several pairing protocols evoked similar spine [Ca2+]i transients but resulted in either LTP or LTD. The sequence dependence of near-coincident presynaptic and postsynaptic activity on the direction of changes in synaptic strength suggested that LTP and LTD were induced by two processes, which were controlled separately by postsynaptic [Ca2+]i levels. Activation of voltage-dependent Ca2+ channels before metabotropic glutamate receptors (mGluRs) resulted in the phospholipase C-dependent (PLC-dependent) synthesis of endocannabinoids, which acted as a retrograde messenger to induce LTD. LTP required a large [Ca2+]i transient evoked by NMDA receptor activation. Blocking mGluRs abolished the induction of LTD and uncovered the Ca2+-dependent induction of LTP. We conclude that the volume-averaged peak elevation of [Ca2+]i in spines of layer 2/3 pyramids determines the magnitude of long-term changes in synaptic efficacy. The direction of the change is controlled, however, via a mGluR-coupled signaling cascade. mGluRs act in conjunction with PLC as sequence-sensitive coincidence detectors when postsynaptic precede presynaptic action potentials to induce LTD. Thus presumably two different Ca2+ sensors in spines control the induction of spike-timing-dependent synaptic plasticity.

Learning real-world stimuli in a neural network with spike-driven synaptic dynamics

We present a model of spike-driven synaptic plasticity inspired by experimental observations and motivated by the desire to build an electronic hardware device that can learn to classify complex stimuli in a semisupervised fashion. During training, patterns of activity are sequentially imposed on the input neurons, and an additional instructor signal drives the output neurons toward the desired activity. The network is made of integrate-and-fire neurons with constant leak and a floor. The synapses are bistable, and they are modified by the arrival of presynaptic spikes. The sign of the change is determined by both the depolarization and the state of a variable that integrates the postsynaptic action potentials. Following the training phase, the instructor signal is removed, and the output neurons are driven purely by the activity of the input neurons weighted by the plastic synapses. In the absence of stimulation, the synapses preserve their internal state indefinitely. Memories are also very robust to the disruptive action of spontaneous activity. A network of 2000 input neurons is shown to be able to classify correctly a large number (thousands) of highly overlapping patterns (300 classes of preprocessed Latex characters, 30 patterns per class, and a subset of the NIST characters data set) and to generalize with performances that are better than or comparable to those of artificial neural networks. Finally we show that the synaptic dynamics is compatible with many of the experimental observations on the induction of long-term modifications (spike-timing-dependent plasticity and its dependence on both the postsynaptic depolarization and the frequency of pre- and postsynaptic neurons).

Spindle burst like spike discharge oscillations in organotypic cultures of neonatal rat cortex

Early network oscillations and spindle bursts are typical patterns of spontaneous rhythmic activity in cortical networks of neonatal rodents in vivo and in vitro. The latter can also be triggered in vivo by stimulation of afferent inputs. The mechanisms underlying such oscillations undergo profound developmental changes in the first postnatal weeks. Their possible role in cortical development is postulated but not known in detail. We have studied spontaneous and evoked patterns of activity in organotypic cultures of slices from neonatal rat cortex grown on multielectrode arrays (MEAs) for extracellular single- and multi-unit recording. Episodes of spontaneous spike discharge oscillations at 7 - 25 Hz lasting for 0.6 - 3 seconds appeared in about half of these cultures spontaneously and could be triggered by electrical stimulation of few distinct electrodes. These oscillations usually covered only restricted areas of the slices. Besides oscillations, single population bursts that spread in a wavelike manner over the whole slice also appeared spontaneously and were triggered by electrical stimulation. In most but not all cultures, population bursts preceded the oscillations. Both population bursts and spike discharge oscillations required intact glutamatergic synaptic transmission since they were suppressed by the AMPA/kainate glutamate receptor antagonist CNQX. The NMDA antagonist d-APV suppressed the oscillations but not the population bursts, suggesting an involvement of NMDA receptors in the oscillations. These findings show that spindle burst like cortical rhythms are reproduced in organotypic cultures of neonatal cortex. The culture model thus allows investigating the role of such rhythms in cortical circuit formation. Supported by SNF grant No. 3100A0-107641/1.

Spike timing dependent plasticity: a consequence of more fundamental learning rules.

Spike timing dependent plasticity (STDP) is a phenomenon in which the precise timing of spikes affects the sign and magnitude of changes in synaptic strength. STDP is often interpreted as the comprehensive learning rule for a synapse - the "first law" of synaptic plasticity. This interpretation is made explicit in theoretical models in which the total plasticity produced by complex spike patterns results from a superposition of the effects of all spike pairs. Although such models are appealing for their simplicity, they can fail dramatically. For example, the measured single-spike learning rule between hippocampal CA3 and CA1 pyramidal neurons does not predict the existence of long-term potentiation one of the best-known forms of synaptic plasticity. Layers of complexity have been added to the basic STDP model to repair predictive failures, but they have been outstripped by experimental data. We propose an alternate first law: neural activity triggers changes in key biochemical intermediates, which act as a more direct trigger of plasticity mechanisms. One particularly successful model uses intracellular calcium as the intermediate and can account for many observed properties of bidirectional plasticity. In this formulation, STDP is not itself the basis for explaining other forms of plasticity, but is instead a consequence of changes in the biochemical intermediate, calcium. Eventually a mechanism-based framework for learning rules should include other messengers, discrete change at individual synapses, spread of plasticity among neighboring synapses, and priming of hidden processes that change a synapse's susceptibility to future change. Mechanism-based models provide a rich framework for the computational representation of synaptic plasticity.

Coreleased orexin and glutamate evoke nonredundant spike outputs and computations in histamine neurons.

Stable wakefulness requires orexin/hypocretin neurons (OHNs) and OHR2 receptors. OHNs sense diverse environmental cues and control arousal accordingly. For unknown reasons, OHNs contain multiple excitatory transmitters, including OH peptides and glutamate. To analyze their cotransmission within computational frameworks for control, we optogenetically stimulated OHNs and examined resulting outputs (spike patterns) in a downstream arousal regulator, the histamine neurons (HANs). OHR2s were essential for sustained HAN outputs. OHR2-dependent HAN output increased linearly during constant OHN input, suggesting that the OHN→HAN(OHR2) module may function as an integral controller. OHN stimulation evoked OHR2-dependent slow postsynaptic currents, similar to midnanomolar OH concentrations. Conversely, glutamate-dependent output transiently communicated OHN input onset, peaking rapidly then decaying alongside OHN→HAN glutamate currents. Blocking glutamate-driven spiking did not affect OH-driven spiking and vice versa, suggesting isolation (low cross-modulation) of outputs. Therefore, in arousal regulators, cotransmitters may translate distinct features of OHN activity into parallel, nonredundant control signals for downstream effectors.

Decision making in beach volleyball defense: Crucial factors derived from interviews with top-level experts

Objectives: In fast ball sports like beach volleyball, decision-making skills are a determining factor for excellent performance. The current investigation aimed to identify factors that influence the decisionmaking process in top-level beach volleyball defense in order to find relevant aspects for further research. For this reason, focused interviews with top players in international beach volleyball were conducted and analyzed with respect to decision-making characteristics. Design: Nineteen world-tour beach volleyball defense players, including seven Olympic or world champions, were interviewed, focusing on decision-making factors, gaze behavior, and interactions between the two. Methods: Verbal data were analyzed by inductive content analysis according to Mayring (2008). This approach allows categories to emerge from the interview material itself instead of forcing data into preset classifications and theoretical concepts. Results: The data analysis showed that, for top-level beach volleyball defense, decision making depends on opponent specifics, external context, situational context, opponent's movements, and intuition. Information on gaze patterns and visual cues revealed general tendencies indicating optimal gaze strategies that support excellent decision making. Furthermore, the analysis highlighted interactions between gaze behavior, visual information, and domain-specific knowledge. Conclusions: The present findings provide information on visual perception, domain-specific knowledge, and interactions between the two that are relevant for decision making in top-level beach volleyball defense. The results can be used to inform sports practice and to further untangle relevant mechanisms underlying decision making in complex game situations.