Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

The elusive S2 state, the S1/S2 splitting, and the excimer states of the benzene dimer

We observe the weak S 0 → S 2 transitions of the T-shaped benzene dimers (Bz)2 and (Bz-d 6)2 about 250 cm−1 and 220 cm−1 above their respective S 0 → S 1 electronic origins using two-color resonant two-photon ionization spectroscopy. Spin-component scaled (SCS) second-order approximate coupled-cluster (CC2) calculations predict that for the tipped T-shaped geometry, the S 0 → S 2 electronic oscillator strength f el (S 2) is ∼10 times smaller than f el (S 1) and the S 2 state lies ∼240 cm−1 above S 1, in excellent agreement with experiment. The S 0 → S 1 (ππ ∗) transition is mainly localized on the “stem” benzene, with a minor stem → cap charge-transfer contribution; the S 0 → S 2 transition is mainly localized on the “cap” benzene. The orbitals, electronic oscillator strengths f el (S 1) and f el (S 2), and transition frequencies depend strongly on the tipping angle ω between the two Bz moieties. The SCS-CC2 calculated S 1 and S 2 excitation energies at different T-shaped, stacked-parallel and parallel-displaced stationary points of the (Bz)2 ground-state surface allow to construct approximate S 1 and S 2 potential energy surfaces and reveal their relation to the “excimer” states at the stacked-parallel geometry. The f el (S 1) and f el (S 2) transition dipole moments at the C 2v -symmetric T-shape, parallel-displaced and stacked-parallel geometries are either zero or ∼10 times smaller than at the tipped T-shaped geometry. This unusual property of the S 0 → S 1 and S 0 → S 2 transition-dipole moment surfaces of (Bz)2 restricts its observation by electronic spectroscopy to the tipped and tilted T-shaped geometries; the other ground-state geometries are impossible or extremely difficult to observe. The S 0 → S 1/S 2 spectra of (Bz)2 are compared to those of imidazole ⋅ (Bz)2, which has a rigid triangular structure with a tilted (Bz)2 subunit. The S 0 → S 1/ S 2 transitions of imidazole-(benzene)2 lie at similar energies as those of (Bz)2, confirming our assignment of the (Bz)2 S 0 → S 2 transition.