Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior, and in particular in behavioral decision making. Such decision making is likely to involve the integration of many synaptic events in space and time. However, using a single reinforcement signal to modulate synaptic plasticity, as suggested in classical reinforcement learning algorithms, a twofold problem arises. Different synapses will have contributed differently to the behavioral decision, and even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike-time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward, but also by a population feedback signal. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference (TD) based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task, the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second task involves an action sequence which is itself extended in time and reward is only delivered at the last action, as it is the case in any type of board-game. The third task is the inspection game that has been studied in neuroeconomics, where an inspector tries to prevent a worker from shirking. Applying our algorithm to this game yields a learning behavior which is consistent with behavioral data from humans and monkeys, revealing themselves properties of a mixed Nash equilibrium. The examples show that our neuronal implementation of reward based learning copes with delayed and stochastic reward delivery, and also with the learning of mixed strategies in two-opponent games.

Possible earthquake trigger for 6th century mass wasting deposit at Lake Ohrid (Macedonia/Albania)

Abstract. Lake Ohrid shared by the Republics of Albania and Macedonia is formed by a tectonically active graben within the south Balkans and suggested to be the oldest lake in Europe. Several studies have shown that the lake provides a valuable record of climatic and environmental changes and a distal tephrostratigraphic record of volcanic eruptions from Italy. Fault structures identified in seismic data demonstrate that sediments have also the potential to record tectonic activity in the region. Here, we provide an example of linking seismic and sedimentological information with tectonic activity and historical documents. Historical documents indicate that a major earthquake destroyed the city of Lychnidus (today: city of Ohrid) in the early 6th century AD. Multichannel seismic profiles, parametric sediment echosounder profiles, and a 10.08m long sediment record from the western part of the lake indicate a 2m thick mass wasting deposit, which is tentatively correlated with this earthquake. The mass wasting deposit is chronologically well constrained, as it directly overlays the AD472/AD 512 tephra. Moreover, radiocarbon dates and cross correlation with other sediment sequences with similar geochemical characteristics of the Holocene indicate that the mass wasting event took place prior to the onset of the Medieval Warm Period, and is attributed it to one of the known earthquakes in the region in the early 6th century AD.

Into the abyss of Lake Geneva: the elemo interdisciplinary field investigation using the MIR submersibles

In summer 2011, the two Russian MIR sub- mersibles were brought to Switzerland to perform deep water dives in Lake Geneva. Research teams from several environmental science institutes, both national and inter- national, participated in this interdisciplinary effort to investigate the deeper parts of Lake Geneva. Using the MIRs allowed the scientists to see and precisely select the sites where they could extract specific sediment cores and carry out detailed in situ measurements at the sediment– water boundary. One focus site was the surrounding of the outlet of the wastewater treatment plant of the City of Lausanne, which discharges into the Vidy Bay. The investigations concentrated on the pollution of the local sediments, pollution-related ecotoxicological risks, micro- bial activity and spreading and removal of the effluents from the bay to the open waters of the lake. The other focus site was the Rhoˆne River delta and its subaquatic canyons, which formed as a result of the long-term interplay of the deposition of river-borne sediments and flood-triggered canyon erosion events.

Human–climate interactions in the central Mediterranean region during the last millennia: The laminated record of Lake Butrint (Albania)

Lake Butrint (39°47 N, 20°1 E) is a ca. 21 m deep, coastal lagoon located in SW Albania where finely-laminated sediments have been continuously deposited during the last millennia. The multi-proxy analysis (sedimentology, high-resolution elemental geochemistry and pollen) of a 12 m long sediment core, supported by seven AMS radiocarbon dates and 137Cs dating, enable a precise reconstruction of the environmental change that occurred in the central Mediterranean region during the last ∼4.5 cal kyrs BP. Sediments consist of triplets of authigenic carbonates, organic matter and clayey laminae. Fluctuations in the thickness and/or presence of these different types of seasonal laminae indicate variations in water salinity, organic productivity and runoff in the lake's catchment, as a result of the complex interplay of tectonics, anthropogenic forcing and climate variability. The progradation of the Pavllo river delta, favoured by variable human activity from the nearby ancient city of Butrint, led to the progressive isolation of this hydrological system from the Ionian Sea. The system evolved from an open bay to a restricted lagoon, which is consistent with archaeological data. An abrupt increase in mass-wasting activity between 1515 and 1450 BC, likely caused by nearby seismic activity, led to the accumulation of 24 homogenites, up to 17 cm thick. They have been deposited during the onset of finely laminated sedimentation, which indicates restricted, anoxic bottom water conditions and higher salinity. Periods of maximum water salinity, biological productivity, and carbonate precipitation coincide with warmer intervals, such as the early Roman Warm Period (RWP) (500 BC–0 AD), the Medieval Climate Anomaly (MCA) (800–1400 AD) and recent times (after 1800 AD). Conversely, lower salinity and more oxic conditions, with higher clastic input were recorded during 1400–500 BC, the Late Roman and the Early Medieval periods (0–800 AD) and during the Little Ice Age (1400–1800 AD). Hydrological fluctuations recorded in Butrint are in phase with most central and western Mediterranean records and correlate with NAO variability. In contrast, opposite hydrological patterns have been recorded in the Eastern Balkans and the Levant during the last millennium, emphasizing a complex spatial variability in the region. Phases of maximum settlement intensity in Butrint (Roman-Late Antique) coincide with warmer and/or stable climate periods (0–800 AD and MCA, respectively), indicating a long-term influence of climatic conditions on human activities. The Late Holocene sedimentary record of Lake Butrint demonstrates the complex interplay of climate variability, tectonics and human impact in the recent evolution of coastal Mediterranean regions.