Spatio-temporal credit assignment in population learning

Learning by reinforcement is important in shaping animal behavior. But behavioral decision making is likely to involve the integration of many synaptic events in space and time. So in using a single reinforcement signal to modulate synaptic plasticity a twofold problem arises. Different synapses will have contributed differently to the behavioral decision and, even for one and the same synapse, releases at different times may have had different effects. Here we present a plasticity rule which solves this spatio-temporal credit assignment problem in a population of spiking neurons. The learning rule is spike time dependent and maximizes the expected reward by following its stochastic gradient. Synaptic plasticity is modulated not only by the reward but by a population feedback signal as well. While this additional signal solves the spatial component of the problem, the temporal one is solved by means of synaptic eligibility traces. In contrast to temporal difference based approaches to reinforcement learning, our rule is explicit with regard to the assumed biophysical mechanisms. Neurotransmitter concentrations determine plasticity and learning occurs fully online. Further, it works even if the task to be learned is non-Markovian, i.e. when reinforcement is not determined by the current state of the system but may also depend on past events. The performance of the model is assessed by studying three non-Markovian tasks. In the first task the reward is delayed beyond the last action with non-related stimuli and actions appearing in between. The second one involves an action sequence which is itself extended in time and reward is only delivered at the last action, as is the case in any type of board-game. The third is the inspection game that has been studied in neuroeconomics. It only has a mixed Nash equilibrium and exemplifies that the model also copes with stochastic reward delivery and the learning of mixed strategies.

Container Terminal Simulation. A case study: port of Valencia.

This paper is concerned with the study of non-Markovian queuing systems in container terminals. The methodology presented has been applied to analyze the ship traffic in the port of Valencia located in the Western Mediterranean. Two container terminals have been studied: the public container terminal of NOATUM and the dedicated container terminal of MSC. This paper contains the results of a simulation model based on queuing theory. The methodology presented is found to be effective in replicating realistic ship traffic operations in port as well as in conducting capacity evaluations. Thus the methodology can be used for capacity planning (long term), tactical planning (medium term) and even for the container terminal design (port enlargement purposes).

Memory landscapes of single-enzyme molecules

Immobilized single horseradish peroxidase enzymes were observed by confocal fluorescence spectroscopy during catalysis of the oxidation reaction of the nonfluorescent dihydrorhodamine 6G substrate into the highly fluorescent product rhodamine 6G. By extracting only the non-Markovian behavior of the spectroscopic two-state process of enzyme-product complex formation and release, memory landscapes were generated for single-enzyme molecules. The memory landscapes can be used to discriminate between different origins of stretched exponential kinetics that are found in the first-order correlation analysis. Memory landscapes of single-enzyme data shows oscillations that are expected in a single-enzyme system that possesses a set of transient states. Alternative origins of the oscillations may not, however, be ruled out. The data and analysis indicate that substrate interaction with the enzyme selects a set of conformational substates for which the enzyme is active.

Scaling of decoherence effects in quantum computers

The scaling of decoherence rates with qubit number N is studied for a simple model of a quantum computer in the situation where N is large. The two state qubits are localized around well-separated positions via trapping potentials and vibrational centre of mass motion of the qubits occurs. Coherent one and two qubit gating processes are controlled by external classical fields and facilitated by a cavity mode ancilla. Decoherence due to qubit coupling to a bath of spontaneous modes, cavity decay modes and to the vibrational modes is treated. A non-Markovian treatment of the short time behaviour of the fidelity is presented, and expressions for the characteristic decoherence time scales obtained for the case where the qubit/cavity mode ancilla is in a pure state and the baths are in thermal states. Specific results are given for the case where the cavity mode is in the vacuum state and gating processes are absent and the qubits are in (a) the Hadamard state (b) the GHZ state.

Dynamics and topographic organization of recursive self-organizing maps

Recently there has been an outburst of interest in extending topographic maps of vectorial data to more general data structures, such as sequences or trees. However, there is no general consensus as to how best to process sequences using topographicmaps, and this topic remains an active focus of neurocomputational research. The representational capabilities and internal representations of the models are not well understood. Here, we rigorously analyze a generalization of the self-organizingmap (SOM) for processing sequential data, recursive SOM (RecSOM) (Voegtlin, 2002), as a nonautonomous dynamical system consisting of a set of fixed input maps. We argue that contractive fixed-input maps are likely to produce Markovian organizations of receptive fields on the RecSOM map. We derive bounds on parameter β (weighting the importance of importing past information when processing sequences) under which contractiveness of the fixed-input maps is guaranteed. Some generalizations of SOM contain a dynamic module responsible for processing temporal contexts as an integral part of the model. We show that Markovian topographic maps of sequential data can be produced using a simple fixed (nonadaptable) dynamic module externally feeding a standard topographic model designed to process static vectorial data of fixed dimensionality (e.g., SOM). However, by allowing trainable feedback connections, one can obtain Markovian maps with superior memory depth and topography preservation. We elaborate on the importance of non-Markovian organizations in topographic maps of sequential data. © 2006 Massachusetts Institute of Technology.

Temporal correlations of local network losses

We introduce a continuum model describing data losses in a single node of a packet-switched network (like the Internet) which preserves the discrete nature of the data loss process. By construction, the model has critical behavior with a sharp transition from exponentially small to finite losses with increasing data arrival rate. We show that such a model exhibits strong fluctuations in the loss rate at the critical point and non-Markovian power-law correlations in time, in spite of the Markovian character of the data arrival process. The continuum model allows for rather general incoming data packet distributions and can be naturally generalized to consider the buffer server idleness statistics.

Random walks in local dynamics of network losses

We suggest a model for data losses in a single node (memory buffer) of a packet-switched network (like the Internet) which reduces to one-dimensional discrete random walks with unusual boundary conditions. By construction, the model has critical behavior with a sharp transition from exponentially small to finite losses with increasing data arrival rate. We show that for a finite-capacity buffer at the critical point the loss rate exhibits strong fluctuations and non-Markovian power-law correlations in time, in spite of the Markovian character of the data arrival process.

O paradoxo da superdifusão de uma caminhada aleatória com memória exponencial

The random walk models with temporal correlation (i.e. memory) are of interest in the study of anomalous diffusion phenomena. The random walk and its generalizations are of prominent place in the characterization of various physical, chemical and biological phenomena. The temporal correlation is an essential feature in anomalous diffusion models. These temporal long-range correlation models can be called non-Markovian models, otherwise, the short-range time correlation counterparts are Markovian ones. Within this context, we reviewed the existing models with temporal correlation, i.e. entire memory, the elephant walk model, or partial memory, alzheimer walk model and walk model with a gaussian memory with profile. It is noticed that these models shows superdiffusion with a Hurst exponent H > 1/2. We study in this work a superdiffusive random walk model with exponentially decaying memory. This seems to be a self-contradictory statement, since it is well known that random walks with exponentially decaying temporal correlations can be approximated arbitrarily well by Markov processes and that central limit theorems prohibit superdiffusion for Markovian walks with finite variance of step sizes. The solution to the apparent paradox is that the model is genuinely non-Markovian, due to a time-dependent decay constant associated with the exponential behavior. In the end, we discuss ideas for future investigations.

Strong quantum memory at resonant Fermi edges revealed by shot noise

Studies of non-equilibrium current fluctuations enable assessing correlations involved in quantum transport through nanoscale conductors. They provide additional information to the mean current on charge statistics and the presence of coherence, dissipation, disorder, or entanglement. Shot noise, being a temporal integral of the current autocorrelation function, reveals dynamical information. In particular, it detects presence of non-Markovian dynamics, i.e., memory, within open systems, which has been subject of many current theoretical studies. We report on low-temperature shot noise measurements of electronic transport through InAs quantum dots in the Fermi-edge singularity regime and show that it exhibits strong memory effects caused by quantum correlations between the dot and fermionic reservoirs. Our work, apart from addressing noise in archetypical strongly correlated system of prime interest, discloses generic quantum dynamical mechanism occurring at interacting resonant Fermi edges.

On the Existence and Characterization of Markovian Equilibrium in Models with Simple Non-Paternalistic Altruism

This paper provides sufficient conditions for existence of Markovian equilibrium in models with non-paternalistic altruism extending to one generation ahead. When utility is non-separable, we show that each equilibrium savings policy correspondence is increasing everywhere and single-valued, except perhaps on a countable number of points. It is also upper hemi-continuous where it is single valued. When utility is separable, we show that the equilibrium is unique, increasing, and continuous, and we provide an algorithm converging uniformly to the equilibrium.

Markovian Processes, Two-Sided Autoregressions and Finite-Sample Inference for Stationary and Nonstationary Autoregressive Processes.

In this paper, we develop finite-sample inference procedures for stationary and nonstationary autoregressive (AR) models. The method is based on special properties of Markov processes and a split-sample technique. The results on Markovian processes (intercalary independence and truncation) only require the existence of conditional densities. They are proved for possibly nonstationary and/or non-Gaussian multivariate Markov processes. In the context of a linear regression model with AR(1) errors, we show how these results can be used to simplify the distributional properties of the model by conditioning a subset of the data on the remaining observations. This transformation leads to a new model which has the form of a two-sided autoregression to which standard classical linear regression inference techniques can be applied. We show how to derive tests and confidence sets for the mean and/or autoregressive parameters of the model. We also develop a test on the order of an autoregression. We show that a combination of subsample-based inferences can improve the performance of the procedure. An application to U.S. domestic investment data illustrates the method.

Segmentation non-supervisée d'images couleur par sur-segmentation Markovienne en régions et procédure de regroupement de régions par graphes pondérés

Mémoire numérisé par la Division de la gestion de documents et des archives de l'Université de Montréal

Semiclassical evolution of dissipative Markovian systems

A semiclassical approximation for an evolving density operator, driven by a `closed` Hamiltonian operator and `open` Markovian Lindblad operators, is obtained. The theory is based on the chord function, i.e. the Fourier transform of the Wigner function. It reduces to an exact solution of the Lindblad master equation if the Hamiltonian operator is a quadratic function and the Lindblad operators are linear functions of positions and momenta. Initially, the semiclassical formulae for the case of Hermitian Lindblad operators are reinterpreted in terms of a (real) double phase space, generated by an appropriate classical double Hamiltonian. An extra `open` term is added to the double Hamiltonian by the non-Hermitian part of the Lindblad operators in the general case of dissipative Markovian evolution. The particular case of generic Hamiltonian operators, but linear dissipative Lindblad operators, is studied in more detail. A Liouville-type equivariance still holds for the corresponding classical evolution in double phase space, but the centre subspace, which supports the Wigner function, is compressed, along with expansion of its conjugate subspace, which supports the chord function. Decoherence narrows the relevant region of double phase space to the neighbourhood of a caustic for both the Wigner function and the chord function. This difficulty is avoided by a propagator in a mixed representation, so that a further `small-chord` approximation leads to a simple generalization of the quadratic theory for evolving Wigner functions.

Aging and stationary properties of non-equilibrium symmetrical three-state models

We consider a non-equilibrium three-state model whose dynamics is Markovian and displays the same symmetry as the three-state Potts model, i.e. the transition rates are invariant under the cyclic permutation of the states. Unlike the Potts model, detailed balance is, in general, not satisfied. The aging and the stationary properties of the model defined on a square lattice are obtained by means of large-scale Monte Carlo simulations. We show that the phase diagram presents a critical line, belonging to the three-state Potts universality class, that ends at a point whose universality class is that of the Voter model. Aging is considered on the critical line, at the Voter point and in the ferromagnetic phase.

QoS management in smart grids: a Markovian approach

In Smart Grids, a variety of new applications are available to users of the electrical system (from consumers to the electric system operators and market operators). Some applications such as the SCADA systems, which control generators or substations, have consequences, for example, with a communication delay. The result of a failure to deliver a control message due to noncompliance of the time constraint can be catastrophic. On the other hand, applications such as smart metering of consumption have fewer restrictions. Since each type of application has different quality of service requirements (importance, delay, and amount of data to transmit) to transmit its messages, the policy to control and share the resources of the data communication network must consider them. In this paper Markov Decision Process Theory is employed to determine optimal policies to explore as much as possible the availability of throughput in order to transmit all kinds of messages, considering the quality of service requirements defined to each kind of message. First a non-preemptive model is formulated and after that a preemptive model is derived. Numerical results are used to compare FIFO, non-preemptive and preemptive policies.