Spatio-temporal credit assignment in population learning

We present a model for plasticity induction in reinforcement learning which is based on a cascade of synaptic memory traces. In the cascade of these so called eligibility traces presynaptic input is first corre lated with postsynaptic events, next with the behavioral decisions and finally with the external reinforcement. A population of leaky integrate and fire neurons endowed with this plasticity scheme is studied by simulation on different tasks. For operant co nditioning with delayed reinforcement, learning succeeds even when the delay is so large that the delivered reward reflects the appropriateness, not of the immediately preceeding response, but of a decision made earlier on in the stimulus - decision sequence . So the proposed model does not rely on the temporal contiguity between decision and pertinent reward and thus provides a viable means of addressing the temporal credit assignment problem. In the same task, learning speeds up with increasing population si ze, showing that the plasticity cascade simultaneously addresses the spatial problem of assigning credit to the different population neurons. Simulations on other task such as sequential decision making serve to highlight the robustness of the proposed sch eme and, further, contrast its performance to that of temporal difference based approaches to reinforcement learning.

Spatio-Temporal Credit Assignment in Neuronal Population Learning

n learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Spatio-temporal Associations Between GOES Aerosol Optical Depth Retrievals and Ground-Level PM2.5

We assess the strength of association between aerosol optical depth (AOD) retrievals from the GOES Aerosol/Smoke Product (GASP) and ground-level fine particulate matter (PM2.5) to assess AOD as a proxy for PM2.5 in the United States. GASP AOD is retrieved from a geostationary platform and therefore provides dense temporal coverage with half-hourly observations every day, in contrast to once per day snapshots from polar-orbiting satellites. However, GASP AOD is based on a less-sophisticated instrument and retrieval algorithm. We find that correlations between GASP AOD and PM2.5 over time at fixed locations are reasonably high, except in the winter and in the western U.S. Correlations over space at fixed times are lower. Simple averaging over time actually reduces correlations over space dramatically, but statistical calibration allows averaging over time that produces strong correlations. These results and the data density of GASP AOD highlight its potential to help improve exposure estimates for epidemiological analyses. On average 40% of days in a month have a GASP AOD retrieval compared to 14% for MODIS and 4% for MISR. Furthermore, GASP AOD has been retrieved since November 1994, providing the possibility of a long-term record that pre-dates the availability of most PM2.5 monitoring data and other satellite instruments.

Semiparametric Latent Variable Regression Models for Spatio-temporal Modeling of Mobile Source Particles in the Greater Boston Area

Traffic particle concentrations show considerable spatial variability within a metropolitan area. We consider latent variable semiparametric regression models for modeling the spatial and temporal variability of black carbon and elemental carbon concentrations in the greater Boston area. Measurements of these pollutants, which are markers of traffic particles, were obtained from several individual exposure studies conducted at specific household locations as well as 15 ambient monitoring sites in the city. The models allow for both flexible, nonlinear effects of covariates and for unexplained spatial and temporal variability in exposure. In addition, the different individual exposure studies recorded different surrogates of traffic particles, with some recording only outdoor concentrations of black or elemental carbon, some recording indoor concentrations of black carbon, and others recording both indoor and outdoor concentrations of black carbon. A joint model for outdoor and indoor exposure that specifies a spatially varying latent variable provides greater spatial coverage in the area of interest. We propose a penalised spline formation of the model that relates to generalised kringing of the latent traffic pollution variable and leads to a natural Bayesian Markov Chain Monte Carlo algorithm for model fitting. We propose methods that allow us to control the degress of freedom of the smoother in a Bayesian framework. Finally, we present results from an analysis that applies the model to data from summer and winter separately

Practical Large-Scale Spatio-Temporal Modeling of Particulate Matter Concentrations

The last two decades have seen intense scientific and regulatory interest in the health effects of particulate matter (PM). Influential epidemiological studies that characterize chronic exposure of individuals rely on monitoring data that are sparse in space and time, so they often assign the same exposure to participants in large geographic areas and across time. We estimate monthly PM during 1988-2002 in a large spatial domain for use in studying health effects in the Nurses' Health Study. We develop a conceptually simple spatio-temporal model that uses a rich set of covariates. The model is used to estimate concentrations of PM10 for the full time period and PM2.5 for a subset of the period. For the earlier part of the period, 1988-1998, few PM2.5 monitors were operating, so we develop a simple extension to the model that represents PM2.5 conditionally on PM10 model predictions. In the epidemiological analysis, model predictions of PM10 are more strongly associated with health effects than when using simpler approaches to estimate exposure. Our modeling approach supports the application in estimating both fine-scale and large-scale spatial heterogeneity and capturing space-time interaction through the use of monthly-varying spatial surfaces. At the same time, the model is computationally feasible, implementable with standard software, and readily understandable to the scientific audience. Despite simplifying assumptions, the model has good predictive performance and uncertainty characterization.

Automation of shear-wave splitting parameter determination of local earthquakes at Yellowstone : application as indicator of crustal stress and temporal variation

Shear-wave splitting can be a useful technique for determining crustal stress fields in volcanic settings and temporal variations associated with activity. Splitting parameters were determined for a subset of local earthquakes recorded from 2000-2010 at Yellowstone. Analysis was automated using an unsupervised cluster analysis technique to determine optimum splitting parameters from 270 analysis windows for each event. Six stations clearly exhibit preferential fast polarization values sub-orthogonal to the direction of minimum horizontal compression. Yellowstone deformation results in a local crustal stress field differing from the regional field dominated by NE-SW extension, and fast directions reflect this difference rotating around the caldera maintaining perpendicularity to the rim. One station exhibits temporal variations concordant with identified periods of caldera subsidence and uplift. From splitting measurements, we calculated a crustal anisotropy of ~17-23% and crack density ~0.12-0.17 possibly resulting from stress-aligned fluid filled microcracks in the upper crust and an active hydrothermal system.