Parametrized actor-critic algorithms for finite-horizon MDPs

Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FHMDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Short-Horizon Asymmetric Mean-Reversion and Overreactions: Evidence from the Nordic Stock Markets

This paper examines the asymmetric behavior of conditional mean and variance. Short-horizon mean-reversion behavior in mean is modeled with an asymmetric nonlinear autoregressive model, and the variance is modeled with an Exponential GARCH in Mean model. The results of the empirical investigation of the Nordic stock markets indicates that negative returns revert faster to positive returns when positive returns generally persist longer. Asymmetry in both mean and variance can be seen on all included markets and are fairly similar. Volatility rises following negative returns more than following positive returns which is an indication of overreactions. Negative returns lead to increased variance and positive returns leads even to decreased variance.

Optimal infinite-horizon feedback laws for a general class of constrained discrete-time systems: stability and moving-horizon approximations

Stability results are given for a class of feedback systems arising from the regulation of time-varying discrete-time systems using optimal infinite-horizon and moving-horizon feedback laws. The class is characterized by joint constraints on the state and the control, a general nonlinear cost function and nonlinear equations of motion possessing two special properties. It is shown that weak conditions on the cost function and the constraints are sufficient to guarantee uniform asymptotic stability of both the optimal infinite-horizon and movinghorizon feedback systems. The infinite-horizon cost associated with the moving-horizon feedback law approaches the optimal infinite-horizon cost as the moving horizon is extended.

An Actor-Critic Algorithm for Finite Horizon Markov Decision Processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Parameter estimation of a two-horizon soil profile by combining crop canopy and surface soil moisture observations using GLUE

Estimation of soil parameters by inverse modeling using observations on either surface soil moisture or crop variables has been successfully attempted in many studies, but difficulties to estimate root zone properties arise when heterogeneous layered soils are considered. The objective of this study was to explore the potential of combining observations on surface soil moisture and crop variables - leaf area index (LAI) and above-ground biomass for estimating soil parameters (water holding capacity and soil depth) in a two-layered soil system using inversion of the crop model STICS. This was performed using GLUE method on a synthetic data set on varying soil types and on a data set from a field experiment carried out in two maize plots in South India. The main results were (i) combination of surface soil moisture and above-ground biomass provided consistently good estimates with small uncertainity of soil properties for the two soil layers, for a wide range of soil paramater values, both in the synthetic and the field experiment, (ii) above-ground biomass was found to give relatively better estimates and lower uncertainty than LAI when combined with surface soil moisture, especially for estimation of soil depth, (iii) surface soil moisture data, either alone or combined with crop variables, provided a very good estimate of the water holding capacity of the upper soil layer with very small uncertainty whereas using the surface soil moisture alone gave very poor estimates of the soil properties of the deeper layer, and (iv) using crop variables alone (else above-ground biomass or LAI) provided reasonable estimates of the deeper layer properties depending on the soil type but provided poor estimates of the first layer properties. The robustness of combining observations of the surface soil moisture and the above-ground biomass for estimating two layer soil properties, which was demonstrated using both synthetic and field experiments in this study, needs now to be tested for a broader range of climatic conditions and crop types, to assess its potential for spatial applications. (C) 2012 Elsevier B.V. All rights reserved.

Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

Prioritizing county-level well-being: Moving toward assessment of Gulf Coast counties impacted by the Deepwater Horizon Industrial Disaster

To develop a portfolio of indicators and measures that could best measure changes in the social, economic, environmental and health dimensions of well-being in coastal counties we convened a group of experts March 8-9, 2011 in Charleston, SC, U.S.A. The region of interest was of the northern Gulf of Mexico, specifically, those coastal counties most impacted during the explosion and subsequent oil spill from the Macondo Prospect wellhead during the summer of 2010. Over the course of the two-day workshop participants moved through presentations and facilitated sessions to identify and prioritize potential indicators and measures deemed most valuable for capturing changes in well-being related to changes in or disruption of ecosystem services. The experts reached consensus on a list of indicators that are now being operationalized by NOAA researchers. The ultimate goal of this research project is to determine whether a meaningful set of social and economic indicators can be developed to document changes in well-being that occur as a result of changes in ecosystem services. The outcomes and outputs from the workshop that is the subject of this report helped us to identify high-quality indicators useful for measuring well-being.

Deepwater Horizon oil spill: assessment of potential impacts on the deep soft-bottom benthos; interim data summary report

A study was initiated in May 2011, under the direction of the Deepwater Horizon (DWH) Natural Resource Damage Assessment (NRDA) Deepwater Benthic Communities Technical Working Group (NRDA Deep Benthic TWG), to assess potential impacts of the DWH oil spill on sediments and resident benthic fauna in deepwater (> 200 meters) areas of the Gulf. Key objectives of the study were to complete the analysis of samples from 65 priority stations sampled in September-October 2010 on two DWH Response cruises (Gyre and Ocean Veritas) and from 38 long-term monitoring sites (including a subset of 35 of the original 65) sampled on a follow-up NRDA cruise in May-June 2011. The present progress report provides a brief summary of results from the initial processing of samples from fall 2010 priority sites (plus three additional historical sites). Data on key macrofaunal, meiofaunal, and abiotic environmental variables are presented for each of these samples and additional maps are included to depict spatial patterns in these variables throughout the study region. The near-field zone within about 3 km of the wellhead, where many of the stations showed evidence of impaired benthic condition (e.g. low taxa richness, high nematode/harpacticoid-copepod ratios), also is an area that contained some of the highest concentrations of total petroleum hydrocarbons (TPH), total polycyclic aromatic hydrocarbons (total PAHs), and barium in sediments (as possible indicators of DWH discharges). There were similar co-occurrences at other sites outside this zone, especially to the southwest of the wellhead out to about 15 km. However, there also were exceptions to this pattern, for example at several farther-field sites in deeper-slope and canyon locations where there was low benthic species richness but no evidence of exposure to DWH discharges. Such cases are consistent with historical patterns of benthic distributions in relation to natural controlling factors such as depth, position within canyons, and availability of organic matter derived from surface-water primary production.