Optimal infinite-horizon feedback laws for a general class of constrained discrete-time systems: stability and moving-horizon approximations

Stability results are given for a class of feedback systems arising from the regulation of time-varying discrete-time systems using optimal infinite-horizon and moving-horizon feedback laws. The class is characterized by joint constraints on the state and the control, a general nonlinear cost function and nonlinear equations of motion possessing two special properties. It is shown that weak conditions on the cost function and the constraints are sufficient to guarantee uniform asymptotic stability of both the optimal infinite-horizon and movinghorizon feedback systems. The infinite-horizon cost associated with the moving-horizon feedback law approaches the optimal infinite-horizon cost as the moving horizon is extended.

Parameter estimation of a two-horizon soil profile by combining crop canopy and surface soil moisture observations using GLUE

Estimation of soil parameters by inverse modeling using observations on either surface soil moisture or crop variables has been successfully attempted in many studies, but difficulties to estimate root zone properties arise when heterogeneous layered soils are considered. The objective of this study was to explore the potential of combining observations on surface soil moisture and crop variables - leaf area index (LAI) and above-ground biomass for estimating soil parameters (water holding capacity and soil depth) in a two-layered soil system using inversion of the crop model STICS. This was performed using GLUE method on a synthetic data set on varying soil types and on a data set from a field experiment carried out in two maize plots in South India. The main results were (i) combination of surface soil moisture and above-ground biomass provided consistently good estimates with small uncertainity of soil properties for the two soil layers, for a wide range of soil paramater values, both in the synthetic and the field experiment, (ii) above-ground biomass was found to give relatively better estimates and lower uncertainty than LAI when combined with surface soil moisture, especially for estimation of soil depth, (iii) surface soil moisture data, either alone or combined with crop variables, provided a very good estimate of the water holding capacity of the upper soil layer with very small uncertainty whereas using the surface soil moisture alone gave very poor estimates of the soil properties of the deeper layer, and (iv) using crop variables alone (else above-ground biomass or LAI) provided reasonable estimates of the deeper layer properties depending on the soil type but provided poor estimates of the first layer properties. The robustness of combining observations of the surface soil moisture and the above-ground biomass for estimating two layer soil properties, which was demonstrated using both synthetic and field experiments in this study, needs now to be tested for a broader range of climatic conditions and crop types, to assess its potential for spatial applications. (C) 2012 Elsevier B.V. All rights reserved.

Optimal simultaneous maximum a posterioriestimation of states, noise statistics and parameters I. Algorithm

The simultaneous state and parameter estimation problem for a linear discrete-time system with unknown noise statistics is treated as a large-scale optimization problem. The a posterioriprobability density function is maximized directly with respect to the states and parameters subject to the constraint of the system dynamics. The resulting optimization problem is too large for any of the standard non-linear programming techniques and hence an hierarchical optimization approach is proposed. It turns out that the states can be computed at the first levelfor given noise and system parameters. These, in turn, are to be modified at the second level.The states are to be computed from a large system of linear equations and two solution methods are considered for solving these equations, limiting the horizon to a suitable length. The resulting algorithm is a filter-smoother, suitable for off-line as well as on-line state estimation for given noise and system parameters. The second level problem is split up into two, one for modifying the noise statistics and the other for modifying the system parameters. An adaptive relaxation technique is proposed for modifying the noise statistics and a modified Gauss-Newton technique is used to adjust the system parameters.

Two-temperature accretion around rotating black holes: a description of the general advective flow paradigm in the presence of various cooling processes to explain low to high luminous sources

We investigate viscous two-temperature accretion disc flows around rotating black holes. We describe the global solution of accretion flows with a sub-Keplerian angular momentum profile, by solving the underlying conservation equations including explicit cooling processes self-consistently. Bremsstrahlung, synchrotron and inverse Comptonization of soft photons are considered as possible cooling mechanisms. We focus on the set of solutions for sub-Eddington, Eddington and super-Eddington mass accretion rates around Schwarzschild and Kerr black holes with a Kerr parameter of 0.998. It is found that the flow, during its infall from the Keplerian to sub-Kepleria transition region to the black hole event horizon, passes through various phases of advection: the general advective paradigm to the radiatively inefficient phase, and vice versa. Hence, the flow governs a much lower electron temperature similar to 10(8)-10(9.5) K, in the range of accretion rate in Eddington units 0.01 less than or similar to (M) over dot less than or similar to 100, compared to the hot protons of temperature similar to 10(10.2)-10(11.8) K. Therefore, the solution may potentially explain the hard X-rays and gamma-rays emitted from active galactic nuclei (AGNs) and X-ray binaries. We then compare the solutions for two different regimes of viscosity. We conclude that a weakly viscous flow is expected to be cooling dominated, particularly at the inner region of the disc, compared to its highly viscous counterpart, which is radiatively inefficient. With all the solutions in hand, we finally reproduce the observed luminosities of the underfed AGNs and quasars (e. g. Sgr A*) to ultraluminous X-ray sources (e. g. SS433), at different combinations of input parameters, such as the mass accretion rate and the ratio of specific heats. The set of solutions also predicts appropriately the luminosity observed in highly luminous AGNs and ultraluminous quasars (e. g. PKS 0743-67).

Parametrized actor-critic algorithms for finite-horizon MDPs

Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FHMDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.

Solution of MDPS using simulation-based value iteration

This article proposes a three-timescale simulation based algorithm for solution of infinite horizon Markov Decision Processes (MDPs). We assume a finite state space and discounted cost criterion and adopt the value iteration approach. An approximation of the Dynamic Programming operator T is applied to the value function iterates. This 'approximate' operator is implemented using three timescales, the slowest of which updates the value function iterates. On the middle timescale we perform a gradient search over the feasible action set of each state using Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates, thus finding the minimizing action in T. On the fastest timescale, the 'critic' estimates, over which the gradient search is performed, are obtained. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is also presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are performed using the proposed algorithm. The results obtained are verified against classical value iteration where the feasible set is suitably discretized. Over such a discretized setting, a variant of the algorithm of [12] is compared and the proposed algorithm is found to converge faster.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Optimal advertisement-innovation mix for maximizing the discounted flow of profit

Due to boom in telecommunications market, there is hectic competition among the cellular handset manufacturers. As cellular manufacturing industry operates in an oligopoly framework, often price-rigidity leads to non-price wars. The handset manufacturing firms indulge in product innovation and also advertise their products in order to achieve their objective of maximizing discounted flow of profit. It is of interest to see what would be the optimal advertisement-innovation mix that would maximize the discounted How of profit for the firms. We used differential game theory to solve this problem. We adopted the open-loop solution methodology. We experimented for various scenarios over a 30 period horizon and derived interesting managerial insights.

Modelling the chemical weathering fluxes at the watershed scale in the Tropics (Mule Hole, South India): Relative contribution of the smectite/kaolinite assemblage versus primary minerals

We investigate the chemical weathering processes and fluxes in a small experimental watershed (SEW) through a modelling approach. The study site is the Mule Hole SEW developed on a gneissic basement located in the climatic gradient of the Western Ghats, South India. The model couples a lumped hydrological model simulating the water budget at the watershed scale to the WITCH model estimating the dissolution/precipitation rates of minerals using laboratory kinetic laws. Forcing functions and parameters of the simulation are defined by the field data. The coupled model is calibrated with stream and groundwater compositions through the testing of a large range of smectite solubility and abundance in the soil horizons. We found that, despite the low abundance of smectite in the dominant soil type of the watershed (4 vol.%), their net dissolution provides 75% of the export of dissolved silica, while primary silicate mineral dissolution releases only 15% of this flux. Overall, smectites (modelled as montmorillonites) are not stable under the present day climatic conditions. Furthermore, the dissolution of trace carbonates in the saprolitic horizon provides 50% of the calcium export at the watershed scale. Modelling results show the contrasted behavior of the two main soil types of the watershed: red soils (88% of the surface) are provider of calcium, while black soils (smectite-rich and characterized by a lower drainage) consumes calcium through overall carbonate precipitation. Our model results stress the key role played by minor/accessory minerals and by the thermodynamic properties of smectite minerals, and by the drainage of the weathering profiles on the weathering budget of a tropical watershed. (C) 2010 Elsevier B.V. All rights reserved.

The top 100 questions of importance to the future of global agriculture

Despite a significant growth in food production over the past half-century, one of the most important challenges facing society today is how to feed an expected population of some nine billion by the middle of the 20th century. To meet the expected demand for food without significant increases in prices, it has been estimated that we need to produce 70-100 per cent more food, in light of the growing impacts of climate change, concerns over energy security, regional dietary shifts and the Millennium Development target of halving world poverty and hunger by 2015. The goal for the agricultural sector is no longer simply to maximize productivity, but to optimize across a far more complex landscape of production, rural development, environmental, social justice and food consumption outcomes. However, there remain significant challenges to developing national and international policies that support the wide emergence of more sustainable forms of land use and efficient agricultural production. The lack of information flow between scientists, practitioners and policy makers is known to exacerbate the difficulties, despite increased emphasis upon evidence-based policy. In this paper, we seek to improve dialogue and understanding between agricultural research and policy by identifying the 100 most important questions for global agriculture. These have been compiled using a horizon-scanning approach with leading experts and representatives of major agricultural organizations worldwide. The aim is to use sound scientific evidence to inform decision making and guide policy makers in the future direction of agricultural research priorities and policy support. If addressed, we anticipate that these questions will have a significant impact on global agricultural practices worldwide, while improving the synergy between agricultural policy, practice and research. This research forms part of the UK Government's Foresight Global Food and Farming Futures project.

On the optimality of exhaustive service policies in multiclass queueing systems with modulated arrivals and switchovers

Consider a single-server multiclass queueing system with K classes where the individual queues are fed by K-correlated interrupted Poisson streams generated in the states of a K-state stationary modulating Markov chain. The service times for all the classes are drawn independently from the same distribution. There is a setup time (and/or a setup cost) incurred whenever the server switches from one queue to another. It is required to minimize the sum of discounted inventory and setup costs over an infinite horizon. We provide sufficient conditions under which exhaustive service policies are optimal. We then present some simulation results for a two-class queueing system to show that exhaustive, threshold policies outperform non-exhaustive policies.

Optimal control of a stochastic hybrid system with discounted cost

We address the optimal control problem of a very general stochastic hybrid system with both autonomous and impulsive jumps. The planning horizon is infinite and we use the discounted-cost criterion for performance evaluation. Under certain assumptions, we show the existence of an optimal control. We then derive the quasivariational inequalities satisfied by the value function and establish well-posedness. Finally, we prove the usual verification theorem of dynamic programming.

Geochemistry of water and ground water in the Nhecolandia, Pantanal of Mato Grosso, Brazil: Variability and associated processes

A distinctive feature of the Nhecolandia, a sub-region of the Pantanal wetland in Brazil, is the presence of both saline and freshwater lakes. Saline lakes used to be attributed to a past and phase during the Pleistocene. However, recent studies have shown that saline and fresh water lakes are linked by a continuous water table, indicating that saline water could come from a contemporary concentration process. This concentration process could also be responsible for the large chemical variability of the waters observed in the area. A regional water sampling has been conducted in surface and sub-surface water and the water table, and the results of the geochemical and statistical analysis are presented. Based on sodium contents, the concentration shows a 1: 4443 ratio. All the samples belong to the same chemical family and evolve in a sodic alkaline manner. Calcite or magnesian calcite precipitates very early in the process of concentration, probably followed by the precipitation of magnesian silicates. The most concentrated solutions remain under-saturated with respect to the sodium carbonate salt, even if this equilibrium is likely reached around the saline lakes. Apparently, significant amounts of sulfate and chloride are lost simultaneously from the solutions, and this cannot be explained solely by evaporative concentration. This could be attributed to the sorption on reduced minerals in a green sub-surface horizon in the "cordilhieira" areas. In the saline lakes, low potassium, phosphate, magnesium, and sulfate are attributed to algal blooms. Under the influence of evaporation, the concentration of solutions and associated chemical precipitations are identified as the main factors responsible for the geochemical variability in this environment (about 92 % of the variance). Therefore, the saline lakes of Nhecolandia have to be managed as landscape units in equilibrium with the present water flows and not inherited from a past and phase. In order to elaborate hydrochemical tracers for a quantitative estimation of water flows, three points have to be investigated more precisely: (1) the quantification of magnesium involved in the Mg-calcite precipitation; (2) the identification of the precise stoichiometry of the Mg-silicate; and (3) the verification of the loss of chloride and sulfate by sorption onto labile iron minerals.

Bidding Dynamics of Rational Advertisers in Sponsored Search Auctions on the Web

In this paper, we address a key problem faced by advertisers in sponsored search auctions on the web: how much to bid, given the bids of the other advertisers, so as to maximize individual payoffs? Assuming the generalized second price auction as the auction mechanism, we formulate this problem in the framework of an infinite horizon alternative-move game of advertiser bidding behavior. For a sponsored search auction involving two advertisers, we characterize all the pure strategy and mixed strategy Nash equilibria. We also prove that the bid prices will lead to a Nash equilibrium, if the advertisers follow a myopic best response bidding strategy. Following this, we investigate the bidding behavior of the advertisers if they use Q-learning. We discover empirically an interesting trend that the Q-values converge even if both the advertisers learn simultaneously.

An Actor-Critic Algorithm for Finite Horizon Markov Decision Processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.