On the Milito-Cruz adaptive control scheme for Markov chains

Milito and Cruz have introduced a novel adaptive control scheme for finite Markov chains when a finite parametrized family of possible transition matrices is available. The scheme involves the minimization of a composite functional of the observed history of the process incorporating both control and estimation aspects. We prove the a.s. optimality of a similar scheme when the state space is countable and the parameter space a compact subset ofR.

Residence time distribution for a class of Gaussian Markov processes

We study the distribution of residence time or equivalently that of "mean magnetization" for a family of Gaussian Markov processes indexed by a positive parameter alpha. The persistence exponent for these processes is simply given by theta=alpha but the residence time distribution is nontrivial. The shape of this distribution undergoes a qualitative change as theta increases, indicating a sharp change in the ergodic properties of the process. We develop two alternate methods to calculate exactly but recursively the moments of the distribution for arbitrary alpha. For some special values of alpha, we obtain closed form expressions of the distribution function. [S1063-651X(99)03306-1].

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Bayesian parameter identification in dynamic state space models using modified measurement equations

When Markov chain Monte Carlo (MCMC) samplers are used in problems of system parameter identification, one would face computational difficulties in dealing with large amount of measurement data and (or) low levels of measurement noise. Such exigencies are likely to occur in problems of parameter identification in dynamical systems when amount of vibratory measurement data and number of parameters to be identified could be large. In such cases, the posterior probability density function of the system parameters tends to have regions of narrow supports and a finite length MCMC chain is unlikely to cover pertinent regions. The present study proposes strategies based on modification of measurement equations and subsequent corrections, to alleviate this difficulty. This involves artificial enhancement of measurement noise, assimilation of transformed packets of measurements, and a global iteration strategy to improve the choice of prior models. Illustrative examples cover laboratory studies on a time variant dynamical system and a bending-torsion coupled, geometrically non-linear building frame under earthquake support motions. (C) 2015 Elsevier Ltd. All rights reserved.

Model and parameter uncertainty in IDF relationships under climate change

Quantifying distributional behavior of extreme events is crucial in hydrologic designs. Intensity Duration Frequency (IDF) relationships are used extensively in engineering especially in urban hydrology, to obtain return level of extreme rainfall event for a specified return period and duration. Major sources of uncertainty in the IDF relationships are due to insufficient quantity and quality of data leading to parameter uncertainty due to the distribution fitted to the data and uncertainty as a result of using multiple GCMs. It is important to study these uncertainties and propagate them to future for accurate assessment of return levels for future. The objective of this study is to quantify the uncertainties arising from parameters of the distribution fitted to data and the multiple GCM models using Bayesian approach. Posterior distribution of parameters is obtained from Bayes rule and the parameters are transformed to obtain return levels for a specified return period. Markov Chain Monte Carlo (MCMC) method using Metropolis Hastings algorithm is used to obtain the posterior distribution of parameters. Twenty six CMIP5 GCMs along with four RCP scenarios are considered for studying the effects of climate change and to obtain projected IDF relationships for the case study of Bangalore city in India. GCM uncertainty due to the use of multiple GCMs is treated using Reliability Ensemble Averaging (REA) technique along with the parameter uncertainty. Scale invariance theory is employed for obtaining short duration return levels from daily data. It is observed that the uncertainty in short duration rainfall return levels is high when compared to the longer durations. Further it is observed that parameter uncertainty is large compared to the model uncertainty. (C) 2015 Elsevier Ltd. All rights reserved.

Spatial B-jump at sudden channel enlargements with abrupt drop

An analytical and experimental study of the hydraulic jump in stilling basins with abrupt drop and sudden enlargement, called the spatial B-jump here, is carried out for finding the sequent depth ratio and resulting energy dissipation. The spatial B-jump studied has its toe downstream of the expansion section, and the stream lines at the toe are characterized by downward curvature. An expression is obtained for the sequent depth ratio based on the momentum equation with suitable assumptions for the extra pressure force term because of the abrupt drop in the bed and sudden enlargement in the basin width. Predictions compare favorably with experiments. It is shown that the spatial B-jump needs less tailwater depth, thereby enhancing the stability of the jump when compared either with spatial jump, which forms in sudden expanding channels, or with B-jump, which forms in a channel with an abrupt drop in bed. It is also shown that there is a significant increase in relative energy loss for the spatial B-jump compared to either the spatial jump or B-jump alone.

Ergodic control of partially observed Markov chains

The ergodic or long-run average cost control problem for a partially observed finite-state Markov chain is studied via the associated fully observed separated control problem for the nonlinear filter. Dynamic programming equations for the latter are derived, leading to existence and characterization of optimal stationary policies.

Massive black-hole binary inspirals: results from the LISA parameter estimation taskforce

The LISA Parameter Estimation Taskforce was formed in September 2007 to provide the LISA Project with vetted codes, source distribution models and results related to parameter estimation. The Taskforce's goal is to be able to quickly calculate the impact of any mission design changes on LISA's science capabilities, based on reasonable estimates of the distribution of astrophysical sources in the universe. This paper describes our Taskforce's work on massive black-hole binaries (MBHBs). Given present uncertainties in the formation history of MBHBs, we adopt four different population models, based on (i) whether the initial black-hole seeds are small or large and (ii) whether accretion is efficient or inefficient at spinning up the holes. We compare four largely independent codes for calculating LISA's parameter-estimation capabilities. All codes are based on the Fisher-matrix approximation, but in the past they used somewhat different signal models, source parametrizations and noise curves. We show that once these differences are removed, the four codes give results in extremely close agreement with each other. Using a code that includes both spin precession and higher harmonics in the gravitational-wave signal, we carry out Monte Carlo simulations and determine the number of events that can be detected and accurately localized in our four population models.

Non-stationary signal analysis by least mean fourth (LMF) adaptive algorithm

A modified least mean fourth (LMF) adaptive algorithm applicable to non-stationary signals is presented. The performance of the proposed algorithm is studied by simulation for non-stationarities in bandwidth, centre frequency and gain of a stochastic signal. These non-stationarities are in the form of linear, sinusoidal and jump variations of the parameters. The proposed LMF adaptation is found to have better parameter tracking capability than the LMS adaptation for the same speed of convergence.

Is one-parameter scaling theory of localisation consistent? Comment on Schuh's comment

It is maintained that the one-parameter scaling theory is inconsistent with the physics of Anderson localisation.

Asymptotic behaviour of point processes with Markov-dependent intervals

This splitting techniques for MARKOV chains developed by NUMMELIN (1978a) and ATHREYA and NEY (1978b) are used to derive an imbedded renewal process in WOLD's point process with MARKOV-correlated intervals. This leads to a simple proof of renewal theorems for such processes. In particular, a key renewal theorem is proved, from which analogues to both BLACKWELL's and BREIMAN's forms of the renewal theorem can be deduced.

An account of chronological developments in control of distributed parameter systems

Control systems arising in many engineering fields are often of distributed parameter type, which are modeled by partial differential equations. Decades of research have lead to a great deal of literature on distributed parameter systems scattered in a wide spectrum.Extensions of popular finite-dimensional techniques to infinite-dimensional systems as well as innovative infinite-dimensional specific control design approaches have been proposed. A comprehensive account of all the developments would probably require several volumes and is perhaps a very difficult task. In this paper, however, an attempt has been made to give a brief yet reasonably representative account of many of these developments in a chronological order. To make it accessible to a wide audience, mathematical descriptions have been completely avoided with the assumption that an interested reader can always find the mathematical details in the relevant references.

Optimal Parameter Trajectory Estimation in Parameterized SDEs: An Algorithmic Procedure

We consider the problem of estimating the optimal parameter trajectory over a finite time interval in a parameterized stochastic differential equation (SDE), and propose a simulation-based algorithm for this purpose. Towards this end, we consider a discretization of the SDE over finite time instants and reformulate the problem as one of finding an optimal parameter at each of these instants. A stochastic approximation algorithm based on the smoothed functional technique is adapted to this setting for finding the optimal parameter trajectory. A proof of convergence of the algorithm is presented and results of numerical experiments over two different settings are shown. The algorithm is seen to exhibit good performance. We also present extensions of our framework to the case of finding optimal parameterized feedback policies for controlled SDE and present numerical results in this scenario as well.

Proline ring conformations corresponding to a bistable jump model from 13C spin-lattice relaxation times

A formalism for extracting the conformations of a proline ring based on the bistable jump model of R. E. London [(1978) J. Am. Chem. Soc. 100, 2678-2685] from 13C spin-lattice relaxation times (T1) is given. The method is such that the relaxation data are only partially used to generate the conformations; these conformations are constrained to satisfy the rest of the relaxation data and to yield acceptable ring geometry. An alternate equation for T1 of 13C nuclei to that of London is given. The formalism is illustrated through an example.