On the Milito-Cruz adaptive control scheme for Markov chains

Milito and Cruz have introduced a novel adaptive control scheme for finite Markov chains when a finite parametrized family of possible transition matrices is available. The scheme involves the minimization of a composite functional of the observed history of the process incorporating both control and estimation aspects. We prove the a.s. optimality of a similar scheme when the state space is countable and the parameter space a compact subset ofR.

Risk Minimizing Option Pricing for a Class of Exotic Options in a Markov-Modulated Market

We address risk minimizing option pricing in a regime switching market where the floating interest rate depends on a finite state Markov process. The growth rate and the volatility of the stock also depend on the Markov process. Using the minimal martingale measure, we show that the locally risk minimizing prices for certain exotic options satisfy a system of Black-Scholes partial differential equations with appropriate boundary conditions. We find the corresponding hedging strategies and the residual risk. We develop suitable numerical methods to compute option prices.

Supercritical age-dependent branching Markov processes and their scaling limits

This paper studies the long-time behavior of the empirical distribution of age and normalized position of an age-dependent supercritical branching Markov process. The motion of each individual during its life is a random function of its age. It is shown that the empirical distribution of the age and the normalized position of all individuals alive at time t converges as t -> infinity to a deterministic product measure.

An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Di- and tri-metallic complexes containing two bridging cyclodiphosphazane ligands: the crystal structure of [MO2(CO)8{µ-cis-[PhNP(OC6H4Me-p)]2}2]

The reactions of the mononuclear cyclodiphosphazane complexes, cis-[Mo(CO)(4){cis-[PhNP(OR)](2)}(2)] with [Mo(CO)(4)(nbd)] (nbd = norbornadiene). [Mo(CO)(4)(NHC5H10)(2)] or [MCl(2)(cod)] (cod = cycloocta-1,5-diene) afforded the homobimetallic complexes; [Mo-2(CO)(8){mu-cis-[PhNP(OR)](2)}(2)] (R = C(5)H(4)Me-p 5 or CH2CF3 6) or the heterobimetallic complexes. [Mo-2(CO)(8){mu-cis-[PhNP(OE)](2)}(2)MCl(2)] (R = C(6)H(4)Me-p; M = Pd 7 or Pt 8). In all the above complexes, the two metal moieties are bridged by two cyclodiphosphazane ligands. The reactions of the mononuclear complexes, cis-[M(CO)(4)(A){cis-[PhNP(OC(6)H(4)Me-p)](2)}] with (M'Cl-2(cod)] afforded the trinuclear complexes, cis-[M'Cl-2[M(CO)(4)(A){cis-[PhNP(OC(6)H(4)Me-p)](2)}](2)] (M' = Pd, M = Mo, A = P(OMe)(3) 10; M' = Pt, M = Mo. A = P(OMe)(3) 11; M' = Pd. M = W. A = NHC5H10 12; M' = Pt, M = W. A = NHC5H10 13). The structure of the complex 5 has been determined by single-crystal X-ray crystallography.

Multiscale stochastic approximation for parametric optimization of hidden Markov models

A two-time scale stochastic approximation algorithm is proposed for simulation-based parametric optimization of hidden Markov models, as an alternative to the traditional approaches to ''infinitesimal perturbation analysis.'' Its convergence is analyzed, and a queueing example is presented.

N-3-benzoyl-2 ',3 '-di-O-benzoyluridine

The structure of N-3-benzoyl-2',3'-di-O-benzoyluridine, C30H24N2O9, has two molecules in the asymmetric unit. The uracil bases of both the molecules are in the anti conformation with respect to the ribose moiety and the furanosyl rings adopt a C3'-endo conformation. The orientation about the C4'-C5' bond is gauche-gauche. The two crystallographically independent molecules are linked through several C-H ... O hydrogen bonds. The nucleoside molecules pack as columns along the a axis and these columns repeat along the c axis.

Residence time distribution for a class of Gaussian Markov processes

We study the distribution of residence time or equivalently that of "mean magnetization" for a family of Gaussian Markov processes indexed by a positive parameter alpha. The persistence exponent for these processes is simply given by theta=alpha but the residence time distribution is nontrivial. The shape of this distribution undergoes a qualitative change as theta increases, indicating a sharp change in the ergodic properties of the process. We develop two alternate methods to calculate exactly but recursively the moments of the distribution for arbitrary alpha. For some special values of alpha, we obtain closed form expressions of the distribution function. [S1063-651X(99)03306-1].

Risk-sensitive optimal control for Markov decision processes with monotone cost

The existence of an optimal feedback law is established for the risk-sensitive optimal control problem with denumerable state space. The main assumptions imposed are irreducibility and a near monotonicity condition on the one-step cost function. A solution can be found constructively using either value iteration or policy iteration under suitable conditions on initial feedback law.

Analysis of link-layer backoff schemes on point-to-point Markov fading links

Backoff algorithms are typically employed in multiple-access networks (e.g., Ethernet) to recover from packet collisions. In this letter, we propose and carry out the analysis for three types of link-layer backoff schemes, namely, linear backoff, exponential backoff, and geometric backoff, on point-to-point wireless fading links where packet errors occur nonindependently. In such a scenario, the backoff schemes are shown to achieve better energy efficiency without compromising much on the link layer throughput performance.

Conformational basis for substrate recognition and regulation of catalytic activity in Staphylococcus aureus nucleoside di-phosphate kinase

Nucleoside diphosphate kinases (NDK) are characterized by high catalytic turnover rates and diverse substrate specificity. These features make this enzyme an effective activator of a pro-drug an application that has been actively pursued for a variety of therapeutic strategies. The catalytic mechanism of this enzyme is governed by a conserved histidine that coordinates a magnesium ion at the active site. Despite substantial structural and biochemical information on NDK, the mechanistic feature of the phospho-transfer that leads to auto-phosphorylation remains unclear. While the role of the histidine residue is well documented, the other active site residues, in particular the conserved serine remains poorly characterized. Studies on some homologues suggest no role for the serine residue at the active site, while others suggest a crucial role for this serine in the regulation and quaternary association of this enzyme in some species. Here we report the biochemical features of the Staphylococcus aureus NDK and the mutant enzymes. We also describe the crystal structures of the apo-NDK, as a transition state mimic with vanadate and in complex with different nucleotide substrates. These structures formed the basis for molecular dynamics simulations to understand the broad substrate specificity of this enzyme and the role of active site residues in the phospho-transfer mechanism and oligomerization. Put together, these data suggest that concerted changes in the conformation of specific residues facilitate the stabilization of nucleotide complexes thereby enabling the steps involved in the ping-pong reaction mechanism without large changes to the overall structure of this enzyme. (C) 2011 Elsevier B.V. All rights reserved.

Complexes of rare-earth perchlorates with ditbutyl amides of di, tri and tetraglycolic acids

New complexes of lanthanide perchlorates with di-t-butyl amides of di, tri and tetraglycolic acids have been synthesised. The complexes have the general formula Ln(DiGA)3(ClO4)3; Ln(TriGA)2 (ClO4)3 and Ln(TetGA)2 (C1O4)3, where Ln = La-Yb and Y and DiGA = N,N′, di-t-butyl diglycolamide, TriGA N,N′, di-t-butyl triglycolamide and TetGA = N,N′ di-t-butyl tetraglycolamide, respectively. The complexes have been characterized by analysis, electrolytic conductance, infrared,1H and13C nuclear magnetic resonance and electronic spectral data.Infrared spectra indicate the coordination of all the available ether oxygens and the amide carbonyls in each of the ligands, to the metal ions. IR and conductance data show that the perchlorate groups in all the complexes are ionic.1H and13C NMR data support the IR data regarding the mode of coordination of ligands to the metal ions. Electronic spectral shapes have been interpreted in terms of nine, eight and ten coordination in DiGA, TriGA and TetGA complexes respectively.

An Actor-Critic Algorithm for Finite Horizon Markov Decision Processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Performance analysis of UDP with energy efficient link layer on Markov fading channels

In this paper, we analyze the throughput and energy efficiency performance of user datagram protocol (UDP) using linear, binary exponential, and geometric backoff algorithms at the link layer (LL) on point-to-point wireless fading links. Using a first-order Markov chain representation of the packet success/failure process on fading channels, we derive analytical expressions for throughput and energy efficiency of UDP/LL with and without LL backoff. The analytical results are verified through simulations. We also evaluate the mean delay and delay variation of voice packets and energy efficiency performance over a wireless link that uses UDP for transport of voice packets and the proposed backoff algorithms at the LL. We show that the proposed LL backoff algorithms achieve energy efficiency improvement of the order of 2-3 dB compared to LL with no backoff, without compromising much on the throughput and delay performance at the UDP layer. Such energy savings through protocol means will improve the battery life in wireless mobile terminals.