Optimal Control of Markov Processes with Age-Dependent Transition Rates

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.

Magnetic-field-induced Fabry-Perot resonances in helical edge states

We study electronic transport across a helical edge state exposed to a uniform magnetic ((B) over right arrow) field over a finite length. We show that this system exhibits Fabry-Perot-type resonances in electronic transport. The intrinsic spin anisotropy of the helical edge states allows us to tune these resonances by changing the direction of the (B) over right arrow field while keeping its magnitude constant. This is in sharp contrast to the case of nonhelical one-dimensional electron gases with a parabolic dispersion, where similar resonances do appear in individual spin channels (up arrow and down arrow) separately which, however, cannot be tuned by merely changing the direction of the (B) over right arrow field. These resonances provide a unique way to probe the helical nature of the theory. We study the robustness of these resonances against a possible static impurity in the channel.

Markov chain modeling of evolution of strains in reinforced concrete flexural beams

From the analysis of experimentally observed variations in surface strains with loading in reinforced concrete beams, it is noted that there is a need to consider the evolution of strains (with loading) as a stochastic process. Use of Markov Chains for modeling stochastic evolution of strains with loading in reinforced concrete flexural beams is studied in this paper. A simple, yet practically useful, bi-level homogeneous Gaussian Markov Chain (BLHGMC) model is proposed for determining the state of strain in reinforced concrete beams. The BLHGMC model will be useful for predicting behavior/response of reinforced concrete beams leading to more rational design.

Nonequilibrium arrhythmic states and transitions in a mathematical model for diffuse fibrosis in human cardiac tissue

We present a comprehensive numerical study of spiral-and scroll-wave dynamics in a state-of-the-art mathematical model for human ventricular tissue with fiber rotation, transmural heterogeneity, myocytes, and fibroblasts. Our mathematical model introduces fibroblasts randomly, to mimic diffuse fibrosis, in the ten Tusscher-Noble-Noble-Panfilov (TNNP) model for human ventricular tissue; the passive fibroblasts in our model do not exhibit an action potential in the absence of coupling with myocytes; and we allow for a coupling between nearby myocytes and fibroblasts. Our study of a single myocyte-fibroblast (MF) composite, with a single myocyte coupled to N-f fibroblasts via a gap-junctional conductance G(gap), reveals five qualitatively different responses for this composite. Our investigations of two-dimensional domains with a random distribution of fibroblasts in a myocyte background reveal that, as the percentage P-f of fibroblasts increases, the conduction velocity of a plane wave decreases until there is conduction failure. If we consider spiral-wave dynamics in such a medium we find, in two dimensions, a variety of nonequilibrium states, temporally periodic, quasiperiodic, chaotic, and quiescent, and an intricate sequence of transitions between them; we also study the analogous sequence of transitions for three-dimensional scroll waves in a three-dimensional version of our mathematical model that includes both fiber rotation and transmural heterogeneity. We thus elucidate random-fibrosis-induced nonequilibrium transitions, which lead to conduction block for spiral waves in two dimensions and scroll waves in three dimensions. We explore possible experimental implications of our mathematical and numerical studies for plane-, spiral-, and scroll-wave dynamics in cardiac tissue with fibrosis.

Damage limit states of reinforced concrete beams subjected to incremental cyclic loading using relaxation ratio analysis of AE parameters

This paper presents an experimental study on damage assessment of reinforced concrete (RC) beams subjected to incremental cyclic loading. During testing acoustic emissions (AEs) were recorded. The analysis of the AE released was carried out by using parameters relaxation ratio, load ratio and calm ratio. Digital image correlation (DIC) technique and tracking with available MATLAB program were used to measure the displacement and surface strains in concrete. Earlier researchers classified the damage in RC beams using Kaiser effect, crack mouth opening displacement and proposed a standard. In general (or in practical situations), multiple cracks occur in reinforced concrete beams. In the present study damage assessment in RC beams was studied according to different limit states specified by the code of practice IS-456:2000 and AE technique. Based on the two ratios namely load ratio and calm ratio and when the deflection reached approximately 85% of the maximum allowable deflection it was observed that the RC beams were heavily damaged. The combination of AE and DIC techniques has the potential to provide the state of damage in RC structures.

Indentation size effect and shear transformation zone size in a bulk metallic glass in two different structural states

The existence of an indentation size effect (ISE) in the onset of yield in a Zr-based bulk metallic glass (BMG) is investigated by employing spherical-tip nanoindentation experiments. Statistically significant data on the load at which the first pop-in in the displacement occurs were obtained for three different tip radii and in two different structural states (as-cast and structurally relaxed) of the BMG. Hertzian contact mechanics were employed to convert the pop-in loads to the maximum shear stress underneath the indenter. Results establish the existence of an ISE in the BMG of both structural states, with shear yield stress increasing with decreasing tip radius. Structural relaxation was found to increase the yield stress and decrease the variability in the data, indicating ``structural homogenization'' with annealing. Statistical analysis of the data was employed to estimate the shear transformation zone (STZ) size. Results of this analysis indicate an STZ size of similar to 25 atoms, which increases to similar to 34 atoms upon annealing. These observations are discussed in terms of internal structure changes that occur during structural relaxation and their interaction with the stressed volumes in spherical indentation of a metallic glass. (C) 2012 Acta Materialia Inc. Published by Elsevier Ltd. All rights reserved.

Methodology for Pavement Design Reliability and Back Analysis Using Markov Chain Monte Carlo Simulation

Given the increasing cost of designing and building new highway pavements, reliability analysis has become vital to ensure that a given pavement performs as expected in the field. Recognizing the importance of failure analysis to safety, reliability, performance, and economy, back analysis has been employed in various engineering applications to evaluate the inherent uncertainties of the design and analysis. The probabilistic back analysis method formulated on Bayes' theorem and solved using the Markov chain Monte Carlo simulation method with a Metropolis-Hastings algorithm has proved to be highly efficient to address this issue. It is also quite flexible and is applicable to any type of prior information. In this paper, this method has been used to back-analyze the parameters that influence the pavement life and to consider the uncertainty of the mechanistic-empirical pavement design model. The load-induced pavement structural responses (e.g., stresses, strains, and deflections) used to predict the pavement life are estimated using the response surface methodology model developed based on the results of linear elastic analysis. The failure criteria adopted for the analysis were based on the factor of safety (FOS), and the study was carried out for different sample sizes and jumping distributions to estimate the most robust posterior statistics. From the posterior statistics of the case considered, it was observed that after approximately 150 million standard axle load repetitions, the mean values of the pavement properties decrease as expected, with a significant decrease in the values of the elastic moduli of the expected layers. An analysis of the posterior statistics indicated that the parameters that contribute significantly to the pavement failure were the moduli of the base and surface layer, which is consistent with the findings from other studies. After the back analysis, the base modulus parameters show a significant decrease of 15.8% and the surface layer modulus a decrease of 3.12% in the mean value. The usefulness of the back analysis methodology is further highlighted by estimating the design parameters for specified values of the factor of safety. The analysis revealed that for the pavement section considered, a reliability of 89% and 94% can be achieved by adopting FOS values of 1.5 and 2, respectively. The methodology proposed can therefore be effectively used to identify the parameters that are critical to pavement failure in the design of pavements for specified levels of reliability. DOI: 10.1061/(ASCE)TE.1943-5436.0000455. (C) 2013 American Society of Civil Engineers.

Non-Stationary Semi-Markov Decision Processes on a Finite Horizon

We introduce and study a class of non-stationary semi-Markov decision processes on a finite horizon. By constructing an equivalent Markov decision process, we establish the existence of a piecewise open loop relaxed control which is optimal for the finite horizon problem.

Dynamic mechanism design for markets with strategic resources

The assignment of tasks to multiple resources becomes an interesting game theoretic problem, when both the task owner and the resources are strategic. In the classical, nonstrategic setting, where the states of the tasks and resources are observable by the controller, this problem is that of finding an optimal policy for a Markov decision process (MDP). When the states are held by strategic agents, the problem of an efficient task allocation extends beyond that of solving an MDP and becomes that of designing a mechanism. Motivated by this fact, we propose a general mechanism which decides on an allocation rule for the tasks and resources and a payment rule to incentivize agents' participation and truthful reports. In contrast to related dynamic strategic control problems studied in recent literature, the problem studied here has interdependent values: the benefit of an allocation to the task owner is not simply a function of the characteristics of the task itself and the allocation, but also of the state of the resources. We introduce a dynamic extension of Mezzetti's two phase mechanism for interdependent valuations. In this changed setting, the proposed dynamic mechanism is efficient, within period ex-post incentive compatible, and within period ex-post individually rational.

Asymptotics of the invariant measure in mean field models with jumps

We consider the asymptotics of the invariant measure for the process of spatial distribution of N coupled Markov chains in the limit of a large number of chains. Each chain reflects the stochastic evolution of one particle. The chains are coupled through the dependence of transition rates on the spatial distribution of particles in the various states. Our model is a caricature for medium access interactions in wireless local area networks. Our model is also applicable in the study of spread of epidemics in a network. The limiting process satisfies a deterministic ordinary differential equation called the McKean-Vlasov equation. When this differential equation has a unique globally asymptotically stable equilibrium, the spatial distribution converges weakly to this equilibrium. Using a control-theoretic approach, we examine the question of a large deviation from this equilibrium.

Packing in molten globules and native states

Close packing of hydrophobic residues in the protein interior is an important determinant of protein stability. Cavities introduced by large to small substitutions are known to destabilize proteins. Conversely, native states of proteins and protein fragments can be stabilized by filling in existing cavities. Molten globules (MGs) were initially used to describe a state of protein which has well-defined secondary structure but little or no tertiary packing. Subsequent studies have shown that MGs do have some degree of native-like topology and specific packing. Wet molten globules (WMGs) with hydrated cores and considerably decreased packing relative to the native state have been studied extensively. Recently there has been renewed interest in identification and characterization of dry molten globules (DMGs). These are slightly expanded forms of the native state which show increased conformational flexibility, native-like main-chain hydrogen bonding and dry interiors. The generality of occurrence of DMGs during protein unfolding and the extent and nature of packing in DMGs remain to be elucidated. Packing interactions in native proteins and MGs can be probed through mutations. Next generation sequencing technologies make it possible to determine relative populations of mutants in a large pool. When this is coupled to phenotypic screens or cell-surface display, it becomes possible to rapidly examine large panels of single-site or multi-site mutants. From such studies, residue specific contributions to protein stability and function can be estimated in a highly parallelized fashion. This complements conventional biophysical methods for characterization of packing in native states and molten globules.

Electronic energy transfer between long-range resonance states of 3-mercaptopropionic acid capped CdTe quantum dots in aqueous media

We demonstrate electronic energy transfer between resonance states of 2 and 2.8 nm CdTe quantum dots in aqueous media using steady-state photoluminescence spectroscopy without using any external linker molecule. With increasing concentration of larger dots, there is subsequent quenching of luminescence in smaller dots accompanied by the enhancement of luminescence in larger dots. Our experimental evidence suggests that there is long-range resonance energy transfer among electronic excitations, specifically from the electronically confined states of the smaller dots to the higher excited states of the larger dots.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

Tunability of electronic states in ultrathin gold nanowires

The electronic state in ultrathin gold nanowires is tuned by careful engineering of the device architecture via a chemical methodology. The electrons are localized to an insulating state (showing variable range hopping transport) by simply bringing them close to the substrate, while the insertion of an interlayer leads to a Tomonaga Luttinger liquid state.

Investigating proline puckering states in diproline segments in proteins

In the current study, the puckering states of the Proline ring occurring in diproline segments (LPro-LPro) in proteins has been investigated with a segregation made on the basis of cis and trans states for the Pro-Pro peptide bond and the conformational states for the diproline segment to investigate the effects of conformation of the diproline segment on the corresponding puckering state of the Proline ring in the segment if any. The value of the endocyclic ring torsional angles of the pyrrolidine ring has been used for calculating and visualizing various puckering states using a proposed new sign convention (+/-) nomenclature. The results have been compared to that obtained in a previous study on peptides from this group. In this study, quite interestingly, the Planar (G) conformation that was present in 14.3% of the cases in peptides, appears to be nearly a rare conformation in the case of proteins (1.9%). The present study indicates that the (C-exo/C-exo), (C-exo/Twisted C-exo-C-endo) and (Twisted C-endo-C-exo/Twisted C-endo-C-exo) categories are the most preferred combinations. For Proline rings in proteins, the states C-exo, Twisted C-exo-C-endo and Twisted C-endo-C-exo are the most preferred states. Within diproline segments, the pyrrolidine ring conformations do not show a strong co-relation to the backbone conformation in which they are observed. It is likely that five-membered rings have a considerable plasticity of structure and are readily deformed to accommodate a variety of energetically preferred backbone conformations.