An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

MicroRNA-155 Is Required for Mycobacterium bovis BCG-Mediated Apoptosis of Macrophages

Pathogenic rnycobacteria, including Mycobacterium tuberculosis and Mycobacterium bovis, cause significant morbidity and mortality worldwide. However, the vaccine strain Mycobacterium bovis BCG, unlike virulent strains, triggers extensive apoptosis of infected macrophages, a step necessary for the elicitation of robust protective immunity. We here demonstrate that M. bovis BCG triggers Toll-like receptor 2 (TLR2)-dependent microRNA-155 (miR-155) expression, which involves signaling cross talk among phosphatidylinositol 3-kinase (PI3K), protein kinase C delta (PKC delta), and mitogen-activated protein kinases (MAPKs) and recruitment of NF-kappa B and c-ETS to miR-155 promoter. Genetic and signaling perturbations presented the evidence that miR-155 regulates PKA signaling by directly targeting a negative regulator of PKA, protein kinase inhibitor alpha (PKI-alpha). Enhanced activation of PKA signaling resulted in the generation of PKA C-alpha; phosphorylation of MSK1, cyclic AMP response element binding protein (CREB), and histone H3; and recruitment of phospho-CREB to the apoptotic gene promoters. The miR-155-triggered activation of caspase-3, BAK1, and cytochrome c translocation involved signaling integration of MAPKs and epigenetic or posttranslational modification of histones or CREB. Importantly, M. bovis BCG infection-induced apoptosis was severely compromised in macrophages derived from miR-155 knockout mice. Gain-of-function and loss-of-function studies validated the requirement of miR-155 for M. bovis BCG's ability to trigger apoptosis. Overall, M. bovis BCG-driven miR-155 dictates cell fate decisions of infected macrophages, strongly implicating a novel role for miR-155 in orchestrating cellular reprogramming during immune responses to mycobacterial infection.

The force distribution function of an oscillator model of polymer stretching at constant velocity

Recent simulations of the stretching of tethered biopolymers at a constant speed v (Ponmurugan and Vemparala, 2011 Phys. Rev. E 84 060101(R)) have suggested that for any time t, the distribution of the fluctuating forces f responsible for chain deformation is governed by a relation of the form P(+ f)/ P(- f) = expgamma f], gamma being a coefficient that is solely a function of v and the temperature T. This result, which is reminiscent of the fluctuation theorems applicable to stochastic trajectories involving thermodynamic variables, is derived in this paper from an analytical calculation based on a generalization of Mazonka and Jarzynski's classic model of dragged particle dynamics Mazonka and Jarzynski, 1999 arXiv:cond-\textbackslashmat/9912121v1]. However, the analytical calculations suggest that the result holds only if t >> 1 and the force fluctuations are driven by white rather than colored noise; they further suggest that the coefficient gamma in the purported theorem varies not as v(0.15)T-(0.7), as indicated by the simulations, but as vT(-1).

MS_RHII-RSD, a Dual-Function RNase HII-(p)ppGpp Synthetase from Mycobacterium smegmatis

In the noninfectious soil saprophyte Mycobacterium smegmatis, intracellular levels of the stress alarmones guanosine tetraphosphate and guanosine pentaphosphate, together termed (p)ppGpp, are regulated by the enzyme Rel(Msm). This enzyme consists of a single, bifunctional polypeptide chain that is capable of both synthesizing and hydrolyzing (p)ppGpp. The rel(Msm), knockout strain of M. smegmatis (Delta rel(Msm)) is expected to show a (p)ppGpp null (p)ppGpp(0)] phenotype. Contrary to this expectation, the strain is capable of synthesizing (p)ppGpp in vivo. In this study, we identify and functionally characterize the open reading frame (ORF), MSMEG_5849, that encodes a second functional (p)ppGpp synthetase in M. smegmatis. In addition to (p)ppGpp synthesis, the 567-amino-acid-long protein encoded by this gene is capable of hydrolyzing RNA(.)DNA hybrids and bears similarity to the conventional RNase HII enzymes. We have classified this protein as actRel(Msm) in accordance with the recent nomenclature proposed and have named it MS_RHII-RSD, indicating the two enzymatic activities present RHII, RNase HII domain, originally identified as (d) under bar omain of (u) under bar nknown (f) under bar unction 429 (DUF429), and RSD, RelA_SpoT nucleotidyl transferase domain, the SYNTH domain responsible for (p)ppGpp synthesis activity]. MS_RHII-RSD is expressed and is constitutively active in vivo and behaves like a monofunctional (p)ppGpp synthetase in vitro. The occurrence of the RNase HII and (p)ppGpp synthetase domains together on the same polypeptide chain is suggestive of an in vivo role for this novel protein as a link connecting the essential life processes of DNA replication, repair, and transcription to the highly conserved stress survival pathway, the stringent response.

Relation between the work function and Young's modulus of RhSi and estimate of Schottky-barrier height at RhSi/Si interface: An ab-initio study

Density-functional calculations are performed to explore the relationship between the work function and Young's modulus of RhSi, and to estimate the p-Schottky-barrier height (SBH) at the Si/RhSi(010) interface. It is shown that the Young's modulus and the workfunction of RhSi satisfy the generic sextic relation, proposed recently for elemental metals. The calculated p-SBH at the Si/RhSi interface is found to differ only by 0.04 eV in opposite limits, viz., no-pinning and strong pinning. We find that the p-SBH is reduced as much as by 0.28 eV due to vacancies at the interface. (C) 2012 American Institute of Physics. http://dx.doi.org/10.1063/1.4761994]

Perturbative expansion of the QCD Adler function improved by renormalization-group summation and analytic continuation in the Borel plane

We examine the large-order behavior of a recently proposed renormalization-group-improved expansion of the Adler function in perturbative QCD, which sums in an analytically closed form the leading logarithms accessible from renormalization-group invariance. The expansion is first written as an effective series in powers of the one-loop coupling, and its leading singularities in the Borel plane are shown to be identical to those of the standard ``contour-improved'' expansion. Applying the technique of conformal mappings for the analytic continuation in the Borel plane, we define a class of improved expansions, which implement both the renormalization-group invariance and the knowledge about the large-order behavior of the series. Detailed numerical studies of specific models for the Adler function indicate that the new expansions have remarkable convergence properties up to high orders. Using these expansions for the determination of the strong coupling from the hadronic width of the tau lepton we obtain, with a conservative estimate of the uncertainty due to the nonperturbative corrections, alpha(s)(M-tau(2)) = 0.3189(-0.0151)(+0.0173), which translates to alpha(s)(M-Z(2)) = 0.1184(-0.0018)(+0.0021). DOI: 10.1103/PhysRevD.87.014008

Transcription factor erect wing (EWG) is involved in indirect flight muscle patterning, development and maintenance in Drosophila

Muscle development is a multistep process which includes myoblast diversification, proliferation, migration, fusion, differentiation and growth. A hierarchical exhibition of myogenic factors is important for dexterous execution of progressive events in muscle formation. EWG (erect wing) is a transcription factor known to have a role in indirect flight muscle development (IFM) in Drosophila. We marked out the precise spatio-temporal expression profile of EWG in the myoblasts, and in the developing muscles. Mutant adult flies null for EWG in myoblasts show variable number of IFM, suggesting that EWG is required for patterning of the IFM. The remnant muscle found in the EWG null flies show proper assembly of the structural proteins, which implies that some myoblasts manage to fuse, develop and differentiate normally indicating that EWG is not required for differentiation program per se. However, when EWG expression is extended beyond its expression window in a wild type background, muscle thinning is observed implying EWG function in protein synthesis inhibition. Mis-expression studies in wing disc myoblasts hinted at its role in myoblast proliferation. We thus conclude that EWG is important for regulating fusion events which in turn decides the IFM pattern. Also IFM in EWG null mutants show clumps containing broken fibres and an altered mitochondrial morphology. The vertebrate homolog of EWG is nuclear respiratory factor1 (NRF1) which is known to have a function in mitochondrial biogenesis and protection against oxidative stress. Gene expression for inner mitochondrial membrane protein, Opa1-like was found to be absent in these mutants. Also, these flies were more sensitive to oxidative stress, indicating a compromised mitochondrial functioning. Our results therefore demonstrate that EWG functions in maintaining muscles’ structural integrity by ensuing proper mitochondrial activity.

A nested linear codes approach to distributed function computation over subspaces

In this paper, we consider a distributed function computation setting, where there are m distributed but correlated sources X1,...,Xm and a receiver interested in computing an s-dimensional subspace generated by [X1,...,Xm]Γ for some (m × s) matrix Γ of rank s. We construct a scheme based on nested linear codes and characterize the achievable rates obtained using the scheme. The proposed nested-linear-code approach performs at least as well as the Slepian-Wolf scheme in terms of sum-rate performance for all subspaces and source distributions. In addition, for a large class of distributions and subspaces, the scheme improves upon the Slepian-Wolf approach. The nested-linear-code scheme may be viewed as uniting under a common framework, both the Korner-Marton approach of using a common linear encoder as well as the Slepian-Wolf approach of employing different encoders at each source. Along the way, we prove an interesting and fundamental structural result on the nature of subspaces of an m-dimensional vector space V with respect to a normalized measure of entropy. Here, each element in V corresponds to a distinct linear combination of a set {Xi}im=1 of m random variables whose joint probability distribution function is given.

Point Spread Function Engineering for Super-Resolution Single-Photon and Multiphoton Fluorescence Microscopy

Super-resolution imaging techniques are of paramount interest for applications in bioimaging and fluorescence microscopy. Recent advances in bioimaging demand application-tailored point spread functions. Here, we present some approaches for generating application-tailored point spread functions along with fast imaging capabilities. Aperture engineering techniques provide interesting solutions for obtaining desired system point spread functions. Specially designed spatial filters—realized by optical mask—are outlined both in a single-lens and 4Pi configuration. Applications include depth imaging, multifocal imaging, and super-resolution imaging. Such an approach is suitable for fruitful integration with most existing state-of-art imaging microscopy modalities.

Primary Microcephaly Gene MCPH1 Shows Signatures of Tumor Suppressors and Is Regulated by miR-27a in Oral Squamous Cell Carcinoma

Mutations in the MCPH1 (microcephalin 1) gene, located at chromosome 8p23.1, result in two autosomal recessive disorders: primary microcephaly and premature chromosome condensation syndrome. MCPH1 has also been shown to be downregulated in breast, prostate and ovarian cancers, and mutated in 1/10 breast and 5/41 endometrial tumors, suggesting that it could also function as a tumor suppressor (TS) gene. To test the possibility of MCPH1 as a TS gene, we first performed LOH study in a panel of 81 matched normal oral tissues and oral squamous cell carcinoma (OSCC) samples, and observed that 14/71 (19.72%) informative samples showed LOH, a hallmark of TS genes. Three protein truncating mutations were identified in 1/15 OSCC samples and 2/5 cancer cell lines. MCPH1 was downregulated at both the transcript and protein levels in 21/41 (51.22%) and 19/25 (76%) OSCC samples respectively. A low level of MCPH1 promoter methylation was also observed in 4/40 (10%) tumor samples. We further observed that overexpression of MCPH1 decreased cellular proliferation, anchorage-independent growth in soft agar, cell invasion and tumor size in nude mice, indicating its tumor suppressive function. Using bioinformatic approaches and luciferase assay, we showed that the 3'-UTR of MCPH1 harbors two non-overlapping functional seed regions for miR-27a which negatively regulated its level. The expression level of miR-27a negatively correlated with the MCPH1 protein level in OSCC. Our study indicates for the first time that, in addition to its role in brain development, MCPH1 also functions as a tumor suppressor gene and is regulated by miR-27a.

Clusters of bound particles in the derivative delta-function Bose gas

In this paper we discuss a novel procedure for constructing clusters of bound particles in the case of a quantum integrable derivative delta-function Bose gas in one dimension. It is shown that clusters of bound particles can be constructed for this Bose gas for some special values of the coupling constant, by taking the quasi-momenta associated with the corresponding Bethe state to be equidistant points on a single circle in the complex momentum plane. We also establish a connection between these special values of the coupling constant and some fractions belonging to the Farey sequences in number theory. This connection leads to a classification of the clusters of bound particles associated with the derivative delta-function Bose gas and allows us to study various properties of these clusters like their size and their stability under the variation of the coupling constant. (C) 2013 Elsevier B.V. All rights reserved.

A novel Q-learning algorithm with function approximation for constrained Markov decision processes

We present a novel multi-timescale Q-learning algorithm for average cost control in a Markov decision process subject to multiple inequality constraints. We formulate a relaxed version of this problem through the Lagrange multiplier method. Our algorithm is different from Q-learning in that it updates two parameters - a Q-value parameter and a policy parameter. The Q-value parameter is updated on a slower time scale as compared to the policy parameter. Whereas Q-learning with function approximation can diverge in some cases, our algorithm is seen to be convergent as a result of the aforementioned timescale separation. We show the results of experiments on a problem of constrained routing in a multistage queueing network. Our algorithm is seen to exhibit good performance and the various inequality constraints are seen to be satisfied upon convergence of the algorithm.

On the zeta function of a periodic-finite-type shift

Periodic-finite-type shifts (PFT's) are sofic shifts which forbid the appearance of finitely many pre-specified words in a periodic manner. The class of PFT's strictly includes the class of shifts of finite type (SFT's). The zeta function of a PET is a generating function for the number of periodic sequences in the shift. For a general sofic shift, there exists a formula, attributed to Manning and Bowen, which computes the zeta function of the shift from certain auxiliary graphs constructed from a presentation of the shift. In this paper, we derive an interesting alternative formula computable from certain ``word-based graphs'' constructed from the periodically-forbidden word description of the PET. The advantages of our formula over the Manning-Bowen formula are discussed.

Cl-35 NQR frequency and spin lattice relaxation time in 3,4-dichlorophenol as a function of pressure and temperature

The pressure dependences of Cl-35 nuclear quadrupole resonance (NQR) frequency, temperature and pressure variation of spin lattice relaxation time (T-1) were investigated in 3,4-dichlorophenol. T-1 was measured in the temperature range 77-300 K. Furthermore, the NQR frequency and T-1 for these compounds were measured as a function of pressure up to 5 kbar at 300 K. The temperature dependence of the average torsional lifetimes of the molecules and the transition probabilities W-1 and W-2 for the Delta m = +/- 1 and Delta m = +/- 2 transitions were also obtained. A nonlinear variation of NQR frequency with pressure has been observed and the pressure coefficients were observed to be positive. A thermodynamic analysis of the data was carried out to determine the constant volume temperature coefficients of the NQR frequency. An attempt is made to compare the torsional frequencies evaluated from NQR data with those obtained by IR spectra. On selecting the appropriate mode from IR spectra, a good agreement with torsional frequency obtained from NQR data is observed. The previously mentioned approach is a good illustration of the supplementary nature of the data from IR studies, in relation to NQR studies of compounds in solid state.

Synthesis and application of weight function for edge cracked semicircular disk (ECSD)

We analyze the utility of edge cracked semicircular disk (ECSD) for rapid assessment of fracture toughness using compressive loading. Continuing our earlier work on ECSD, a theoretical examination here leads to a novel way for synthesizing weight functions using two distinct form factors. The efficacy of ECSD mode-I weight function synthesized using displacement and form factor methods is demonstrated by comparing with finite element results. Theory of elasticity in conjunction with finite element method is utilized to analyze crack opening potency of ECSD under eccentric compression to explore newer configurations of ECSD for fracture testing.