An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

MS_RHII-RSD, a Dual-Function RNase HII-(p)ppGpp Synthetase from Mycobacterium smegmatis

In the noninfectious soil saprophyte Mycobacterium smegmatis, intracellular levels of the stress alarmones guanosine tetraphosphate and guanosine pentaphosphate, together termed (p)ppGpp, are regulated by the enzyme Rel(Msm). This enzyme consists of a single, bifunctional polypeptide chain that is capable of both synthesizing and hydrolyzing (p)ppGpp. The rel(Msm), knockout strain of M. smegmatis (Delta rel(Msm)) is expected to show a (p)ppGpp null (p)ppGpp(0)] phenotype. Contrary to this expectation, the strain is capable of synthesizing (p)ppGpp in vivo. In this study, we identify and functionally characterize the open reading frame (ORF), MSMEG_5849, that encodes a second functional (p)ppGpp synthetase in M. smegmatis. In addition to (p)ppGpp synthesis, the 567-amino-acid-long protein encoded by this gene is capable of hydrolyzing RNA(.)DNA hybrids and bears similarity to the conventional RNase HII enzymes. We have classified this protein as actRel(Msm) in accordance with the recent nomenclature proposed and have named it MS_RHII-RSD, indicating the two enzymatic activities present RHII, RNase HII domain, originally identified as (d) under bar omain of (u) under bar nknown (f) under bar unction 429 (DUF429), and RSD, RelA_SpoT nucleotidyl transferase domain, the SYNTH domain responsible for (p)ppGpp synthesis activity]. MS_RHII-RSD is expressed and is constitutively active in vivo and behaves like a monofunctional (p)ppGpp synthetase in vitro. The occurrence of the RNase HII and (p)ppGpp synthetase domains together on the same polypeptide chain is suggestive of an in vivo role for this novel protein as a link connecting the essential life processes of DNA replication, repair, and transcription to the highly conserved stress survival pathway, the stringent response.

Tradeoff of average power and average delay for a point-to-point link with fading

We consider a discrete time system with packets arriving randomly at rate lambda per slot to a fading point-to-point link, for which the transmitter can control the number of packets served in a slot by varying the transmit power. We provide an asymptotic characterization of the minimum average delay of the packets, when average transmitter power is a small positive quantity V more than the minimum average power required for queue stability. We show that the minimum average delay will grow either as log (1/V) or 1/V when V down arrow 0, for certain sets of values of lambda. These sets are determined by the distribution of fading gain, the maximum number of packets which can be transmitted in a slot, and the assumed transmit power function, as a function of the fading gain and the number of packets transmitted. We identify a case where the above behaviour of the tradeoff differs from that obtained from a previously considered model, in which the random queue length process is assumed to evolve on the non-negative real line.

Unraveling the Intricate Organization of Mammalian Mitochondrial Presequence Translocases: Existence of Multiple Translocases for Maintenance of Mitochondrial Function

Mitochondria are indispensable organelles implicated in multiple aspects of cellular processes, including tumorigenesis. Heat shock proteins play a critical regulatory role in accurately delivering the nucleus-encoded proteins through membrane-bound presequence translocase (Tim23 complex) machinery. Although altered expression of mammalian presequence translocase components had been previously associated with malignant phenotypes, the overall organization of Tim23 complexes is still unsolved. In this report, we show the existence of three distinct Tim23 complexes, namely, B1, B2, and A, involved in the maintenance of normal mitochondrial function. Our data highlight the importance of Magmas as a regulator of translocase function and in dynamically recruiting the J-proteins DnaJC19 and DnaJC15 to individual translocases. The basic housekeeping function involves translocases B1 and B2 composed of Tim17b isoforms along with DnaJC19, whereas translocase A is nonessential and has a central role in oncogenesis. Translocase B, having a normal import rate, is essential for constitutive mitochondrial functions such as maintenance of electron transport chain complex activity, organellar morphology, iron-sulfur cluster protein biogenesis, and mitochondrial DNA. In contrast, translocase A, though dispensable for housekeeping functions with a comparatively lower import rate, plays a specific role in translocating oncoproteins lacking presequence, leading to reprogrammed mitochondrial functions and hence establishing a possible link between the TIM23 complex and tumorigenicity.

Blocking central opiate function modulates hedonic impact and anterior cingulate response to rewards and losses.

Reward processing is linked to specific neuromodulatory systems with a dopaminergic contribution to reward learning and motivational drive being well established. Neuromodulatory influences on hedonic responses to actual receipt of reward, or punishment, referred to as experienced utility are less well characterized, although a link to the endogenous opioid system is suggested. Here, in a combined functional magnetic resonance imaging-psychopharmacological investigation, we used naloxone to block central opioid function while subjects performed a gambling task associated with rewards and losses of different magnitudes, in which the mean expected value was always zero. A graded influence of naloxone on reward outcome was evident in an attenuation of pleasure ratings for larger reward outcomes, an effect mirrored in attenuation of brain activity to increasing reward magnitude in rostral anterior cingulate cortex. A more striking effect was seen for losses such that under naloxone all levels of negative outcome were rated as more unpleasant. This hedonic effect was associated with enhanced activity in anterior insula and caudal anterior cingulate cortex, areas implicated in aversive processing. Our data indicate that a central opioid system contributes to both reward and loss processing in humans and directly modulates the hedonic experience of outcomes.

Spectroscopic investigation of the function of aqueous 2-hydroxyethylmethacrylate/glutaraldehyde solution as a dentin desensitizer

Fourier-transform (FT)-Raman and -infrared (IR) spectroscopy were employed to investigate the function of the aqueous 2-hydroxyethylmethacrylate/glutaraldehyde solution (Gluma) as a desensitizer. 2-Hydroxyethylmethacrylate (HEMA), glutaraldehyde (GA), and the mixture of HEMA/GA (i.e. Gluma) were used to interact with dentin, collagen, hydroxyapatite (HAP), and bovine serum albumin (BSA) individually. All the interactions were monitored by an FT-Raman spectrometer. FT-IR spectroscopy was also used in this study. The results show that HEMA could be absorbed by dentin and collagen; GA could cross-link collagen and BSA; and when BSA was added to Gluma, polymerization of HEMA occurred. The results suggest that Gluma acts as a desensitizer whereby, first, GA reacts with part of the serum albumin in dentinal fluid, which induces a precipitation of serum albumin, then, second, a reaction of GA with serum albumin induces polymerization of HEMA. The function of Gluma as a desensitizer to block dentinal tubules occurs via these two reactions.

Semi-Supervised OWA Aggregation for Link-Based Similarity Evaluation and Alias Detection

T.Boongoen and Q. Shen. Semi-Supervised OWA Aggregation for Link-Based Similarity Evaluation and Alias Detection. Proceedings of the 18th International Conference on Fuzzy Systems (FUZZ-IEEE'09), pp. 288-293, 2009. Sponsorship: EPSRC

The function of the self-attention network

This commentary links Humphrey and Sui’s proposed Self-attention Network (SAN) to the memory advantage associated with self-relevant information (i.e., the self-reference effect). Articulating this link elucidates the functional quality of the SAN in ensuring that information of potential importance to self is not lost. This adaptive system for self-processing mirrors the cognitive response to threat stimuli, which also elicit attentional biases and produce characteristically enhanced, episodic representations in memory. Understanding the link between the SAN and memory is key to comprehending more broadly the operation of the self in cognition.

Leptin metabolically licenses T cells for activation to link nutrition and immunity.

Immune responses are highly energy-dependent processes. Activated T cells increase glucose uptake and aerobic glycolysis to survive and function. Malnutrition and starvation limit nutrients and are associated with immune deficiency and increased susceptibility to infection. Although it is clear that immunity is suppressed in times of nutrient stress, mechanisms that link systemic nutrition to T cell function are poorly understood. We show in this study that fasting leads to persistent defects in T cell activation and metabolism, as T cells from fasted animals had low glucose uptake and decreased ability to produce inflammatory cytokines, even when stimulated in nutrient-rich media. To explore the mechanism of this long-lasting T cell metabolic defect, we examined leptin, an adipokine reduced in fasting that regulates systemic metabolism and promotes effector T cell function. We show that leptin is essential for activated T cells to upregulate glucose uptake and metabolism. This effect was cell intrinsic and specific to activated effector T cells, as naive T cells and regulatory T cells did not require leptin for metabolic regulation. Importantly, either leptin addition to cultured T cells from fasted animals or leptin injections to fasting animals was sufficient to rescue both T cell metabolic and functional defects. Leptin-mediated metabolic regulation was critical, as transgenic expression of the glucose transporter Glut1 rescued cytokine production of T cells from fasted mice. Together, these data demonstrate that induction of T cell metabolism upon activation is dependent on systemic nutritional status, and leptin links adipocytes to metabolically license activated T cells in states of nutritional sufficiency.

ROLE OF ICAM-1-MEDIATED ENDOCYTOSIS IN ENDOTHELIAL FUNCTION AND IMPLICATIONS FOR CARRIER-ASSISTED DRUG DELIVERY

Intercellular adhesion molecule 1 (ICAM-1) is a transmembrane protein found on the surface of vascular endothelial cells (ECs). Its expression is upregulated at inflammatory sites, allowing for targeted delivery of therapeutics using ICAM-1-binding drug carriers. Engagement of multiple copies of ICAM-1 by these drug carriers induces cell adhesion molecule (CAM)-mediated endocytosis, which results in trafficking of carriers to lysosomes and across ECs. Knowledge about the regulation behind CAM-mediated endocytosis can help improve drug delivery, but questions remain about these regulatory mechanisms. Furthermore, little is known about the natural function of this endocytic pathway. To address these gaps in knowledge, we focused on two natural binding partners of ICAM-1 that potentially elicit CAM-mediated endocytosis: leukocytes (which bind ICAM-1 via β₂ integrins) and fibrin polymers (a main component of blood clots which binds ICAM-1 via the γ3 sequence). First, inspired by properties of these natural binding partners, we varied the size and targeting moiety of model drug carriers to determine how these parameters affect CAM-mediated endocytosis. Increasing ICAM-1-targeted carrier size slowed carrier uptake kinetics, reduced carrier trafficking to lysosomes, and increased carrier transport across ECs. Changing targeting moieties from antibodies to peptides decreased particle binding and uptake, lowered trafficking to lysosomes, and increased transport across ECs. Second, using cell culture models of leukocyte/EC interactions, inhibiting regulatory elements of the CAM-mediated pathway disrupted leukocyte sampling, a process crucial to leukocyte crossing of endothelial layers (transmigration). This inhibition also decreased leukocyte transmigration across ECs, specifically through the transcellular route, which occurs through a single EC without disassembly of cell-cell junctions. Third, fibrin meshes, which mimic blood clot fragments/remnants, bound to ECs at ICAM-1-enriched sites and were internalized by the endothelium. Inhibiting the CAM-mediated pathway disrupted this uptake. Following endocytosis, fibrin meshes trafficked to lysosomes where they were degraded. In mouse models, CAM-mediated endocytosis of fibrin meshes appeared to remove fibrin remnants at the endothelial surface, preventing re-initiation of the coagulation cascade. Overall, these results support a link between CAM-mediated endocytosis and leukocyte transmigration as well as uptake of fibrin materials by ECs. Furthermore, these results will guide the future design of ICAM-1-targeted carrier-assisted therapies.

The Impact of Polystyrene Microplastics on Feeding, Function and Fecundity in the Marine CopepodCalanus helgolandicus

Microscopic plastic debris, termed “microplastics”, are of increasing environmental concern. Recent studies have demonstrated that a range of zooplankton, including copepods, can ingest microplastics. Copepods are a globally abundant class of zooplankton that form a key trophic link between primary producers and higher trophic marine organisms. Here we demonstrate that ingestion of microplastics can significantly alter the feeding capacity of the pelagic copepod Calanus helgolandicus. Exposed to 20 μm polystyrene beads (75 microplastics mL–1) and cultured algae ([250 μg C L–1) for 24 h, C. helgolandicus ingested 11% fewer algal cells (P = 0.33) and 40% less carbon biomass (P < 0.01). There was a net downward shift in the mean size of algal prey consumed (P < 0.001), with a 3.6 fold increase in ingestion rate for the smallest size class of algal prey (11.6–12.6 μm), suggestive of postcapture or postingestion rejection. Prolonged exposure to polystyrene microplastics significantly decreased reproductive output, but there were no significant differences in egg production rates, respiration or survival. We constructed a conceptual energetic (carbon) budget showing that microplastic-exposed copepods suffer energetic depletion over time. We conclude that microplastics impede feeding in copepods, which over time could lead to sustained reductions in ingested carbon biomass.

Clinical traits of LRRK2-associated parkinson's disease in ireland: a link between familial and idiopathic PD

The role of genetics in parkinsonism has been confirmed over the last decade with the identification of genetic variation in seven genes, which are causative in familial forms of the disorder. A number of pathogenic mutations have been identified in the latest gene LRRK2, with a Gly2019Ser amino acid substitution identified in two siblings and one patient with idiopathic Parkinson's disease from Ireland. The clinical features resemble the idiopathic variant with a tremor predominant clinical picture shared by the siblings, slow progression of symptoms, and no observation of cognitive disturbance in all. The family and the sporadic individual were apparently not related and originated from different regions of Ireland, although haplotype analysis does suggest they share a common founder. The influence of the G2019S substitution on protein function and disease phenotype has yet to be fully resolved, but its elucidation will undoubtedly further our understanding of the mechanisms underlying Parkinson's disease.

Comparing whole-link travel time models used in DTA

In a model commonly used in dynamic traffic assignment the link travel time for a vehicle entering a link at time t is taken as a function of the number of vehicles on the link at time t. In an alternative recently introduced model, the travel time for a vehicle entering a link at time t is taken as a function of an estimate of the flow in the immediate neighbourhood of the vehicle, averaged over the time the vehicle is traversing the link. Here we compare the solutions obtained from these two models when applied to various inflow profiles. We also divide the link into segments, apply each model sequentially to the segments and again compare the results. As the number of segments is increased, the discretisation refined to the continuous limit, the solutions from the two models converge to the same solution, which is the solution of the Lighthill, Whitham, Richards (LWR) model for traffic flow. We illustrate the results for different travel time functions and patterns of inflows to the link. In the numerical examples the solutions from the second of the two models are closer to the limit solutions. We also show that the models converge even when the link segments are not homogeneous, and introduce a correction scheme in the second model to compensate for an approximation error, hence improving the approximation to the LWR model.

Selection for cuticular melanism reveals immune function and life-history trade-offs in Spodoptera littoralis

Several insect species show an increase in cuticular melanism in response to high densities. In some species, there is evidence that this melanism is correlated with an up-regulation of certain immune system components, particularly phenoloxidase (PO) activity, and with the down-regulation of lysozyme activity, suggesting a trade-off between the two traits. As melanism has a genetic component, we selected both melanic and nonmelanic lines of the phase-polyphenic lepidopteran, Spodoptera littoralis, in order to test for a causative genetic link between melanism, PO activity and lysozyme activity, and to establish if there are any life-history costs associated with the melanic response. We found that, in fact, melanic lines had lower PO activity and higher lysozyme activity than nonmelanic lines, confirming a genetic trade-off between the two immune responses, but also indicating a genetic trade-off between melanism and PO activity. In addition, we found that lines with high PO activity had slower development rates suggesting that investment in PO, rather than in melanism, is costly.

Estimating trophic link density from quantitative but incomplete diet data

The trophic link density and the stability of food webs are thought to be related, but the nature of this relation is controversial. This article introduces a method for estimating the link density from diet tables which do not cover the complete food web and do not resolve all diet items to species level. A simple formula for the error of this estimate is derived. Link density is determined as a function of a threshold diet fraction below which diet items are ignored (