An Online Actor-Critic Algorithm with Function Approximation for Constrained Markov Decision Processes

We develop an online actor-critic reinforcement learning algorithm with function approximation for a problem of control under inequality constraints. We consider the long-run average cost Markov decision process (MDP) framework in which both the objective and the constraint functions are suitable policy-dependent long-run averages of certain sample path functions. The Lagrange multiplier method is used to handle the inequality constraints. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal solution. We also provide the results of numerical experiments on a problem of routing in a multi-stage queueing network with constraints on long-run average queue lengths. We observe that our algorithm exhibits good performance on this setting and converges to a feasible point.

Training-Based Antenna Selection for PER Minimization: A POMDP Approach

This paper considers the problem of receive antenna selection (AS) in a multiple-antenna communication system having a single radio-frequency (RF) chain. The AS decisions are based on noisy channel estimates obtained using known pilot symbols embedded in the data packets. The goal here is to minimize the average packet error rate (PER) by exploiting the known temporal correlation of the channel. As the underlying channels are only partially observed using the pilot symbols, the problem of AS for PER minimization is cast into a partially observable Markov decision process (POMDP) framework. Under mild assumptions, the optimality of a myopic policy is established for the two-state channel case. Moreover, two heuristic AS schemes are proposed based on a weighted combination of the estimated channel states on the different antennas. These schemes utilize the continuous valued received pilot symbols to make the AS decisions, and are shown to offer performance comparable to the POMDP approach, which requires one to quantize the channel and observations to a finite set of states. The performance improvement offered by the POMDP solution and the proposed heuristic solutions relative to existing AS training-based approaches is illustrated using Monte Carlo simulations.

A building-up process in open-framework metal carboxylates that involves a progressive increase in dimensionality

Extensive research work has been carried out in the last few years on the synthesis and characterization of several families of open-framework materials, including aluminosilicates,[1] phosphates,[2] and carboxylates.[3] These studies have shown the occurrence of a variety of three dimensional (3D) architectures containing channels and other features.

Understanding the building-up process of three dimensional open-framework metal phosphates: Acid degradation of the 3D structures to lower dimensional structures

Acid degradation of 3D zinc phosphates primarily yields a one-dimensional ladder compound, an observation that is significant considering that the latter forms 3D structures on heating in water.

Different phenomenological theories and their abilities to describe the interdiffusion process in a binary system during multiphase growth

There are essentially two different phenomenological models available to describe the interdiffusion process in binary systems in the olid state. The first of these, which is used more frequently, is based on the theory of flux partitioning. The second model, developed much more recently, uses the theory of dissociation and reaction. Although the theory of flux partitioning has been widely used, we found that this theory does not account for the mobility of both species and therefore is not suitable for use in most interdiffusion systems. We have first modified this theory to take into account the mobility of both species and then further extended it to develop relations or the integrated diffusion coefficient and the ratio of diffusivities of the species. The versatility of these two different models is examined in the Co-Si system with respect to different end-member compositions. From our analysis, we found that the applicability of the theory of flux partitioning is rather limited but the theory of dissociation and reaction can be used in any binary system.

A Novel Cache Architecture and Placement Framework for Packet Forwarding Engines

Packet forwarding is a memory-intensive application requiring multiple accesses through a trie structure. With the requirement to process packets at line rates, high-performance routers need to forward millions of packets every second with each packet needing up to seven memory accesses. Earlier work shows that a single cache for the nodes of a trie can reduce the number of external memory accesses. It is observed that the locality characteristics of the level-one nodes of a trie are significantly different from those of lower level nodes. Hence, we propose a heterogeneously segmented cache architecture (HSCA) which uses separate caches for level-one and lower level nodes, each with carefully chosen sizes. Besides reducing misses, segmenting the cache allows us to focus on optimizing the more frequently accessed level-one node segment. We find that due to the nonuniform distribution of nodes among cache sets, the level-one nodes cache is susceptible t high conflict misses. We reduce conflict misses by introducing a novel two-level mapping-based cache placement framework. We also propose an elegant way to fit the modified placement function into the cache organization with minimal increase in access time. Further, we propose an attribute preserving trace generation methodology which emulates real traces and can generate traces with varying locality. Performanc results reveal that our HSCA scheme results in a 32 percent speedup in average memory access time over a unified nodes cache. Also, HSC outperforms IHARC, a cache for lookup results, with as high as a 10-fold speedup in average memory access time. Two-level mappin further enhances the performance of the base HSCA by up to 13 percent leading to an overall improvement of up to 40 percent over the unified scheme.

A solution framework for scheduling a BPM with non-identical job dimensions

A Batch Processing Machine (BPM) is one which processes a number of jobs simultaneously as a batch with common beginning and ending times. Also, a BPM, once started cannot be interrupted in between (Pre-emption not allowed). This research is motivated by a BPM in steel casting industry. There are three main stages in any steel casting industry viz., pre-casting stage, casting stage and post-casting stage. A quick overview of the entire process, is shown in Figure 1. There are two BPMs : (1) Melting furnace in the pre-casting stage and (2) Heat Treatment Furnace (HTF) in the post casting stage of steel casting manufacturing process. This study focuses on scheduling the latter, namely HTF. Heat-treatment operation is one of the most important stages of steel casting industries. It determines the final properties that enable components to perform under demanding service conditions such as large mechanical load, high temperature and anti-corrosive processing. In general, different types of castings have to undergo more than one type of heat-treatment operations, where the total heat-treatment processing times change. To have a better control, castings are primarily classified into a number of job-families based on the alloy type such as low-alloy castings and high alloy castings. For technical reasons such as type of alloy, temperature level and the expected combination of heat-treatment operations, the castings from different families can not be processed together in the same batch.

Two- and Three-Dimensional Open-Framework Uranium Arsenates: Synthesis, Structure, and Characterization

Hydrothermal reactions between uranium salts and arsenic pentoxide in the presence of two different amines yielded six new uranium arsenate phases exhibiting open-framework structures, ethylenediamine (en): [C2N2H9]-[(UO2)(ASO(4))] I; [C2N2H10][(UO2)F(HASO(4))]2 center dot 4H(2)O, II; [C2N2H9][U2F5(HASO(4))(2)], III; [C2N2H9][UF2(ASO(4))], IV; diethylenetriamine (DETA), [C4N3H16][U2F3(ASO(4))(2)(HAsO4)] V; and [C4N3H16][U2F6(AsO4)(HAsO4)], VI. The structures were determined using single crystal studies, which revealed two- (I, II, V) and three-dimensional (III, IV, VI) structures for the uranium arsenates. The uranium atom, in these compounds, exhibits considerable variations in the coordination (6 to 9) that appears to have some correlation with the synthetic conditions. The water molecules in [C2N2H10][(UO2)F(HAsO4)](2 center dot)4H(2)O, II, could be reversibly removed, and the dehydrated phase, [C2N2H10][(UO2)F(HAsO4)](2), IIa, was also characterized using single crystal studies. The observation of many mineralogical structures in the present compounds suggests that the hydrothermal method could successfully replicate the geothermal conditions. As part of this study, we have observed autunite, Ca[(UO2)(PO4)](2)(H2O)(11), metavauxite, [Fe(H2O)(6)][Al(OH)(H2O)(PO4)](2), finarite, PbCU(SO4)(OH)(2), and tancoite, LiNa2H[Al(PO4)(2)(OH)], structures. The repeated observation of the secondary building unit, SBU-4, in many of the uranium arsenate structures suggests that these are viable building units. Optical studies on the uranium arsenate compound, [C4N3H16][U2F6(AsO4)(HASO(4))), VI, containing uranium in the +4 oxidation state indicates a blue emission through an upconversion process. The compound also exhibits antiferromagnetic behavior.

A general thermodynamic framework for understanding the behaviour of absorption chillers

In this article, a general definition of the process average temperature has been developed, and the impact of the various dissipative mechanisms on 1/COP of the chiller evaluated. The present component-by-component black box analysis removes the assumptions regarding the generator outlet temperature(s) and the component effective thermal conductances. Mass transfer resistance is also incorporated into the absorber analysis to arrive at a more realistic upper limit to the cooling capacity. Finally, the theoretical foundation for the absorption chiller T-s diagram is derived. This diagrammatic approach only requires the inlet and outlet conditions of the chiller components and can be employed as a practical tool for system analysis and comparison. (C) 2000 Elsevier Science Ltd and IIR. All rights reserved.

Voltage and Temperature Aware Statistical Leakage Analysis Framework Using Artificial Neural Networks

Artificial neural networks (ANNs) have shown great promise in modeling circuit parameters for computer aided design applications. Leakage currents, which depend on process parameters, supply voltage and temperature can be modeled accurately with ANNs. However, the complex nature of the ANN model, with the standard sigmoidal activation functions, does not allow analytical expressions for its mean and variance. We propose the use of a new activation function that allows us to derive an analytical expression for the mean and a semi-analytical expression for the variance of the ANN-based leakage model. To the best of our knowledge this is the first result in this direction. Our neural network model also includes the voltage and temperature as input parameters, thereby enabling voltage and temperature aware statistical leakage analysis (SLA). All existing SLA frameworks are closely tied to the exponential polynomial leakage model and hence fail to work with sophisticated ANN models. In this paper, we also set up an SLA framework that can efficiently work with these ANN models. Results show that the cumulative distribution function of leakage current of ISCAS'85 circuits can be predicted accurately with the error in mean and standard deviation, compared to Monte Carlo-based simulations, being less than 1% and 2% respectively across a range of voltage and temperature values.

A novel filtering framework through Girsanov correction for the identification of nonlinear dynamical systems

Using a Girsanov change of measures, we propose novel variations within a particle-filtering algorithm, as applied to the inverse problem of state and parameter estimations of nonlinear dynamical systems of engineering interest, toward weakly correcting for the linearization or integration errors that almost invariably occur whilst numerically propagating the process dynamics, typically governed by nonlinear stochastic differential equations (SDEs). Specifically, the correction for linearization, provided by the likelihood or the Radon-Nikodym derivative, is incorporated within the evolving flow in two steps. Once the likelihood, an exponential martingale, is split into a product of two factors, correction owing to the first factor is implemented via rejection sampling in the first step. The second factor, which is directly computable, is accounted for via two different schemes, one employing resampling and the other using a gain-weighted innovation term added to the drift field of the process dynamics thereby overcoming the problem of sample dispersion posed by resampling. The proposed strategies, employed as add-ons to existing particle filters, the bootstrap and auxiliary SIR filters in this work, are found to non-trivially improve the convergence and accuracy of the estimates and also yield reduced mean square errors of such estimates vis-a-vis those obtained through the parent-filtering schemes.

Heat and mass transfer and chemical transformation in a cerium nitrate droplet

This paper deals with the thermo-physical changes that a droplet undergoes when it is radiatively heated in a levitated environment. The heat and mass transport model has been developed along with chemical kinetics within a cerium nitrate droplet. The chemical transformation of cerium nitrate to ceria during the process is predicted using Kramers' reaction mechanism which justifies the formation of ceria at a very low temperature as observed in experiments. The rate equation modeled by Kramers is modified suitably to be applicable within the framework of a droplet, and predicts experimental results well in both bulk form of cerium nitrate and in aqueous cerium nitrate droplet. The dependence of dissociation reaction rate on droplet size is determined and the transient mass concentration of unreacted cerium nitrate is reported. The model is validated with experiments both for liquid phase vaporization and chemical reaction. Vaporization and chemical conversion are simulated for different ambient conditions. The competitive effects of sensible heating rate and the rate of vaporization with diffusion of cerium nitrate is seen to play a key role in determining the mass fraction of ceria formed within the droplet. Spatially resolved modeling of the droplet enables the understanding of the conversion of chemical species in more detail.

Multivariate Granger causality: an estimation framework based on factorization of the spectral density matrix

Granger causality is increasingly being applied to multi-electrode neurophysiological and functional imaging data to characterize directional interactions between neurons and brain regions. For a multivariate dataset, one might be interested in different subsets of the recorded neurons or brain regions. According to the current estimation framework, for each subset, one conducts a separate autoregressive model fitting process, introducing the potential for unwanted variability and uncertainty. In this paper, we propose a multivariate framework for estimating Granger causality. It is based on spectral density matrix factorization and offers the advantage that the estimation of such a matrix needs to be done only once for the entire multivariate dataset. For any subset of recorded data, Granger causality can be calculated through factorizing the appropriate submatrix of the overall spectral density matrix.

A framework for post-silicon realization of arbitrary instruction extensions on reconfigurable data-paths

In this paper we present a framework for realizing arbitrary instruction set extensions (IE) that are identified post-silicon. The proposed framework has two components viz., an IE synthesis methodology and the architecture of a reconfigurable data-path for realization of the such IEs. The IE synthesis methodology ensures maximal utilization of resources on the reconfigurable data-path. In this context we present the techniques used to realize IEs for applications that demand high throughput or those that must process data streams. The reconfigurable hardware called HyperCell comprises a reconfigurable execution fabric. The fabric is a collection of interconnected compute units. A typical use case of HyperCell is where it acts as a co-processor with a host and accelerates execution of IEs that are defined post-silicon. We demonstrate the effectiveness of our approach by evaluating the performance of some well-known integer kernels that are realized as IEs on HyperCell. Our methodology for realizing IEs through HyperCells permits overlapping of potentially all memory transactions with computations. We show significant improvement in performance for streaming applications over general purpose processor based solutions, by fully pipelining the data-path. (C) 2014 Elsevier B.V. All rights reserved.

Estimating successive pK(a) values of polyprotic acids from ab initio molecular dynamics using metadynamics: the dissociation of phthalic acid and its isomers

Estimation of the dissociation constant, or pK(a), of weak acids continues to be a central goal in theoretical chemistry. Here we show that ab initio Car-Parrinello molecular dynamics simulations in conjunction with metadynamics calculations of the free energy profile of the dissociation reaction can provide reasonable estimates of the successive pK(a) values of polyprotic acids. We use the distance-dependent coordination number of the protons bound to the hydroxyl oxygen of the carboxylic group as the collective variable to explore the free energy profile of the dissociation process. Water molecules, sufficient to complete three hydration shells surrounding the acid molecule, were included explicitly in the computation procedure. Two distinct minima corresponding to the dissociated and un-dissociated states of the acid are observed and the difference in their free energy values provides the estimate for pK(a), the acid dissociation constant. We show that the method predicts the pK(a) value of benzoic acid in good agreement with experiment and then show using phthalic acid (benzene dicarboxylic acid) as a test system that both the first and second pK(a) values as well, as the subtle difference in their values for different isomers can be predicted in reasonable agreement with experimental data.