An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes

We develop in this article the first actor-critic reinforcement learning algorithm with function approximation for a problem of control under multiple inequality constraints. We consider the infinite horizon discounted cost framework in which both the objective and the constraint functions are suitable expected policy-dependent discounted sums of certain sample path functions. We apply the Lagrange multiplier method to handle the inequality constraints. Our algorithm makes use of multi-timescale stochastic approximation and incorporates a temporal difference (TD) critic and an actor that makes a gradient search in the space of policy parameters using efficient simultaneous perturbation stochastic approximation (SPSA) gradient estimates. We prove the asymptotic almost sure convergence of our algorithm to a locally optimal policy. (C) 2010 Elsevier B.V. All rights reserved.

Study Of Acoustic Source Localization Algorithm For Planar Arrays

The source localization algorithms in the earlier works, mostly used non-planar arrays. If we consider scenarios like human-computer communication, or human-television communication where the microphones need to be placed on the computer monitor or television front panel, i.e we need to use the planar arrays. The algorithm proposed in 1], is a Linear Closed Form source localization algorithm (LCF algorithm) which is based on Time Difference of Arrivals (TDOAs) that are obtained from the data collected using the microphones. It assumes non-planar arrays. The LCF algorithm is applied to planar arrays in the current work. The relationship between the error in the source location estimate and the perturbation in the TDOAs is derived using first order perturbation analysis and validated using simulations. If the TDOAs are erroneous, both the coefficient matrix and the data matrix used for obtaining source location will be perturbed. So, the Total least squares solution for source localization is proposed in the current work. The sensitivity analysis of the source localization algorithm for planar arrays and non-planar arrays is done by introducing perturbation in the TDOAs and the microphone locations. It is shown that the error in the source location estimate is less when we use planar array instead of the particular non-planar array considered for same perturbation in the TDOAs or microphone location. The location of the reference microphone is proved to be important for getting an accurate source location estimate if we are using the LCF algorithm.

Effect of heat-treatment on strength of clays

The main objective of this investigation was to understand the strength development of clays below fusion or vitrification temperatures of 900°C. The other objective was to establish threshold temperatures to produce a satisfactory construction material from clayey sediments from the Western Beaufort Sea for shore protection of artificial islands with minimum expense of thermal energy. Studies were, therefore, conducted using kaolinite, bentonite, and a clayey sediment from the Beaufort Sea. Unconfined-compressive-strength tests were conducted on clay samples heat treated from 110 to 700°C. Furthermore, to understand the factors responsible for strength-development-thermogravimetric studies and pore-size analysis, using mercury porosimetry, were also conducted. A gradual increase in strength was obtained with an increase in firing temperature. However, substantial and permanent increase in strength occurred only after dehydroxylation of all the clays studied; Clay samples heated to temperatures above dehydroxylation became resistant to disintegration upon immersion in water. Results indicate that the clayey sediments from Western Beaufort Sea have to be heat treated to about 600°C to produce granular material for use as a fill or shore-protection material for artificial islands.

Multi-Agent Rendezvous Algorithm with Rectilinear Decision Domain

The aim of this paper is to develop a computationally efficient decentralized rendezvous algorithm for a group of autonomous agents. The algorithm generalizes the notion of sensor domain and decision domain of agents to enable implementation of simple computational algorithms. Specifically, the algorithm proposed in this paper uses a rectilinear decision domain (RDD) as against the circular decision domain assumed in earlier work. Because of this, the computational complexity of the algorithm reduces considerably and, when compared to the standard Ando's algorithm available in the literature, the RDD algorithm shows very significant improvement in convergence time performance. Analytical results to prove convergence and supporting simulation results are presented in the paper.

Layered Tabu Search Algorithm for Large-MIMO Detection and a Lower Bound on ML Performance

In this paper, we are concerned with low-complexity detection in large multiple-input multiple-output (MIMO) systems with tens of transmit/receive antennas. Our new contributions in this paper are two-fold. First, we propose a low-complexity algorithm for large-MIMO detection based on a layered low-complexity local neighborhood search. Second, we obtain a lower bound on the maximum-likelihood (ML) bit error performance using the local neighborhood search. The advantages of the proposed ML lower bound are i) it is easily obtained for MIMO systems with large number of antennas because of the inherent low complexity of the search algorithm, ii) it is tight at moderate-to-high SNRs, and iii) it can be tightened at low SNRs by increasing the number of symbols in the neighborhood definition. Interestingly, the proposed detection algorithm based on the layered local search achieves bit error performances which are quite close to this lower bound for large number of antennas and higher-order QAM. For e. g., in a 32 x 32 V-BLAST MIMO system, the proposed detection algorithm performs close to within 1.7 dB of the proposed ML lower bound at 10(-3) BER for 16-QAM (128 bps/Hz), and close to within 4.5 dB of the bound for 64-QAM (192 bps/Hz).

Performance of a pipelined ring algorithm for Fast Fourier Transform on transputer arrays

In this paper a pipelined ring algorithm is presented for efficient computation of one and two dimensional Fast Fourier Transform (FFT) on a message passing multiprocessor. The algorithm has been implemented on a transputer based system and experiments reveal that the algorithm is very efficient. A model for analysing the performance of the algorithm is developed from its computation-communication characteristics. Expressions for execution time, speedup and efficiency are obtained and these expressions are validated with experimental results obtained on a four transputer system. The analytical model is then used to estimate the performance of the algorithm for different number of processors, and for different sizes of the input data.

Stabilizing trench's algorithm to invert symmetric toeplitz matrices

Presented here is a stable algorithm that uses Zohar's formulation of Trench's algorithm and computes the inverse of a symmetric Toeplitz matrix including those with vanishing or nearvanishing leading minors. The algorithm is based on a diagonal modification of the matrix, and exploits symmetry and persymmetry properties of the inverse matrix.

Effective drug dosage design for treatment of infectious diseases using dynamic inversion

A generic nonlinear mathematical model describing the human immunological dynamics is used to design an effective automatic drug administration scheme. Even though the model describes the effects of various drugs on the dynamic system, this work is confined to the drugs that kill the invading pathogen and heal the affected organ. From a system theoretic point of view, the drug inputs can be interpreted as control inputs, which can be designed based on control theoretic concepts. The controller is designed based on the principle of dynamic inversion and is found to be effective in curing the �nominal model patient� by killing the invading microbes and healing the damaged organ. A major advantage of this technique is that it leads to a closed-form state feedback form of control. It is also proved from a rigorous mathematical analysis that the internal dynamics of the system remains stable when the proposed controller is applied. A robustness study is also carried out for testing the effectiveness of the drug administration scheme for parameter uncertainties. It is observed from simulation studies that the technique has adequate robustness for many �realistic model patients� having off-nominal parameter values as well.

Integral treatment for the representation of thermodynamic properties in multicomponent systems using interaction parameters

The partial thermodynamic functions of the solvent component of a ternary system have been deduced in terms of the interaction parameters by integration of several series which emerge from the Maclaurin infinite series based on the integral property of the system and subjected to appropriate boundary conditions. The series integration shows that the resulting partial functions are suitable for interpreting the thermodynamic properties of the system and are independent of compositional paths. In the present analysis, the higher order terms of these series are found to make insignificant contributions.

Fast algorithm for data exchange in reconfigurable tree structures

This paper presents a fast algorithm for data exchange in a network of processors organized as a reconfigurable tree structure. For a given data exchange table, the algorithm generates a sequence of tree configurations in which the data exchanges are to be executed. A significant feature of the algorithm is that each exchange is executed in a tree configuration in which the source and destination nodes are adjacent to each other. It has been proved in a theorem that for every pair of nodes in the reconfigurable tree structure, there always exists two and only two configurations in which these two nodes are adjacent to each other. The algorithm utilizes this fact and determines the solution so as to optimize both the number of configurations required and the time to perform the data exchanges. Analysis of the algorithm shows that it has linear time complexity, and provides a large reduction in run-time as compared to a previously proposed algorithm. This is well-confirmed from the experimental results obtained by executing a large number of randomly-generated data exchange tables. Another significant feature of the algorithm is that the bit-size of the routing information code is always two bits, irrespective of the number of nodes in the tree. This not only increases the speed of the algorithm but also results in simpler hardware inside each node.

Pipelined ring algorithm for matrix multiplication on transputer networks: performance analysis and estimation

A parallel matrix multiplication algorithm is presented, and studies of its performance and estimation are discussed. The algorithm is implemented on a network of transputers connected in a ring topology. An efficient scheme for partitioning the input matrices is introduced which enables overlapping computation with communication. This makes the algorithm achieve near-ideal speed-up for reasonably large matrices. Analytical expressions for the execution time of the algorithm have been derived by analysing its computation and communication characteristics. These expressions are validated by comparing the theoretical results of the performance with the experimental values obtained on a four-transputer network for both square and irregular matrices. The analytical model is also used to estimate the performance of the algorithm for a varying number of transputers and varying problem sizes. Although the algorithm is implemented on transputers, the methodology and the partitioning scheme presented in this paper are quite general and can be implemented on other processors which have the capability of overlapping computation with communication. The equations for performance prediction can also be extended to other multiprocessor systems.

Passivation of surface and bulk defects in p-GaSb by hydrogenated amorphous silicon treatment

Passivation of point and extended defects in GaSb has been observed as a result of hydrogenated amorphous silicon (a-Si:H) treatment by the glow discharge technique. Cathodoluminescence (CL) images recorded at various depths in the samples clearly show passivation of defects on the surface as well as in the bulk region. The passivation of various recombination centers in the bulk is attributed to the formation of hydrogen-impurity complexes by diffusion of hydrogen ions from the plasma a-Si:H acts as a protective cap layer and prevents surface degradation which is usually encountered by bare exposure to hydrogen plasma. An enhancement in luminescence intensity up to 20 times is seen due to the passivation of nonradiative recombination centers. The passivation efficiency is found to improve with an increase in a-Si:H deposition temperature. The relative passivation efficiency of donors and acceptors by hydrogen in undoped and Te-compensated p-GaSb has been evaluated by CL and by the temperature dependence of photoluminescence intensities. Most notably, effective passivation of minority dopants in tellurium compensated p-GaSb is evidenced for the first time. (C) 1996 American Institute of Physics.

Self-consistent microscopic treatment of the effects of self-motion of the probe on ionic and dipolar solvation dynamics

A simple but self-consistent microscopic theory for the time dependent solvation energy of both ions and dipoles is presented which includes, for the first time, the details of the self-motion of the probe on its own solvation dynamics. The theory leads to several interesting predictions. The most important of them is that, for dipolar solvation, both the rotational and the translational motions of the dipolar solute probe can significantly accelerate the rate of solvation. In addition, the rotational self-motion of the solute can also give rise to an additional mechanism of nonexponentiality in solvation time correlation functions in otherwise slow liquids. A comparison between the present theoretical predictions and the recent experimental studies of Maroncelli et al. on solvation dynamics of aniline in l-propanol seems to indicate that the said experiments have missed the initial solvent response up to about 45 ps. After mapping the experimental results on the redefined time scale, the theoretical results can explain the experimental results for solvation of aniline in 1-propanol very well. For ionic solvation, the translational motion is significant for light solutes only. For example, for Li+ in water, translational motion speeds up the solvation by about 20%. The present theory demonstrates that in dipolar solvation the partial quenching of the self-motion due to the presence of specific solute-solvent interactions (such as H-bonding) may lead to a much slower solvation than that when the self-motion is present. This point has been discussed. In addition, we present the theoretical results for solvation of aniline in propylene carbonate, Here, the solvation is predicted to be complete within 15-20 ps.

Treatment of electroplating effluent through emulsion-free liquid embrane

Separation of dissolved heavy metals such-as Cr(VI) and Cu(II) from electroplating effluents using a new technique of emulsion-free liquid membrane (EFLM) has been studied. Experimental results show that nearly 95% extraction is obtained resulting in stripping phase enrichment up to 50 times relative to feed. It is also found that emulsion-free liquid membranes are highly efficient and superior to other types of liquid membranes.

An Efficient Robust Algorithm for the Surface-Potential Calculation of Independent DG MOSFET

Although the recently proposed single-implicit-equation-based input voltage equations (IVEs) for the independent double-gate (IDG) MOSFET promise faster computation time than the earlier proposed coupled-equations-based IVEs, it is not clear how those equations could be solved inside a circuit simulator as the conventional Newton-Raphson (NR)-based root finding method will not always converge due to the presence of discontinuity at the G-zero point (GZP) and nonremovable singularities in the trigonometric IVE. In this paper, we propose a unique algorithm to solve those IVEs, which combines the Ridders algorithm with the NR-based technique in order to provide assured convergence for any bias conditions. Studying the IDG MOSFET operation carefully, we apply an optimized initial guess to the NR component and a minimized solution space to the Ridders component in order to achieve rapid convergence, which is very important for circuit simulation. To reduce the computation budget further, we propose a new closed-form solution of the IVEs in the near vicinity of the GZP. The proposed algorithm is tested with different device parameters in the extended range of bias conditions and successfully implemented in a commercial circuit simulator through its Verilog-A interface.