Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

The nature of the charge-ordered state in Y0.5Ca0.5MnO3 with a very small average radius of the A-site cations

A detailed investigation of Y0.5Ca0.5MnO3 with a very small radius of the A-site cations ([r(A)] approximate to 1.13 Angstrom reveals the occurrence of a charge-ordering transition in the paramagnetic state, at a relatively high temperature of 260 K. The orthorhombic lattice distortion, as measured by the dimensionless index D, is large (similar to 1.75%) over the entire 300-100 K range, but the antiferromagnetic interactions become prominent only at low temperatures (< 160 K). The charge-ordering gap in Y0.5Ca0.5MnO3, measured by low-temperature vacuum tunnelling spectroscopy, is large (similar to 0.5 eV) and the charge-ordered state is unaffected by the application of a magnetic field of 6 T. The study indicates that the nature of charge-ordering in Y0.5Ca0.5MnO3 which is dominated by the cooperative Jahn-Teller effect and the associated lattice distortion is distinctly different from analogous manganates with larger [r(A)].

Electric-field-dependent average resistance and the resistance fluctuation in one-dimensional disordered systems

Following an invariant-imbedding approach, we obtain analytical expressions for the ensemble-averaged resistance (ρ) and its Sinai’s fluctuations for a one-dimensional disordered conductor in the presence of a finite electric field F. The mean resistance shows a crossover from the exponential to the power-law length dependence with increasing field strength in agreement with known numerical results. More importantly, unlike the zero-field case the resistance distribution saturates to a Poissonian-limiting form proportional to A‖F‖exp(-A‖F‖ρ) for large sample lengths, where A is constant.

Average Orientation of Molecules Dissolved in Nematic Liquid Crystals of Opposite Diamagnetic Anisotropies

ZLI-1167 is a ternary mixture of nematic liquid crystals with negative diamagnetic anisotropy. It has, therefore, been used as a solvent where the spinning of the samples around the vertical axis in the conventional electromagnets without destroying the orientation of the dissolved molecules is possible in NMR experiments. This results in sharp lines with widths up to 1 Hz in the spectra.1,2 In an NMR system using a superconducting magnet (where the magnetic field direction is along the axis of spinning of the sample), it is possible to use even the nematic liquid crystals with positive diamagnetic anisotropy such as N-(p'-methoxybenzylidene)-p-n-butylaniline (MBBA) or N-(p'-ethoxybenzylidene)-p-n-butylaniline (EBBA) to obtain the spectra with sample spinning with equally sharp lines.3 The orientational behaviour of the dissolved molecules as a function of relative concentrations of the two solvents is investigated and the results are reported in the present communication.

An optimal weighted-average congestion based pricing scheme for enhanced QoS

Pricing is an effective tool to control congestion and achieve quality of service (QoS) provisioning for multiple differentiated levels of service. In this paper, we consider the problem of pricing for congestion control in the case of a network of nodes under a single service class and multiple queues, and present a multi-layered pricing scheme. We propose an algorithm for finding the optimal state dependent price levels for individual queues, at each node. The pricing policy used depends on a weighted average queue length at each node. This helps in reducing frequent price variations and is in the spirit of the random early detection (RED) mechanism used in TCP/IP networks. We observe in our numerical results a considerable improvement in performance using our scheme over that of a recently proposed related scheme in terms of both throughput and delay performance. In particular, our approach exhibits a throughput improvement in the range of 34 to 69 percent in all cases studied (over all routes) over the above scheme.

Average-Delay Optimal Policies for the Point-to-Point Channel

Average-delay optimal scheduflng of messages arriving to the transmitter of a point-to-point channel is considered in this paper. We consider a discrete time batch-arrival batch-service queueing model for the communication scheme, with service time that may be a function of batch size. The question of delay optimality is addressed within the semi-Markov decision-theoretic framework. Approximations to the average-delay optimal policy are obtained.

Mapping of Average Shear Wave Velocity for Bangalore Region: A Case Study

Mapping the shear wave velocity profile is an important part in seismic hazard and microzonation studies. The shear wave velocity of soil in the city of Bangalore was mapped using the Multichannel Analysis of Surface Wave (MASW) technique. An empirical relationship was found between the Standard Penetration Test (SPT) corrected N value ((N1)60cs) and measured shear wave velocity (Vs). The survey points were selected in such a way that the results represent the entire Bangalore region, covering an area of 220 km2. Fifty-eight 1-D and 20 2-D MASW surveys were performed and their velocity profiles determined. The average shear wave velocity of Bangalore soils was evaluated for depths of 5 m, 10 m, 15 m, 20 m, 25 m and 30 m. The sub-soil classification was made for seismic local site effect evaluation based on average shear wave velocity of 30-m depth (Vs30) of sites using the National Earthquake Hazards Reduction Program (NEHRP) and International Building Code (IBC) classification. Mapping clearly indicates that the depth of soil obtained from MASW closely matches with the soil layers identified in SPT bore holes. Estimation of local site effects for an earthquake requires knowledge of the dynamic properties of soil, which is usually expressed in terms of shear wave velocity. Hence, to make use of abundant SPT data available on many geotechnical projects in Bangalore, an attempt was made to develop a relationship between Vs (m/s) and (N1)60cs. The measured shear wave velocity at 38 locations close to SPT boreholes was used to generate the correlation between the corrected N values and shear wave velocity. A power fit model correlation was developed with a regression coefficient (R2) of 0.84. This relationship between shear wave velocity and corrected SPT N values correlates well with the Japan Road Association equations.

On the Average Case Communication Complexity for Detection in Sensor Networks

The problem of sensor-network-based distributed intrusion detection in the presence of clutter is considered. It is argued that sensing is best regarded as a local phenomenon in that only sensors in the immediate vicinity of an intruder are triggered. In such a setting, lack of knowledge of intruder location gives rise to correlated sensor readings. A signal-space viewpoint is introduced in which the noise-free sensor readings associated to intruder and clutter appear as surfaces $\mathcal{S_I}$ and $\mathcal{S_C}$ and the problem reduces to one of determining in distributed fashion, whether the current noisy sensor reading is best classified as intruder or clutter. Two approaches to distributed detection are pursued. In the first, a decision surface separating $\mathcal{S_I}$ and $\mathcal{S_C}$ is identified using Neyman-Pearson criteria. Thereafter, the individual sensor nodes interactively exchange bits to determine whether the sensor readings are on one side or the other of the decision surface. Bounds on the number of bits needed to be exchanged are derived, based on communication complexity (CC) theory. A lower bound derived for the two-party average case CC of general functions is compared against the performance of a greedy algorithm. The average case CC of the relevant greater-than (GT) function is characterized within two bits. In the second approach, each sensor node broadcasts a single bit arising from appropriate two-level quantization of its own sensor reading, keeping in mind the fusion rule to be subsequently applied at a local fusion center. The optimality of a threshold test as a quantization rule is proved under simplifying assumptions. Finally, results from a QualNet simulation of the algorithms are presented that include intruder tracking using a naive polynomial-regression algorithm.

Denumerable state stochastic games with limiting average payoff

We study stochastic games with countable state space, compact action spaces, and limiting average payoff. ForN-person games, the existence of an equilibrium in stationary strategies is established under a certain Liapunov stability condition. For two-person zero-sum games, the existence of a value and optimal strategies for both players are established under the same stability condition.

Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.

Average lattices and aperiodic structures

Statistically averaged lattices provide a common basis to understand the diffraction properties of structures displaying deviations from regular crystal structures. An average lattice is defined and examples are given in one and two dimensions along with their diffraction patterns. The absence of periodicity in reciprocal space corresponding to aperiodic structures is shown to arise out of different projected spacings that are irrationally related, when the grid points are projected along the chosen coordinate axes. It is shown that the projected length scales are important factors which determine the existence or absence of observable periodicity in the diffraction pattern more than the sequence of arrangement.

Brownian dynamics simulation of dense binary colloidal mixtures. II. Translational and bond-orientational order

We report the Brownian dynamics simulation results on the translational and bond-angle-orientational correlations for charged colloidal binary suspensions as the interparticle interactions are increased to form a crystalline (for a volume fraction phi = 0.2) or a glassy (phi = 0.3) state. The translational order is quantified in terms of the two- and four-point density autocorrelation functions whose comparisons show that there is no growing correlation length near the glass transition. The nearest-neighbor orientational order is determined in terms of the quadratic rotational invariant Q(l) and the bond-orientational correlation functions g(l)(t). The l dependence of Q(l) indicates that icosahedral (l = 6) order predominates at the cost of the cubic order (l = 4) near the glass as well as the crystal transition. The density and orientational correlation functions for a supercooled liquid freezing towards a glass fit well to the streched-exponential form exp[-(t/tau)(beta)]. The average relaxation times extracted from the fitted stretched-exponential functions as a function of effective temperatures T* obey the Arrhenius law for liquids freezing to a crystal whereas these obey the Vogel-Tamman-Fulcher law exp[AT(0)*/(T* - T-0*)] for supercooled Liquids tending towards a glassy state. The value of the parameter A suggests that the colloidal suspensions are ''fragile'' glass formers like the organic and molecular liquids.

Statistical thermodynamics of lattice models in zeolites: Implications of local versus global mean field interactions

The statistical thermodynamics of adsorption in caged zeolites is developed by treating the zeolite as an ensemble of M identical cages or subsystems. Within each cage adsorption is assumed to occur onto a lattice of n identical sites. Expressions for the average occupancy per cage are obtained by minimizing the Helmholtz free energy in the canonical ensemble subject to the constraints of constant M and constant number of adsorbates N. Adsorbate-adsorbate interactions in the Brag-Williams or mean field approximation are treated in two ways. The local mean field approximation (LMFA) is based on the local cage occupancy and the global mean field approximation (GMFA) is based on the average coverage of the ensemble. The GMFA is shown to be equivalent in formulation to treating the zeolite as a collection of interacting single site subsystems. In contrast, the treatment in the LMFA retains the description of the zeolite as an ensemble of identical cages, whose thermodynamic properties are conveniently derived in the grand canonical ensemble. For a z coordinated lattice within the zeolite cage, with epsilon(aa) as the adsorbate-adsorbate interaction parameter, the comparisons for different values of epsilon(aa)(*)=epsilon(aa)z/2kT, and number of sites per cage, n, illustrate that for -1 0. We compare the isotherms predicted with the LMFA with previous GMFA predictions [K. G. Ayappa, C. R. Kamala, and T. A. Abinandanan, J. Chem. Phys. 110, 8714 (1999)] (which incorporates both the site volume reduction and a coverage-dependent epsilon(aa)) for xenon and methane in zeolite NaA. In all cases the predicted isotherms are very similar, with the exception of a small steplike feature present in the LMFA for xenon at higher coverages. (C) 1999 American Institute of Physics. [S0021-9606(99)70333-8].

A new parallel overlapped domain decomposition method for nonlinear dynamic finite element analysis

In this paper a new parallel algorithm for nonlinear transient dynamic analysis of large structures has been presented. An unconditionally stable Newmark-beta method (constant average acceleration technique) has been employed for time integration. The proposed parallel algorithm has been devised within the broad framework of domain decomposition techniques. However, unlike most of the existing parallel algorithms (devised for structural dynamic applications) which are basically derived using nonoverlapped domains, the proposed algorithm uses overlapped domains. The parallel overlapped domain decomposition algorithm proposed in this paper has been formulated by splitting the mass, damping and stiffness matrices arises out of finite element discretisation of a given structure. A predictor-corrector scheme has been formulated for iteratively improving the solution in each step. A computer program based on the proposed algorithm has been developed and implemented with message passing interface as software development environment. PARAM-10000 MIMD parallel computer has been used to evaluate the performances. Numerical experiments have been conducted to validate as well as to evaluate the performance of the proposed parallel algorithm. Comparisons have been made with the conventional nonoverlapped domain decomposition algorithms. Numerical studies indicate that the proposed algorithm is superior in performance to the conventional domain decomposition algorithms. (C) 2003 Elsevier Ltd. All rights reserved.