Reinforcement Learning With Function Approximation for Traffic Signal Control

We propose, for the first time, a reinforcement learning (RL) algorithm with function approximation for traffic signal control. Our algorithm incorporates state-action features and is easily implementable in high-dimensional settings. Prior work, e. g., the work of Abdulhai et al., on the application of RL to traffic signal control requires full-state representations and cannot be implemented, even in moderate-sized road networks, because the computational complexity exponentially grows in the numbers of lanes and junctions. We tackle this problem of the curse of dimensionality by effectively using feature-based state representations that use a broad characterization of the level of congestion as low, medium, or high. One advantage of our algorithm is that, unlike prior work based on RL, it does not require precise information on queue lengths and elapsed times at each lane but instead works with the aforementioned described features. The number of features that our algorithm requires is linear to the number of signaled lanes, thereby leading to several orders of magnitude reduction in the computational complexity. We perform implementations of our algorithm on various settings and show performance comparisons with other algorithms in the literature, including the works of Abdulhai et al. and Cools et al., as well as the fixed-timing and the longest queue algorithms. For comparison, we also develop an RL algorithm that uses full-state representation and incorporates prioritization of traffic, unlike the work of Abdulhai et al. We observe that our algorithm outperforms all the other algorithms on all the road network settings that we consider.

Signal compression through spatial frequency-based motion estimation

The interest in low bit rate video coding has increased considerably. Despite rapid progress in storage density and digital communication system performance, demand for data-transmission bandwidth and storage capacity continue to exceed the capabilities of available technologies. The growth of data-intensive digital audio, video applications and the increased use of bandwidth-limited media such as video conferencing and full motion video have not only sustained the need for efficient ways to encode analog signals, but made signal compression central to digital communication and data-storage technology. In this paper we explore techniques for compression of image sequences in a manner that optimizes the results for the human receiver. We propose a new motion estimator using two novel block match algorithms which are based on human perception. Simulations with image sequences have shown an improved bit rate while maintaining ''image quality'' when compared to conventional motion estimation techniques using the MAD block match criteria.

Distributed intrusion detection in the presence of correlated sensor readings: Signal-space and communication-complexity view-point

The problem of sensor-network-based distributed intrusion detection in the presence of clutter is considered. It is argued that sensing is best regarded as a local phenomenon in that only sensors in the immediate vicinity of an intruder are triggered. In such a setting, lack of knowledge of intruder location gives rise to correlated sensor readings. A signal-space view-point is introduced in which the noise-free sensor readings associated to intruder and clutter appear as surfaces f(s) and f(g) and the problem reduces to one of determining in distributed fashion, whether the current noisy sensor reading is best classified as intruder or clutter. Two approaches to distributed detection are pursued. In the first, a decision surface separating f(s) and f(g) is identified using Neyman-Pearson criteria. Thereafter, the individual sensor nodes interactively exchange bits to determine whether the sensor readings are on one side or the other of the decision surface. Bounds on the number of bits needed to be exchanged are derived, based on communication-complexity (CC) theory. A lower bound derived for the two-party average case CC of general functions is compared against the performance of a greedy algorithm. Extensions to the multi-party case is straightforward and is briefly discussed. The average case CC of the relevant greaterthan (CT) function is characterized within two bits. Under the second approach, each sensor node broadcasts a single bit arising from appropriate two-level quantization of its own sensor reading, keeping in mind the fusion rule to be subsequently applied at a local fusion center. The optimality of a threshold test as a quantization rule is proved under simplifying assumptions. Finally, results from a QualNet simulation of the algorithms are presented that include intruder tracking using a naive polynomial-regression algorithm. 2010 Elsevier B.V. All rights reserved.

GNSS Signal Detection Under Noise Uncertainty

High sensitivity detection techniques are required for indoor navigation using Global Navigation Satellite System (GNSS) receivers, and typically, a combination of coherent and non- coherent integration is used as the test statistic for detection. The coherent integration exploits the deterministic part of the signal and is limited due to the residual frequency error, navigation data bits and user dynamics, which are not known apriori. So, non- coherent integration, which involves squaring of the coherent integration output, is used to improve the detection sensitivity. Due to this squaring, it is robust against the artifacts introduced due to data bits and/or frequency error. However, it is susceptible to uncertainty in the noise variance, and this can lead to fundamental sensitivity limits in detecting weak signals. In this work, the performance of the conventional non-coherent integration-based GNSS signal detection is studied in the presence of noise uncertainty. It is shown that the performance of the current state of the art GNSS receivers is close to the theoretical SNR limit for reliable detection at moderate levels of noise uncertainty. Alternate robust post-coherent detectors are also analyzed, and are shown to alleviate the noise uncertainty problem. Monte-Carlo simulations are used to confirm the theoretical predictions.

Electrochemical processing of ocean manganese nodules with microbial enhancement to recover valuable metals

In this paper, electroleaching and electrobioleaching of ocean manganese nodules are discussed along with the role of galvanic interactions in bioleaching. Polarization studies using a manganese nodule slurry electrode system indicated that the maximum dissolution of iron and manganese due to electrochemical reduction occurred at negative DC potentials of -600 and -1,400 mV(SCE). Electroleaching and electrobioleaching of ocean manganese nodules in the presence of Acidithiobacillus ferrooxidans and Acidithiobacillus thiooxidans at the above negative applied DC potentials resulted insignificant dissolution of copper, nickel and cobalt in 1 M H2SO4 and in sulfuric acid solution at pH 0.5 and 2.0. Mechanisms involved in electrobioleaching of ocean manganese nodules are discussed. Galvanic leaching of ocean manganese nodules in the presence of externally added pyrite and pyrolusite for enhancement of dissolution was also studied. Various electrochemical and biochemical parameters were optimized, and the electroleaching and galvanic processes thus developed are shown to yield almost complete dissolution of all metal values. This electrobioleaching process developed in the laboratory may be cost effective, energy efficient and environmentally friendly.

Role of interface states associated with transitional nanophase precipitates in the photoluminescence enhancement of SrTiO3 : Pr3+,Al3+

SrTiO3:Pr3+,Al3+ phosphor samples with varying ratios of Sr/Ti/Al were prepared by the gel-carbonate method and the mechanism of enhancement of the red photoluminescence intensity therein was investigated. The photoluminescence (PL) spectra of SrTiO3:Pr3+ show both D-1(2) --> H-3(4) and P-3(0) --> H-3(4) emission in the red and blue spectral regions, respectively, with comparable intensity. The emission intensity of D-1(2) --> H-3(4) is drastically enhanced by the incorporation of Al3+ and excess Ti4+ in the compositional range Sr(Ti,Al-y)(O3+3y/2):Pr3+ (0.2 less than or equal to y less than or equal to 0.4) and SrTi1+xAlyO3+z:Pr3+ (0.2 less than or equal to x less than or equal to 0.5; 0.05 less than or equal to y less than or equal to 0.1; z = 2x + 3y/2) with the complete disappearance of the blue band. This cannot be explained by the simple point defect model as the EPR studies do not show any evidence for the presence of electron or hole centers. TEM investigations show the presence of exsolved nanophases of SrAl12O19 and/or TiO2 in the grain boundary region as well as grain interiors as lamellae which, in turn, form the solid-state defects, namely, dislocation networks, stacking faults and crystallographic shear planes whereby the framework of corner shared TiO6 octehedra changes over to edge-sharing TiO5-AlO5 strands as indicated from the Al-27 MAS NMR studies. The presence of transitional nanophases and the associated defects modify the excitation-emission processes by way of formation of electronic sub-levels at 3.40 and 4.43 eV, leading to magnetic-dipole related red emission with enhanced intensity. This is evidenced by the fact that SrAl12O19:Pr3+,Ti4+ shows bright red emission whereas SrAl12O19:Pr3+ does not show red photoluminescence.

Albedo enhancement of marine clouds to counteract global warming: impacts on the hydrological cycle

Recent studies have shown that changes in solar radiation affect the hydrological cycle more strongly than equivalent CO(2) changes for the same change in global mean surface temperature. Thus, solar radiation management ``geoengineering'' proposals to completely offset global mean temperature increases by reducing the amount of absorbed sunlight might be expected to slow the global water cycle and reduce runoff over land. However, proposed countering of global warming by increasing the albedo of marine clouds would reduce surface solar radiation only over the oceans. Here, for an idealized scenario, we analyze the response of temperature and the hydrological cycle to increased reflection by clouds over the ocean using an atmospheric general circulation model coupled to a mixed layer ocean model. When cloud droplets are reduced in size over all oceans uniformly to offset the temperature increase from a doubling of atmospheric CO(2), the global-mean precipitation and evaporation decreases by about 1.3% but runoff over land increases by 7.5% primarily due to increases over tropical land. In the model, more reflective marine clouds cool the atmospheric column over ocean. The result is a sinking motion over oceans and upward motion over land. We attribute the increased runoff over land to this increased upward motion over land when marine clouds are made more reflective. Our results suggest that, in contrast to other proposals to increase planetary albedo, offsetting mean global warming by reducing marine cloud droplet size does not necessarily lead to a drying, on average, of the continents. However, we note that the changes in precipitation, evaporation and P-E are dominated by small but significant areas, and given the highly idealized nature of this study, a more thorough and broader assessment would be required for proposals of altering marine cloud properties on a large scale.

Supramolecular polymer for explosives sensing: role of H-bonding in enhancement of sensitivity in the solid state

A pi-electron rich supramolecular polymer as an efficient fluorescent sensor for electron deficient nitroaromatic explosives has been synthesized, and the role of H-bonding in dramatic amplification of sensitivity/fluorescence quenching efficiency in the solid state has been established.

Subspace Based Speech Enhancement Using Gaussian Mixture Model

Traditional subspace based speech enhancement (SSE)methods use linear minimum mean square error (LMMSE) estimation that is optimal if the Karhunen Loeve transform (KLT) coefficients of speech and noise are Gaussian distributed. In this paper, we investigate the use of Gaussian mixture (GM) density for modeling the non-Gaussian statistics of the clean speech KLT coefficients. Using Gaussian mixture model (GMM), the optimum minimum mean square error (MMSE) estimator is found to be nonlinear and the traditional LMMSE estimator is shown to be a special case. Experimental results show that the proposed method provides better enhancement performance than the traditional subspace based methods.Index Terms: Subspace based speech enhancement, Gaussian mixture density, MMSE estimation.

Two stage iterative wiener filtering for speech enhancement

We formulate a two-stage Iterative Wiener filtering (IWF) approach to speech enhancement, bettering the performance of constrained IWF, reported in literature. The codebook constrained IWF (CCIWF) has been shown to be effective in achieving convergence of IWF in the presence of both stationary and non-stationary noise. To this, we include a second stage of unconstrained IWF and show that the speech enhancement performance can be improved in terms of average segmental SNR (SSNR), Itakura-Saito (IS) distance and Linear Prediction Coefficients (LPC) parameter coincidence. We also explore the tradeoff between the number of CCIWF iterations and the second stage IWF iterations.

Finite Signal-set Capacity of Two-user Gaussian Multiple Access Channel

The capacity region of a two-user Gaussian Multiple Access Channel (GMAC) with complex finite input alphabets and continuous output alphabet is studied. When both the users are equipped with the same code alphabet, it is shown that, rotation of one of the user’s alphabets by an appropriate angle can make the new pair of alphabets not only uniquely decodable, but will result in enlargement of the capacity region. For this set-up, we identify the primary problem to be finding appropriate angle(s) of rotation between the alphabets such that the capacity region is maximally enlarged. It is shown that the angle of rotation which provides maximum enlargement of the capacity region also minimizes the union bound on the probability of error of the sumalphabet and vice-verse. The optimum angle(s) of rotation varies with the SNR. Through simulations, optimal angle(s) of rotation that gives maximum enlargement of the capacity region of GMAC with some well known alphabets such as M-QAM and M-PSK for some M are presented for several values of SNR. It is shown that for large number of points in the alphabets, capacity gains due to rotations progressively reduce. As the number of points N tends to infinity, our results match the results in the literature wherein the capacity region of the Gaussian code alphabet doesn’t change with rotation for any SNR.

Network Coding for TCP Throughput Enhancement over a Multi-Hop wireless Network

The poor performance of TCP over multi-hop wireless networks is well known. In this paper we explore to what extent network coding can help to improve the throughput performance of TCP controlled bulk transfers over a chain topology multi-hop wireless network. The nodes use a CSMA/ CA mechanism, such as IEEE 802.11’s DCF, to perform distributed packet scheduling. The reverse flowing TCP ACKs are sought to be X-ORed with forward flowing TCP data packets. We find that, without any modification to theMAC protocol, the gain from network coding is negligible. The inherent coordination problem of carrier sensing based random access in multi-hop wireless networks dominates the performance. We provide a theoretical analysis that yields a throughput bound with network coding. We then propose a distributed modification of the IEEE 802.11 DCF, based on tuning the back-off mechanism using a feedback approach. Simulation studies show that the proposed mechanism when combined with network coding, improves the performance of a TCP session by more than 100%.