Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

Network pricing for QoS: A 'Regulation' approach

V. S. Borkar’s work was supported in part by grant number III.5(157)/99-ET from the Department of Science and Technology, Government of India. D. Manjunath’s work was supported in part by grant number 1(1)/2004-E-Infra from the Ministry of Information Technology, Government of India.

Design and Evaluation of a Robust Optical Beam-Interruption-Based Vehicle Classifier System

This paper presents the design and development of a novel optical vehicle classifier system, which is based on interruption of laser beams, that is suitable for use in places with poor transportation infrastructure. The system can estimate the speed, axle count, wheelbase, tire diameter, and the lane of motion of a vehicle. The design of the system eliminates the need for careful optical alignment, whereas the proposed estimation strategies render the estimates insensitive to angular mounting errors and to unevenness of the road. Strategies to estimate vehicular parameters are described along with the optimization of the geometry of the system to minimize estimation errors due to quantization. The system is subsequently fabricated, and the proposed features of the system are experimentally demonstrated. The relative errors in the estimation of velocity and tire diameter are shown to be within 0.5% and to change by less than 17% for angular mounting errors up to 30 degrees. In the field, the classifier demonstrates accuracy better than 97.5% and 94%, respectively, in the estimation of the wheelbase and lane of motion and can classify vehicles with an average accuracy of over 89.5%.

Rollover-Preventive Force Synthesis at Active Suspensions in a Vehicle Performing a Severe Maneuver With Wheels Lifted Off

Among the intelligent safety technologies for road vehicles, active suspensions controlled by embedded computing elements for preventing rollover have received a lot of attention. The existing models for synthesizing and allocating forces in such suspensions are conservatively based on the constraints that are valid until no wheels lift off the ground. However, the fault tolerance of the rollover-preventive systems can be enhanced if the smart/active suspensions can intervene in the more severe situation in which the wheels have just lifted off the ground. The difficulty in computing control in the last situation is that the vehicle dynamics then passes into the regime that yields a model involving disjunctive constraints on the dynamics. Simulation of dynamics with disjunctive constraints in this context becomes necessary to estimate, synthesize, and allocate the intended hardware realizable forces in an active suspension. In this paper, we give an algorithm for the previously mentioned problem by solving it as a disjunctive dynamic optimization problem. Based on this, we synthesize and allocate the roll-stabilizing time-dependent active suspension forces in terms of sensor output data. We show that the forces obtained from disjunctive dynamics are comparable with existing force allocations and, hence, are possibly realizable in the existing hardware framework toward enhancing the safety and fault tolerance.

Hardware Accelerator for 3D Method of Moments based Parasitic Extraction

A Field Programmable Gate Array (FPGA) based hardware accelerator for multi-conductor parasitic capacitance extraction, using Method of Moments (MoM), is presented in this paper. Due to the prohibitive cost of solving a dense algebraic system formed by MoM, linear complexity fast solver algorithms have been developed in the past to expedite the matrix-vector product computation in a Krylov sub-space based iterative solver framework. However, as the number of conductors in a system increases leading to a corresponding increase in the number of right-hand-side (RHS) vectors, the computational cost for multiple matrix-vector products present a time bottleneck, especially for ill-conditioned system matrices. In this work, an FPGA based hardware implementation is proposed to parallelize the iterative matrix solution for multiple RHS vectors in a low-rank compression based fast solver scheme. The method is applied to accelerate electrostatic parasitic capacitance extraction of multiple conductors in a Ball Grid Array (BGA) package. Speed-ups up to 13x over equivalent software implementation on an Intel Core i5 processor for dense matrix-vector products and 12x for QR compressed matrix-vector products is achieved using a Virtex-6 XC6VLX240T FPGA on Xilinx's ML605 board.

Multi-agent Reinforcement Learning for Traffic Signal Control

Optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users amidst the rapid increase in the usage of vehicles. In this paper, we formulate the TSC problem as a discounted cost Markov decision process (MDP) and apply multi-agent reinforcement learning (MARL) algorithms to obtain dynamic TSC policies. We model each traffic signal junction as an independent agent. An agent decides the signal duration of its phases in a round-robin (RR) manner using multi-agent Q-learning with either is an element of-greedy or UCB 3] based exploration strategies. It updates its Q-factors based on the cost feedback signal received from its neighbouring agents. This feedback signal can be easily constructed and is shown to be effective in minimizing the average delay of the vehicles in the network. We show through simulations over VISSIM that our algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm 15] over two real road networks.

Tomorrow's cities - a framework for building resilience

Urbanisation is the great driving force of the twenty-first century. Cities are associated with both productivity and creativity, and the benefits offered by closely connected and high density living and working contribute to sustainability. At the same time, cities need extensive infrastructure – like water, power, sanitation and transportation systems – to operate effectively. Cities therefore comprise multiple components, forming both static and dynamic systems that are interconnected directly and indirectly on a number of levels, all forming the backdrop for the interaction of people and processes. Bringing together large numbers of people and complex products in rich interactions can lead to vulnerability from hazards, threats and even trends, whether natural hazards, epidemics, political upheaval, demographic changes, economic instability and/or mechanical failures; The key to countering vulnerability is the identification of critical systems and clear understanding of their interactions and dependencies. Critical systems can be assessed methodically to determine the implications of their failure and their interconnectivities with other systems to identify options. The overriding need is to support resilience – defined here as the degree to which a system or systems can continue to function effectively in a changing environment. Cities need to recognise the significance of devising adaptation strategies and processes to address a multitude of uncertainties relating to climate, economy, growth and demography. In this paper we put forward a framework to support cities in understanding the hazards, threats and trends that can make them vulnerable to unexpected changes and unpredictable shocks. The framework draws on an asset model of the city, in which components that contribute to resilience include social capital, economic assets, manufactured assets, and governance. The paper reviews the field, and draws together an overarching framework intended to help cities plan a robust trajectory towards increased resilience through flexibility, resourcefulness and responsiveness. It presents some brief case studies demonstrating the applicability of the proposed framework to a wide variety of circumstances.