Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

Network pricing for QoS: A 'Regulation' approach

V. S. Borkar’s work was supported in part by grant number III.5(157)/99-ET from the Department of Science and Technology, Government of India. D. Manjunath’s work was supported in part by grant number 1(1)/2004-E-Infra from the Ministry of Information Technology, Government of India.

Design and Evaluation of a Robust Optical Beam-Interruption-Based Vehicle Classifier System

This paper presents the design and development of a novel optical vehicle classifier system, which is based on interruption of laser beams, that is suitable for use in places with poor transportation infrastructure. The system can estimate the speed, axle count, wheelbase, tire diameter, and the lane of motion of a vehicle. The design of the system eliminates the need for careful optical alignment, whereas the proposed estimation strategies render the estimates insensitive to angular mounting errors and to unevenness of the road. Strategies to estimate vehicular parameters are described along with the optimization of the geometry of the system to minimize estimation errors due to quantization. The system is subsequently fabricated, and the proposed features of the system are experimentally demonstrated. The relative errors in the estimation of velocity and tire diameter are shown to be within 0.5% and to change by less than 17% for angular mounting errors up to 30 degrees. In the field, the classifier demonstrates accuracy better than 97.5% and 94%, respectively, in the estimation of the wheelbase and lane of motion and can classify vehicles with an average accuracy of over 89.5%.

Rollover-Preventive Force Synthesis at Active Suspensions in a Vehicle Performing a Severe Maneuver With Wheels Lifted Off

Among the intelligent safety technologies for road vehicles, active suspensions controlled by embedded computing elements for preventing rollover have received a lot of attention. The existing models for synthesizing and allocating forces in such suspensions are conservatively based on the constraints that are valid until no wheels lift off the ground. However, the fault tolerance of the rollover-preventive systems can be enhanced if the smart/active suspensions can intervene in the more severe situation in which the wheels have just lifted off the ground. The difficulty in computing control in the last situation is that the vehicle dynamics then passes into the regime that yields a model involving disjunctive constraints on the dynamics. Simulation of dynamics with disjunctive constraints in this context becomes necessary to estimate, synthesize, and allocate the intended hardware realizable forces in an active suspension. In this paper, we give an algorithm for the previously mentioned problem by solving it as a disjunctive dynamic optimization problem. Based on this, we synthesize and allocate the roll-stabilizing time-dependent active suspension forces in terms of sensor output data. We show that the forces obtained from disjunctive dynamics are comparable with existing force allocations and, hence, are possibly realizable in the existing hardware framework toward enhancing the safety and fault tolerance.

Multi-agent Reinforcement Learning for Traffic Signal Control

Optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users amidst the rapid increase in the usage of vehicles. In this paper, we formulate the TSC problem as a discounted cost Markov decision process (MDP) and apply multi-agent reinforcement learning (MARL) algorithms to obtain dynamic TSC policies. We model each traffic signal junction as an independent agent. An agent decides the signal duration of its phases in a round-robin (RR) manner using multi-agent Q-learning with either is an element of-greedy or UCB 3] based exploration strategies. It updates its Q-factors based on the cost feedback signal received from its neighbouring agents. This feedback signal can be easily constructed and is shown to be effective in minimizing the average delay of the vehicles in the network. We show through simulations over VISSIM that our algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm 15] over two real road networks.

Tomorrow's cities - a framework for building resilience

Urbanisation is the great driving force of the twenty-first century. Cities are associated with both productivity and creativity, and the benefits offered by closely connected and high density living and working contribute to sustainability. At the same time, cities need extensive infrastructure – like water, power, sanitation and transportation systems – to operate effectively. Cities therefore comprise multiple components, forming both static and dynamic systems that are interconnected directly and indirectly on a number of levels, all forming the backdrop for the interaction of people and processes. Bringing together large numbers of people and complex products in rich interactions can lead to vulnerability from hazards, threats and even trends, whether natural hazards, epidemics, political upheaval, demographic changes, economic instability and/or mechanical failures; The key to countering vulnerability is the identification of critical systems and clear understanding of their interactions and dependencies. Critical systems can be assessed methodically to determine the implications of their failure and their interconnectivities with other systems to identify options. The overriding need is to support resilience – defined here as the degree to which a system or systems can continue to function effectively in a changing environment. Cities need to recognise the significance of devising adaptation strategies and processes to address a multitude of uncertainties relating to climate, economy, growth and demography. In this paper we put forward a framework to support cities in understanding the hazards, threats and trends that can make them vulnerable to unexpected changes and unpredictable shocks. The framework draws on an asset model of the city, in which components that contribute to resilience include social capital, economic assets, manufactured assets, and governance. The paper reviews the field, and draws together an overarching framework intended to help cities plan a robust trajectory towards increased resilience through flexibility, resourcefulness and responsiveness. It presents some brief case studies demonstrating the applicability of the proposed framework to a wide variety of circumstances.

Experimental investigation on TBAB clathrate hydrate slurry flows in a horizontal tube: Forced convective heat transfer behaviors

Tetra-n-butyl-ammonium bromide (TBAB) clathrate hydrate slurry (CHS) is one kind of secondary refrigerants, which is promising to be applied into air-conditioning or latent-heat transportation systems as a thermal storage or cold carrying medium for energy saving. It is a solid-liquid two phase mixture which is easy to produce and has high latent heat and good fluidity. In this paper, the heat transfer characteristics of TBAB slurry were investigated in a horizontal stainless steel tube under different solid mass fractions and flow velocities with constant heat flux. One velocity region of weakened heat transfer was found. Moreover, TBAB CHS was treated as a kind of Bingham fluids, and the influences of the solid particles, flow velocity and types of flow on the forced convective heat transfer coefficients of TBAB CHS were investigated. At last, criterial correlations of Nusselt number for laminar and turbulent flows in the form of power function were summarized, and the error with experimental results was within 20%.

A Structured Approach to Simulate Manufacturing System In Virtual Environment

Nowadays, the realization of the Virtual Factory (VF) is the strategic goal of many manufacturing enterprises for the coming years. The industrial scenario is characterized by the dynamics of innovations increment and the product life cycle became shorter. Furthermore products and the corresponding manufacturing processes get more and more complex. Therefore, companies need new methods for the planning of manufacturing systems.
To date, the efforts have focused on the creation of an integrated environment to design and manage the manufacturing process of a new product. The future goal is to integrate Virtual Reality (VR) tools into the Product Lifecycle Management of the manufacturing industries.
In order to realize this goal the authors have conducted a study to perform VF simulation steps for a supplier of Industrial Automation Systems and have provided a structured approach focusing on interaction between simulation software and VR hardware tools in order to simulate both robotic and
manual work cells.
The first results of the study in progress have been carried out in the VR Laboratory of the Competence Regional Centre for the qualification of the Transportation Systems that has been founded by Campania Region.

Preparing for disaster: a comparative analysis of education for critical infrastructure collapse

This article explores policy approaches to educating populations for potential critical infrastructure collapse in five different countries: the UK, the US, Germany, Japan and New Zealand. ‘Critical infrastructure’ is not always easy to define, and indeed is defined slightly differently across countries – it includes entities vital to life, such as utilities (water, energy), transportation systems and communications, and may also include social and cultural infrastructure. The article is a mapping exercise of different approaches to critical infrastructure protection and preparedness education by the five countries. The exercise facilitates a comparison of the countries and enables us to identify distinctive characteristics of each country’s approach. We argue that contrary to what most scholars of security have argued, these national approaches diverge greatly, suggesting that they are shaped more by internal politics and culture than by global approaches.

Análise de eficiência energética de transportes rodoviários

Projecto para obtenção do grau de Mestre em Engenharia Informática e de computadores