Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

Influence of tool rake angle on the quality of pure magnesium chip-consolidated product

Chips were produced by orthogonal Cutting of cast pure magnesium billet with three different tool rake angles viz., -15 degrees, -5 degrees and +15 degrees on a lathe. Chip consolidation by solid state recycling technique involved cold compaction followed by hot extrusion. The extruded products were characterized for microstructure and mechanical properties. Chip-consolidated products from -15 degrees rake angle tools showed 19% increase in tensile strength, 60% reduction ingrain size and 12% increase in hardness compared to +15 degrees rake chip-consolidated product indicating better chip bonding and grain refinement. Microstructure of the fracture specimen Supports the abovefinding. On the overall, the present work high lights the importance of tool take angle in determining the quality of the chip-consolidated products. (C) 2009 Elsevier B.V. All rights reserved.

Synthesis based on cyclohexadienes. Part 15. Total synthesis of (±)-prezizaene, (±)-prezizanol and (±)-jinkohol II

A new methodology for the synthesis of the complex ring system tricyclo[6.2.1.0(1.5)]undecane. present in the zizaene group of sesquiterpenes, is described. Acid-catalysed rearrangement of the endo alcohol 20 afforded the enone 12, which was transformed stereoselectively into the key intermediate. (+/-)-norprezizanone 10. The features of the synthesis are the transformation of a bicycle[2.2.2] octane framework into a bicycle[3.2.1] octane system by an acid-catalysed rearrangement and a stereoselective conjugate addition of a methyl group on an alpha,beta-unsaturated keto ester at -100 degrees C. Norprezizanone was converted into the sesquiterpenes (+/-)-prezizanol 5 and(+/-)-prezizaene 4. The first total synthesis of (+/-)-jinkohol II 6 is also presented.

Synthesis of chiral bicyclo[2.2.2]oct-5-en-2-ones via an intramolecular alkylation reaction

Generation of the thermodynamic dienolate of 9-bromocarvone derivatives 5, 7 and 11 furnished the chiral bicycle[2.2.2] octenones 6, 8 and 9 and 12 and 13 containing a bridgehead methyl group via an intramolecular alkylation reaction. In an analogous manner intramolecular alkylation reaction of the bromo enones 15a-e, obtained from carvone 2 by 1,3-alkylative enone transposition (-->14) followed by a regiospecific bromoetherification reaction, furnished the bicyclo[2.2.2]oct-5-en-2-ones 16a-e and 17a-e.

Rhythmic egg-laying behaviour in virgin females of fruit flies Drosophila melanogaster

Fruit fly Drosophila melanogaster females display rhythmic egg-laying under 12: 12 h light/dark (LD) cycles which persists with near 24 h periodicity under constant darkness (DD). We have shown previously that persistence of this rhythm does not require the neurons expressing pigment dispersing factor (PDF), thought to be the canonical circadian pacemakers, and proposed that it could be controlled by peripheral clocks or regulated/triggered by the act of mating. We assayed egg-laying behaviour of wild-type Canton S (CS) females under LD, DD and constant light (LL) conditions in three different physiological states; as virgins, as females allowed to mate with males for 1 day and as females allowed to mate for the entire duration of the assay. Here, we report the presence of a circadian rhythm in egg-laying in virgin D. melanogaster females. We also found that egg-laying behaviour of 70 and 90% females from all the three male presence/absence protocols follows circadian rhythmicity under DD and LL, with periods ranging between 18 and 30 h. The egg-laying rhythm of all virgin females synchronized to LD cycles with a peak occurring soon after lights-off. The rhythm in virgins was remarkably robust with maximum number of eggs deposited immediately after lights-off in contrast to mated females which show higher egg-laying during the day. These results suggest that the egg-laying rhythm of D. melanogaster is endogenously driven and is neither regulated nor triggered by the act of mating; instead, the presence of males results in reduction in entrainment to LD cycles.

Multi-agent Reinforcement Learning for Traffic Signal Control

Optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users amidst the rapid increase in the usage of vehicles. In this paper, we formulate the TSC problem as a discounted cost Markov decision process (MDP) and apply multi-agent reinforcement learning (MARL) algorithms to obtain dynamic TSC policies. We model each traffic signal junction as an independent agent. An agent decides the signal duration of its phases in a round-robin (RR) manner using multi-agent Q-learning with either is an element of-greedy or UCB 3] based exploration strategies. It updates its Q-factors based on the cost feedback signal received from its neighbouring agents. This feedback signal can be easily constructed and is shown to be effective in minimizing the average delay of the vehicles in the network. We show through simulations over VISSIM that our algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm 15] over two real road networks.