Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

Fuzzy inference based data preprocessing for VBR video traffic prediction

Prediction of variable bit rate compressed video traffic is critical to dynamic allocation of resources in a network. In this paper, we propose a technique for preprocessing the dataset used for training a video traffic predictor. The technique involves identifying the noisy instances in the data using a fuzzy inference system. We focus on three prediction techniques, namely, linear regression, neural network and support vector regression and analyze their performance on H.264 video traces. Our experimental results reveal that data preprocessing greatly improves the performance of linear regression and neural network, but is not effective on support vector regression.

An instantaneous control model for flow control in an ATM network

This paper is concerned with the optimal flow control of an ATM switching element in a broadband-integrated services digital network. We model the switching element as a stochastic fluid flow system with a finite buffer, a constant output rate server, and a Gaussian process to characterize the input, which is a heterogeneous set of traffic sources. The fluid level should be maintained between two levels namely b1 and b2 with b1network is considered

Efficient Regular Expression Pattern Matching for Network Intrusion Detection Systems using Modified Word-based Automata

Network Intrusion Detection Systems (NIDS) intercept the traffic at an organization's network periphery to thwart intrusion attempts. Signature-based NIDS compares the intercepted packets against its database of known vulnerabilities and malware signatures to detect such cyber attacks. These signatures are represented using Regular Expressions (REs) and strings. Regular Expressions, because of their higher expressive power, are preferred over simple strings to write these signatures. We present Cascaded Automata Architecture to perform memory efficient Regular Expression pattern matching using existing string matching solutions. The proposed architecture performs two stage Regular Expression pattern matching. We replace the substring and character class components of the Regular Expression with new symbols. We address the challenges involved in this approach. We augment the Word-based Automata, obtained from the re-written Regular Expressions, with counter-based states and length bound transitions to perform Regular Expression pattern matching. We evaluated our architecture on Regular Expressions taken from Snort rulesets. We were able to reduce the number of automata states between 50% to 85%. Additionally, we could reduce the number of transitions by a factor of 3 leading to further reduction in the memory requirements.

Mobile agent based TCP attacker identification in MANET using the traffic history (MAITH)

TCP attacks are the major problem faced by Mobile Ad hoc Networks (MANETs) due to its limited network and host resources. Attacker traceback is a promising solution which allows a victim to identify the exact location of the attacker and hence enables the victim to take proper countermeasure near attack origins, for forensics and to discourage attackers from launching the attacks. However, attacker traceback in MANET is a challenging problem due to dynamic network topology, limited network and host resources such as memory, bandwidth and battery life. We introduce a novel method of TCP attacker Identification in MANET using the Traffic History - MAITH. Based on the comprehensive evaluation based on simulations, we showed that MAITH can successfully track down the attacker under diverse mobile multi-hop network environment with low communication, computation, and memory overhead.

Performance analysis of data and voice connections in a cognitive radio network

We study the performance of cognitive (secondary) users in a cognitive radio network which uses a channel whenever the primary users are not using the channel. The usage of the channel by the primary users is modelled by an ON-OFF renewal process. The cognitive users may be transmitting data using TCP connections and voice traffic. The voice traffic is given priority over the data traffic. We theoretically compute the mean delay of TCP and voice packets and also the mean throughput of the different TCP connections. We compare the theoretical results with simulations.

Traffic engineered NoC for streaming applications

Streaming applications demand hard bandwidth and throughput guarantees in a multiprocessor environment amidst resource competing processes. We present a Label Switching based Network-on-Chip (LS-NoC) motivated by throughput guarantees offered by bandwidth reservation. Label switching is a packet relaying technique in which individual packets carry route information in the form of labels. A centralized LS-NoC Management framework engineers traffic into Quality of Service (QoS) guaranteed routes. LS-NoC caters to the requirements of streaming applications where communication channels are fixed over the lifetime of the application. The proposed NoC framework inherently supports heterogeneous and ad hoc system-on-chips. The LS-NoC can be used in conjunction with conventional best effort NoC as a QoS guaranteed communication network or as a replacement to the conventional NoC. A multicast, broadcast capable label switched router for the LS-NoC has been designed. A 5 port, 256 bit data bus, 4 bit label router occupies 0.431 mm(2) in 130 nm and delivers peak bandwidth of 80 Gbits/s per link at 312.5 MHz. Bandwidth and latency guarantees of LS-NoC have been demonstrated on traffic from example streaming applications and on constant and variable bit rate traffic patterns. LS-NoC was found to have a competitive AreaxPower/Throughput figure of merit with state-of-the-art NoCs providing QoS. Circuit switching with link sharing abilities and support for asynchronous operation make LS-NoC a desirable choice for QoS servicing in chip multiprocessors. (C) 2013 Elsevier B.V. All rights reserved.

Greening the IP Over Optical Backbone Network

In this paper we examine the energy consumption of IP Over Optical WDM Networks. As the number of Internet users increases the Internet expands in reach and capacity. This results in increased energy consumption of the network. Minimizing the power consumption, termed as ``Greening the Internet'', is desirable to help service providers (SP) operate their networks and provide services more efficiently in terms of power consumption. Minimizing the operational power typically depends on the strategy (e. g., lightpath bypass, lightpath non-bypass and traffic grooming) and operations (e. g., electronic domain versus optical domain). We consider a typical optical backbone network model, and develop a model which minimizes the power consumption. Performance calculation shows that our method consumes less power compared to traffic grooming approach.

An asymptotically optimal push-pull method for multicasting over a random network

We consider multicast flow problems where either all of the nodes or only a subset of the nodes may be in session. Traffic from each node in the session has to be sent to every other node in the session. If the session does not consist of all the nodes, the remaining nodes act as relays. The nodes are connected by undirected edges whose capacities are independent and identically distributed random variables. We study the asymptotics of the capacity region (with network coding) in the limit of a large number of nodes, and show that the normalized sum rate converges to a constant almost surely. We then provide a decentralized push-pull algorithm that asymptotically achieves this normalized sum rate.

Measurement Based Impromptu Deployment of a Multi-Hop Wireless Relay Network

We study the problem of optimal sequential (''as-you-go'') deployment of wireless relay nodes, as a person walks along a line of random length (with a known distribution). The objective is to create an impromptu multihop wireless network for connecting a packet source to be placed at the end of the line with a sink node located at the starting point, to operate in the light traffic regime. In walking from the sink towards the source, at every step, measurements yield the transmit powers required to establish links to one or more previously placed nodes. Based on these measurements, at every step, a decision is made to place a relay node, the overall system objective being to minimize a linear combination of the expected sum power (or the expected maximum power) required to deliver a packet from the source to the sink node and the expected number of relay nodes deployed. For each of these two objectives, two different relay selection strategies are considered: (i) each relay communicates with the sink via its immediate previous relay, (ii) the communication path can skip some of the deployed relays. With appropriate modeling assumptions, we formulate each of these problems as a Markov decision process (MDP). We provide the optimal policy structures for all these cases, and provide illustrations of the policies and their performance, via numerical results, for some typical parameters.

6PANview: Application performance conscious network monitoring for 6LoWPAN based WSNs

6PANview[1] is a Wireless Sensor Network(WSN) monitoring system for 6LoWPAN/RPL networks which we developed as an overlay network for a WSN application. A monitoring system, while performing its operations for maintaining the health of the monitored network, must also be conscious of its impact on the application performance, and must strive to minimize this impact. To this end, we propose a centralized scheduling algorithm within 6PANview which non-intrusively analyzes application traffic arrival patterns at the base station, identifies network idle periods and schedules monitoring activities. The proposed algorithm finds those periodic sequences which are likely to have given rise to the pattern of arrivals seen at the base station. Parts of those sequences are then extended to coarsely predict future traffic and find epochs where low traffic is predicted, in order to schedule monitoring traffic or other activities at these times. We present simulation results for the proposed prediction and scheduling algorithm and its implementation as part of 6PANview. As an enhancement, we briefly talk about using 6PANview's overlay network architecture for distributed scheduling.

Providing QoS in a cognitive radio network

We consider the problem of wireless channel allocation (whenever the channels are free) to multiple cognitive radio users in a Cognitive Radio Network (CRN) so as to satisfy their Quality of Service (QoS) requirements efficiently. The CRN base station may not know the channel states of all the users. The multiple channels are available at random times. In this setup Opportunistic Splitting can be an attractive solution. A disadvantage of this algorithm is that it requires the metrics of all users to be an independent, identically distributed sequence. However we use a recently generalized version of this algorithm in which the optimal parameters are learnt on-line through stochastic approximation and metrics can be Markov. We provide scheduling algorithms which maximize weighted-sum system throughput or are throughput or delay optimal. We also consider the scenario when some traffic streams are delay sensitive.

A shortest path tree based algorithm for relay placement in a wireless sensor network and its performance analysis

In this paper, we study a problem of designing a multi-hop wireless network for interconnecting sensors (hereafter called source nodes) to a Base Station (BS), by deploying a minimum number of relay nodes at a subset of given potential locations, while meeting a quality of service (QoS) objective specified as a hop count bound for paths from the sources to the BS. The hop count bound suffices to ensure a certain probability of the data being delivered to the BS within a given maximum delay under a light traffic model. We observe that the problem is NP-Hard. For this problem, we propose a polynomial time approximation algorithm based on iteratively constructing shortest path trees and heuristically pruning away the relay nodes used until the hop count bound is violated. Results show that the algorithm performs efficiently in various randomly generated network scenarios; in over 90% of the tested scenarios, it gave solutions that were either optimal or were worse than optimal by just one relay. We then use random graph techniques to obtain, under a certain stochastic setting, an upper bound on the average case approximation ratio of a class of algorithms (including the proposed algorithm) for this problem as a function of the number of source nodes, and the hop count bound. To the best of our knowledge, the average case analysis is the first of its kind in the relay placement literature. Since the design is based on a light traffic model, we also provide simulation results (using models for the IEEE 802.15.4 physical layer and medium access control) to assess the traffic levels up to which the QoS objectives continue to be met. (C) 2014 Elsevier B.V. All rights reserved.

Multi-agent Reinforcement Learning for Traffic Signal Control

Optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users amidst the rapid increase in the usage of vehicles. In this paper, we formulate the TSC problem as a discounted cost Markov decision process (MDP) and apply multi-agent reinforcement learning (MARL) algorithms to obtain dynamic TSC policies. We model each traffic signal junction as an independent agent. An agent decides the signal duration of its phases in a round-robin (RR) manner using multi-agent Q-learning with either is an element of-greedy or UCB 3] based exploration strategies. It updates its Q-factors based on the cost feedback signal received from its neighbouring agents. This feedback signal can be easily constructed and is shown to be effective in minimizing the average delay of the vehicles in the network. We show through simulations over VISSIM that our algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm 15] over two real road networks.