Simultaneous Perturbation Newton Algorithms for Simulation Optimization

We present a new Hessian estimator based on the simultaneous perturbation procedure, that requires three system simulations regardless of the parameter dimension. We then present two Newton-based simulation optimization algorithms that incorporate this Hessian estimator. The two algorithms differ primarily in the manner in which the Hessian estimate is used. Both our algorithms do not compute the inverse Hessian explicitly, thereby saving on computational effort. While our first algorithm directly obtains the product of the inverse Hessian with the gradient of the objective, our second algorithm makes use of the Sherman-Morrison matrix inversion lemma to recursively estimate the inverse Hessian. We provide proofs of convergence for both our algorithms. Next, we consider an interesting application of our algorithms on a problem of road traffic control. Our algorithms are seen to exhibit better performance than two Newton algorithms from a recent prior work.

Expediency of Penetration Ratio and Evaluation of Mean Throughput for Safety and Commercial Applications in VANETs

Vehicular ad hoc network (VANET) applications are principally categorized into safety and commercial applications. Efficient traffic management for routing an emergency vehicle is of paramount importance in safety applications of VANETs. In the first case, a typical example of a high dense urban scenario is considered to demonstrate the role of penetration ratio for achieving reduced travel time between source and destination points. The major requirement for testing these VANET applications is a realistic simulation approach which would justify the results prior to actual deployment. A Traffic Simulator coupled with a Network Simulator using a feedback loop feature is apt for realistic simulation of VANETs. Thus, in this paper, we develop the safety application using traffic control interface (TraCI), which couples SUMO (traffic simulator) and NS2 (network simulator). Likewise, the mean throughput is one of the necessary performance measures for commercial applications of VANETs. In the next case, commercial applications have been considered wherein the data is transferred amongst vehicles (V2V) and between roadside infrastructure and vehicles (I2V), for which the throughput is assessed.

Reinforcement Learning With Function Approximation for Traffic Signal Control

We propose, for the first time, a reinforcement learning (RL) algorithm with function approximation for traffic signal control. Our algorithm incorporates state-action features and is easily implementable in high-dimensional settings. Prior work, e. g., the work of Abdulhai et al., on the application of RL to traffic signal control requires full-state representations and cannot be implemented, even in moderate-sized road networks, because the computational complexity exponentially grows in the numbers of lanes and junctions. We tackle this problem of the curse of dimensionality by effectively using feature-based state representations that use a broad characterization of the level of congestion as low, medium, or high. One advantage of our algorithm is that, unlike prior work based on RL, it does not require precise information on queue lengths and elapsed times at each lane but instead works with the aforementioned described features. The number of features that our algorithm requires is linear to the number of signaled lanes, thereby leading to several orders of magnitude reduction in the computational complexity. We perform implementations of our algorithm on various settings and show performance comparisons with other algorithms in the literature, including the works of Abdulhai et al. and Cools et al., as well as the fixed-timing and the longest queue algorithms. For comparison, we also develop an RL algorithm that uses full-state representation and incorporates prioritization of traffic, unlike the work of Abdulhai et al. We observe that our algorithm outperforms all the other algorithms on all the road network settings that we consider.

Throughput Maximization of Delay-Constrained Traffic in Wireless Energy Harvesting Sensors

A wireless Energy Harvesting Sensor (EHS) needs to send data packets arriving in its queue over a fading channel at maximum possible throughput while ensuring acceptable packet delays. At the same time, it needs to ensure that energy neutrality is satisfied, i.e., the average energy drawn from a battery should equal the amount of energy deposited in it minus the energy lost due to the inefficiency of the battery. In this work, a framework is developed under which a system designer can optimize the performance of the EHS node using power control based on the current channel state information, when the EHS node employs a single modulation and coding scheme and the channel is Rayleigh fading. Optimal system parameters for throughput optimal, delay optimal and delay-constrained throughput optimal policies that ensure energy neutrality are derived. It is seen that a throughput optimal (maximal) policy is packet delay-unbounded and an average delay optimal (minimal) policy achieves negligibly small throughput. Finally, the influence of the harvested energy profile on the performance of the EHS is illustrated through the example of solar energy harvesting.

Reinforcement learning with average cost for adaptive control of traffic lights at intersections

We propose for the first time two reinforcement learning algorithms with function approximation for average cost adaptive control of traffic lights. One of these algorithms is a version of Q-learning with function approximation while the other is a policy gradient actor-critic algorithm that incorporates multi-timescale stochastic approximation. We show performance comparisons on various network settings of these algorithms with a range of fixed timing algorithms, as well as a Q-learning algorithm with full state representation that we also implement. We observe that whereas (as expected) on a two-junction corridor, the full state representation algorithm shows the best results, this algorithm is not implementable on larger road networks. The algorithm PG-AC-TLC that we propose is seen to show the best overall performance.

Performance of cellular CDMA with voice/data traffic with an SIR based admission control

We analyze the performance of an SIR based admission control strategy in cellular CDMA systems with both voice and data traffic. Most studies In the current literature to estimate CDMA system capacity with both voice and data traf-Bc do not take signal-tlFlnterference ratio (SIR) based admission control into account In this paper, we present an analytical approach to evaluate the outage probability for voice trafllc, the average system throughput and the mean delay for data traffic for a volce/data CDMA system which employs an SIR based admission controL We show that for a dataaniy system, an improvement of about 25% In both the Erlang capacity as well as the mean delay performance is achieved with an SIR based admission control as compared to code availability based admission control. For a mixed voice/data srtem with 10 Erlangs of voice traffic, the Lmprovement in the mean delay performance for data Is about 40%.Ah, for a mean delay of 50 ms with 10 Erlangs voice traffic, the data Erlang capacity improves by about 9%.

Multi-agent Reinforcement Learning for Traffic Signal Control

Optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users amidst the rapid increase in the usage of vehicles. In this paper, we formulate the TSC problem as a discounted cost Markov decision process (MDP) and apply multi-agent reinforcement learning (MARL) algorithms to obtain dynamic TSC policies. We model each traffic signal junction as an independent agent. An agent decides the signal duration of its phases in a round-robin (RR) manner using multi-agent Q-learning with either is an element of-greedy or UCB 3] based exploration strategies. It updates its Q-factors based on the cost feedback signal received from its neighbouring agents. This feedback signal can be easily constructed and is shown to be effective in minimizing the average delay of the vehicles in the network. We show through simulations over VISSIM that our algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm 15] over two real road networks.

Biomethanation of herbaceous biomass residues using 3-zone plug flow like digesters - A case study from India

Biomethanation of herbaceous biomass feedstock has the potential to provide clean energy source for cooking and other activities in areas where such biomass availability predominates. A biomethanation concept that involves fermentation of biomass residues in three steps, occurring in three zones of the fermentor is described. This approach while attempting take advantage of multistage reactors simplifies the reactor operation and obviates the need for a high degree of process control or complex reactor design. Typical herbaceous biomass decompose with a rapid VFA flux initially (with a tendency to float) followed by a slower decomposition showing balanced process of VFA generation and its utilization by methanogens that colonize biomass slowly. The tendency to float at the initial stages is suppressed by allowing previous days feed to hold it below digester liquid which permits VFA to disperse into the digester liquid without causing process inhibition. This approach has been used to build and operate simple biomass digesters to provide cooking gas in rural areas with weed and agro-residues. With appropriate modifications, the same concept has been used for digesting municipal solid wastes in small towns where large fermentors are not viable. With further modifications this concept has been used for solid-liquid feed fermentors. Methanogen colonized leaf biomass has been used as biofilm support to treat coffee processing wastewater as well as crop litter alternately in a year. During summer it functions as a biomass based biogas plants operating in the three-zone mode while in winter, feeding biomass is suspended and high strength coffee processing wastewater is let into the fermentor achieving over 90% BOD reduction. The early field experience of these fermentors is presented.

Some aspects of multilevel load-frequency control of a power system

The application of multilevel control strategies for load-frequency control of interconnected power systems is assuming importance. A large multiarea power system may be viewed as an interconnection of several lower-order subsystems, with possible change of interconnection pattern during operation. The solution of the control problem involves the design of a set of local optimal controllers for the individual areas, in a completely decentralised environment, plus a global controller to provide the corrective signal to account for interconnection effects. A global controller, based on the least-square-error principle suggested by Siljak and Sundareshan, has been applied for the LFC problem. A more recent work utilises certain possible beneficial aspects of interconnection to permit more desirable system performances. The paper reports the application of the latter strategy to LFC of a two-area power system. The power-system model studied includes the effects of excitation system and governor controls. A comparison of the two strategies is also made.

Experiences With WM: A Centralised Scheduling Approach for Performance Management of IEEE 802.11 Wireless LANs

In our earlier work ([1]) we proposed WLAN Manager (or WM) a centralised controller for QoS management of infrastructure WLANs based on the IEEE 802.11 DCF standards. The WM approach is based on queueing and scheduling packets in a device that sits between all traffic flowing between the APs and the wireline LAN, requires no changes to the AP or the STAs, and can be viewed as implementing a "Split-MAC" architecture. The objectives of WM were to manage various TCP performance related issues (such as the throughput "anomaly" when STAs associate with an AP with mixed PHY rates, and upload-download unfairness induced by finite AP buffers), and also to serve as the controller for VoIP admission control and handovers, and for other QoS management measures. In this paper we report our experiences in implementing the proposals in [1]: the insights gained, new control techniques developed, and the effectiveness of the WM approach in managing TCP performance in an infrastructure WLAN. We report results from a hybrid experiment where a physical WM manages actual TCP controlled packet flows between a server and clients, with the WLAN being simulated, and also from a small physical testbed with an actual AP.

Some aspects of multilevel load-frequency control of a power system

The application of multilevel control strategies for load-frequency control of interconnected power systems is assuming importance. A large multiarea power system may be viewed as an interconnection of several lower-order subsystems, with possible change of interconnection pattern during operation. The solution of the control problem involves the design of a set of local optimal controllers for the individual areas, in a completely decentralised environment, plus a global controller to provide the corrective signal to account for interconnection effects. A global controller, based on the least-square-error principle suggested by Siljak and Sundareshan, has been applied for the LFC problem. A more recent work utilises certain possible beneficial aspects of interconnection to permit more desirable system performances. The paper reports the application of the latter strategy to LFC of a two-area power system. The power-system model studied includes the effects of excitation system and governor controls. A comparison of the two strategies is also made.

Optimal Nonlinear Control and Estimation for a Reusable Launch Vehicle During Reentry Phase

In this paper a nonlinear optimal controller has been designed for aerodynamic control during the reentry phase of the Reusable Launch Vehicle (RLV). The controller has been designed based on a recently developed technique Optimal Dynamic Inversion (ODI). For full state feedback the controller has required full information about the system states. In this work an Extended Kalman filter (EKF) is developed to estimate the states. The vehicle (RLV) has been has been consider as a nonlinear Six-Degree-Of-Freedom (6-DOF) model. The simulation results shows that EKF gives a very good estimation of the states and it is working well with ODI. The resultant trajectories are very similar to those obtained by perfect state feedback using ODI only.

Optimal threshold policies for admission control in communication networks via discrete parameter stochastic approximation

The problem of admission control of packets in communication networks is studied in the continuous time queueing framework under different classes of service and delayed information feedback. We develop and use a variant of a simulation based two timescale simultaneous perturbation stochastic approximation (SPSA) algorithm for finding an optimal feedback policy within the class of threshold type policies. Even though SPSA has originally been designed for continuous parameter optimization, its variant for the discrete parameter case is seen to work well. We give a proof of the hypothesis needed to show convergence of the algorithm on our setting along with a sketch of the convergence analysis. Extensive numerical experiments with the algorithm are illustrated for different parameter specifications. In particular, we study the effect of feedback delays on the system performance.

Switching frequency variation control in hysteresis PWM controller for IM drives using variable parabolic bands for current error space phasor

Variation of switching frequency over the entire operating speed range of an induction motor (M drive is the major problem associated with conventional two-level three-phase hysteresis controller as well as the space phasor based PWM hysteresis controller. This paper describes a simple hysteresis current controller for controlling the switching frequency variation in the two-level PWM inverter fed IM drives for various operating speeds. A novel concept of continuously variable hysteresis boundary of current error space phasor with the varying speed of the IM drive is proposed in the present work. The variable parabolic boundary for the current error space phasor is suggested for the first time in this paper for getting the switching frequency pattern with the hysteresis controller, similar to that of the constant switching frequency voltage-controlled space vector PWM (VC-SVPWM) based inverter fed IM drive. A generalized algorithm is also developed to determine parabolic boundary for controlling the switching frequency variation, for any IM load. Only the adjacent inverter voltage vectors forming a triangular sector, in which tip of the machine voltage vector ties, are switched to keep current error space vector within the parabolic boundary. The controller uses a self-adaptive sector identification logic, which provides smooth transition between the sectors and is capable of taldng the inverter up to six-step mode of operation, if demanded by drive system. The proposed scheme is simulated and experimentally verified on a 3.7 kW IM drive.

Vector control of permanent magnet synchronous motor using DSP controller

PMSM drive with high dynamic response is the attractive solution for servo applications like robotics, machine tools, electric vehicles. Vector control is widely accepted control strategy for PMSM control, which enables decoupled control of torque and flux, this improving the transient response of torque and speed. As the vector control demands exhaustive real time computations, so the present work is implemented using TI DSP 320C240. Presently position and speed controller have been successfully tested. The feedback information used is shaft (rotor) position from the incremental encoder and two motor currents. We conclude with the hope to extend the present experimental set up for further research related to PMSM applications.