Solution of MDPS using simulation-based value iteration

This article proposes a three-timescale simulation based algorithm for solution of infinite horizon Markov Decision Processes (MDPs). We assume a finite state space and discounted cost criterion and adopt the value iteration approach. An approximation of the Dynamic Programming operator T is applied to the value function iterates. This 'approximate' operator is implemented using three timescales, the slowest of which updates the value function iterates. On the middle timescale we perform a gradient search over the feasible action set of each state using Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates, thus finding the minimizing action in T. On the fastest timescale, the 'critic' estimates, over which the gradient search is performed, are obtained. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is also presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are performed using the proposed algorithm. The results obtained are verified against classical value iteration where the feasible set is suitably discretized. Over such a discretized setting, a variant of the algorithm of [12] is compared and the proposed algorithm is found to converge faster.

Optimal threshold policies for admission control in communication networks via discrete parameter stochastic approximation

The problem of admission control of packets in communication networks is studied in the continuous time queueing framework under different classes of service and delayed information feedback. We develop and use a variant of a simulation based two timescale simultaneous perturbation stochastic approximation (SPSA) algorithm for finding an optimal feedback policy within the class of threshold type policies. Even though SPSA has originally been designed for continuous parameter optimization, its variant for the discrete parameter case is seen to work well. We give a proof of the hypothesis needed to show convergence of the algorithm on our setting along with a sketch of the convergence analysis. Extensive numerical experiments with the algorithm are illustrated for different parameter specifications. In particular, we study the effect of feedback delays on the system performance.

Moment-Matched Lognormal Modeling of Uplink Interference with Power Control and Cell Selection

We develop an alternate characterization of the statistical distribution of the inter-cell interference power observed in the uplink of CDMA systems. We show that the lognormal distribution better matches the cumulative distribution and complementary cumulative distribution functions of the uplink interference than the conventionally assumed Gaussian distribution and variants based on it. This is in spite of the fact that many users together contribute to uplink interference, with the number of users and their locations both being random. Our observations hold even in the presence of power control and cell selection, which have hitherto been used to justify the Gaussian distribution approximation. The parameters of the lognormal are obtained by matching moments, for which detailed analytical expressions that incorporate wireless propagation, cellular layout, power control, and cell selection parameters are developed. The moment-matched lognormal model, while not perfect, is an order of magnitude better in modeling the interference power distribution.

A reinforcement learning based algorithm for finite horizon Markov decision processes

We develop a simulation based algorithm for finite horizon Markov decision processes with finite state and finite action space. Illustrative numerical experiments with the proposed algorithm are shown for problems in flow control of communication networks and capacity switching in semiconductor fabrication.

Switching frequency variation control in hysteresis PWM controller for IM drives using variable parabolic bands for current error space phasor

Variation of switching frequency over the entire operating speed range of an induction motor (M drive is the major problem associated with conventional two-level three-phase hysteresis controller as well as the space phasor based PWM hysteresis controller. This paper describes a simple hysteresis current controller for controlling the switching frequency variation in the two-level PWM inverter fed IM drives for various operating speeds. A novel concept of continuously variable hysteresis boundary of current error space phasor with the varying speed of the IM drive is proposed in the present work. The variable parabolic boundary for the current error space phasor is suggested for the first time in this paper for getting the switching frequency pattern with the hysteresis controller, similar to that of the constant switching frequency voltage-controlled space vector PWM (VC-SVPWM) based inverter fed IM drive. A generalized algorithm is also developed to determine parabolic boundary for controlling the switching frequency variation, for any IM load. Only the adjacent inverter voltage vectors forming a triangular sector, in which tip of the machine voltage vector ties, are switched to keep current error space vector within the parabolic boundary. The controller uses a self-adaptive sector identification logic, which provides smooth transition between the sectors and is capable of taldng the inverter up to six-step mode of operation, if demanded by drive system. The proposed scheme is simulated and experimentally verified on a 3.7 kW IM drive.

Robust fixed order lateral H-2 controller for micro air vehicle

This paper presents a robust fixed order H-2 controller design using Strengthened discrete optimal projection equations, which approximate the first order necessary optimality condition. The novelty of this work is the application of the robust H-2 controller to a micro aerial vehicle named Sarika2 developed in house. The controller is designed in discrete domain for the lateral dynamics of Sarika2 in the presence of low frequency atmospheric turbulence (gust) and high frequency sensor noise. The design specification includes simultaneous stabilization, disturbance rejection and noise attenuation over the entire flight envelope of the vehicle. The resulting controller performance is comprehensively analyzed by means of simulation.

Generation of parabolic trajectories for current error space phasor similar to an SVPWM controller for control of switching frequency variation in current hysteresis PWM controlled IM drives

Switching frequency variation over a fundamental period is a major problem associated with hysteresis controller based VSI fed IM drives. This paper describes a novel concept of generating parabolic trajectories for current error space phasor for controlling the switching frequency variation in the hysteresis controller based two-level inverter fed IM drives. A generalized algorithm is developed to determine unique set of parabolic trajectories for different speeds of operation for any given IM load. Proposed hysteresis controller provides the switching frequency spectrum of inverter output voltage, similar to that of the constant switching frequency VC-SVPWM based IM drive. The scheme is extensively simulated and experimentally verified on a 3.7 kW IM drive for steady state and transient performance.

Adaptive load control of the central processor in a distributed system with a star topology

The author presents adaptive control techniques for controlling the flow of real-time jobs from the peripheral processors (PPs) to the central processor (CP) of a distributed system with a star topology. He considers two classes of flow control mechanisms: (1) proportional control, where a certain proportion of the load offered to each PP is sent to the CP, and (2) threshold control, where there is a maximum rate at which each PP can send jobs to the CP. The problem is to obtain good algorithms for dynamically adjusting the control level at each PP in order to prevent overload of the CP, when the load offered by the PPs is unknown and varying. The author formulates the problem approximately as a standard system control problem in which the system has unknown parameters that are subject to change. Using well-known techniques (e.g., naive-feedback-controller and stochastic approximation techniques), he derives adaptive controls for the system control problem. He demonstrates the efficacy of these controls in the original problem by using the control algorithms in simulations of a queuing model of the CP and the load controls.

Multiple contour extraction from graylevel images using an artificial neural network

For active contour modeling (ACM), we propose a novel self-organizing map (SOM)-based approach, called the batch-SOM (BSOM), that attempts to integrate the advantages of SOM- and snake-based ACMs in order to extract the desired contours from images. We employ feature points, in the form of ail edge-map (as obtained from a standard edge-detection operation), to guide the contour (as in the case of SOM-based ACMs) along with the gradient and intensity variations in a local region to ensure that the contour does not "leak" into the object boundary in case of faulty feature points (weak or broken edges). In contrast with the snake-based ACMs, however, we do not use an explicit energy functional (based on gradient or intensity) for controlling the contour movement. We extend the BSOM to handle extraction of contours of multiple objects, by splitting a single contour into as many subcontours as the objects in the image. The BSOM and its extended version are tested on synthetic binary and gray-level images with both single and multiple objects. We also demonstrate the efficacy of the BSOM on images of objects having both convex and nonconvex boundaries. The results demonstrate the superiority of the BSOM over others. Finally, we analyze the limitations of the BSOM.

Learning dynamic prices in multiseller electronic retail markets with price sensitive customers, stochastic demands, and inventory replenishments

In this paper, we use reinforcement learning (RL) as a tool to study price dynamics in an electronic retail market consisting of two competing sellers, and price sensitive and lead time sensitive customers. Sellers, offering identical products, compete on price to satisfy stochastically arriving demands (customers), and follow standard inventory control and replenishment policies to manage their inventories. In such a generalized setting, RL techniques have not previously been applied. We consider two representative cases: 1) no information case, were none of the sellers has any information about customer queue levels, inventory levels, or prices at the competitors; and 2) partial information case, where every seller has information about the customer queue levels and inventory levels of the competitors. Sellers employ automated pricing agents, or pricebots, which use RL-based pricing algorithms to reset the prices at random intervals based on factors such as number of back orders, inventory levels, and replenishment lead times, with the objective of maximizing discounted cumulative profit. In the no information case, we show that a seller who uses Q-learning outperforms a seller who uses derivative following (DF). In the partial information case, we model the problem as a Markovian game and use actor-critic based RL to learn dynamic prices. We believe our approach to solving these problems is a new and promising way of setting dynamic prices in multiseller environments with stochastic demands, price sensitive customers, and inventory replenishments.

A simple low-cost electrical switching analyzer for I-V characterization of switching samples and devices

The phenomena of nonlinear I-V behavior and electrical switching find extensive applications in power control, information storage, oscillators, etc. The study of I-V characteristics and switching parameters is necessary for the proper application of switching materials and devices. In the present work, a simple low-cost electrical switching analyzer has been developed for the measurement of the electrical characteristics of switching materials and devices. The system developed consists of a microcontroller-based excitation source and a high-speed data acquisition system. The design details of the excitation source, its interface with the high-speed data acquisition system and personal computer, and the details of the application software developed for automated measurements are described. Typical I-V characteristics and switching curves obtained with the system developed are also presented to illustrate the capability of the instrument developed.

Design of crosspoint-irredundant PLAs using minimal number of control inputs

The problem of determining a minimal number of control inputs for converting a programmable logic array (PLA) with undetectable faults to crosspoint-irredundant PLA for testing has been formulated as a nonstandard set covering problem. By representing subsets of sets as cubes, this problem has been reformulated as familiar problems. It is noted that this result has significance because a crosspoint-irredundant PLA can be converted to a completely testable PLA in a straightforward fashion, thus achieving very good fault coverage and easy testability.

Vector control of permanent magnet synchronous motor using DSP controller

PMSM drive with high dynamic response is the attractive solution for servo applications like robotics, machine tools, electric vehicles. Vector control is widely accepted control strategy for PMSM control, which enables decoupled control of torque and flux, this improving the transient response of torque and speed. As the vector control demands exhaustive real time computations, so the present work is implemented using TI DSP 320C240. Presently position and speed controller have been successfully tested. The feedback information used is shaft (rotor) position from the incremental encoder and two motor currents. We conclude with the hope to extend the present experimental set up for further research related to PMSM applications.

Sliding Modes for the Simulation of Mechanical and Electrical Systems Defined by Differential-Algebraic Equations

We offer a technique, motivated by feedback control and specifically sliding mode control, for the simulation of differential-algebraic equations (DAEs) that describe common engineering systems such as constrained multibody mechanical structures and electric networks. Our algorithm exploits the basic results from sliding mode control theory to establish a simulation environment that then requires only the most primitive of numerical solvers. We circumvent the most important requisite for the conventionalsimulation of DAEs: the calculation of a set of consistent initial conditions. Our algorithm, which relies on the enforcement and occurrence of sliding mode, will ensure that the algebraic equation is satisfied by the dynamic system even for inconsistent initial conditions and for all time thereafter. [DOI:10.1115/1.4001904]

Medium access control schemes for local area networks with multiple priority function

This paper deals with the development and performance evaluation of three modified versions of a scheme proposed for medium access control in local area networks. The original scheme implements a collision-free and fair medium arbitration by using a control wire in conjunction with a data bus. The modifications suggested in this paper are intended to realize the multiple priority function in local area networks.