Diffusion Gradient Temporal Difference for Cooperative Reinforcement Learning with Linear Function Approximation

We introduce a diffusion-based algorithm in which multiple agents cooperate to predict a common and global statevalue function by sharing local estimates and local gradient information among neighbors. Our algorithm is a fully distributed implementation of the gradient temporal difference with linear function approximation, to make it applicable to multiagent settings. Simulations illustrate the benefit of cooperation in learning, as made possible by the proposed algorithm.

A unified framework for linear function approximation of value functions in stochastic control

This paper contributes with a unified formulation that merges previ- ous analysis on the prediction of the performance ( value function ) of certain sequence of actions ( policy ) when an agent operates a Markov decision process with large state-space. When the states are represented by features and the value function is linearly approxi- mated, our analysis reveals a new relationship between two common cost functions used to obtain the optimal approximation. In addition, this analysis allows us to propose an efficient adaptive algorithm that provides an unbiased linear estimate. The performance of the pro- posed algorithm is illustrated by simulation, showing competitive results when compared with the state-of-the-art solutions.

Optimal Piecewise Linear Function Approximation for GPU-based Applications

Many computer vision and human-computer interaction applications developed in recent years need evaluating complex and continuous mathematical functions as an essential step toward proper operation. However, rigorous evaluation of this kind of functions often implies a very high computational cost, unacceptable in real-time applications. To alleviate this problem, functions are commonly approximated by simpler piecewise-polynomial representations. Following this idea, we propose a novel, efficient, and practical technique to evaluate complex and continuous functions using a nearly optimal design of two types of piecewise linear approximations in the case of a large budget of evaluation subintervals. To this end, we develop a thorough error analysis that yields asymptotically tight bounds to accurately quantify the approximation performance of both representations. It provides an improvement upon previous error estimates and allows the user to control the trade-off between the approximation error and the number of evaluation subintervals. To guarantee real-time operation, the method is suitable for, but not limited to, an efficient implementation in modern Graphics Processing Units (GPUs), where it outperforms previous alternative approaches by exploiting the fixed-function interpolation routines present in their texture units. The proposed technique is a perfect match for any application requiring the evaluation of continuous functions, we have measured in detail its quality and efficiency on several functions, and, in particular, the Gaussian function because it is extensively used in many areas of computer vision and cybernetics, and it is expensive to evaluate.

New methods for the estimation of Takagi-Sugeno model based extended Kalman filter and its applications to optimal control for nonlinear systems

This paper describes new approaches to improve the local and global approximation (matching) and modeling capability of Takagi–Sugeno (T-S) fuzzy model. The main aim is obtaining high function approximation accuracy and fast convergence. The main problem encountered is that T-S identification method cannot be applied when the membership functions are overlapped by pairs. This restricts the application of the T-S method because this type of membership function has been widely used during the last 2 decades in the stability, controller design of fuzzy systems and is popular in industrial control applications. The approach developed here can be considered as a generalized version of T-S identification method with optimized performance in approximating nonlinear functions. We propose a noniterative method through weighting of parameters approach and an iterative algorithm by applying the extended Kalman filter, based on the same idea of parameters’ weighting. We show that the Kalman filter is an effective tool in the identification of T-S fuzzy model. A fuzzy controller based linear quadratic regulator is proposed in order to show the effectiveness of the estimation method developed here in control applications. An illustrative example of an inverted pendulum is chosen to evaluate the robustness and remarkable performance of the proposed method locally and globally in comparison with the original T-S model. Simulation results indicate the potential, simplicity, and generality of the algorithm. An illustrative example is chosen to evaluate the robustness. In this paper, we prove that these algorithms converge very fast, thereby making them very practical to use.

A new approach to fuzzy estimation of Takagi-Sugeno model and its applications to optimal control for nonlinear systems

An efficient approach is presented to improve the local and global approximation and modelling capability of Takagi-Sugeno (T-S) fuzzy model. The main aim is obtaining high function approximation accuracy. The main problem is that T-S identification method cannot be applied when the membership functions are overlapped by pairs. This restricts the use of the T-S method because this type of membership function has been widely used during the last two decades in the stability, controller design and are popular in industrial control applications. The approach developed here can be considered as a generalized version of T-S method with optimized performance in approximating nonlinear functions. A simple approach with few computational effort, based on the well known parameters' weighting method is suggested for tuning T-S parameters to improve the choice of the performance index and minimize it. A global fuzzy controller (FC) based Linear Quadratic Regulator (LQR) is proposed in order to show the effectiveness of the estimation method developed here in control applications. Illustrative examples of an inverted pendulum and Van der Pol system are chosen to evaluate the robustness and remarkable performance of the proposed method and the high accuracy obtained in approximating nonlinear and unstable systems locally and globally in comparison with the original T-S model. Simulation results indicate the potential, simplicity and generality of the algorithm.

Ripple cancellation technique applied to a synchronous buck converter to achieve very high bandwidth and very high efficiency envelope amplifier

This work presents a single stage converter for a high bandwidth and a high efficiency envelope amplifier. The current ripple cancellation technique is applied to a synchronous buck converter to cancel the output current ripple and to decrease the switching frequency without a reduction in the large signal bandwidth. The converter is modeled and the new design with ripple cancellation circuit is detailed. The advantages of the proposed design are presented and validated experimentally. The transfer function of the output filter of the buck converter with ripple cancellation circuit has been modeled and compared to measurements, showing a good correspondence. Experimental validation is provided at 4MHz of switching frequency for DC and variable output voltage for a sinusoidal and a 64QAM signal. Additional experimental validation of the efficiency improvement is provided, compared to the equivalent design (same bandwidth and output voltage ripple) of the conventional buck converter.

Wide bandwidth envelope trackers with reduced switching frequency for RF power amplifier

El requerimiento de proveer alta frecuencia de datos en los modernos sistema de comunicación inalámbricos resulta en complejas señales moduladas de radio-frequencia (RF) con un gran ancho de banda y alto ratio pico-promedio (PAPR). Para garantizar la linealidad del comportamiento, los amplificadores lineales de potencia comunes funcionan típicamente entre 4 y 10 dB de back-o_ desde la máxima potencia de salida, ocasionando una baja eficiencia del sistema. La eliminación y restauración de la evolvente (EER) y el seguimiento de la evolvente (ET) son dos prometedoras técnicas para resolver el problema de la eficiencia. Tanto en EER como en ET, es complicado diseñar un amplificador de potencia que sea eficiente para señales de RF de alto ancho de banda y alto PAPR. Una propuesta común para los amplificadores de potencia es incluir un convertidor de potencia de muy alta eficiencia operando a frecuencias más altas que el ancho de banda de la señal RF. En este caso, la potencia perdida del convertidor ocasionado por la alta frecuencia desaconseja su práctica cuando el ancho de banda es muy alto. La solución a este problema es el enfoque de esta disertación que presenta dos arquitecturas de amplificador evolvente: convertidor híbrido-serie con una técnica de evolvente lenta y un convertidor multinivel basado en un convertidor reductor multifase con control de tiempo mínimo. En la primera arquitectura, una topología híbrida está compuesta de una convertidor reductor conmutado y un regulador lineal en serie que trabajan juntos para ajustar la tensión de salida para seguir a la evolvente con precisión. Un algoritmo de generación de una evolvente lenta crea una forma de onda con una pendiente limitada que es menor que la pendiente máxima de la evolvente original. La salida del convertidor reductor sigue esa forma de onda en vez de la evolvente original usando una menor frecuencia de conmutación, porque la forma de onda no sólo tiene una pendiente reducida sino también un menor ancho de banda. De esta forma, el regulador lineal se usa para filtrar la forma de onda tiene una pérdida de potencia adicional. Dependiendo de cuánto se puede reducir la pendiente de la evolvente para producir la forma de onda, existe un trade-off entre la pérdida de potencia del convertidor reductor relacionada con la frecuencia de conmutación y el regulador lineal. El punto óptimo referido a la menor pérdida de potencia total del amplificador de evolvente es capaz de identificarse con la ayuda de modelo preciso de pérdidas que es una combinación de modelos comportamentales y analíticos de pérdidas. Además, se analiza el efecto en la respuesta del filtro de salida del convertidor reductor. Un filtro de dampeo paralelo extra es necesario para eliminar la oscilación resonante del filtro de salida porque el convertidor reductor opera en lazo abierto. La segunda arquitectura es un amplificador de evolvente de seguimiento de tensión multinivel. Al contrario que los convertidores que usan multi-fuentes, un convertidor reductor multifase se emplea para generar la tensión multinivel. En régimen permanente, el convertidor reductor opera en puntos del ciclo de trabajo con cancelación completa del rizado. El número de niveles de tensión es igual al número de fases de acuerdo a las características del entrelazamiento del convertidor reductor. En la transición, un control de tiempo mínimo (MTC) para convertidores multifase es novedosamente propuesto y desarrollado para cambiar la tensión de salida del convertidor reductor entre diferentes niveles. A diferencia de controles convencionales de tiempo mínimo para convertidores multifase con inductancia equivalente, el propuesto MTC considera el rizado de corriente por cada fase basado en un desfase fijo que resulta en diferentes esquemas de control entre las fases. La ventaja de este control es que todas las corrientes vuelven a su fase en régimen permanente después de la transición para que la siguiente transición pueda empezar muy pronto, lo que es muy favorable para la aplicación de seguimiento de tensión multinivel. Además, el control es independiente de la carga y no es afectado por corrientes de fase desbalanceadas. Al igual que en la primera arquitectura, hay una etapa lineal con la misma función, conectada en serie con el convertidor reductor multifase. Dado que tanto el régimen permanente como el estado de transición del convertidor no están fuertemente relacionados con la frecuencia de conmutación, la frecuencia de conmutación puede ser reducida para el alto ancho de banda de la evolvente, la cual es la principal consideración de esta arquitectura. La optimización de la segunda arquitectura para más alto anchos de banda de la evolvente es presentada incluyendo el diseño del filtro de salida, la frecuencia de conmutación y el número de fases. El área de diseño del filtro está restringido por la transición rápida y el mínimo pulso del hardware. La rápida transición necesita un filtro pequeño pero la limitación del pulso mínimo del hardware lleva el diseño en el sentido contrario. La frecuencia de conmutación del convertidor afecta principalmente a la limitación del mínimo pulso y a las pérdidas de potencia. Con una menor frecuencia de conmutación, el ancho de pulso en la transición es más pequeño. El número de fases relativo a la aplicación específica puede ser optimizado en términos de la eficiencia global. Otro aspecto de la optimización es mejorar la estrategia de control. La transición permite seguir algunas partes de la evolvente que son más rápidas de lo que el hardware puede soportar al precio de complejidad. El nuevo método de sincronización de la transición incrementa la frecuencia de la transición, permitiendo que la tensión multinivel esté más cerca de la evolvente. Ambas estrategias permiten que el convertidor pueda seguir una evolvente con un ancho de banda más alto que la limitación de la etapa de potencia. El modelo de pérdidas del amplificador de evolvente se ha detallado y validado mediante medidas. El mecanismo de pérdidas de potencia del convertidor reductor tiene que incluir las transiciones en tiempo real, lo cual es diferente del clásico modelos de pérdidas de un convertidor reductor síncrono. Este modelo estima la eficiencia del sistema y juega un papel muy importante en el proceso de optimización. Finalmente, la segunda arquitectura del amplificador de evolvente se integra con el amplificador de clase F. La medida del sistema EER prueba el ahorro de energía con el amplificador de evolvente propuesto sin perjudicar la linealidad del sistema. ABSTRACT The requirement of delivering high data rates in modern wireless communication systems results in complex modulated RF signals with wide bandwidth and high peak-to-average ratio (PAPR). In order to guarantee the linearity performance, the conventional linear power amplifiers typically work at 4 to 10 dB back-off from the maximum output power, leading to low system efficiency. The envelope elimination and restoration (EER) and envelope tracking (ET) are two promising techniques to overcome the efficiency problem. In both EER and ET, it is challenging to design efficient envelope amplifier for wide bandwidth and high PAPR RF signals. An usual approach for envelope amplifier includes a high-efficiency switching power converter operating at a frequency higher than the RF signal's bandwidth. In this case, the power loss of converter caused by high switching operation becomes unbearable for system efficiency when signal bandwidth is very wide. The solution of this problem is the focus of this dissertation that presents two architectures of envelope amplifier: a hybrid series converter with slow-envelope technique and a multilevel converter based on a multiphase buck converter with the minimum time control. In the first architecture, a hybrid topology is composed of a switched buck converter and a linear regulator in series that work together to adjust the output voltage to track the envelope with accuracy. A slow envelope generation algorithm yields a waveform with limited slew rate that is lower than the maximum slew rate of the original envelope. The buck converter's output follows this waveform instead of the original envelope using lower switching frequency, because the waveform has not only reduced slew rate but also reduced bandwidth. In this way, the linear regulator used to filter the waveform has additional power loss. Depending on how much reduction of the slew rate of envelope in order to obtain that waveform, there is a trade-off between the power loss of buck converter related to the switching frequency and the power loss of linear regulator. The optimal point referring to the lowest total power loss of this envelope amplifier is identified with the help of a precise power loss model that is a combination of behavioral and analytic loss model. In addition, the output filter's effect on the response is analyzed. An extra parallel damping filter is needed to eliminate the resonant oscillation of output filter L and C, because the buck converter operates in open loop. The second architecture is a multilevel voltage tracking envelope amplifier. Unlike the converters using multi-sources, a multiphase buck converter is employed to generate the multilevel voltage. In the steady state, the buck converter operates at complete ripple cancellation points of duty cycle. The number of the voltage levels is equal to the number of phases according the characteristics of interleaved buck converter. In the transition, a minimum time control (MTC) for multiphase converter is originally proposed and developed for changing the output voltage of buck converter between different levels. As opposed to conventional minimum time control for multiphase converter with equivalent inductance, the proposed MTC considers the current ripple of each phase based on the fixed phase shift resulting in different control schemes among the phases. The advantage of this control is that all the phase current return to the steady state after the transition so that the next transition can be triggered very soon, which is very favorable for the application of multilevel voltage tracking. Besides, the control is independent on the load condition and not affected by the unbalance of phase current. Like the first architecture, there is also a linear stage with the same function, connected in series with the multiphase buck converter. Since both steady state and transition state of the converter are not strongly related to the switching frequency, it can be reduced for wide bandwidth envelope which is the main consideration of this architecture. The optimization of the second architecture for wider bandwidth envelope is presented including the output filter design, switching frequency and the number of phases. The filter design area is restrained by fast transition and the minimum pulse of hardware. The fast transition needs small filter but the minimum pulse of hardware limitation pushes the filter in opposite way. The converter switching frequency mainly affects the minimum pulse limitation and the power loss. With lower switching frequency, the pulse width in the transition is smaller. The number of phases related to specific application can be optimized in terms of overall efficiency. Another aspect of optimization is improving control strategy. Transition shift allows tracking some parts of envelope that are faster than the hardware can support at the price of complexity. The new transition synchronization method increases the frequency of transition, allowing the multilevel voltage to be closer to the envelope. Both control strategies push the converter to track wider bandwidth envelope than the limitation of power stage. The power loss model of envelope amplifier is detailed and validated by measurements. The power loss mechanism of buck converter has to include the transitions in real time operation, which is different from classical power loss model of synchronous buck converter. This model estimates the system efficiency and play a very important role in optimization process. Finally, the second envelope amplifier architecture is integrated with a Class F amplifier. EER system measurement proves the power saving with the proposed envelope amplifier without disrupting the linearity performance.