Dynamic equilibrium through reinforcement learning

Reinforcement Learning is an area of Machine Learning that deals with how an agent should take actions in an environment such as to maximize the notion of accumulated reward. This type of learning is inspired by the way humans learn and has led to the creation of various algorithms for reinforcement learning. These algorithms focus on the way in which an agent’s behaviour can be improved, assuming independence as to their surroundings. The current work studies the application of reinforcement learning methods to solve the inverted pendulum problem. The importance of the variability of the environment (factors that are external to the agent) on the execution of reinforcement learning agents is studied by using a model that seeks to obtain equilibrium (stability) through dynamism – a Cart-Pole system or inverted pendulum. We sought to improve the behaviour of the autonomous agents by changing the information passed to them, while maintaining the agent’s internal parameters constant (learning rate, discount factors, decay rate, etc.), instead of the classical approach of tuning the agent’s internal parameters. The influence of changes on the state set and the action set on an agent’s capability to solve the Cart-pole problem was studied. We have studied typical behaviour of reinforcement learning agents applied to the classic BOXES model and a new form of characterizing the environment was proposed using the notion of convergence towards a reference value. We demonstrate the gain in performance of this new method applied to a Q-Learning agent.

Analysis of ground reaction force and electromyographic activity of the gastrocnemius muscle during double support

O documento em anexo encontra-se na versão post-print (versão corrigida pelo editor).

On the robustness of the slotine-Li and the FPT/SVD-based adaptive controllers

A comparative study concerning the robustness of a novel, Fixed Point Transformations/Singular Value Decomposition (FPT/SVD)-based adaptive controller and the Slotine-Li (S&L) approach is given by numerical simulations using a three degree of freedom paradigm of typical Classical Mechanical systems, the cart + double pendulum. The effects of the imprecision of the available dynamical model, presence of dynamic friction at the axles of the drives, and the existence of external disturbance forces unknown and not modeled by the controller are considered. While the Slotine-Li approach tries to identify the parameters of the formally precise, available analytical model of the controlled system with the implicit assumption that the generalized forces are precisely known, the novel one makes do with a very rough, affine form and a formally more precise approximate model of that system, and uses temporal observations of its desired vs. realized responses. Furthermore, it does not assume the lack of unknown perturbations caused either by internal friction and/or external disturbances. Its another advantage is that it needs the execution of the SVD as a relatively time-consuming operation on a grid of a rough system-model only one time, before the commencement of the control cycle within which it works only with simple computations. The simulation examples exemplify the superiority of the FPT/SVD-based control that otherwise has the deficiency that it can get out of the region of its convergence. Therefore its design and use needs preliminary simulation investigations. However, the simulations also exemplify that its convergence can be guaranteed for various practical purposes.

Improved Numerical Simulation for a Novel Adaptive Control Using Fractional Order Derivatives

A novel control technique is investigated in the adaptive control of a typical paradigm, an approximately and partially modeled cart plus double pendulum system. In contrast to the traditional approaches that try to build up ”complete” and ”permanent” system models it develops ”temporal” and ”partial” ones that are valid only in the actual dynamic environment of the system, that is only within some ”spatio-temporal vicinity” of the actual observations. This technique was investigated for various physical systems via ”preliminary” simulations integrating by the simplest 1st order finite element approach for the time domain. In 2004 INRIA issued its SCILAB 3.0 and its improved numerical simulation tool ”Scicos” making it possible to generate ”professional”, ”convenient”, and accurate simulations. The basic principles of the adaptive control, the typical tools available in Scicos, and others developed by the authors, as well as the improved simulation results and conclusions are presented in the contribution.

Application of wavelets to the study of political history

\The idea that social processes develop in a cyclical manner is somewhat like a `Lorelei'. Researchers are lured to it because of its theoretical promise, only to become entangled in (if not wrecked by) messy problems of empirical inference. The reasoning leading to hypotheses of some kind of cycle is often elegant enough, yet the data from repeated observations rarely display the supposed cyclical pattern. (...) In addition, various `schools' seem to exist which frequently arrive at di erent conclusions on the basis of the same data." (van der Eijk and Weber 1987:271). Much of the empirical controversies around these issues arise because of three distinct problems: the coexistence of cycles of di erent periodicities, the possibility of transient cycles and the existence of cycles without xed periodicity. In some cases, there are no reasons to expect any of these phenomena to be relevant. Seasonality caused by Christmas is one such example (Wen 2002). In such cases, researchers mostly rely on spectral analysis and Auto-Regressive Moving-Average (ARMA) models to estimate the periodicity of cycles.1 However, and this is particularly true in social sciences, sometimes there are good theoretical reasons to expect irregular cycles. In such cases, \the identi cation of periodic movement in something like the vote is a daunting task all by itself. When a pendulum swings with an irregular beat (frequency), and the extent of the swing (amplitude) is not constant, mathematical functions like sine-waves are of no use."(Lebo and Norpoth 2007:73) In the past, this di culty has led to two di erent approaches. On the one hand, some researchers dismissed these methods altogether, relying on informal alternatives that do not meet rigorous standards of statistical inference. Goldstein (1985 and 1988), studying the severity of Great power wars is one such example. On the other hand, there are authors who transfer the assumptions of spectral analysis (and ARMA models) into fundamental assumptions about the nature of social phenomena. This type of argument was produced by Beck (1991) who, in a reply to Goldstein (1988), claimed that only \ xed period models are meaningful models of cyclic phenomena".We argue that wavelet analysis|a mathematical framework developed in the mid-1980s (Grossman and Morlet 1984; Goupillaud et al. 1984) | is a very viable alternative to study cycles in political time-series. It has the advantage of staying close to the frequency domain approach of spectral analysis while addressing its main limitations. Its principal contribution comes from estimating the spectral characteristics of a time-series as a function of time, thus revealing how its di erent periodic components may change over time. The rest of article proceeds as follows. In the section \Time-frequency Analysis", we study in some detail the continuous wavelet transform and compare its time-frequency properties with the more standard tool for that purpose, the windowed Fourier transform. In the section \The British Political Pendulum", we apply wavelet analysis to essentially the same data analyzed by Lebo and Norpoth (2007) and Merrill, Grofman and Brunell (2011) and try to provide a more nuanced answer to the same question discussed by these authors: do British electoral politics exhibit cycles? Finally, in the last section, we present a concise list of future directions.

Pimentas do gênero Capsicum cultivadas em Roraima, Amazônia Brasileira. I. Espécies domesticadas

Foram inventariadas as pimentas domesticadas do gênero Capsicum que são cultivadas no Estado de Roraima, extremo norte da Amazônia brasileira. O levantamento foi realizado em comunidades indígenas e não indígenas. Dos 163 acessos registrados, C. chinense Jacq. (76,7%) foi a espécie com o maior número, seguida de C. frutescens L. (9,8%), C. annuum L. (8,0%) e C. baccatum v. pendulum Wild. (5,5%). As formas de fruto mais encontradas foram "alongada" (42,9%) e "ovalada" (27,0%). C. chinense apresentou a maior diversidade de formas enquanto que as demais estavam concentradas na forma "alongada". A cor predominante dos frutos maduros foi a vermelha (64,4%). Isoladamente, C. chinense foi melhor distribuída entre as cores básicas amarela (44,8%) e vermelha (55,2%), independente das diferentes tonalidades assumidas por cada acesso (alaranjado, vermelho-escuro, etc). O nível de pungência sensorial com maior número de registros foi o "alto" (62,6%), seguido do "médio" (16,0%), "baixo" (15,3%) e "muito alto" (6,1%). Dos 105 acessos de coloração vermelha, 67,6% possuía pungência "alta" ou "muito alta". C. chinense do tipo "murupi" e "olho-de-peixe", juntamente com "malagueta" (C. frutescens) são os morfotipos mais tradicionalmente consumidos entre as comunidades indígenas locais.

Bimodal information increases spontaneous interpersonal synchronization of goal directed upper limb movements

When interacting with each other, people often synchronize spontaneously their movements, e.g. during pendulum swinging, chair rocking[5], walking [4][7], and when executing periodic forearm movements[3].Although the spatiotemporal information that establishes the coupling, leading to synchronization, might be provided by several perceptual systems, the systematic study of different sensory modalities contribution is widely neglected. Considering a) differences in the sensory dominance on the spatial and temporal dimension[5] , b) different cue combination and integration strategies [1][2], and c) that sensory information might provide different aspects of the same event, synchronization should be moderated by the type of sensory modality. Here, 9 naïve participants placed a bottle periodically between two target zones, 40 times, in 12 conditions while sitting in front of a confederate executing the same task. The participant could a) see and hear, b) see , c) hear the confederate, d) or audiovisual information about the movements of the confederate was absent. The couple started in 3 different relative positions (i.e., in-phase, anti-phase, out of phase). A retro-reflective marker was attached to the top of the bottles. Bottle displacement was captured by a motion capture system. We analyzed the variability of the continuous relative phase reflecting the degree of synchronization. Results indicate the emergence of spontaneous synchronization, an increase with bimodal information, and an influence of the initial phase relation on the particular synchronization pattern. Results have theoretical implication for studying cue combination in interpersonal coordination and are consistent with coupled oscillator models.

Demonstrative example of application

"Series title: Springerbriefs in applied sciences and technology, ISSN 2191-530X"

Demonstrative application examples

"Series: Solid mechanics and its applications, vol. 226"

On the frictional contacts in multibody system dynamics

"Series title: Computational methods in applied sciences, ISSN1871-3033, vol. 42"

Estudi: microcontrolador LPC2119 - ARM. Cas pràctic: anàlisi, disseny i construcció d'un robot amb pèndul invertit

Estudi de l'arquitectura i prestacions del microcontrolador LPC2119 tot implementant la proposta d’un cas pràctic. En la besant teòrica, es fa una anàlisi acurada del dispositiu LPC2119, enumerant les principals característiques i exposant les seves parts, aprofundint sobretot en l’arquitectura i core ARM que incorpora. En l'àmbit pràctic, s'introdueix el problema del pèndul invertit com a proposta per a ser integrada sobre un robot que exploti les funcionalitats del dispositiu integrat presentades a l'estudi teòric.

La correspondencia entre Robert Moray y Christiaan Huygens (1661-1669): encuentros y desencuentros en la promoción del conocimiento

Sir Robert Moray (1608/9-1673) fue un soldado, cortesano y "hombre de ciencia" escocés, que estuvo en el exilio durante el período de Oliver Cromwell. Poco después de su regreso a Inglaterra en 1660 y gracias en gran medida a su amistad con Carlos II, aparece vinculado al grupo que formará la Royal Society de Londres y será nombrado el primer presidente de la institución durante los primeros meses. Moray ha sido reconocido como una figura imprescindible para entender la consolidación de la Royal Society. Establece, además, una correspondencia muy importante con Christiaan Huygens, en donde aparecen tratados temas de gran relevancia en la década de 1660, tales como la determinación de la longitud en el mar mediante el uso del reloj de péndulo y la construcción y experimentación con la máquina neumática (emblema del proyecto experimental de Robert Boyle). En esta correspondencia aparecen reflejadas, así mismo, las tensiones sobre los problemas de prioridad en diferentes áreas de conocimiento. Una de estas agrias polémicas es la que enfrenta a James Gregory y a Huygens, que acabará con la relación epistolar entre Moray y el sabio holandés.

Estudio del péndulo doble

L'objectiu del projecte “Estudi del pèndol doble” és analitzar eldesenvolupament d'aquest sistema físic i del seu comportament dinàmicmitjançant aplicacions informàtiques.En l'estudi es van utilitzar dues aplicacions informàtiques: MATLAB en laseva versió 2009b per a l'anàlisi del pèndol simple, i la setzena versió deMAPLE per a la resolució i l'anàlisi gràfica de les equacions diferencials quegovernen el sistema.Un cop analitzat el sistema s'aplicarà a un problema mecànic, que seràla creació d'una atracció de fira basada en aquest mecanisme.L'anàlisi del problema mecànic es tindrà en compte optant per diversesconfiguracions i, un cop escollits els paràmetres més adequats, escorregiran en funció de les necessitats de fabricació per a tornar a obtenirles dades i poder analitzar si són vàlides.

Disseny i construcció d'un pèndol invertit

Aquest projecte consisteix en l’estudi del problema clàssic del pèndol invertit en l’àrea de control. Per tal de poder realitzar aquest estudi s’ha construït un prototip que simuli el comportament d’un pèndol invertit. Seguidament es dissenyen uns controladors PID i LQR per aquest prototip. Finalment s’escull el controlador LQR, que és per al qual es realitzen les simulacions i es programa el prototip real.