Aprendizado por Reforço com Valores deInfluência em Sistemas Multi-Agente

We propose a new paradigm for collective learning in multi-agent systems (MAS) as a solution to the problem in which several agents acting over the same environment must learn how to perform tasks, simultaneously, based on feedbacks given by each one of the other agents. We introduce the proposed paradigm in the form of a reinforcement learning algorithm, nominating it as reinforcement learning with influence values. While learning by rewards, each agent evaluates the relation between the current state and/or action executed at this state (actual believe) together with the reward obtained after all agents that are interacting perform their actions. The reward is a result of the interference of others. The agent considers the opinions of all its colleagues in order to attempt to change the values of its states and/or actions. The idea is that the system, as a whole, must reach an equilibrium, where all agents get satisfied with the obtained results. This means that the values of the state/actions pairs match the reward obtained by each agent. This dynamical way of setting the values for states and/or actions makes this new reinforcement learning paradigm the first to include, naturally, the fact that the presence of other agents in the environment turns it a dynamical model. As a direct result, we implicitly include the internal state, the actions and the rewards obtained by all the other agents in the internal state of each agent. This makes our proposal the first complete solution to the conceptual problem that rises when applying reinforcement learning in multi-agent systems, which is caused by the difference existent between the environment and agent models. With basis on the proposed model, we create the IVQ-learning algorithm that is exhaustive tested in repetitive games with two, three and four agents and in stochastic games that need cooperation and in games that need collaboration. This algorithm shows to be a good option for obtaining solutions that guarantee convergence to the Nash optimum equilibrium in cooperative problems. Experiments performed clear shows that the proposed paradigm is theoretical and experimentally superior to the traditional approaches. Yet, with the creation of this new paradigm the set of reinforcement learning applications in MAS grows up. That is, besides the possibility of applying the algorithm in traditional learning problems in MAS, as for example coordination of tasks in multi-robot systems, it is possible to apply reinforcement learning in problems that are essentially collaborative

Técnicas de inteligência artificial para a geração dinâmica de set points para uma coluna de destilação

Artificial Intelligence techniques are applied to improve performance of a simulated oil distillation system. The chosen system was a debutanizer column. At this process, the feed, which comes to the column, is segmented by heating. The lightest components become steams, by forming the LPG (Liquefied Petroleum Gas). The others components, C5+, continue liquid. In the composition of the LPG, ideally, we have only propane and butanes, but, in practice, there are contaminants, for example, pentanes. The objective of this work is to control pentane amount in LPG, by means of intelligent set points (SP s) determination for PID controllers that are present in original instrumentation (regulatory control) of the column. A fuzzy system will be responsible for adjusting the SP's, driven by the comparison between the molar fraction of the pentane present in the output of the plant (LPG) and the desired amount. However, the molar fraction of pentane is difficult to measure on-line, due to constraints such as: long intervals of measurement, high reliability and low cost. Therefore, an inference system was used, based on a multilayer neural network, to infer the pentane molar fraction through secondary variables of the column. Finally, the results shown that the proposed control system were able to control the value of pentane molar fraction under different operational situations

Dinâmica da Plasticidade Sináptica em neurônios do hipocampo durante ciclos de sono: um estudo computacional

Several research lines show that sleep favors memory consolidation and learning. It has been proposed that the cognitive role of sleep is derived from a global scaling of synaptic weights, able to homeostatically restore the ability to learn new things, erasing memories overnight. This phenomenon is typical of slow-wave sleep (SWS) and characterized by non-Hebbian mechanisms, i.e., mechanisms independent of synchronous neuronal activity. Another view holds that sleep also triggers the specific enhancement of synaptic connections, carrying out the embossing of certain mnemonic traces within a lattice of synaptic weights rescaled each night. Such an embossing is understood as the combination of Hebbian and non-Hebbian mechanisms, capable of increasing and decreasing respectively the synaptic weights in complementary circuits, leading to selective memory improvement and a restructuring of synaptic configuration (SC) that can be crucial for the generation of new behaviors ( insights ). The empirical findings indicate that initiation of Hebbian plasticity during sleep occurs in the transition of the SWS to the stage of rapid eye movement (REM), possibly due to the significant differences between the firing rates regimes of the stages and the up-regulation of factors involved in longterm synaptic plasticity. In this study the theories of homeostasis and embossing were compared using an artificial neural network (ANN) fed with action potentials recorded in the hippocampus of rats during the sleep-wake cycle. In the simulation in which the ANN did not apply the long-term plasticity mechanisms during sleep (SWS-transition REM), the synaptic weights distribution was re-scaled inexorably, for its mean value proportional to the input firing rate, erasing the synaptic weights pattern that had been established initially. In contrast, when the long-term plasticity is modeled during the transition SWSREM, an increase of synaptic weights were observed in the range of initial/low values, redistributing effectively the weights in a way to reinforce a subset of synapses over time. The results suggest that a positive regulation coming from the long-term plasticity can completely change the role of sleep: its absence leads to forgetting; its presence leads to a positive mnemonic change

Algoritmo SOM com estrutura hierárquica e dinâmica aplicado a compressão de imagens

ln this work the implementation of the SOM (Self Organizing Maps) algorithm or Kohonen neural network is presented in the form of hierarchical structures, applied to the compression of images. The main objective of this approach is to develop an Hierarchical SOM algorithm with static structure and another one with dynamic structure to generate codebooks (books of codes) in the process of the image Vector Quantization (VQ), reducing the time of processing and obtaining a good rate of compression of images with a minimum degradation of the quality in relation to the original image. Both self-organizing neural networks developed here, were denominated HSOM, for static case, and DHSOM, for the dynamic case. ln the first form, the hierarchical structure is previously defined and in the later this structure grows in an automatic way in agreement with heuristic rules that explore the data of the training group without use of external parameters. For the network, the heuristic mIes determine the dynamics of growth, the pruning of ramifications criteria, the flexibility and the size of children maps. The LBO (Linde-Buzo-Oray) algorithm or K-means, one ofthe more used algorithms to develop codebook for Vector Quantization, was used together with the algorithm of Kohonen in its basic form, that is, not hierarchical, as a reference to compare the performance of the algorithms here proposed. A performance analysis between the two hierarchical structures is also accomplished in this work. The efficiency of the proposed processing is verified by the reduction in the complexity computational compared to the traditional algorithms, as well as, through the quantitative analysis of the images reconstructed in function of the parameters: (PSNR) peak signal-to-noise ratio and (MSE) medium squared error

Utilização da lógica difusa no controle inteligente de sistemas eletro-hidráulicos

Electro-hydraulic servo-systems are widely employed in industrial applications such as robotic manipulators, active suspensions, precision machine tools and aerospace systems. They provide many advantages over electric motors, including high force to weight ratio, fast response time and compact size. However, precise control of electro-hydraulic systems, due to their inherent nonlinear characteristics, cannot be easily obtained with conventional linear controllers. Most flow control valves can also exhibit some hard nonlinearities such as deadzone due to valve spool overlap on the passage´s orifice of the fluid. This work describes the development of a nonlinear controller based on the feedback linearization method and including a fuzzy compensation scheme for an electro-hydraulic actuated system with unknown dead-band. Numerical results are presented in order to demonstrate the control system performance

Controle inteligente de sistemas eletroidráulicos utilizando redes neurais artificiais

This work describes the development of a nonlinear control strategy for an electro-hydraulic actuated system. The system to be controlled is represented by a third order ordinary differential equation subject to a dead-zone input. The control strategy is based on a nonlinear control scheme, combined with an artificial intelligence algorithm, namely, the method of feedback linearization and an artificial neural network. It is shown that, when such a hard nonlinearity and modeling inaccuracies are considered, the nonlinear technique alone is not enough to ensure a good performance of the controller. Therefore, a compensation strategy based on artificial neural networks, which have been notoriously used in systems that require the simulation of the process of human inference, is used. The multilayer perceptron network and the radial basis functions network as well are adopted and mathematically implemented within the control law. On this basis, the compensation ability considering both networks is compared. Furthermore, the application of new intelligent control strategies for nonlinear and uncertain mechanical systems are proposed, showing that the combination of a nonlinear control methodology and artificial neural networks improves the overall control system performance. Numerical results are presented to demonstrate the efficacy of the proposed control system

Simulação do caminhamento da água da chuva e herbicidas em palhadas utilizadas em sistemas de plantio direto

Um experimento foi conduzido no NuPAM/FCA/UNESP, Botucatu-SP, objetivando avaliar a dinâmica de retenção de água e o caminhamento de um traçante (simulando um herbicida) em diferentes coberturas mortas. Os tratamentos foram constituídos pelo monitoramento do traçante FD&C-1 pulverizado sobre coberturas mortas de cevada, trigo, aveia-preta colhida, aveia-preta rolada, azevém, milheto e capim-braquiária, nas quantidades de 3.000, 6.000 e 9.000 kg ha-1, antes e após simulação de chuvas. As repetições constituíram-se de oito conjuntos de PVC + funil + béquer com palha, onde, através da chuva lixiviada pelas palhadas e do peso dos suportes de PVC, foram estimadas a retenção e transposição da água, assim como quantificado o traçante extraído, através de procedimentos espectrofotométricos. Os diferentes tipos de resíduos culturais mostraram-se similares quanto à retenção da água da chuva, ocorrendo uniformização entre os primeiros 7,5 e 15 mm de precipitação. A formação de pontos secos associados a canais preferenciais de escorrimento induziu menor capacidade de embebição e retenção da água das chuvas pelas palhadas. As máximas capacidades médias de retenção da chuva pelas coberturas foram de 1,22, 1,99 e 2,59 mm para 3.000, 6.000 e 9.000 kg de matéria seca ha-1, respectivamente. As precipitações iniciais entre 10 e 20 mm foram fundamentais para o molhamento uniforme das palhadas e carregamento do traçante até o solo, independentemente do tipo e da quantidade de palha. Esse comportamento indica ser viável a utilização de programas similares de controle de plantas daninhas para diferentes tipos e quantidades de palha em sistemas de plantio direto.

Determinação de dados de equilíbrio de fases para sistemas aquosos com eletrólitos

In the present work are established initially the fundamental relationships of thermodynamics that govern the equilibrium between phases, the models used for the description of the behavior non ideal of the liquid and vapor phases in conditions of low pressures. This work seeks the determination of vapor-liquid equilibrium (VLE) data for a series of multicomponents mixtures of saturated aliphatic hydrocarbons, prepared synthetically starting from substances with analytical degree and the development of a new dynamic cell with circulation of the vapor phase. The apparatus and experimental procedures developed are described and applied for the determination of VLE data. VLE isobarics data were obtained through a Fischer s ebulliometer of circulation of both phases, for the systems pentane + dodecane, heptane + dodecane and decane + dodecane. Using the two new dynamic cells especially projected, of easy operation and low cost, with circulation of the vapor phase, data for the systems heptane + decane + dodecane, acetone + water, tween 20 + dodecane, phenol + water and distillation curves of a gasoline without addictive were measured. Compositions of the equilibrium phases were found by densimetry, chromatography, and total organic carbon analyzer. Calibration curves of density versus composition were prepared from synthetic mixtures and the behavior excess volumes were evaluated. The VLE data obtained experimentally for the hydrocarbon and aqueous systems were submitted to the test of thermodynamic consistency, as well as the obtained from the literature data for another binary systems, mainly in the bank DDB (Dortmund Data Bank), where the Gibbs-Duhem equation is used obtaining a satisfactory data base. The results of the thermodynamic consistency tests for the binary and ternary systems were evaluated in terms of deviations for applications such as model development. Later, those groups of data (tested and approved) were used in the KijPoly program for the determination of the binary kij parameters of the cubic equations of state original Peng-Robinson and with the expanded alpha function. These obtained parameters can be applied for simulation of the reservoirs petroleum conditions and of the several distillation processes found in the petrochemistry industry, through simulators. The two designed dynamic cells used equipments of national technology for the determination of VLE data were well succeed, demonstrating efficiency and low cost. Multicomponents systems, mixtures of components of different molecular weights and also diluted solutions may be studied in these developed VLE cells

Os Polímeros como Sistemas Complexos

Neste trabalho, através de simulações computacionais, identificamos os fenômenos físicos associados ao crescimento e a dinâmica de polímeros como sistemas complexos exibindo comportamentos não linearidades, caos, criticalidade auto-organizada, entre outros. No primeiro capítulo, iniciamos com uma breve introdução onde descrevemos alguns conceitos básicos importantes ao entendimento do nosso trabalho. O capítulo 2 consiste na descrição do nosso estudo da distribuição de segmentos num polímero ramificado. Baseado em cálculos semelhantes aos usados em cadeias poliméricas lineares, utilizamos o modelo de crescimento para polímeros ramificados (Branched Polymer Growth Model - BPGM) proposto por Lucena et al., e analisamos a distribuição de probabilidade dos monômeros num polímero ramificado em 2 dimensões, até então desconhecida. No capítulo seguinte estudamos a classe de universalidade dos polímeros ramificados gerados pelo BPGM. Utilizando simulações computacionais em 3 dimensões do modelo proposto por Lucena et al., calculamos algumas dimensões críticas (dimensões fractal, mínima e química) para tentar elucidar a questão da classe de universalidade. Ainda neste Capítulo, descrevemos um novo modelo para a simulação de polímeros ramificados que foi por nós desenvolvido de modo a poupar esforço computacional. Em seguida, no capítulo 4 estudamos o comportamento caótico do crescimento de polímeros gerados pelo BPGM. Partimos de polímeros criticamente organizados e utilizamos uma técnica muito semelhante aquela usada em transições de fase em Modelos de Ising para estudar propagação de danos chamada de Distância de Hamming. Vimos que a distância de Hamming para o caso dos polímeros ramificados se comporta como uma lei de potência, indicando um caráter não-extensivo na dinâmica de crescimento. No Capítulo 5 analisamos o movimento molecular de cadeias poliméricas na presença de obstáculos e de gradientes de potenciais. Usamos um modelo generalizado de reptação para estudar a difusão de polímeros lineares em meios desordenados. Investigamos a evolução temporal destas cadeias em redes quadradas e medimos os tempos característicos de transporte t. Finalizamos esta dissertação com um capítulo contendo a conclusão geral denoss o trabalho (Capítulo 6), mais dois apêndices (Apêndices A e B) contendo a fenomenologia básica para alguns conceitos que utilizaremos ao longo desta tese (Fractais e Percolação respectivamente) e um terceiro e ´ultimo apêndice (Apêndice C) contendo uma descrição de um programa de computador para simular o crescimentos de polímeros ramificados em uma rede quadrada

Dinâmica Estocástica e Cosmologia: alguns resultados analíticos

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior

Propagação de danos em sistemas cooperativos: propriedades termodinâmicas

High-precision calculations of the correlation functions and order parameters were performed in order to investigate the critical properties of several two-dimensional ferro- magnetic systems: (i) the q-state Potts model; (ii) the Ashkin-Teller isotropic model; (iii) the spin-1 Ising model. We deduced exact relations connecting specific damages (the difference between two microscopic configurations of a model) and the above mentioned thermodynamic quanti- ties which permit its numerical calculation, by computer simulation and using any ergodic dynamics. The results obtained (critical temperature and exponents) reproduced all the known values, with an agreement up to several significant figures; of particular relevance were the estimates along the Baxter critical line (Ashkin-Teller model) where the exponents have a continuous variation. We also showed that this approach is less sensitive to the finite-size effects than the standard Monte-Carlo method. This analysis shows that the present approach produces equal or more accurate results, as compared to the usual Monte Carlo simulation, and can be useful to investigate these models in circumstances for which their behavior is not yet fully understood

Dinâmica de tempos curtos aplicado ao modelo de ising diluído por sítio em duas dimensões

Existem vários métodos de simulação para calcular as propriedades críticas de sistemas; neste trabalho utilizamos a dinâmica de tempos curtos, com o intuito de testar a eficiência desta técnica aplicando-a ao modelo de Ising com diluição de sítios. A Dinâmica de tempos curtos em combinação com o método de Monte Carlos verificou que mesmo longe do equilíbrio termodinâmico o sistema já se mostra insensível aos detalhes microscópicos das interações locais e portanto, o seu comportamento universal pode ser estudado ainda no regime de não-equilíbrio, evitando-se o problema do alentecimento crítico ( critical slowing down ) a que sistema em equilíbrio fica submetido quando está na temperatura crítica. O trabalho de Huse e Janssen mostrou um comportamento universal e uma lei de escala nos sistemas críticos fora do equilíbrio e identificou a existência de um novo expoente crítico dinâmico θ, associado ao comportamento anômalo da magnetização. Fazemos uima breve revisão das transições de fase e fenômeno críticos. Descrevemos o modelo de Ising, a técnica de Monte Carlo e por final, a dinâmica de tempos curtos. Aplicamos a dinâmica de tempos curtos para o modelo de Insing ferromagnéticos em uma rede quadrada com diluição de sítios. Calculamos o expoente dinâmicos θ e z, onde verificamos que existe quebra de classe de universilidade com relação às diferentes concentrações de sítios (p=0.70,0.75,0.80,0.85,0.90,0.95,1.00). calculamos também os expoentes estáticos β e v, onde encontramos pequenas variações com a desordem. Finalmente, apresentamos nossas conclusões e possíveis extensões deste trabalho

Estudo de sistemas complexos com interações de longo alcance : percolação, redes e tráfego

In this thesis we investigate physical problems which present a high degree of complexity using tools and models of Statistical Mechanics. We give a special attention to systems with long-range interactions, such as one-dimensional long-range bondpercolation, complex networks without metric and vehicular traffic. The flux in linear chain (percolation) with bond between first neighbor only happens if pc = 1, but when we consider long-range interactions , the situation is completely different, i.e., the transitions between the percolating phase and non-percolating phase happens for pc < 1. This kind of transition happens even when the system is diluted ( dilution of sites ). Some of these effects are investigated in this work, for example, the extensivity of the system, the relation between critical properties and the dilution, etc. In particular we show that the dilution does not change the universality of the system. In another work, we analyze the implications of using a power law quality distribution for vertices in the growth dynamics of a network studied by Bianconi and Barabási. It incorporates in the preferential attachment the different ability (fitness) of the nodes to compete for links. Finally, we study the vehicular traffic on road networks when it is submitted to an increasing flux of cars. In this way, we develop two models which enable the analysis of the total flux on each road as well as the flux leaving the system and the behavior of the total number of congested roads

Estudo de propriedades críticas em sistemas complexos: método de procura automática em sistemas tipo Ising e percolação de longo alcance

The new technique for automatic search of the order parameters and critical properties is applied to several well-know physical systems, testing the efficiency of such a procedure, in order to apply it for complex systems in general. The automatic-search method is combined with Monte Carlo simulations, which makes use of a given dynamical rule for the time evolution of the system. In the problems inves¬tigated, the Metropolis and Glauber dynamics produced essentially equivalent results. We present a brief introduction to critical phenomena and phase transitions. We describe the automatic-search method and discuss some previous works, where the method has been applied successfully. We apply the method for the ferromagnetic fsing model, computing the critical fron¬tiers and the magnetization exponent (3 for several geometric lattices. We also apply the method for the site-diluted ferromagnetic Ising model on a square lattice, computing its critical frontier, as well as the magnetization exponent f3 and the susceptibility exponent 7. We verify that the universality class of the system remains unchanged when the site dilution is introduced. We study the problem of long-range bond percolation in a diluted linear chain and discuss the non-extensivity questions inherent to long-range-interaction systems. Finally we present our conclusions and possible extensions of this work

Propagação de danos: importância da atualização dinâmica, efeitos da diluição e propriedades termodinâmicas

Neste trabalho investigamos aspectos da propagação de danos em sistemas cooperativos, descritos por modelos de variáveis discretas (spins), mutuamente interagentes, distribuídas nos sítios de uma rede regular. Os seguintes casos foram examinados: (i) A influência do tipo de atualização (paralela ou sequencial) das configurações microscópicas, durante o processo de simulação computacional de Monte Carlo, no modelo de Ising em uma rede triangular. Observamos que a atualização sequencial produz uma transição de fase dinâmica (Caótica- Congelada) a uma temperatura TD ≈TC (Temperatura de Curie), para acoplamentos ferromagnéticos (TC=3.6409J/Kb) e antiferromagnéticos (TC=0). A atualização paralela, que neste caso é incapaz de diferenciar os dois tipos de acoplamentos, leva a uma transição em TD ≠TC; (ii) Um estudo do modelo de Ising na rede quadrada, com diluição temperada de sítios, mostrou que a técnica de propagação de danos é um eficiente método para o cálculo da fronteira crítica e da dimensão fractal do aglomerado percolante, já que os resultados obtidos (apesar de um esforço computacional relativamente modesto), são comparáveis àqueles resultantes da aplicação de outros métodos analíticos e/ou computacionais de alto empenho; (iii) Finalmente, apresentamos resultados analíticos que mostram como certas combinações especiais de danos podem ser utilizadas para o cálculo de grandezas termodinâmicas (parâmetros de ordem, funções de correlação e susceptibilidades) do modelo Nα x Nβ, o qual contém como casos particulares alguns dos modelos mais estudados em Mecânica Estatística (Ising, Potts, Ashkin Teller e Cúbico)