Dynamic equilibrium through reinforcement learning

Reinforcement Learning is an area of Machine Learning that deals with how an agent should take actions in an environment such as to maximize the notion of accumulated reward. This type of learning is inspired by the way humans learn and has led to the creation of various algorithms for reinforcement learning. These algorithms focus on the way in which an agent’s behaviour can be improved, assuming independence as to their surroundings. The current work studies the application of reinforcement learning methods to solve the inverted pendulum problem. The importance of the variability of the environment (factors that are external to the agent) on the execution of reinforcement learning agents is studied by using a model that seeks to obtain equilibrium (stability) through dynamism – a Cart-Pole system or inverted pendulum. We sought to improve the behaviour of the autonomous agents by changing the information passed to them, while maintaining the agent’s internal parameters constant (learning rate, discount factors, decay rate, etc.), instead of the classical approach of tuning the agent’s internal parameters. The influence of changes on the state set and the action set on an agent’s capability to solve the Cart-pole problem was studied. We have studied typical behaviour of reinforcement learning agents applied to the classic BOXES model and a new form of characterizing the environment was proposed using the notion of convergence towards a reference value. We demonstrate the gain in performance of this new method applied to a Q-Learning agent.

Análise funcional comparativa de algoritmos de aprendizagem por reforço

De entre todos os paradigmas de aprendizagem actualmente identificados, a Aprendizagem por Reforço revela-se de especial interesse e aplicabilidade nos inúmeros processos que nos rodeiam: desde a solitária sonda que explora o planeta mais remoto, passando pelo programa especialista que aprende a apoiar a decisão médica pela experiencia adquirida, até ao cão de brincar que faz as delícias da criança interagindo com ela e adaptando-se aos seus gostos, e todo um novo mundo que nos rodeia e apela crescentemente a que façamos mais e melhor nesta área. Desde o aparecimento do conceito de aprendizagem por reforço, diferentes métodos tem sido propostos para a sua concretização, cada um deles abordando aspectos específicos. Duas vertentes distintas, mas complementares entre si, apresentam-se como características chave do processo de aprendizagem por reforço: a obtenção de experiência através da exploração do espaço de estados e o aproveitamento do conhecimento obtido através dessa mesma experiência. Esta dissertação propõe-se seleccionar alguns dos métodos propostos mais promissores de ambas as vertentes de exploração e aproveitamento, efectuar uma implementação de cada um destes sobre uma plataforma modular que permita a simulação do uso de agentes inteligentes e, através da sua aplicação na resolução de diferentes configurações de ambientes padrão, gerar estatísticas funcionais que permitam inferir conclusões que retractem entre outros aspectos a sua eficiência e eficácia comparativas em condições específicas.

Sensor de glicose polarimétrico

Esta tese tem como principal objectivo a investigação teórica e experimental do desempenho de um sensor polarimétrico baseado num cristal líquido para medição da concentração de glicose. Recentemente uma série de sensores polarimétricos baseados em cristais líquidos foram propostos na literatura e receberam considerável interesse devido as suas características únicas. De facto, em comparação com outros moduladores electro-ópticos, o cristal líquido funciona com tensões mais baixas, tem baixo consumo de energia e maior ângulo de rotação. Além disso, este tipo de polarímetro pode ter pequenas dimensões que é uma característica interessante para dispositivos portáteis e compactos. Existem por outro lado algumas desvantagens, nomeadamente o facto do desempenho do polarímetro ser fortemente dependente do tipo de cristal líquido e da tensão a ele aplicada o que coloca desafios na escolha dos parâmetros óptimos de operação. Esta tese descreve o desenvolvimento do sensor polarimétrico, incluindo a integração dos componentes de óptica e electrónica, os algoritmos de processamento de sinal e um interface gráfico que facilita a programação de diversos parâmetros de operação e a calibração do sensor. Após a optimização dos parâmetros de operação verificou-se que o dispositivo mede a concentração da glicose em amostras com uma concentração de 8 mg/ml, com uma percentagem de erro inferior a 6% e um desvio padrão de 0,008o. Os resultados foram obtidos para uma amostra com percurso óptico de apenas 1 cm.

Aspiração manual de trombos no enfarte agudo do miocárdio com supradesnivelamento segmento ST: impacto da técnica de aspiração manual de trombos na reperfusão do miocárdio durante a intervenção coronária percutânea primária

Mestrado em Tecnologia de Diagnóstico e Intervenção Cardiovascular. Área de especialização: Intervenção Cardiovascular.

Integração de reacção e deliberação em agentes inteligentes

Trabalho Final de Mestrado para obtenção do grau de Mestre em Engenharia de Redes de Comunicação e Multimédia

Viabilidade do reforço sísmico de um edifício de pequeno porte em alvenaria de pedra ordinária

Trabalho Final de Mestrado para obtenção do grau de Mestre em Engenharia Civil na Área de Especialização de Estruturas

Self-scheduling and bidding strategies of thermal units with stochastic emission constraints

This paper is on the self-scheduling problem for a thermal power producer taking part in a pool-based electricity market as a price-taker, having bilateral contracts and emission-constrained. An approach based on stochastic mixed-integer linear programming approach is proposed for solving the self-scheduling problem. Uncertainty regarding electricity price is considered through a set of scenarios computed by simulation and scenario-reduction. Thermal units are modelled by variable costs, start-up costs and technical operating constraints, such as: forbidden operating zones, ramp up/down limits and minimum up/down time limits. A requirement on emission allowances to mitigate carbon footprint is modelled by a stochastic constraint. Supply functions for different emission allowance levels are accessed in order to establish the optimal bidding strategy. A case study is presented to illustrate the usefulness and the proficiency of the proposed approach in supporting biding strategies. (C) 2014 Elsevier Ltd. All rights reserved.

Non-selective optical wavelength-division multiplexing devices based on a-SiC:H multilayer heterostuctures

In this paper we present results on the optimization of multilayered a-SiC:H heterostructures for wavelength-division (de) multiplexing applications. The non selective WDM device is a double heterostructure in a glass/ITO/a-SiC:H (p-i-n) /a-SiC:H(-p) /a-Si:H(-i')/a-SiC:H (-n')/ITO configuration. The single or the multiple modulated wavelength channels are passed through the device, and absorbed accordingly to its wavelength, giving rise to a time dependent wavelength electrical field modulation across it. The effect of single or multiple input signals is converted to an electrical signal to regain the information (wavelength, intensity and frequency) of the incoming photogenerated carriers. Here, the (de) multiplexing of the channels is accomplished electronically, not optically. This approach offers advantages in terms of cost since several channels share the same optical components; and the electrical components are typically less expensive than the optical ones. An electrical model gives insight into the device operation.

Stacked photo-sensing devices based on SiC alloys A non-pixelled architecture for imagers and demultiplexing devices

In this review paper different designs based on stacked p-i'-n-p-i-n heterojunctions are presented and compared with the single p-i-n sensing structures. The imagers utilise self-field induced depletion layers for light detection and a modulated laser beam for sequential readout. The effect of the sensing element structure, cell configurations (single or tandem), and light source properties (intensity and wavelength) are correlated with the sensor output characteristics (light-to-dark sensivity, spatial resolution, linearity and S/N ratio). The readout frequency is optimized showing that scans speeds up to 104 lines per second can be achieved without degradation in the resolution. Multilayered p-i'-n-p-i-n heterostructures can also be used as wavelength-division multiplexing /demultiplexing devices in the visible range. Here the sensor element faces the modulated light from different input colour channels, each one with a specific wavelength and bit rate. By reading out the photocurrent at appropriated applied bias, the information is multiplexed or demultiplexed and can be transmitted or recovered again. Electrical models are present to support the sensing methodologies.

Perspectives on the electrically induced properties of electrospun cellulose/liquid crystal devices

A mat of electrospun cellulose fibers are deposed on transparent conductive oxide covered glass, and two such plates enclose a nematic liquid crystal. Thus two new types of Cellulose based Polymer Dispersed Liquid Crystal devices, based on hydroxypropylcellulose and Cellulose Acetate and the nematic liquid crystal E7 have been obtained. The current-voltage characteristics indicates ionic type conduction. Heating-cooling cycles have been applied on the samples and the activation energies have been determined. Simultaneously with the thermo-stimulated currents, the optical transmission dependence on the d.c. electric field and temperature was registered. ON-OFF switching times have been determined for different control voltages. (C) 2011 Elsevier B.V. All rights reserved.

Multilayer Architectures Based on a-SiC:H Material: Tunable Wavelength Filters in Optical Processing Devices

The characteristics of tunable wavelength filters based on a-SiC:H multilayered stacked pin cells are studied both theoretically and experimentally. The optical transducers were produced by PECVD and tested for a proper fine tuning of the cyan and yellow fluorescent proteins emission. The active device consists of a p-i'(a-SiC:H)-n/p-i(a-Si:H)-n heterostructures sandwiched between two transparent contacts. Experimental data on spectral response analysis, current-voltage characteristics and color and transmission rate discrimination are reported. Cyan and yellow fluorescent input channels were transmitted together, each one with a specific transmission rate and different intensities. The multiplexed optical signal was analyzed by reading out, under positive and negative applied voltages, the generated photocurrents. Results show that the optimized optical transducer has the capability of combining the transient fluorescent signals onto a single output signal without losing any specificity (color and intensity). It acts as a voltage controlled optical filter: when the applied voltages are chosen appropriately the transducer can select separately the cyan and yellow channel emissions (wavelength and frequency) and also to quantify their relative intensities. A theoretical analysis supported by a numerical simulation is presented.

Optoelectronic Logic Functions Based on Reconfigurable SiC Multilayer Devices

WDM multilayered SiC/Si devices based on a-Si:H and a-SiC:H filter design are approached from a reconfigurable point of view. Results show that the devices, under appropriated optical bias, act as reconfigurable active filters that allow optical switching and optoelectronic logic functions development. Under front violet irradiation the magnitude of the red and green channels are amplified and the blue and violet reduced. Violet back irradiation cuts the red channel, slightly influences the magnitude of the green and blue ones and strongly amplifies de violet channel. This nonlinearity provides the possibility for selective removal of useless wavelengths. Particular attention is given to the amplification coefficient weights, which allow taking into account the wavelength background effects when a band needs to be filtered from a wider range of mixed signals, or when optical active filter gates are used to select and filter input signals to specific output ports in WDM communication systems. A truth table of an encoder that performs 8-to-1 multiplexer (MUX) function is presented.

Optical processing devices based on a-SiCH multilayer architectures

Red, green and blue optical signals were directed to an a-SiC:H multilayered device, each one with a specific transmission rate. The combined optical signal was analyzed by reading out, under different applied voltages, the generated photocurrent. Results show that when a chromatic time dependent wavelength combination with different transmission rates irradiates the multilayered structure, the device operates as a tunable wavelength filter and can be used in wavelength division multiplexing systems for short range communications. An application to fluorescent proteins detection is presented. (C) 2010 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim

Logic functions based on optical bias controlled SIC Tandem devices

The purpose of this paper is the design of an optoelectronic circuit based on a-SiC technology, able to act simultaneously as a 4-bit binary encoder or a binary decoder in a 4-to-16 line configurations and show multiplexer-based logical functions. The device consists of a p-i'(a-SiC:H)-n/p-i(a-Si:H)-n multilayered structure produced by PECVD. To analyze it under information-modulated wave (color channels) and uniform irradiation (background) four monochromatic pulsed lights (input channels): red, green, blue and violet shine on the device. Steady state optical bias was superimposed separately from the front and the back sides, and the generated photocurrent was measured. Results show that the devices, under appropriate optical bias, act as reconfigurable active filters that allow optical switching and optoelectronic logic functions development providing the possibility for selective removal of useless wavelengths. The logic functions needed to construct any other complex logic functions are the NOT, and both or either an AND or an OR. Any other complex logic function that might be found can also be used as building blocks to achieve the functions needed for the retrieval of channels within the WDM communication link. (C) 2014 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim

Demux devices based on A-SiC:H

In this paper we present results about the functioning of a multilayered a-SiC:H heterostructure as a device for wavelength-division demultiplexing of optical signals. The device is composed of two stacked p-i-n photodiodes, both optimized for the selective collection of photogenerated carriers. Band gap engineering was used to adjust the photogeneration and recombination rates profiles of the intrinsic absorber regions of each photodiode to short and long wavelength absorption and carrier collection in the visible spectrum. The photocurrent signal using different input optical channels was analyzed at reverse and forward bias and under steady state illumination. This photocurrent is used as an input for a demux algorithm based on the voltage controlled sensitivity of the device. The device functioning is explained with results obtained by numerical simulation of the device, which permit an insight to the internal electric configuration of the double heterojunction.These results address the explanation of the device functioning in the frequency domain to a wavelength tunable photocapacitance due to the accumulation of space charge localized at the internal junction. The existence of a direct relation between the experimentally observed capacitive effects of the double diode and the quality of the semiconductor materials used to form the internal junction is highlighted.