Trajectory Planning in UAV-Aided Wireless Networks Via Reinforcement Learning

This thesis is focused on the design of a flexible, dynamic and innovative telecommunication's system for future 6G applications on vehicular communications. The system is based on the development of drones acting as mobile base stations in an urban scenario to cope with the increasing traffic demand and avoid network's congestion conditions. In particular, the exploitation of Reinforcement Learning algorithms is used to let the drone learn autonomously how to behave in a scenario full of obstacles with the goal of tracking and serve the maximum number of moving vehicles, by at the same time, minimizing the energy consumed to perform its tasks. This project is an extraordinary opportunity to open the doors to a new way of applying and develop telecommunications in an urban scenario by mixing it to the rising world of the Artificial Intelligence.

Cooperative framework for open real-time systems

Actualmente, os sistemas embebidos estão presentes em toda a parte. Embora grande parte da população que os utiliza não tenha a noção da sua presença, na realidade, se repentinamente estes sistemas deixassem de existir, a sociedade iria sentir a sua falta. A sua utilização massiva deve-se ao facto de estarem practicamente incorporados em quase os todos dispositivos electrónicos de consumo, telecomunicações, automação industrial e automóvel. Influenciada por este crescimento, a comunidade científica foi confrontada com novos problemas distribuídos por vários domínios científicos, dos quais são destacados a gestão da qualidade de serviço e gestão de recursos - domínio encarregue de resolver problemas relacionados com a alocação óptima de recursos físicos, tais como rede, memória e CPU. Existe na literatura um vasto conjunto de modelos que propõem soluções para vários problemas apresentados no contexto destes domínios científicos. No entanto, não é possível encontrar modelos que lidem com a gestão de recursos em ambientes de execução cooperativos e abertos com restrições temporais utilizando coligações entre diferentes nós, de forma a satisfazer os requisitos não funcionais das aplicações. Devido ao facto de estes sistemas serem dinâmicos por natureza, apresentam a característica de não ser possível conhecer, a priori, a quantidade de recursos necessários que uma aplicação irá requerer do sistema no qual irá ser executada. Este conhecimento só é adquirido aquando da execução da aplicação. De modo a garantir uma gestão eficiente dos recursos disponíveis, em sistemas que apresentam um grande dinamismo na execução de tarefas com e sem restrições temporais, é necessário garantir dois aspectos fundamentais. O primeiro está relacionado com a obtenção de garantias na execução de tarefas de tempo-real. Estas devem sempre ser executadas dentro da janela temporal requirida. O segundo aspecto refere a necessidade de garantir que todos os recursos necessários à execução das tarefas são fornecidos, com o objectivo de manter os níveis de performance quer das aplicações, quer do próprio sistema. Tendo em conta os dois aspectos acima mencionados, o projecto CooperatES foi especificado com o objectivo de permitir a dispositivos com poucos recursos uma execução colectiva de serviços com os seus vizinhos, de modo a cumprir com as complexas restrições de qualidade de serviço impostas pelos utilizadores ou pelas aplicações. Decorrendo no contexto do projecto CooperatES, o trabalho resultante desta tese tem como principal objectivo avaliar a practicabilidade dos conceitos principais propostos no âmbito do projecto. O trabalho em causa implicou a escolha e análise de uma plataforma, a análise de requisitos, a implementação e avaliação de uma framework que permite a execução cooperativa de aplicações e serviços que apresentem requisitos de qualidade de serviço. Do trabalho desenvolvido resultaram as seguintes contribuições: Análise das plataformas de código aberto que possam ser utilizadas na implementação dos conceitos relacionados com o projecto CooperatES; Critérios que influenciaram a escolha da plataforma Android e um estudo focado na análise da plataforma sob uma perspectiva de sistemas de tempo-real; Experiências na implementação dos conceitos do projecto na plataforma Android; Avaliação da practicabilidade dos conceitos propostos no projecto CooperatES; Proposta de extensões que permitam incorporar características de sistemas de tempo real abertos na plataforma Android.

A preliminary idea for an 8-competitive, log2 DMAX + log2 log2 1/U asymptotic-space, interface generation algorithm for two-level hierarchical scheduling of constrained-deadline sporadic tasks on a uniprocessor

Consider a single processor and a software system. The software system comprises components and interfaces where each component has an associated interface and each component comprises a set of constrained-deadline sporadic tasks. A scheduling algorithm (called global scheduler) determines at each instant which component is active. The active component uses another scheduling algorithm (called local scheduler) to determine which task is selected for execution on the processor. The interface of a component makes certain information about a component visible to other components; the interfaces of all components are used for schedulability analysis. We address the problem of generating an interface for a component based on the tasks inside the component. We desire to (i) incur only a small loss in schedulability analysis due to the interface and (ii) ensure that the amount of space (counted in bits) of the interface is small; this is because such an interface hides as much details of the component as possible. We present an algorithm for generating such an interface.

A pseudo-medium-wide 8-competitive interface for two-level compositional real-time scheduling of constrained- deadline sporadic tasks on a uniprocessor

Compositional real-time scheduling clearly requires that ”normal” real-time scheduling challenges are addressed but challenges intrinsic to compositionality must be addressed as well, in particular: (i) how should interfaces be described? and (ii) how should numerical values be assigned to parameters constituting the interfaces? The real-time systems community has traditionally used narrow interfaces for describing a component (for example, a utilization/bandwidthlike metric and the distribution of this bandwidth in time). In this paper, we introduce the concept of competitive ratio of an interface and show that typical narrow interfaces cause poor performance for scheduling constrained-deadline sporadic tasks (competitive ratio is infinite). Therefore, we explore more expressive interfaces; in particular a class called medium-wide interfaces. For this class, we propose an interface type and show how the parameters of the interface should be selected. We also prove that this interface is 8-competitive.

Building adaptable, QoS-aware dependable embedded systems

Most of today’s embedded systems are required to work in dynamic environments, where the characteristics of the computational load cannot always be predicted in advance. Furthermore, resource needs are usually data dependent and vary over time. Resource constrained devices may need to cooperate with neighbour nodes in order to fulfil those requirements and handle stringent non-functional constraints. This paper describes a framework that facilitates the distribution of resource intensive services across a community of nodes, forming temporary coalitions for a cooperative QoSaware execution. The increasing need to tailor provided service to each application’s specific needs determines the dynamic selection of peers to form such a coalition. The system is able to react to load variations, degrading its performance in a controlled fashion if needed. Isolation between different services is achieved by guaranteeing a minimal service quality to accepted services and by an efficient overload control that considers the challenges and opportunities of dynamic distributed embedded systems.

Time-bounded distributed QoS-aware service configuration in heterogeneous cooperative environments

The scarcity and diversity of resources among the devices of heterogeneous computing environments may affect their ability to perform services with specific Quality of Service constraints, particularly in dynamic distributed environments where the characteristics of the computational load cannot always be predicted in advance. Our work addresses this problem by allowing resource constrained devices to cooperate with more powerful neighbour nodes, opportunistically taking advantage of global distributed resources and processing power. Rather than assuming that the dynamic configuration of this cooperative service executes until it computes its optimal output, the paper proposes an anytime approach that has the ability to tradeoff deliberation time for the quality of the solution. Extensive simulations demonstrate that the proposed anytime algorithms are able to quickly find a good initial solution and effectively optimise the rate at which the quality of the current solution improves at each iteration, with an overhead that can be considered negligible.

A hybrid genetic algorithm for the multi-mode resource-constrained project scheduling

A construction project is a group of discernible tasks or activities that are conduct-ed in a coordinated effort to accomplish one or more objectives. Construction projects re-quire varying levels of cost, time and other resources. To plan and schedule a construction project, activities must be defined sufficiently. The level of detail determines the number of activities contained within the project plan and schedule. So, finding feasible schedules which efficiently use scarce resources is a challenging task within project management. In this context, the well-known Resource Constrained Project Scheduling Problem (RCPSP) has been studied during the last decades. In the RCPSP the activities of a project have to be scheduled such that the makespan of the project is minimized. So, the technological precedence constraints have to be observed as well as limitations of the renewable resources required to accomplish the activities. Once started, an activity may not be interrupted. This problem has been extended to a more realistic model, the multi-mode resource con-strained project scheduling problem (MRCPSP), where each activity can be performed in one out of several modes. Each mode of an activity represents an alternative way of combining different levels of resource requirements with a related duration. Each renewable resource has a limited availability for the entire project such as manpower and machines. This paper presents a hybrid genetic algorithm for the multi-mode resource-constrained pro-ject scheduling problem, in which multiple execution modes are available for each of the ac-tivities of the project. The objective function is the minimization of the construction project completion time. To solve the problem, is applied a two-level genetic algorithm, which makes use of two separate levels and extend the parameterized schedule generation scheme. It is evaluated the quality of the schedules and presents detailed comparative computational re-sults for the MRCPSP, which reveal that this approach is a competitive algorithm.

A two-level genetic algorithm for the multi-mode resource-constrained project scheduling problem

This paper presents a genetic algorithm for the multimode resource-constrained project scheduling problem (MRCPSP), in which multiple execution modes are available for each of the activities of the project. The objective function is the minimization of the construction project completion time. To solve the problem, is applied a two-level genetic algorithm, which makes use of two separate levels and extend the parameterized schedule generation scheme by introducing an improvement procedure. It is evaluated the quality of the schedule and present detailed comparative computational results for the MRCPSP, which reveal that this approach is a competitive algorithm.

Hierarchical Reinforcement Learning in Behavior and the Brain

Dissertation presented to obtain the Ph.D degree in Biology, Neuroscience

The Nursing and Midwifery Resource - Guidance for Best Practice on the Recruitment of Overseas Nurses and Midwives

Guidance for Best Practice on the Recruitment of Overseas Nurses and Midwives is prepared specifically for employers. The document includes a set of principles which are fundamental to the development of a model of transcultural nursing and midwifery. The document is based on the assumption that each potential employee has the qualifications and experience required to meet the criteria for registration as a nurse or midwife in Ireland Download the Report here

Autonomous underwater vehicle control using reinforcement learning policy search methods

Autonomous underwater vehicles (AUV) represent a challenging control problem with complex, noisy, dynamics. Nowadays, not only the continuous scientific advances in underwater robotics but the increasing number of subsea missions and its complexity ask for an automatization of submarine processes. This paper proposes a high-level control system for solving the action selection problem of an autonomous robot. The system is characterized by the use of reinforcement learning direct policy search methods (RLDPS) for learning the internal state/action mapping of some behaviors. We demonstrate its feasibility with simulated experiments using the model of our underwater robot URIS in a target following task

Towards Direct Policy Search Reinforcement Learning for Robot Control

This paper proposes a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot. Although the dominant approach, when using RL, has been to apply value function based algorithms, the system here detailed is characterized by the use of direct policy search methods. Rather than approximating a value function, these methodologies approximate a policy using an independent function approximator with its own parameters, trying to maximize the future expected reward. The policy based algorithm presented in this paper is used for learning the internal state/action mapping of a behavior. In this preliminary work, we demonstrate its feasibility with simulated experiments using the underwater robot GARBI in a target reaching task

Opportunities, constraints and constrained opportunities – A study on mothers’ working time patterns in 22 European countries

The aim of this study was to analyse mothers’ working time patterns across 22 European countries. The focus was on three questions: how much mothers prefer to work, how much they actually work, and to what degree their preferred and actual working times are (in)consistent with each other. The focus was on cross-national differences in mothers’ working time patterns, comparison of mothers’ working times to that of childless women and fathers, as well as on individual- and country-level factors that explain the variation between them. In the theoretical background, the departure point was an integrative theoretical approach where the assumption is that there are various kinds of explanations for the differences in mothers’ working time patterns – namely structural, cultural and institutional – , and that these factors are laid in two levels: individual- and country-levels. Data were extracted from the European Social Survey (ESS) 2010 / 2011. The results showed that mothers’ working time patterns, both preferred and actual working times, varied across European countries. Four clusters were formed to illustrate the differences. In the full-time pattern, full-time work was the most important form of work, leaving all other working time forms marginal. The full-time pattern was perceived in terms of preferred working times in Bulgaria and Portugal. In polarised pattern countries, fulltime work was also important, but it was accompanied by a large share of mothers not working at all. In the case of preferred working times, many Eastern and Southern European countries followed it whereas in terms of actual working times it included all Eastern and Southern European countries as well as Finland. The combination pattern was characterised by the importance of long part-time hours and full-time work. It was the preferred working time pattern in the Nordic countries, France, Slovenia, and Spain, but Belgium, Denmark, France, Norway, and Sweden followed it in terms of actual working times. The fourth cluster that described mothers’ working times was called the part-time pattern, and it was illustrated by the prevalence of short and long part-time work. In the case of preferred working times, it was followed in Belgium, Germany, Ireland, the Netherlands and Switzerland. Besides Belgium, the part-time pattern was followed in the same countries in terms of actual working times. The consistency between preferred and actual working times was rather strong in a majority of countries. However, six countries fell under different working time patterns when preferred and actual working times were compared. Comparison of working mothers’, childless women’s, and fathers’ working times showed that differences between these groups were surprisingly small. It was only in part-time pattern countries that working mothers worked significantly shorter hours than working childless women and fathers. Results therefore revealed that when mothers’ working times are under study, an important question regarding the population examined is whether it consists of all mothers or only working mothers. Results moreover supported the use of the integrative theoretical approach when studying mothers’ working time patterns. Results indicate that mothers’ working time patterns in all countries are shaped by various opportunities and constraints, which are comprised of structural, cultural, institutional, and individual-level factors.

Reinforcement Learning approaches to Economic Dispatch problem

This paper presents Reinforcement Learning (RL) approaches to Economic Dispatch problem. In this paper, formulation of Economic Dispatch as a multi stage decision making problem is carried out, then two variants of RL algorithms are presented. A third algorithm which takes into consideration the transmission losses is also explained. Efficiency and flexibility of the proposed algorithms are demonstrated through different representative systems: a three generator system with given generation cost table, IEEE 30 bus system with quadratic cost functions, 10 generator system having piecewise quadratic cost functions and a 20 generator system considering transmission losses. A comparison of the computation times of different algorithms is also carried out.

A Reinforcement Learning Approach to Economic Dispatch using Neural Networks

This paper presents a Reinforcement Learning (RL) approach to economic dispatch (ED) using Radial Basis Function neural network. We formulate the ED as an N stage decision making problem. We propose a novel architecture to store Qvalues and present a learning algorithm to learn the weights of the neural network. Even though many stochastic search techniques like simulated annealing, genetic algorithm and evolutionary programming have been applied to ED, they require searching for the optimal solution for each load demand. Also they find limitation in handling stochastic cost functions. In our approach once we learn the Q-values, we can find the dispatch for any load demand. We have recently proposed a RL approach to ED. In that approach, we could find only the optimum dispatch for a set of specified discrete values of power demand. The performance of the proposed algorithm is validated by taking IEEE 6 bus system, considering transmission losses