Efficient learning of reactive robot behaviors with a Neural-Q_learning approach

The purpose of this paper is to propose a Neural-Q_learning approach designed for online learning of simple and reactive robot behaviors. In this approach, the Q_function is generalized by a multi-layer neural network allowing the use of continuous states and actions. The algorithm uses a database of the most recent learning samples to accelerate and guarantee the convergence. Each Neural-Q_learning function represents an independent, reactive and adaptive behavior which maps sensorial states to robot control actions. A group of these behaviors constitutes a reactive control scheme designed to fulfill simple missions. The paper centers on the description of the Neural-Q_learning based behaviors showing their performance with an underwater robot in a target following task. Real experiments demonstrate the convergence and stability of the learning system, pointing out its suitability for online robot learning. Advantages and limitations are discussed

Disseny de controladors òptims per al robot Pioneer

Aquest projecte titulat: “Disseny de controladors òptims per al robot Pioneer”, té com a funció incloure en la recerca, que ja està iniciada, del control del Robot Pioneer 2DX, una nova versió d’agents go to per al funcionament del robot. La problemàtica que ens trobem és sobretot per al primer controlador. Fins ara el sistema multi-agent fet, feia servir un agent go to que generava la trajectòria a seguir i la controlava mitjançant un PID. Introduint un mètode geomètric com és el cas del pure pursuit la cosa es complica ja que és més complex l’ajustament del funcionament d’aquest. Centrant-nos en canvi el cas del segon controlador el problema es simplifica ja que l’ajustatge d’aquest mateix es pot realitzar de manera empírica i la problemàtica per a la situació en concret es millora amb major facilitat. És per aquest motiu, sobretot pel primer controlador, que s’han hagut de realitzar algunes modificacions en el plantejament del projecte al llarg d’aquest. En un principi estava pensat crear aquest controlador a través de Matlab® mitjançant l’eina Simulink® però per problemes de software en un moment donat hem hagut de redirigir el projecte cap al llenguatge base de l’estructura multi-agent com és el C++. Per aquest motiu també s’ha hagut de prescindir de la implementació d’aquests també en l’estructura LabView®.

A formalization for multi-agent decision support in cooperative environments. A framework for situated agents

La tesis propone un marco de trabajo para el soporte de la toma de decisiones adecuado para soportar la ejecución distribuida de acciones cooperativas en entornos multi-agente dinámicos y complejos. Soporte para la toma de decisiones es un proceso que intenta mejorar la ejecución de la toma de decisiones en escenarios cooperativos. Este proceso ocurre continuamente en la vida diaria. Los humanos, por ejemplo, deben tomar decisiones acerca de que ropa usar, que comida comer, etc. En este sentido, un agente es definido como cualquier cosa que está situada en un entorno y que actúa, basado en su observación, su interpretación y su conocimiento acerca de su situación en tal entorno para lograr una acción en particular.Por lo tanto, para tomar decisiones, los agentes deben considerar el conocimiento que les permita ser consientes en que acciones pueden o no ejecutar. Aquí, tal proceso toma en cuenta tres parámetros de información con la intención de personificar a un agente en un entorno típicamente físico. Así, el mencionado conjunto de información es conocido como ejes de decisión, los cuales deben ser tomados por los agentes para decidir si pueden ejecutar correctamente una tarea propuesta por otro agente o humano. Los agentes, por lo tanto, pueden hacer mejores decisiones considerando y representando apropiadamente tal información. Los ejes de decisión, principalmente basados en: las condiciones ambientales, el conocimiento físico y el valor de confianza del agente, provee a los sistemas multi-agente un confiable razonamiento para alcanzar un factible y exitoso rendimiento cooperativo.Actualmente, muchos investigadores tienden a generar nuevos avances en la tecnología agente para incrementar la inteligencia, autonomía, comunicación y auto-adaptación en escenarios agentes típicamente abierto y distribuidos. En este sentido, esta investigación intenta contribuir en el desarrollo de un nuevo método que impacte tanto en las decisiones individuales como colectivas de los sistemas multi-agente. Por lo tanto, el marco de trabajo propuesto ha sido utilizado para implementar las acciones concretas involucradas en el campo de pruebas del fútbol robótico. Este campo emula los juegos de fútbol real, donde los agentes deben coordinarse, interactuar y cooperar entre ellos para solucionar tareas complejas dentro de un escenario dinámicamente cambiante y competitivo, tanto para manejar el diseño de los requerimientos involucrados en las tareas como para demostrar su efectividad en trabajos colectivos. Es así que los resultados obtenidos tanto en el simulador como en el campo real de experimentación, muestran que el marco de trabajo para el soporte de decisiones propuesto para agentes situados es capaz de mejorar la interacción y la comunicación, reflejando en un adecuad y confiable trabajo en equipo dentro de entornos impredecibles, dinámicos y competitivos. Además, los experimentos y resultados también muestran que la información seleccionada para generar los ejes de decisión para situar a los agentes, es útil cuando tales agentes deben ejecutar una acción o hacer un compromiso en cada momento con la intención de cumplir exitosamente un objetivo colectivo. Finalmente, algunas conclusiones enfatizando las ventajas y utilidades del trabajo propuesto en la mejora del rendimiento colectivo de los sistemas multi-agente en situaciones tales como tareas coordinadas y asignación de tareas son presentadas.

Multi-level planning for semi-autonomous vehicles in traffic scenarios based on separation maximization

The planning of semi-autonomous vehicles in traffic scenarios is a relatively new problem that contributes towards the goal of making road travel by vehicles free of human drivers. An algorithm needs to ensure optimal real time planning of multiple vehicles (moving in either direction along a road), in the presence of a complex obstacle network. Unlike other approaches, here we assume that speed lanes are not present and that different lanes do not need to be maintained for inbound and outbound traffic. Our basic hypothesis is to carry forward the planning task to ensure that a sufficient distance is maintained by each vehicle from all other vehicles, obstacles and road boundaries. We present here a 4-layer planning algorithm that consists of road selection (for selecting the individual roads of traversal to reach the goal), pathway selection (a strategy to avoid and/or overtake obstacles, road diversions and other blockages), pathway distribution (to select the position of a vehicle at every instance of time in a pathway), and trajectory generation (for generating a curve, smooth enough, to allow for the maximum possible speed). Cooperation between vehicles is handled separately at the different levels, the aim being to maximize the separation between vehicles. Simulated results exhibit behaviours of smooth, efficient and safe driving of vehicles in multiple scenarios; along with typical vehicle behaviours including following and overtaking.

Cooperação e multi-level governance: o caso do Grande Recife Consórcio de Transporte Metropolitano

As primeiras regiões metropolitanas brasileiras foram instituídas de maneira vertical e autoritária como parte da estratégia de desenvolvimento nacional promovida pelo governo militar. Percebidas como instituições não-democráticas e rejeitadas como possível quarto ente federativo, as regiões metropolitanas, desde a Constituição de 1988, foram gradualmente esvaziadas dos seus propósitos originais. Em sua orfandade, os problemas socioeconômicos proliferaram e foram acentuados, e passaram a predominar relações intergovernamentais competitivas em vez de cooperativas. Um dos principais desafios enfrentados pelo modelo federalista brasileiro, em especial quando se trata destas regiões, está relacionado à necessidade de estabelecer maior cooperação e coordenação, tidas como imprescindíveis para garantir um relacionamento mais equilibrado entre os entes federativos, assim como para a efetiva implementação de políticas de enfrentamento das desigualdades e exclusão social nas aglomerações urbanas. Este trabalho analisa o Grande Recife Consórcio Metropolitano de Transportes (CMT), empresa pública multifederativa estabelecida em 2008 entre os governos municipais e estadual da Região Metropolitana de Recife (RMR). Responsável pelo planejamento, gestão e implementação compartilhada da política de transporte público coletivo na RMR, o Grande Recife se tornou realidade com a aprovação e regulamentação da Lei Federal nº 11.107 de 2005, conhecida como a Lei de Consórcios Públicos. O Grande Recife é uma experiência pioneira e inovadora, demonstrando que é possível encontrar uma maneira de superar conflitos e desafios comuns e, ao mesmo tempo, garantir a preservação da autonomia de cada ente, bem como os direitos cidadãos. Neste trabalho consideramos essa experiência de cooperação intergovernamental como um exemplo de multi-level governance (MLG), uma vez que é ilustrativa de um novo arranjo institucional democrático entre distintas esferas governamentais para a gestão compartilhada de um serviço público.

The influence of the supply chain agents on the new product development's performance: an analysis based on the multi-group moderation

Este estudo buscou verificar a influencia dos agentes da cadeia de suprimentos no desempenho do desenvolvimento de novos produtos quando os agentes são analisados em conjunto. A motivação desta pesquisa veio de estudos que alertaram para a consideração da integração da cadeia de suprimentos como um constructo multidimensional, englobando o envolvimento da manufatura, fornecedores e clientes no desenvolvimento de novos produtos; e devido à falta de informação sobre as influencias individuais destes agentes no desenvolvimento de novos produtos. Sob essas considerações, buscou-se construir um modelo analítico baseado na Teoria do Capital Social e Capacidade Absortiva, construir hipóteses a partir da revisão da literatura e conectar constructos como cooperação, envolvimento do fornecedor no desenvolvimento de novos produtos (DNP), envolvimento do cliente no DNP, envolvimento da manufatura no DNP, antecipação de novas tecnologias, melhoria contínua, desempenho operacional do DNP, desempenho de mercado do NPD e desempenho de negócio do DNP. Para testar as hipóteses foram consideradas três variáveis moderadoras, tais como turbulência ambiental (baixa, média e alta), indústria (eletrônicos, maquinários e equipamentos de transporte) e localização (América, Europa e Ásia). Para testar o modelo foram usados dados do projeto High Performance Manufacturing que contém 339 empresas das indústrias de eletrônicos, maquinários e equipamentos de transporte, localizadas em onze países. As hipóteses foram testadas por meio da Análise Fatorial Confirmatória (AFC) incluindo a moderação muti-grupo para as três variáveis moderadoras mencionadas anteriormente. Os principais resultados apontaram que as hipóteses relacionadas com cooperação foram confirmadas em ambientes de média turbulência, enquanto as hipóteses relacionadas ao desempenho no DNP foram confirmadas em ambientes de baixa turbulência ambiental e em países asiáticos. Adicionalmente, sob as mesmas condições, fornecedores, clientes e manufatura influenciam diferentemente no desempenho de novos produtos. Assim, o envolvimento de fornecedores influencia diretamente no desempenho operacional e indiretamente no desempenho de mercado e de negócio em baixos níveis de turbulência ambiental, na indústria de equipamentos de transporte em países da Americanos e Europeus. De igual forma, o envolvimento do cliente influenciou diretamente no desempenho operacional e indiretamente no desempenho de mercado e do negócio em médio nível de turbulência ambiental, na indústria de maquinários e em países Asiáticos. Fornecedores e clientes não influenciam diretamente no desempenho de mercado e do negócio e não influenciam indiretamente no desempenho operacional. O envolvimento da manufatura não influenciou nenhum tipo de desempenho do desenvolvimento de novos produtos em todos os cenários testados.

Multi-resolução com fóvea móvel para redução e abstração de dados em tempo real

We propose a new approach to reduction and abstraction of visual information for robotics vision applications. Basically, we propose to use a multi-resolution representation in combination with a moving fovea for reducing the amount of information from an image. We introduce the mathematical formalization of the moving fovea approach and mapping functions that help to use this model. Two indexes (resolution and cost) are proposed that can be useful to choose the proposed model variables. With this new theoretical approach, it is possible to apply several filters, to calculate disparity and to obtain motion analysis in real time (less than 33ms to process an image pair at a notebook AMD Turion Dual Core 2GHz). As the main result, most of time, the moving fovea allows the robot not to perform physical motion of its robotics devices to keep a possible region of interest visible in both images. We validate the proposed model with experimental results

ANN statistical image recognition method for computer vision in agricultural mobile robot navigation

The main application area in this project, is to deploy image processing and segmentation techniques in computer vision through an omnidirectional vision system to agricultural mobile robots (AMR) used for trajectory navigation problems, as well as localization matters. Thereby, computational methods based on the JSEG algorithm were used to provide the classification and the characterization of such problems, together with Artificial Neural Networks (ANN) for image recognition. Hence, it was possible to run simulations and carry out analyses of the performance of JSEG image segmentation technique through Matlab/Octave computational platforms, along with the application of customized Back-propagation Multilayer Perceptron (MLP) algorithm and statistical methods as structured heuristics methods in a Simulink environment. Having the aforementioned procedures been done, it was practicable to classify and also characterize the HSV space color segments, not to mention allow the recognition of segmented images in which reasonably accurate results were obtained. © 2010 IEEE.

Cross-layer optimizations in multi-hop ad hoc networks

Unlike traditional wireless networks, characterized by the presence of last-mile, static and reliable infrastructures, Mobile ad Hoc Networks (MANETs) are dynamically formed by collections of mobile and static terminals that exchange data by enabling each other's communication. Supporting multi-hop communication in a MANET is a challenging research area because it requires cooperation between different protocol layers (MAC, routing, transport). In particular, MAC and routing protocols could be considered mutually cooperative protocol layers. When a route is established, the exposed and hidden terminal problems at MAC layer may decrease the end-to-end performance proportionally with the length of each route. Conversely, the contention at MAC layer may cause a routing protocol to respond by initiating new routes queries and routing table updates. Multi-hop communication may also benefit the presence of pseudo-centralized virtual infrastructures obtained by grouping nodes into clusters. Clustering structures may facilitate the spatial reuse of resources by increasing the system capacity: at the same time, the clustering hierarchy may be used to coordinate transmissions events inside the network and to support intra-cluster routing schemes. Again, MAC and clustering protocols could be considered mutually cooperative protocol layers: the clustering scheme could support MAC layer coordination among nodes, by shifting the distributed MAC paradigm towards a pseudo-centralized MAC paradigm. On the other hand, the system benefits of the clustering scheme could be emphasized by the pseudo-centralized MAC layer with the support for differentiated access priorities and controlled contention. In this thesis, we propose cross-layer solutions involving joint design of MAC, clustering and routing protocols in MANETs. As main contribution, we study and analyze the integration of MAC and clustering schemes to support multi-hop communication in large-scale ad hoc networks. A novel clustering protocol, named Availability Clustering (AC), is defined under general nodes' heterogeneity assumptions in terms of connectivity, available energy and relative mobility. On this basis, we design and analyze a distributed and adaptive MAC protocol, named Differentiated Distributed Coordination Function (DDCF), whose focus is to implement adaptive access differentiation based on the node roles, which have been assigned by the upper-layer's clustering scheme. We extensively simulate the proposed clustering scheme by showing its effectiveness in dominating the network dynamics, under some stressing mobility models and different mobility rates. Based on these results, we propose a possible application of the cross-layer MAC+Clustering scheme to support the fast propagation of alert messages in a vehicular environment. At the same time, we investigate the integration of MAC and routing protocols in large scale multi-hop ad-hoc networks. A novel multipath routing scheme is proposed, by extending the AOMDV protocol with a novel load-balancing approach to concurrently distribute the traffic among the multiple paths. We also study the composition effect of a IEEE 802.11-based enhanced MAC forwarding mechanism called Fast Forward (FF), used to reduce the effects of self-contention among frames at the MAC layer. The protocol framework is modelled and extensively simulated for a large set of metrics and scenarios. For both the schemes, the simulation results reveal the benefits of the cross-layer MAC+routing and MAC+clustering approaches over single-layer solutions.

Teoria del consenso e applicazione al problema del coordinamento del moto di robot.

Il sempre crescente numero di applicazioni di reti di sensori, robot cooperanti e formazioni di veicoli, ha fatto sì che le problematiche legate al coordinamento di sistemi multi-agente (MAS) diventassero tra le più studiate nell’ambito della teoria dei controlli. Esistono numerosi approcci per affrontare il problema, spesso profondamente diversi tra loro. La strategia studiata in questa tesi è basata sulla Teoria del Consenso, che ha una natura distribuita e completamente leader-less; inoltre il contenuto informativo scambiato tra gli agenti è ridotto al minimo. I primi 3 capitoli introducono ed analizzano le leggi di interazione (Protocolli di Consenso) che permettono di coordinare un Network di sistemi dinamici. Nel capitolo 4 si pensa all'applicazione della teoria al problema del "loitering" circolare di più robot volanti attorno ad un obiettivo in movimento. Si sviluppa a tale scopo una simulazione in ambiente Matlab/Simulink, che genera le traiettorie di riferimento di raggio e centro impostabili, a partire da qualunque posizione iniziale degli agenti. Tale simulazione è stata utilizzata presso il “Center for Research on Complex Automated Systems” (CASY-DEI Università di Bologna) per implementare il loitering di una rete di quadrirotori "CrazyFlie". I risultati ed il setup di laboratorio sono riportati nel capitolo 5. Sviluppi futuri si concentreranno su algoritmi locali che permettano agli agenti di evitare collisioni durante i transitori: il controllo di collision-avoidance dovrà essere completamente indipendente da quello di consenso, per non snaturare il protocollo di Consenso stesso.

Multi-node TOA-DOA cooperative LOS-NLOS localization : enabling high accuracy and reliability

This dissertation investigates high performance cooperative localization in wireless environments based on multi-node time-of-arrival (TOA) and direction-of-arrival (DOA) estimations in line-of-sight (LOS) and non-LOS (NLOS) scenarios. Here, two categories of nodes are assumed: base nodes (BNs) and target nodes (TNs). BNs are equipped with antenna arrays and capable of estimating TOA (range) and DOA (angle). TNs are equipped with Omni-directional antennas and communicate with BNs to allow BNs to localize TNs; thus, the proposed localization is maintained by BNs and TNs cooperation. First, a LOS localization method is proposed, which is based on semi-distributed multi-node TOA-DOA fusion. The proposed technique is applicable to mobile ad-hoc networks (MANETs). We assume LOS is available between BNs and TNs. One BN is selected as the reference BN, and other nodes are localized in the coordinates of the reference BN. Each BN can localize TNs located in its coverage area independently. In addition, a TN might be localized by multiple BNs. High performance localization is attainable via multi-node TOA-DOA fusion. The complexity of the semi-distributed multi-node TOA-DOA fusion is low because the total computational load is distributed across all BNs. To evaluate the localization accuracy of the proposed method, we compare the proposed method with global positioning system (GPS) aided TOA (DOA) fusion, which are applicable to MANETs. The comparison criterion is the localization circular error probability (CEP). The results confirm that the proposed method is suitable for moderate scale MANETs, while GPS-aided TOA fusion is suitable for large scale MANETs. Usually, TOA and DOA of TNs are periodically estimated by BNs. Thus, Kalman filter (KF) is integrated with multi-node TOA-DOA fusion to further improve its performance. The integration of KF and multi-node TOA-DOA fusion is compared with extended-KF (EKF) when it is applied to multiple TOA-DOA estimations made by multiple BNs. The comparison depicts that it is stable (no divergence takes place) and its accuracy is slightly lower than that of the EKF, if the EKF converges. However, the EKF may diverge while the integration of KF and multi-node TOA-DOA fusion does not; thus, the reliability of the proposed method is higher. In addition, the computational complexity of the integration of KF and multi-node TOA-DOA fusion is much lower than that of EKF. In wireless environments, LOS might be obstructed. This degrades the localization reliability. Antenna arrays installed at each BN is incorporated to allow each BN to identify NLOS scenarios independently. Here, a single BN measures the phase difference across two antenna elements using a synchronized bi-receiver system, and maps it into wireless channel’s K-factor. The larger K is, the more likely the channel would be a LOS one. Next, the K-factor is incorporated to identify NLOS scenarios. The performance of this system is characterized in terms of probability of LOS and NLOS identification. The latency of the method is small. Finally, a multi-node NLOS identification and localization method is proposed to improve localization reliability. In this case, multiple BNs engage in the process of NLOS identification, shared reflectors determination and localization, and NLOS TN localization. In NLOS scenarios, when there are three or more shared reflectors, those reflectors are localized via DOA fusion, and then a TN is localized via TOA fusion based on the localization of shared reflectors.

A Cooperative and Adaptive Variable Neighborhood Search for the Multi Depot Vehicle Routing Problem with Time Windows

In this paper we propose two cooperation schemes to compose new parallel variants of the Variable Neighborhood Search (VNS). On the one hand, a coarse-grained cooperation scheme is introduced which is well suited for being enhanced with a solution warehouse to store and manage the so far best found solutions and a self-adapting mechanism for the most important search parameters. This makes an a priori parameter tuning obsolete. On the other hand, a fine-grained scheme was designed to reproduce the successful properties of the sequential VNS. In combination with the use of parallel exploration threads all of the best solutions and 11 out of 20 new best solutions for the Multi Depot Vehicle Routing Problem with Time Windows were found.

A multi-level stakeholder approach to sustainable land management

Soils provide us with over 90% of all human food, livestock feed, fibre and fuel on Earth. Soils, however, have more than just productive functions. The key challenge in coming years will be to address the diverse and potentially conflicting demands now being made by human societies and other forms of life, while ensuring that future generations have the same potential to use soils and land of comparable quality. In a multi-level stakeholder approach, down-to-earth action will have to be supplemented with measures at various levels, from households to communities, and from national policies to international conventions. Knowledge systems, both indigenous and scientific, and related research and learning processes must play a central role. Ongoing action can be enhanced through a critical assessment of the impact of past achievements, and through better cooperation between people and institutions.

An Economic Derivation on Trade Coefficients under the Framework of Multi-regional I-O Analysis

The gravity model, entropy model, potential type model and others like these have been adopted to formulate interregional trade coefficients under the framework of Multi-Regional I-O (MRIO) analysis. Since most of these models are based upon analogies in physics or on statistical principles, they do not provide a theoretical explanation from the view of a firm's or individual's rational and deterministic decision making. In this paper, according to the deterministic choice theory, not only is an alternative formulation of the trade coefficients presented, but also a discussion of an appropriate definition for purchasing prices indices. Since this formulation is consistent with the MRIO system, it can be employed as a useful model-building tool in multi-regional models such as the spatial CGE model.

Cooperative Task Execution between Modular Robots Based on Tight-Loose Cooperation Strategies

The complexity in the execution of cooperative tasks is high due to the fact that a robot team requires movement coordination at the beginning of the mission and continuous coordination during the execution of the task. A variety of techniques have been proposed to give a solution to this problem assuming standard mobile robots. This work focuses on presenting the execution of a cooperative task by a modular robot team. The complexity of the task execution increases due to the fact that each robot is composed of modules which have to be coordinated in a proper way to successfully work. A combined tight and loose cooperation strategy is presented and a bar-pushing example is used as a cooperative task to show the performance of this type of system.