Reinforcement Learning by Policy Search

One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.

On the Convergence of Stochastic Iterative Dynamic Programming Algorithms

Recent developments in the area of reinforcement learning have yielded a number of new algorithms for the prediction and control of Markovian environments. These algorithms, including the TD(lambda) algorithm of Sutton (1988) and the Q-learning algorithm of Watkins (1989), can be motivated heuristically as approximations to dynamic programming (DP). In this paper we provide a rigorous proof of convergence of these DP-based learning algorithms by relating them to the powerful techniques of stochastic approximation theory via a new convergence theorem. The theorem establishes a general class of convergent algorithms to which both TD(lambda) and Q-learning belong.

Scaling algorithms for distributed max flow

Bibliography: p. 22-24.

Plugin de routing para JOSM

Presentació del plugin de càlcul de rutes per a JOSM (Java OpenStreetMap)

QoS online routing and MPLS multilevel protection: a survey

A survey of MPLS protection methods and their utilization in combination with online routing methods is presented in this article. Usually, fault management methods pre-establish backup paths to recover traffic after a failure. In addition, MPLS allows the creation of different backup types, and hence MPLS is a suitable method to support traffic-engineered networks. In this article, an introduction of several label switch path backup types and their pros and cons are pointed out. The creation of an LSP involves a routing phase, which should include QoS aspects. In a similar way, to achieve a reliable network the LSP backups must also be routed by a QoS routing method. When LSP creation requests arrive one by one (a dynamic network scenario), online routing methods are applied. The relationship between MPLS fault management and QoS online routing methods is unavoidable, in particular during the creation of LSP backups. Both aspects are discussed in this article. Several ideas on how these actual technologies could be applied together are presented and compared

Improving Clustering Algorithms for Image Segmentation using Contour and Region Information

In image segmentation, clustering algorithms are very popular because they are intuitive and, some of them, easy to implement. For instance, the k-means is one of the most used in the literature, and many authors successfully compare their new proposal with the results achieved by the k-means. However, it is well known that clustering image segmentation has many problems. For instance, the number of regions of the image has to be known a priori, as well as different initial seed placement (initial clusters) could produce different segmentation results. Most of these algorithms could be slightly improved by considering the coordinates of the image as features in the clustering process (to take spatial region information into account). In this paper we propose a significant improvement of clustering algorithms for image segmentation. The method is qualitatively and quantitative evaluated over a set of synthetic and real images, and compared with classical clustering approaches. Results demonstrate the validity of this new approach

An improved method for discovering link criticality in transport networks

Quantitatively assessing the importance or criticality of each link in a network is of practical value to operators, as that can help them to increase the network's resilience, provide more efficient services, or improve some other aspect of the service. Betweenness is a graph-theoretical measure of centrality that can be applied to communication networks to evaluate link importance. However, as we illustrate in this paper, the basic definition of betweenness centrality produces inaccurate estimations as it does not take into account some aspects relevant to networking, such as the heterogeneity in link capacity or the difference between node-pairs in their contribution to the total traffic. A new algorithm for discovering link centrality in transport networks is proposed in this paper. It requires only static or semi-static network and topology attributes, and yet produces estimations of good accuracy, as verified through extensive simulations. Its potential value is demonstrated by an example application. In the example, the simple shortest-path routing algorithm is improved in such a way that it outperforms other more advanced algorithms in terms of blocking ratio

All-Optical Unicast/Multicast Routing in WDM Networks

Optical transport networks (OTN) must be prepared in terms of better resource utilization, for accommodating unicast and multicast traffic together. Light-trees have been proposed for supporting multicast connections in OTN. Nevertheless when traffic grooming is applied in light-trees, resources can be underutilized as traffic can be routed to undesirable destinations in order to avoid optical-electrical-optical (OEO) conversions. In this paper, a novel architecture named S/G light- tree for supporting unicast/multicast connections is proposed. The architecture allows traffic dropping and aggregation in different wavelengths without performing OEO conversions. A heuristic that routes traffic demands using less wavelengths by taking advantage of the proposed architecture is designed as well. Simulation results show that the architecture can minimize the number of used wavelengths and OEO conversions when compared to light-trees

All-Optical Label Stacking: Easing the Trade-offs Between Routing and Architecture Cost in All-Optical Packet Switching

All-optical label swapping (AOLS) forms a key technology towards the implementation of all-optical packet switching nodes (AOPS) for the future optical Internet. The capital expenditures of the deployment of AOLS increases with the size of the label spaces (i.e. the number of used labels), since a special optical device is needed for each recognized label on every node. Label space sizes are affected by the way in which demands are routed. For instance, while shortest-path routing leads to the usage of fewer labels but high link utilization, minimum interference routing leads to the opposite. This paper studies all-optical label stacking (AOLStack), which is an extension of the AOLS architecture. AOLStack aims at reducing label spaces while easing the compromise with link utilization. In this paper, an integer lineal program is proposed with the objective of analyzing the softening of the aforementioned trade-off due to AOLStack. Furthermore, a heuristic aiming at finding good solutions in polynomial-time is proposed as well. Simulation results show that AOLStack either a) reduces the label spaces with a low increase in the link utilization or, similarly, b) uses better the residual bandwidth to decrease the number of labels even more

Hiding routing information

This is the original "onion routing" paper, it explains in the details how onions are built and work. This is optional reading, although I strongly advise you to read the Introduction and section 3 "Onions"

Problemas en la implementación de algoritmos de routing de alta complejidad en dispositivos móviles: el caso Itiner@

La aparición de terminales de telefonía móvil cada vez más potentes abre un nuevo abanico de posibilidades en cuanto a usos y aplicaciones. Sin embargo, y dadas las limitaciones tanto de memoria como de CPU que tienen estos dispositivos, algunas de las aplicaciones potenciales resultan muy difíciles o incluso imposibles de llevar a la práctica. Este es el caso, por ejemplo, de aplicaciones de cálculo de rutas. En el contexto del proyecto Itiner@, un asistente para rutas turísticas completamente autónomo que debe funcionar incluso sin conexión a Internet, todos los procesos deben ejecutarse íntegramente de forma local en el dispositivo móvil. Dado que es un proyecto orientado al ocio, es importante que la experiencia del usuario sea satisfactoria, por lo que además de poder ejecutar el algoritmo de cálculo de rutas, el sistema debe hacerlo de forma rápida. En este sentido, los algoritmos recursivos habituales son demasiado costosos o lentos para su uso en Itiner@ y ha sido necesario reinventar este tipo de algoritmos en función de las limitaciones que tienen estos dispositivos. En el presente trabajo se presenta el proceso seguido y las dificultades encontradas para implementar un algoritmo recursivo de cálculo de rutas que se ejecute íntegramente en un dispositivo móvil Android de forma eficiente. Así, finalmente se llega a un algoritmo recursivo de cálculo de rutas para dispositivos móviles que se ejecuta de forma más eficiente frente a algoritmos directamente portados a dispositivos móviles. La principal contribución del trabajo es doble: por un lado ofrece algunas guías útiles al desarrollo de algoritmos más eficientes para dispositivos móviles; y por el otro, muestra un algoritmo de cálculo de rutas que funciona con un tiempo de respuesta aceptable, en un entorno exigente, como es el de las aplicaciones de turismo en móviles

Congestion probability routing in virtual path ATM network

This dissertation focuses on the problem of providing mechanisms for routing point to point and multipoint connections in ATM networks. In general the notion of multipoint connection refers to connections that involve a group of users with more than two members. The main objective of this dissertation is to contribute to design efficient routing protocols with alterative routes in fully connected VP-based ATM Networks for call establishment of point to point and multipoint VC connections. An efficient route should be computed during this connection establishment phase.