Reinforcement Learning by Policy Search

One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.

Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

Autonomous underwater vehicle control using reinforcement learning policy search methods

Autonomous underwater vehicles (AUV) represent a challenging control problem with complex, noisy, dynamics. Nowadays, not only the continuous scientific advances in underwater robotics but the increasing number of subsea missions and its complexity ask for an automatization of submarine processes. This paper proposes a high-level control system for solving the action selection problem of an autonomous robot. The system is characterized by the use of reinforcement learning direct policy search methods (RLDPS) for learning the internal state/action mapping of some behaviors. We demonstrate its feasibility with simulated experiments using the model of our underwater robot URIS in a target following task

Towards Direct Policy Search Reinforcement Learning for Robot Control

This paper proposes a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot. Although the dominant approach, when using RL, has been to apply value function based algorithms, the system here detailed is characterized by the use of direct policy search methods. Rather than approximating a value function, these methodologies approximate a policy using an independent function approximator with its own parameters, trying to maximize the future expected reward. The policy based algorithm presented in this paper is used for learning the internal state/action mapping of a behavior. In this preliminary work, we demonstrate its feasibility with simulated experiments using the underwater robot GARBI in a target reaching task

Maternal Mortality in Colombia: Some Thoughs on the Situation and Public Policy During Last Decade

In this article we make a critical analysis about the situation of maternal mortality in Colombia, considering some of the most current rates of this problem and the public policy rules adopted in the last years to decrease its prevalence. We think that the rate of maternal mortality in Colombia is too hight because the State does not guarantee the right health care to Colombian women. In this way, the structure and resources distribution established by the policies on sexual and reproductive health have not reflected the population necessities, specially in the case of young mothers left aside by the social security system.

The long-term EU budget: size or flexibility? Bruegel Policy Contribution 2012/20, November 2012

The EU is in the process of negotiating its 2014-20 financial framework. Failure to reach an agreement would imply a delay in the preparation of the strategic plans each member state puts together to explain how it will use Structural and Cohesion Funds. Even if solutions are found – for example annual renewals of the budget based on the previous year's figures – there will be political and institutional costs. EU leaders have too often and too forcefully advocated the use of the EU budget for growth to be able to drop the idea without consequences. • The overwhelming attention paid to the size of the budget is misplaced. EU leaders should instead aim to make the EU budget more flexible, safeguard it from future political power struggles, and reinforce assessment of the impact of EU funded growth policies. • To improve flexibility a commitment device should be created that places the EU budget above continuous political disagreement. We suggest the creation of a European Growth Fund, on the basis of which the European Commission should be allowed to borrow on capital markets to anticipate pre-allocated EU expenditure, such as Structural and Cohesion Funds. Markets would thus be a factor in EU budget policymaking, with a potentially disciplining effect. Attaching conditionality to this type of disbursement appears legitimate, as capital delivered in this way is a form of assistance.

Monetary policy and banking supervision: coordination instead of separation. CEPS Policy Brief No. 286, 12 December 2012

Introduction. Following the June 2012 European Council decision to place the ‘Single Supervisory Mechanism’ (SSM) within the European Central Bank, the general presumption in the policy discussions has been that there should be ‘Chinese walls’ between the supervisory and monetary policy arms of the ECB. The current legislative proposal, in fact, is explicit on this account. On the contrary, however, this paper finds that there is no need to impose a strict separation between these two functions. The authors argue, in fact, that a strict separation of supervision and monetary policy is not even desirable during a financial crisis when the systemic stability of the financial system represents the biggest threat to a monetary policy that aims at price stability. In their view, the key problem hampering the ECB today is that it lacks detailed information on the state of health of the banking system, which is often highly confidential. Chinese walls would not solve this problem. Moreover, in light of the fact that the new, proposed Supervisory Board will be composed to a large extent of representatives of the same institutions that also dominate the Governing Council, the paper finds that it does not make sense to have Chinese walls between two boards with largely overlapping memberships. In addition, it recommends that some members of the Supervisory Boards should be “independents” in order to reduce the tendency of supervisors to unduly delay the recognition of losses.

Performance of long-term care systems in Europe. ENEPRI Policy Brief No. 13, December 2012

The evaluation of long-term care (LTC) systems carried out in Work Package 7 of the ANCIEN project shows which performance criteria are important and – based on the available information – how European countries score on those criteria. This paper summarises the results and discusses the policy implications. An overall evaluation was carried out for four representative countries: Germany, the Netherlands, Spain and Poland. Of the four countries, the Dutch system has the highest scores on quality of life of LTC users, quality of care and equity of the LTC system, and it performs the secondbest after Poland in terms of the total burden of care (consisting of the financial burden and the burden of informal caregiving). The German system has somewhat lower scores than the Dutch on all four dimensions. The Polish system excels in having a low total burden of care, but it scores the lowest on quality of care and equity. The Spanish system has few extreme scores. Some important lessons are the following. The performance of a LTC system is a complex concept where many dimensions have to be included. Specifically, the impact of informal caregiving on the caregivers and on society should not be forgotten. The role of the state in funding and organising LTC versus individual responsibilities is one of the most important differences among countries. Choices concerning private funding and the role of informal care have a large effect not only on the public expenditures but also on the fairness of the system. International research into the relative preferences for the different performance criteria could produce a sound basis for the weights used in the overall evaluation.

On the use of headline indices to link environmental quality and income at the level of the nation state

This paper deconstructs the relationship between the Environmental Sustainability Index (ESI) and national income. The ESI attempts to provide a single figure which encapsulates environmental sustainability' for each country included in the analysis, and this allied with a 'league table' format so as to name and shame bad performers, has resulted in widespread reporting within the popular presses of a number of countries. In essence, the higher the value of the ESI then the more 'environmentally sustainable' a country is deemed to be. A logical progression beyond the use of the ESI to publicise environmental sustainability is its use within a more analytical context. Thus an index designed to simplify in order to have an impact on policy is used to try and understand causes of good and bad performance in environmental sustainability. For example the creators of the ESI claim that ESI is related to GDP/capita (adjusted for Purchasing Power Parity) such that the ESI increases linearly with wealth. While this may in a sense be a comforting picture, do the variables within the ESI allow for alternatives to the story, and if they do then what are the repercussions for those producing such indices for broad consumption amongst the policy makers, mangers, the press, etc.? The latter point is especially important given the appetite for such indices amongst non-specialists, and for all their weaknesses the ESI and other such aggregated indices will not go away. (C) 2007 Elsevier Ltd. All rights reserved.

Ideational change in the WTO and its impacts on EU agricultural policy institutions and the CAP

This paper argues that the Uruguay Round Agreement on Agriculture (URAA) introduced the market liberal paradigm as the ideational underpinning of the new farm trade regime. Though the immediate consequences in terms of limitations on agricultural support and protection were very modest, the Agreement did impact on the way in which domestic farm policy evolves. It forced EU agricultural policy makers to consider the agricultural negotiations when reforming the Common Agricultural Policy (CAP). The new paradigm in global farm trade resulted in a process of institutional layering in which concerns raised in the World Trade Organization (WTO) were gradually incorporated in EU agricultural institutions. This has resulted in gradual reform of the CAP in which policy instruments have been changed in order to make the CAP more WTO compatible. The underlying paradigm, the state-assisted paradigm, has been sustained though it has been rephrased by introducing the concept of multifunctionality.

An assessment of the benefits and limitations of the shamba agroforestry system in Kenya and of management and policy requirements for its successful and sustainable reintroduction

The shamba system involves farmers tending tree saplings on state-owned forest land in return for being permitted to intercrop perennial food crops until canopy closure. At one time the system was used throughout all state-owned forest lands in Kenya, accounting for a large proportion of some 160,000 ha. The system should theoretically be mutually beneficial to both local people and the government. However the system has had a chequered past in Kenya due to widespread malpractice and associated environmental degradation. It was last banned in 2003 but in early 2008 field trials were initiated for its reintroduction. This study aimed to: assess the benefits and limitations of the shamba system in Kenya; assess the main influences on the extent to which the limitations and benefits are realised and; consider the management and policy requirements for the system's successful and sustainable operation. Information was obtained from 133 questionnaires using mainly open ended questions and six participatory workshops carried out in forest-adjacent communities on the western slopes of Mount Kenya in Nyeri district. In addition interviews were conducted with key informants from communities and organisations. There was strong desire amongst local people for the system's reintroduction given that it had provided significant food, income and employment. Local perceptions of the failings of the system included firstly mismanagement by government or forest authorities and secondly abuse of the system by shamba farmers and outsiders. Improvements local people considered necessary for the shamba system to work included more accountability and transparency in administration and better rules with respect to plot allocation and stewardship. Ninety-seven percent of respondents said they would like to be more involved in management of the forest and 80% that they were willing to pay for the use of a plot. The study concludes that the structural framework laid down by the 2005 Forests Act, which includes provision for the reimplementation of the shamba system under the new plantation establishment and livelihood improvement scheme (PELIS) [It should be noted that whilst the shamba system was re-branded in 2008 under the acronym PELIS, for the sake of simplicity the authors continue to refer to the 'shamba system' and 'shamba farmers' throughout this paper.], is weakened because insufficient power is likely to be devolved to local people, casting them merely as 'forest users' and the shamba system as a 'forest user right'. In so doing the system's potential to both facilitate and embody the participation of local people in forest management is limited and the long-term sustainability of the new system is questionable. Suggested instruments to address this include some degree of sharing of profits from forest timber, performance related guarantees for farmers to gain a new plot and use of joint committees consisting of local people and the forest authorities for long term management of forests.

Explaining the 'health check' of the Common Agricultural Policy: budgetary politics, globalisation and paradigm change revisited

Three potential explanations of past reforms of the Common Agricultural Policy (CAP) can be identified in the literature: a budget constraint, pressure from General Agreement on Tariffs and Trade/World Trade Organization (GATT/WTO) negotiations or commitments and a paradigm shift emphasising agriculture’s provision of public goods. This discussion on the driving forces of CAP reform links to broader theoretical questions on the role of budgetary politics, globalisation of public policy and paradigm shift in explaining policy change. In this article, the Health Check reforms of 2007/2008 are assessed. They were probably more ambitious than first supposed, although it was a watered-down package agreed by ministers in November 2008. We conclude that the Health Check was not primarily driven by budget concerns or by the supposed switch from the state-assisted to the multifunctional policy paradigm. The European Commission’s wish to adopt an offensive negotiating stance in the closing phases of the Doha Round was a more likely explanatory factor. The shape and purpose of the CAP post-2013 is contested with divergent views among the Member States.

Policy interventions to promote healthy eating: a review of what works, what does not and what is promising

Unhealthy diets can lead to various diseases, which in turn can translate into a bigger burden for the state in the form of health services and lost production. Obesity alone has enormous costs and claims thousands of lives every year. Although diet quality in the European Union has improved across countries, it still falls well short of conformity with the World Health Organization dietary guidelines. In this review, we classify types of policy interventions addressing healthy eating and identify through a literature review what specific policy interventions are better suited to improve diets. Policy interventions are classified into two broad categories: information measures and measures targeting the market environment. Using this classification, we summarize a number of previous systematic reviews, academic papers, and institutional reports and draw some conclusions about their effectiveness. Of the information measures, policy interventions aimed at reducing or banning unhealthy food advertisements generally have had a weak positive effect on improving diets, while public information campaigns have been successful in raising awareness of unhealthy eating but have failed to translate the message into action. Nutritional labeling allows for informed choice. However, informed choice is not necessarily healthier; knowing or being able to read and interpret nutritional labeling on food purchased does not necessarily result in consumption of healthier foods. Interventions targeting the market environment, such as fiscal measures and nutrient, food, and diet standards, are rarer and generally more effective, though more intrusive. Overall, we conclude that measures to support informed choice have a mixed and limited record of success. On the other hand, measures to target the market environment are more intrusive but may be more effective.