Reinforcement Learning by Policy Search

One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.

Policy gradient based Reinforcement Learning for real autonomous underwater cable tracking

This paper proposes a field application of a high-level reinforcement learning (RL) control system for solving the action selection problem of an autonomous robot in cable tracking task. The learning system is characterized by using a direct policy search method for learning the internal state/action mapping. Policy only algorithms may suffer from long convergence times when dealing with real robotics. In order to speed up the process, the learning phase has been carried out in a simulated environment and, in a second step, the policy has been transferred and tested successfully on a real robot. Future steps plan to continue the learning process on-line while on the real robot while performing the mentioned task. We demonstrate its feasibility with real experiments on the underwater robot ICTINEU AUV

The institutional environment and its influence on political options to Protect Winnipeg’s Female garments Workers against the impact of globalization

The principal objective of this paper is to identify the relationship between the re­sults of the Canadian policies implemented to protect female workers against the impact of globalization on the garment industry and the institutional setting in which this labour market is immersed in Winnipeg. This research paper begins with a brief summary of the institutional theory appro­ach that sheds light on the analysis of the effects of institutions on the policy options to protect female workers of the Winnipeg garment industry. Next, this paper identi­fies the set of beliefs, formal procedures, routines, norms and conventions that cha­racterize the institutional environment of the female workers of Winnipeg’s garment industry. Subsequently, this paper descri­bes the impact of free trade policies on the garment industry of Winnipeg. Afterward, this paper presents an analysis of the ba­rriers that the institutional features of the garment sector in Winnipeg can set to the successful achievement of policy options addressed to protect the female workforce of this sector. Three policy options are considered: ethical purchasing; training/retraining programs and social engage­ment support for garment workers; and protection of migrated workers through promoting and facilitating bonds between Canada’s trade unions and trade unions of the labour sending countries. Finally, this paper concludes that the formation of isolated cultural groups inside of factories; the belief that there is gender and race discrimination on the part of the garment industry management against workers; the powerless social conditions of immi­grant women; the economic rationality of garment factories’ managers; and the lack of political will on the part of Canada and the labour sending countries to set effective bilateral agreements to protect migrate wor­kers, are the principal barriers that divide the actors involved in the garment industry in Winnipeg. This division among the prin­cipal actors of Winnipeg’s garment industry impedes the change toward more efficient institutions and, hence, the successful achievement of policy options addressed to protect women workers. 

Establishing a community of practice of researchers, practitioners, policy-makers and communities to sustainably manage environmental health risks in Ecuador

The Sustainably Managing Environmental Health Risk in Ecuador project was launched in 2004 as a partnership linking a large Canadian university with leading Cuban and Mexican institutes to strengthen the capacities of four Ecuadorian universities for leading community-based learning and research in areas as diverse as pesticide poisoning, dengue control, water and sanitation, and disaster preparedness. By 2009, train-the-trainer project initiation involved 27 participatory action research Master’s theses in 15 communities where 1200 community learners participated in the implementation of associated interventions. This led to establishment of innovative Ecuadorian-led master’s and doctoral programs, and a Population Health Observatory on Collective Health, Environment and Society for the Andean region based at the Universidad Andina Simon Bolivar. Building on this network, numerous initiatives were begun, such as an internationally funded research project to strengthen dengue control in the coastal community of Machala, and establishment of a local community eco-health centre focusing on determinants of health near Cuenca. Alliances of academic and non-academic partners from the South and North provide a promising orientation for learning together about ways of addressing negative trends of development. Assessing the impacts and sustainability of such processes, however, requires longer term monitoring of results and related challenges.

Key Issues on the Brazilian Foreign Policy (Coyuntura)

El artículo trata sobre diferentes visiones de la política exterior brasilera. Primero introduce las principales directivas de la política exterior: desarrollo económico y social, así como delimitación limítrofe con los países de América del Sur. La segunda establece la situación de las relaciones de Brasil luego de la guerra fría, con algunos países y regiones en el mundo: Estados Unidos, Unión Europea, América Latina, China, África, India y otras regiones. Finalmente se refiere a los intereses nacionales brasileros en las Naciones Unidas en campos como energía, medio ambiente y defensa.

Breaking down the barriers to firm growth in Europe: The fourth EFIGE policy report. Bruegel Blueprint 18, 28 August 2012

Large firms contribute disproportionately to the economic performance of countries: they are more productive, pay higher wages, enjoy higher profits and are more successful in international markets. The differences between European countries in terms of the size of their firms are stark. Firms in Italy and Spain, for example, are on average 40 percent smaller than firms in Germany. The low average firm size translates into a chronic lack of large firms. In Italy and Spain, a mere 5 percent of manufacturing firms have more than 250 employees, compared to a much higher 11 percent in Germany. Understanding the roots of these differences is key to improving the economic performance of Europe’s lagging economies. So why is there so much variation in firm size in different European countries? What are the barriers that keep firms in some countries from growing? And which policies are likely to be most effective in breaking down those barriers? This policy report aims to answer these questions by developing a quantitative model of the seven European countries covered by the EFIGE survey (Austria, France, Germany, Hungary, Italy, Spain and the UK). The EFIGE survey asked 14,444 firms in those countries about their performance, their modes of internationalisation, their staffing decisions, their financing structure, and their competitive environment, among other topics.

Agricultural Policy and Nest Success of Prairie Ducks in Canada and the United States

The Prairie Pothole Region of North America has been modified by agriculture during the past 100 yr, resulting in habitat loss, fragmentation, and degradation that have reduced the abundance and productivity of many wildlife species. The 1985 U.S. Farm Bill provided economic incentives to agriculture that are considered by many to be beneficial to nesting waterfowl and other wildlife. Canada has not experienced an equally comprehensive legislative initiative, which would seem to indicate that benefits to waterfowl in Canada should lag behind those in the United States. However, with the removal of some agricultural subsidies in Canada during the 1990s, the amount of perennial cover in the Canadian prairies increased to levels similar to those of the 1970s. Therefore, it is unclear whether and how the U.S. and Canadian prairies might differ with regard to habitat quality for nesting waterfowl. We used historical and contemporary data to compare temporal trends in duck nest success between the United States and Canada and to assess how mean nest success varied with proportion of cropland and wetland density. The data best supported models with nonlinear temporal trends that varied between the two countries and suggested that mean nest success in Canada declined from its high point in 1930s and remained below the long-term value of 0.16 until the end of the time series in 2005. Mean nest success in the United States also declined from its high point in the 1930s, but increased to above the long-term value of 0.25 during the early 2000s. Mean nest success varied negatively with proportion of cropland in both the United States and Canada. Mean nest success was positively correlated with pond density at Canadian sites, but showed only a weak association with pond density at U.S. sites. All models explained the low proportions of the variation in nest success, suggesting that unmeasured factors such as the abundance and identity of nest predators may have strong effects on nest success. Nonetheless, these results support earlier suggestions that agricultural policy that encourages permanent cover positively influences duck reproductive success. We also found that, for reasons that are not entirely clear, nest success for the same intensity of row cropping was generally higher in the United States than in Canada. Further research is required to elucidate the exact nature of the composition, size, and distribution of permanent cover that coincides with greater average nest success by dabbling ducks in the United States. In addition, the data suggest that the benefits that might accrue from increases in the amount of perennial cover in Canada would be better realized if these efforts are accompanied by strong measures to conserve wetlands.

Construction contract policy: do we mean what we say?

Recent developments in contracting practice in the UK have built upon recommendations contained in highprofile reports, such as those by Latham and Egan. However, the New Engineering Contract (NEC), endorsed by Latham, is based upon principles of contract drafting that seem open to question. Any contract operates in the context of its legislative environment and current working practices. This report identifies eight contentious hypotheses in the literature on construction contracts and tests their validity in a sample survey that attracted 190 responses. The survey shows, among other things, that while partnership is a positive and useful idea, authoritative contract management is considered more effective and that “win-win” contracts, while desirable, are basically impractical. Further, precision and fairness in contracts are not easy to achieve simultaneously. While participants should know what is in their contracts, they should not routinely resort to legal action; and standard-form contracts should not seek to be universally applicable. Fundamental changes to drafting policy should be undertaken within the context of current legal contract doctrine and with a sensitivity to the way that contracts are used in contemporary practice. Attitudes to construction contracting may seem to be changing on the surface, but detailed analysis of what lies behind apparent agreement on new ways of working reveals that attitudes are changing much more slowly than they appear to be.

Environmental assessment and the built environment : a highway engineer's perspective

This paper is an engineer's appreciation of environmental assessment with particular reference to highway development. While scheme-related Environmental Assessment for individual development may identify particular potential impacts, and may avoid or minimise some of the problems, in many cases it may be too late to take such actions. Ideally, Environmental Assessment should commence at the Strategic Level to cover policies, plan and programmes, and the scheme-related Environmental Assessments for individual projects should supplement those in the framework of Strategic Level. The utimate target is to assess the policy for their contribution to effecting sustainable development. Whole Life Environmental Impacts should be considered. These are the full impact consideration from planning, design and choice of materials, construction, operation and finally decommission. Most of the Environmental Assessments have not included the Whole Life Environmental Impacts. There is only limited monitoring in the operation stage after the construction of the scheme is complete, therefore, subsequent Environmental Assessments cannot benefit from the feedback of the scheme. No development should cost the Earth, hence Environmental Assessments have to be carried out thoroughly to serve as one of the instruments to meet the need of sustainable development.

Abatement of mercury pollution in the small-scale gold mining industry: Restructuring the policy and research agendas

This paper critiques contemporary research and policy approaches taken toward the analysis and abatement of mercury pollution in the small-scale gold mining sector. Unmonitored releases of mercury from gold amalgamation have caused considerable environmental contamination and human health complications in rural reaches of sub-Saharan Africa, Latin America and Asia. Whilst these problems have caught the attention of the scientific community over the past 15-20 years, the research that has since been undertaken has failed to identify appropriate mitigation measures, and has done little to advance understanding of why contamination persists. Moreover, the strategies used to educate operators about the impacts of acute mercury exposure, and the technologies implemented to prevent farther pollution, have been marginally effective at best. The mercury pollution problem will not be resolved until governments and donor agencies commit to carrying out research aimed at improving understanding of the dynamics of small scale gold mining communities. Acquisition of this knowledge is the key to designing and implementing appropriate support and abatement measures. (c) 2005 Elsevier B.V. All rights reserved.

Intentions of UK farmers toward biofuel crop production: Implications for policy targets and land use change

The United States and the European Union have set targets for biofuel production to decrease reliance on fossil fuels and to reduce fossil carbon emissions. Attainment of biofuel targets d6pends upon policy and infrastructure development but also on production of suitable raw materials. Production of relevant crops relies on the decisions that farmers make in their economic and political environment. We need to identify any farmer-related barriers to biofuel production and to determine whether novel policy and technology are required to meet targets. These aspects of the emerging biofuel industry are relevant across international barriers and have notyet been addressed quantitatively. We describe a case study from the UK of farmers' intentions toward producing two biofuel crops for which refining capacity either exists or is under construction. Given farmers' intentions, current land use, and conversion efficiency, we estimate potential biofuel production. These estimates indicate that EU targets are not achievable using domestically grown raw materials without policy intervention, use of alternative feedstocks, and either significant improvements in processing efficiency or largescale changes in land use.

Potential economic impacts of terminator technologies: policy implications for developing countries

The terminator gene can render seeds sterile, so forcing farmers to purchase fresh seed every year. It is a technological solution to the problem of market failure that could increase the appropriability of R&D investment more effectively than intellectual property rights legislation or patents. This paper shows that appropriability should be more than tripled and that this leads to greater private R&D investment, which may be expected to double or triple. This would bring open-pollinating varieties into line with F1 hybrids, for which seed cannot be saved. In turn, the increased investment should raise yield increases to levels similar to those for hybrid crops. Thus, there are benefits to set against the possible ecological and environmental costs and the clear distributional and social consequences. The paper discusses the way the seed market is developing, the possible impacts, especially from a developing country viewpoint, and considers the policy changes that are needed.

Private investment in the countryside: an assessment of the role of new houses and estates in sustaining the rural economy and environment

The British countryside has been shaped and sustained over the years by the establishment of landed estates. Some of our best known, and now most protected, landmarks derive from this tradition by which money, that was often sourced from outside the rural economy, was invested in land. Whilst there was some reversal in this trend during the last century, there is again a widespread desire among people of means to invest in new country property. Paragraph 3.21 of Planning Policy Guidance Note 7: The Countryside - Environmental Quality and Economic and Social Development was introduced in 1997 as a means of perpetuating the historic tradition of innovation in the countryside through the construction of fine individual houses in landscaped grounds. That it was considered necessary to use a special provision of this kind reflects the prevailing presumption of planning authorities against allowing private residential development in open countryside. The Government is currently reviewing rural planning policy and is focusing on higher density housing, affordable homes and the use of brownfield sites. There is an underlying conception that individual private house developments contribute nothing and are seen as the least attractive option for most development sites. The purpose of paragraph 3.21 lies outside the government’s priorities and its particular provisions may therefore be excluded in forthcoming ‘policy statements’. This paper seeks to examine the role of private investors wishing to build new houses in the countryside, and the impact that that might have on local economies. It explores the interpretation placed on PPG7 through an investigation of appeal sites, and concludes by making recommendations for the review process, including the retention of some form of exceptions policy for new build houses.