Reinforcement Learning by Policy Search

One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.

Factors influencing adoption of improved grassland management by small-scale dairy farmers in central Mexico and the implications for future research on smallholder adoption in developing countries

There have been limited recent advances in understanding of what influences uptake of innovations despite the current international focus on smallholder agriculture as a means of achieving food security and rural development. This paper provides a rigorous study of factors influencing adoption by smallholders in central Mexico and builds on findings to identify a broad approach to significantly improve research on and understanding of factors influencing adoption by smallholders in developing countries. Small-scale dairy systems play an important role in providing income, employment and nutrition in the highlands of central Mexico. A wide variety of practices and technologies have been promoted by the government public services to increase milk production and economic efficiency, but there have been very low levels of uptake of most innovations, with the exception of improving grassland through introduction of grass varieties together with management practices. A detailed study was conducted with 80 farmers who are already engaged with the use of this innovation to better understand the process of adoption and identify socioeconomic and farm variables, cognitive (beliefs), and social–psychological (social norms) factors associated with farmers' use of improved grassland. The Theory of Reasoned Action (TRA) was used as a theoretical framework and Spearman Rank Order correlation was conducted to analyse the data. Most farmers (92.5%) revealed strong intention to continue to use improved grassland (which requires active management and investment of resources) for the next 12 months; whereas 7.5% of farmers were undecided and showed weak intention, which was associated with farmers whose main income was from non-farm activities as well as with farmers who had only recently started using improved grassland. Despite farmers' experience of using improved grassland (mean of 18 years) farmers' intentions to continue to adopt it was influenced almost as much by salient referents (mainly male relatives) as by their own attitudes. The hitherto unnoticed longevity of the role social referents play in adoption decisions is an important finding and has implications for further research and for the design of extension approaches. The study demonstrates the value and importance of using TRA or TPB approaches to understand social cognitive (beliefs) and social–psychological (social norms) factors in the study of adoption. However, other factors influencing adoption processes need to be included to provide fuller understanding. An approach that would enable this, and the development of more generalisable findings than from location specific case studies, and contribute to broader conceptualisation, is proposed.

Public debt management, monetary policy and financial institutions

Despite the large size of the Brazilian debt market, as well the large diversity of its bonds, the picture that emerges is of a market that has not yet completed its transition from the role it performed during the megainflation years, namely that of providing a liquid asset that provided positive real returns. This unfinished transition is currently placing the market under severe stress, as fears of a possible default from the next administration grow larger. This paper analyzes several aspects pertaining to the management of the domestic public debt. The causes for the extremely large and fast growth ofthe domestic public debt during the seven-year period that President Cardoso are discussed in Section 2. Section 3 computes Value at Risk and Cash Flow at Risk measures for the domestic public debt. The rollover risk is introduced in a mean-variance framework in Section 4. Section 5 discusses a few issues pertaining to the overlap between debt management and monetary policy. Finally, Section 6 wraps up with policy discussion and policy recommendations.