Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Ensuring student success : the role of support services in improving the quality of the student learning experience

This paper is based on the premise that universities have an obligation to provide adequate student support services, such as learning assistance (that is, assistance with academic writing and other study skills) and that in order to be effective such services must be responsive to the wider policy and social implications of student attrition and retention. The paper outlines briefly some of the factors that have influenced the development of learning assistance practices in Australia and America. This is followed by an account of experiences at one Australian metropolitan university where learning assistance service provision shifted from a decentralised, faculty-based model to a centralised model of service delivery. This shift was in response to concerns about lack of quality and consistency in a support model dependent upon faculty resources yet a follow up study identified other problems in the centralised delivery of learning assistance services. These problems, clustered under the heading contextualised versus decontextualised learning assistance, include the relevance of generic learning assistance services to students struggling with specific course related demands; the apparent tensions between challenging students and assisting students at risk of failure; and variations in the level of collaboration between learning advisers and academic staff in supporting students in the learning environment. These problems are analysed using the theoretical modelling derived from the tools made available through cultural historical activity theory and expansive visibilisation (Engeström & Miettinen, 1999).

The Maori social science academy and evidence-based policy

This paper focuses on recent moves to forge stronger linkages between the Māori social science academy and the policy industry. A critical appraisal of this development is offered, with particular attention given to the desirability of enhancing the academy’s role in the policy process, given the policy industry’s continued privileging of Eurocentric theory and research methodologies within the developing evidence-based environment. The paper ends with a discussion of the possibilities and problems associated with engagement with the policy industry, particularly as these relate to the various roles members can (or are forced to) take; either as ‘insiders’ (such as policy workers and contract researchers), or independent, critical ‘outsiders’. The author concludes that the best that insiders can hope for are incremental, largely ineffective changes to Māori policy, while independent members of the academy are best placed to speak on behalf of Māori, Māori communities, hapu and iwi.

Collaborative inquiries into literacy, place and identity in changing policy contexts : implications for teacher development

In this chapter we describe a history of collaboration between university-based literacy researchers and school-based teachers in teacher development programs and practitioner inquiries designed to improve literacy outcomes for students living in low-socioeconomic circumstances. We consider how an inquiry stance has informed teachers working for social justice through curriculum and pedagogy designed to connect children’s developing literacy repertoires with their changing material, social and linguistic contexts. We use examples from the practices of two of our long-term teacher-collaborators to show what has been possible to achieve, even in radically different policy contexts, because of teachers’ continued commitment to themes of place and belonging, and language and identity.

Having a yarn about smoking : using action research to develop a 'no smoking' policy within an Aboriginal health organisation

Objectives: This article reports on a culturally appropriate process of development of a smoke-free workplace policy within the peak Aboriginal Controlled Community Health Organisation in Victoria, Australia. Smoking is acknowledged as being responsible for at least 20% of all deaths in Aboriginal communities in Australia, and many Aboriginal health workers smoke. Methods: The smoke-free workplace policy was developed using the iterative, discursive and experience-based methodology of Participatory Action Research, combined with the culturally embedded concept of ‘having a yarn’. Results: Staff members initially identified smoking as a topic to be avoided within workplace discussions. This was due, in part, to grief (everyone had suffered a smoking related bereavement). Further, there was anxiety that discussing smoking would result in culturally difficult conflict. The use of yarning opened up a safe space for discussion and debate,enabling development of a policy that was accepted across the organisation. Conclusions: Within Aboriginal organisations, it is not sufficient to focus on the outcomes of policy development. Rather, due attention must be paid to the process employed in development of policy, particularly when that policy is directly related to an emotionally and communally weighted topic such as smoking.

Assessment policy change in relation to English language teaching and learning in China : a study of perspectives from two universities

In response to concerns about the quality of English Language Learning (ELL) education at tertiary level, the Chinese Ministry of Education (CMoE) launched the College English Reform Program (CERP) in 2004. By means of a press release (CMoE, 2005) and a guideline document titled College English Curriculum Requirements (CECR) (CMoE, 2007), the CERP proposed two major changes to the College English assessment policy, which were: (1) the shift to optional status for the compulsory external test, the College English Test Band 4 (CET4); and (2) the incorporation of formative assessment into the existing summative assessment framework. This study investigated the interactions between the College English assessment policy change, the theoretical underpinnings, and the assessment practices within two Chinese universities (one Key University and one Non-Key University). It adopted a sociocultural theoretical perspective to examine the implementation process as experienced by local actors of institutional and classroom levels. Systematic data analysis using a constant comparative method (Merriam, 1998) revealed that contextual factors and implementation issues did not lead to significant differences in the two cases. Lack of training in assessment and the sociocultural factors such as the traditional emphasis on the product of learning and hierarchical teacher/students relationship are decisive and responsible for the limited effect of the reform.

National Cultural Policy submission

This was my submission to the Australian Federal Government’s call for submissions in response to the National Cultural Policy Discussion Paper.

Social outcomes in the construction industry : the case of the Western Australia ‘Percent for Art’ policy

Social outcomes, in particular intangible social outcomes, are generally difficult to achieve in the construction industry due to the predominantly episodic, fragmented and heavily regulated nature of construction that presupposes a tendency towards mainstream construction processes and design. The Western Australian ‘Percent for Art’ policy is recognized for stimulating social outcomes, by creating richer and more aesthetically pleasing social environments through the incorporation of artwork into public buildings. A case study of four Percent for Art projects highlights the role of the Artwork Selection Committee in incorporating artwork into construction. A total of 20 semi-structured interviews were conducted with committee members and policy officers. Data analysis involved a combination of pattern coding and matrix categorization, and resulted in the identification of the committee’s three key elements of collaborative communication, democratic decision-making and project champions. The findings suggest these key elements foster the interaction, communication and relationships needed to facilitate feedback, enhance relationships, create cross-functional teams and lower project resistance, which are all necessary to overcome constraints to social outcomes in construction. The findings provide greater insight into the mechanisms for achieving social outcomes and a basis for future discussion about the processes for achieving social outcomes in the construction industry.

Preventing injuries on a state-wide basis : researchers and parliamentary committees are partners in turning injury prevention and safety research into practice and policy

Context: Parliamentary committees established in Westminster parliaments, such as Queensland, provide a cross-party structure that enables them to recommend policy and legislative changes that may otherwise be difficult for one party to recommend. The overall parliamentary committee process tends to be more cooperative and less adversarial than the main chamber of parliament and, as a result, this process permits parliamentary committees to make recommendations more on the available research evidence and less on political or party considerations. Objectives: This paper considers the contributions that parliamentary committees in Queensland have made in the past in the areas of road safety, drug use as well as organ and tissue donation. The paper also discusses the importance of researchers actively engaging with parliamentary committees to ensure the best evidence based policy outcomes. Key messages: In the past, parliamentary committees have successfully facilitated important safety changes with many committee recommendations based on research results. In order to maximise the benefits of the parliamentary committee process it is essential that researchers inform committees about their work and become key stakeholders in the inquiry process. Researchers can keep committees informed by making submissions to their inquiries, responding to requests for information and appearing as witnesses at public hearings. Researchers should emphasise the key findings and implications of their research as well as considering the jurisdictional implications and political consequences. It is important that researchers understand the differences between lobbying and providing informed recommendations when interacting with committees. Discussion and conclusions: Parliamentary committees in Queensland have successfully assisted in the introduction of evidence based policy and legislation. In order to present best practice recommendations, committees rely on the evidence presented to them including the results of researchers. Actively engaging with parliamentary committees will help researchers to turn their results into practice with a corresponding decrease in injuries and fatalities. Developing an understanding of parliamentary committees, and the typical inquiry process used by these committees, will help researchers to present their research results in a manner that will encourage the adoption of their ideas by parliamentary committees, the presentation of these results as recommendations within the report and the subsequent enactment of the committee’s recommendations by the government.

Retrofitting commercial office buildings for sustainability : tenants’ expectations and experiences

Introduction Buildings, which account for approximately half of all annual energy and greenhouse gas emissions, are an important target area for any strategy addressing climate change. Whilst new commercial buildings increasingly address sustainability considerations, incorporating green technology in the refurbishment process of older buildings is technically, financially and socially challenging. This research explores the expectations and experiences of commercial office building tenants, whose building was under-going green refurbishment. Methodology Semi-structured in-depth interviews with seven residents and neighbours of a large case-study building under-going green refurbishment in Melbourne, Australia. Built in 1979, the 7,008m² ‘B’ grade building consists of 11 upper levels of office accommodation, ground floor retail, and a basement area leased as a licensed restaurant. After refurbishment, which included the installation of chilled water pumps, solar water heating, waterless urinals, insulation, disabled toilets, and automatic dimming lights, it was expected that the environmental performance of the building would move from a non-existent zero ABGR (Australian Building Greenhouse Rating) star rating to 3.5 stars, with a 40% reduction in water consumption and 20% reduction in energy consumption. Interviews were transcribed, with responses analysed using a thematic approach, identifying categories, themes and patterns. Results Commercial property tenants are on a journey to sustainability - they are interested and willing to engage in discussions about sustainability initiatives, but the process, costs and benefits need to be clear. Critically, whilst sustainability was an essential and non-negotiable criterion in building selection for government and larger corporate tenants, sustainability was not yet a core business value for smaller organisations – whilst they could see it as an emerging issue, they wanted detailed cost-benefit analyses, pay-back calculations of proposed technologies and, ideally, wished they could trial the technology first-hand in some way. Although extremely interested in learning more, most participants reported relatively minimal knowledge of specific sustainability features, designs or products. In discussions about different sustainable technologies (e.g., waterless urinals, green-rated carpets), participants frequently commented that they knew little about the technology, had not heard of it or were not sure exactly how it worked. Whilst participants viewed sustainable commercial buildings as the future, they had varied expectations about the fate of existing older buildings – most felt that they would have to be retrofitted at some point to meet market expectations and predicted the emergence of a ‘non-sustainability discount’ for residing in a building without sustainable features. Discussion This research offers a beginning point for understanding the difficulty of integrating green technology in older commercial buildings. Tenants currently have limited understandings of technology and potential building performance outcomes, which ultimately could impede the implementation of sustainable initiatives in older buildings. Whilst the commercial property market is interested in learning about sustainability in the built environment, the findings highlight the importance of developing a strong business case, communication and transition plan for implementing sustainability retrofits in existing commercial buildings.