Meta-evaluation to improve learning, evaluation capacity development and sustainability : Findings from a participatory evaluation project in Nepal

Recent studies suggest that meta-evaluation can be valuable in developing new approaches to evaluation, building evaluation capacities, and enhancing organizational learning. These new extensions of the concept of meta-evaluation are significant, given the growing emphasis on improving the quality and effectiveness of evaluation practices in the South Asian region. Following a review of the literature, this paper presents a case study of the use of concurrent meta-evaluation in the four-year project Assessing Communication for Social Change which developed and trialled a participatory impact assessment methodology in collaboration with a development communication Non-government organization (NGO) in Nepal. Key objectives of the meta-evaluation included to: continuously develop, adapt and improve the impact assessment methodology, Monitoring and Evaluation (M&E) systems and process and other project activities; identify impacts of the project; and build capacities in critical reflection and review. Our analysis indicates that this meta-evaluation was essential to understanding various constraints related to the organizational context that affected the success of the project and the development of improved M&E systems and capacities within the NGO. We identified several limitations of our meta-evaluation methods, which were balanced by the strengths of other methods. Our case study suggests that as well as assessing the quality, credibility and value of evaluation practices, meta-evaluations need to focus on important contextual issues that can have significant impacts on the outcomes of participatory evaluation projects. They include hierarchical organizational cultures, communication barriers, power/knowledge relations, and the time and resources available. Meta-evaluations also need to consider wider issues such as the sustainability of evaluation systems and approaches.

Libraries as co-working spaces : understanding user motivations and perceived barriers to social learning

This paper aims to inform design strategies for smart space technology to enhance libraries as environments for co-working and informal social learning. The focus is on understanding user motivations, behaviour, and activities in the library when there is no programmed agenda. The study analyses gathered data over five months of ethnographic research at ‘The Edge’ – a bookless library space at the State Library of Queensland in Brisbane, Australia, that is explicitly dedicated to co-working, social learning, peer collaboration, and creativity around digital culture and technology. The results present five personas that embody people’s main usage patterns as well as motivations, attitudes, and perceived barriers to social learning. It appears that most users work individually or within pre-organised groups, but usually do not make new connections with co-present, unacquainted users. Based on the personas, four hybrid design dimensions are suggested to improve the library as a social interface for shared learning encounters across physical and digital spaces. The findings in this paper offer actionable knowledge for managers, decision makers, and designers of technology-enhanced library spaces and similar collaboration and co-working spaces.

Professional practice research : ensuring teacher development through a critical approach to professional learning

Currently the development of a national system for the ongoing enhancement of teacher professionalism across Australia is underway. The initiative led by Australian Institute of Teaching and School Leadership (AITSL) on behalf of the Ministerial Council for Education, Early Childhood Development and Youth Affairs (now SCSEEC) is progressing rapidly with a finalised set of Professional Standards for Teachers and a set of Professional Standards for Principals approved by Ministers in 2011. It is clear that there is an inextricable link between the newly proposed professional standards and the professional education of teachers and principals across Australia. Further, it is imperative that the education sector will need to work in a unified manner through ongoing consultations to ensure the standards truly reflect what teachers and principals desire of the profession, in terms of teacher preparation, professional learning and training, and professional recognition.

A learner perspective on barriers to e-learning

This study aims to identify and categorize barriers to e-learning adoption and the relative impact of those barriers on learners. It contributes to the understanding of learner perceptions of barriers, the different types of barriers and their relative importance. This study used a quantitative methodology grounded in previous literature. The study is based on a self-administered questionnaire conducted with employees in a rail organisation. This research demonstrates there are three key factors that represent barriers to e-learning: the nature of e-learning as a learning approach, the use of technology, and concerns about lack of time and potential interruptions when trying to complete e-learning.

Evaluating the use of online synchronous communication to enhance learning in statistics

According to social constructivists, learners are active participants in constructing new knowledge in a social process where they interact with others. In these social settings teachers or more knowledgeable peers provide support. This research study investigated the contribution that an online synchronous tutorial makes to support teaching and learning of undergraduate introductory statistics offered by an Australian regional university at a distance. The introductory statistics course which served as a research setting in this study was a requirement of a variety of programs at the University, including psychology, business and science. Often students in these programs perceive this course to be difficult and irrelevant to their programs of study. Negative attitudes and associated anxiety mean that students often struggle with the content. While asynchronous discussion forums have been shown to provide a level of interaction and support, it was anticipated that online synchronous tutorials would offer immediate feedback to move students forward through ―stuck places.‖ At the beginning of the semester the researcher offered distance students in this course the opportunity to participate in a weekly online synchronous tutorial which was an addition to the usual support offered by the teaching team. This tutorial was restricted to 12 volunteers to allow sufficient interaction to occur for each of the participants. The researcher, as participant-observer, conducted the weekly tutorials using the University's interactive online learning platform, Wimba Classroom, whereby participants interacted using audio, text chat and a virtual whiteboard. Prior to the start of semester, participants were surveyed about their previous mathematical experiences, their perceptions of the introductory statistics course and why they wanted to participate in the online tutorial. During the semester, they were regularly asked pertinent research questions related to their personal outcomes from the tutorial sessions. These sessions were recorded using screen capture software and the participants were interviewed about their experiences at the end of the semester. Analysis of these data indicated that the perceived value of online synchronous tutorial lies in the interaction with fellow students and a content expert and with the immediacy of feedback given. The collaborative learning environment offered the support required to maintain motivation, enhance confidence and develop problemsolving skills in these distance students of introductory statistics. Based on these findings a model of online synchronous learning is proposed.

E-motion diaries : blogging emotional responses to online learning in higher education

Emotions are inherently social, and are central to learning, online interaction and literacy practices (Shen, Wang, & Shen, 2009). Demonstrating the dynamic sociality of literacy practice, we used e-motion diaries or web logs to explore the emotional states of pre-service high school teachers’ experiences of online learning activities. This is because the methods of communication used by university educators in online learning and writing environments play an important role in fulfilling students’ need for social interaction and inclusion (McInnerney & Roberts, 2004). Feelings of isolation and frustration are common emotions experienced by students in many online learning environments, and are associated with the success or failure of online interactions and learning (Su, et al., 2005). The purpose of the study was to answer the research question: What are the trajectories of pre-service teachers’ emotional states during online learning experiences? This is important because emotions are central to learning, and the current trend toward Massive Open Online Courses (MOOCs) needs research about students’ emotional connections in online learning environments (Kop, 2011). The project was conducted with a graduate class of 64 high school science pre-service teachers in Science Education Curriculum Studies in a large Australian university, including males and females from a variety of cultural backgrounds, aged 22-55 years. Online activities involved the students watching a series of streamed live lectures for the first 5 weeks providing a varied set of learning experiences, such as viewing science demonstrations (e.g., modeling the use of discrepant events). Each week, students provided feedback on learning by writing and posting an e-motion diary or web log about their emotional response. Students answered the question: What emotions did you experience during this learning experience? The descriptive data set included 284 online posts, with students contributing multiple entries. Linguistic appraisal theory, following Martin and White (2005), was used to regroup the 22 different discrete emotions reported by students into the six main affect groups – three positive and three negative: unhappiness/happiness, insecurity/security, and dissatisfaction/satisfaction. The findings demonstrated that the pre-service teachers’ emotional responses to the streamed lectures tended towards happiness, security, and satisfaction within the typology of affect groups – un/happiness, in/security, and dis/satisfaction. Fewer students reported that the streamed lectures triggered negative feelings of frustration, powerlessness, and inadequacy, and when this occurred, it often pertained to expectations of themselves in the forthcoming field experience in classrooms. Exceptions to this pattern of responses occurred in relation to the fifth streamed lecture presented in a non-interactive slideshow format that compressed a large amount of content. Many students responded to the content of the lecture rather than providing their emotional responses to this lecture, and one student felt “completely disengaged”. The social practice of online writing as blogs enabled the students to articulate their emotions. The findings primarily contribute new understanding about students' wide range of differing emotional states, both positive and negative, experienced in response to streamed live lectures and other learning activities in higher education external coursework. The is important because the majority of previous studies have focused on particular negative emotions, such as anxiety in test taking. The research also highlights the potentials of appraisal theory for studying human emotions in online learning and writing.

Dilemmatic spaces : high-stakes testing and the possibilities of collaborative knowledge work to generate learning innovations

This paper examines collaborative researcher-practitioner knowledge work around assessment data in culturally diverse, low- socioeconomic school communities in Queensland, Australia. Specifically, the paper draws on interview accounts about the work of a bridging knowledge flows between a local university and a cluster of schools. We draw on Bernstein’s (2000) concept of recontextualisation to explore the processes of knowledge mediation in dialogues around student assessment data to design instructional innovations. We argue that critical policy studies need to explore the complex ways in which neoliberal education policies are enacted in local sites. Moreover, we suggest that an analysis of collaborative knowledge work designed to improve student learning outcomes in low-socioeconomic school communities necessitates attention to the principles regulating knowledge flows across boundaries. In addition, it necessitates attention to the ways in which mediators navigate dilemmatic spaces, anxieties and affects/feelings in order to generate innovative learning designs in the current global context of high-stakes national testing and accountability regimes.

Investigating a disciplinary approach to literacy learning in a secondary school

This paper presents findings from a Design-Based Research (DBR) project undertaken in a large regional high school in Queensland. The study focused on an intervention involving explicit teaching using systemic functional grammar in assessed writing across two Year 8 subjects: English and History. The study’s findings demonstrate that, despite efforts at the whole school and classroom level to support a disciplinary literacy approach to subject learning, there are considerable constraints that need to be considered and overcome in order for students to develop appropriate writing capabilities for particular discipline areas.

Designing rich information experiences to shape learning outcomes

Students in higher education typically learn to use information as part of their course of study, which is intended to support ongoing academic, personal and professional growth. Informing the development of effective information literacy education, this research uses a phenomenographic approach to investigate the experiences of a teacher and students engaged in lessons focused on exploring language and gender topics by tracing and analyzing their evolution through scholarly discourse. The findings suggest that the way learners use information influences content-focused learning outcomes, and reveal how teachers may enact lessons that enable students to learn to use information in ways that foster a specific understanding of the topic they are investigating.

Can teacher collaboration overcome barriers to interdisciplinary learning in a disciplinary university? A case study using climate change

A teacher network was formed at an Australian university in order to better promote interdisciplinary student learning on the complex social-environmental problem of climate change. Rather than leaving it to students to piece together disciplinary responses, eight teaching academics collaborated on the task of exposing students to different types of knowledge in a way that was more than the summing of disciplinary parts. With a part-time network facilitator providing cohesion, network members were able to teach into each other’s classes, and share material and student activities across a range of units that included business, zoology, marine science, geography and education. Participants reported that the most positive aspects of the project were the collegiality and support for teaching innovation provided by peers. However, participants also reported being time-poor and overworked. Maintaining the collaboration beyond the initial one year project proved difficult because without funding for the network facilitator, participants were unable to dedicate the time required to meet and collaborate on shared activities. In order to strengthen teacher collaboration in a university whose administrative structures are predominantly discipline-based, there is need for recognition of the benefits of interdisciplinary learning to be matched by recognition of the need for financial and other resources to support collaborative teaching initiatives.