Learning, evaluation, action and reflection for new technologies, empowerment and rural sustainability (The LEARNERS project)

-

Meta-evaluation to improve learning, evaluation capacity development and sustainability : Findings from a participatory evaluation project in Nepal

Recent studies suggest that meta-evaluation can be valuable in developing new approaches to evaluation, building evaluation capacities, and enhancing organizational learning. These new extensions of the concept of meta-evaluation are significant, given the growing emphasis on improving the quality and effectiveness of evaluation practices in the South Asian region. Following a review of the literature, this paper presents a case study of the use of concurrent meta-evaluation in the four-year project Assessing Communication for Social Change which developed and trialled a participatory impact assessment methodology in collaboration with a development communication Non-government organization (NGO) in Nepal. Key objectives of the meta-evaluation included to: continuously develop, adapt and improve the impact assessment methodology, Monitoring and Evaluation (M&E) systems and process and other project activities; identify impacts of the project; and build capacities in critical reflection and review. Our analysis indicates that this meta-evaluation was essential to understanding various constraints related to the organizational context that affected the success of the project and the development of improved M&E systems and capacities within the NGO. We identified several limitations of our meta-evaluation methods, which were balanced by the strengths of other methods. Our case study suggests that as well as assessing the quality, credibility and value of evaluation practices, meta-evaluations need to focus on important contextual issues that can have significant impacts on the outcomes of participatory evaluation projects. They include hierarchical organizational cultures, communication barriers, power/knowledge relations, and the time and resources available. Meta-evaluations also need to consider wider issues such as the sustainability of evaluation systems and approaches.

Design education online : learning delivery and evaluation

Online learning has been recognised as an effective pedagogical method and tool, and is broadly integrated into various types of teaching and learning strategies in higher education. In practice, the use of Virtual Learning Environment (VLE) in higher education has become an integral strategy for quality education. The field of design education however has not been researched extensively in regard to online learning, delivery and evaluation. This paper discusses design education from an online learning perspective. It proposes an integrated framework with three key components for online learning via VLE including an interactive delivery structure, communication channels, and learning evaluation. Additionally, the paper describes and evaluates how VLE sites for two design units were built based on an integrated framework and student learning experiences. The results indicate that online design education should be integrated with various educational values and functional features in a systematic manner, and requires designing learning evaluation protocols as part of learning activities and communicative forms within online-based learning sites.

Course evaluation : reconfigurations for learning with learning management systems

The introduction of online delivery platforms such as learning management systems (LMS) in tertiary education has changed the methods and modes of curriculum delivery and communication. While course evaluation methods have also changed from paper-based in-class-administered methods to largely online-administered methods, the data collection instruments have remained unchanged. This paper reports on a small exploratory study of two tertiary-level courses. The study investigated why design of the instruments and methods to administer surveys in the courses are ineffective measures against the intrinsic characteristics of online learning. It reviewed the students' response rates of the conventional evaluations for the courses over an eight-year period. It then compared a newly developed online evaluation and the conventional methods over a two-year period. The results showed the response rates with the new evaluation method increased by more than 80% from the average of the conventional evaluations (below 30%), and the students' written feedback was more detailed and comprehensive than in the conventional evaluations. The study demonstrated the possibility that the LMS-based learning evaluation can be effective and efficient in terms of the quality of students' participation and engagement in their learning, and for an integrated pedagogical approach in an online learning environment.

Ensuring student success : the role of support services in improving the quality of the student learning experience

This paper is based on the premise that universities have an obligation to provide adequate student support services, such as learning assistance (that is, assistance with academic writing and other study skills) and that in order to be effective such services must be responsive to the wider policy and social implications of student attrition and retention. The paper outlines briefly some of the factors that have influenced the development of learning assistance practices in Australia and America. This is followed by an account of experiences at one Australian metropolitan university where learning assistance service provision shifted from a decentralised, faculty-based model to a centralised model of service delivery. This shift was in response to concerns about lack of quality and consistency in a support model dependent upon faculty resources yet a follow up study identified other problems in the centralised delivery of learning assistance services. These problems, clustered under the heading contextualised versus decontextualised learning assistance, include the relevance of generic learning assistance services to students struggling with specific course related demands; the apparent tensions between challenging students and assisting students at risk of failure; and variations in the level of collaboration between learning advisers and academic staff in supporting students in the learning environment. These problems are analysed using the theoretical modelling derived from the tools made available through cultural historical activity theory and expansive visibilisation (Engeström & Miettinen, 1999).

Critical success elements for the design and implementation of organisational e-learning

Organisations are engaging in e-learning as a mechanism for delivering flexible learning to meet the needs of individuals and organisations. In light of the increasing use and organisational investment in e-learning, the need for methods to evaluate the success of its design and implementation seems more important than ever. To date, developing a standard for the evaluation of e-learning appears to have eluded both academics and practitioners. The currently accepted evaluation methods for e-learning are traditional learning and development models, such as Kirkpatrick’s model (1976). Due to the technical nature of e-learning it is important to broaden the scope and consider other evaluation models or techniques, such as the DeLone and McLean Information Success Model, that may be applicable to the e-learning domain. Research into the use of e-learning courses has largely avoided considering the applicability of information systems research. Given this observation, it is reasonable to conclude that e-learning implementation decisions and practice could be overlooking useful or additional viewpoints. This research investigated how existing evaluation models apply in the context of organisational e-learning, and resulted in an Organisational E-learning success Framework, which identifies the critical elements for success in an e-learning environment. In particular this thesis highlights the critical importance of three e-learning system creation elements; system quality, information quality, and support quality. These elements were explored in depth and the nature of each element is described in detail. In addition, two further elements were identified as factors integral to the success of an e-learning system; learner preferences and change management. Overall, this research has demonstrated the need for a holistic approach to e-learning evaluation. Furthermore, it has shown that the application of both traditional training evaluation approaches and the D&M IS Success Model are appropriate to the organisational e-learning context, and when combined can provide this holistic approach. Practically, this thesis has reported the need for organisations to consider evaluation at all stages of e-learning from design through to implementation.

REFRAME : a new approach to evaluation in higher education

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.

Student success : the identification and support of first year university students at risk of attrition

The engagement behaviour of 1,524 student-enrolments (“students”) in five first year units was monitored and 608 (39.9%) were classified as “at risk” using the criterion of not submitting or failing their first assignment. Of these, 327 (53.8%) were successfully contacted (i.e., spoken to by phone) and provided with advice and/or referral to learning and personal support services while the remaining 281 (46.2%) could not be contacted. Nine hundred and sixteen students (60.1%) were classified as “not at risk.” Overall, the at risk group who were contacted achieved significantly higher end-of-semester final grades than, and persisted (completed the unit) at more than twice the rate of, the at risk group who were not contacted. There were variations among the units which were explained by the timing of the first assignment, specific teaching-learning processes and the structure of the curriculum. Implications for curriculum design and supporting first year students within a personal, social and academic framework are discussed.

Supervising practice based research

Practice based research appears to have emerged within several Higher Education agendas including the professional doctorates and the teacher as researcher. One way of thinking about this methodological approach is to consider its research paradigm – a practice based epistemology, and from this perspective to consider what special application to research supervision the paradigm invites. Within a “supervision as pedagogy” agenda these applications can be considered as pedagogies. This paper has been written in the style of practice based research, drawing on the author’s own experiences of supervising students undertaking practice based research. It adopts a position that research supervision is pedagogy and draws on the model of ‘Productive Pedagogies” to articulate strategies to help novice research students develop a research proposal.

Making use of pedagogic models as reflective catalysts for investigating pedagogic practice

The notion of pedagogy for anyone in the teaching profession is innocuous. The term itself, is steeped in history but the details of the practice can be elusive. What does it mean for an academic to be embracing pedagogy? The problem is not limited to academics; most teachers baulk at the introduction of a pedagogic agenda and resist attempts to have them reflect on their classroom teaching practice, where ever that classroom might be constituted. This paper explores the application of a pedagogic model (Education Queensland, 2001) which was developed in the context of primary and secondary teaching and was part of a schooling agenda to improve pedagogy. As a teacher educator I introduced the model to classroom teachers (Hill, 2002) using an Appreciative Inquiry (Cooperrider and Srivastva 1987) model and at the same time applied the model to my own pedagogy as an academic. Despite being instigated as a model for classroom teachers, I found through my own practitioner investigation that the model was useful for exploring my own pedagogy as a university academic (Hill, 2007, 2008). Cooperrider, D.L. and Srivastva, S. (1987) Appreciative inquiry in organisational life, in Passmore, W. and Woodman, R. (Eds) Research in Organisational Changes and Development (Vol 1) Greenwich, CT: JAI Press. Pp 129-69 Education Queensland (2001) School Reform Longitudinal Study (QSRLS), Brisbane, Queensland Government. Hill, G. (2002, December ) Reflecting on professional practice with a cracked mirror: Productive Pedagogy experiences. Australian Association for Research in Education Conference. Brisbane, Australia. Hill, G. (2007) Making the assessment criteria explicit through writing feedback: A pedagogical approach to developing academic writing. International Journal of Pedagogies and Learning 3(1), 59-66. Hill, G. (2008) Supervising Practice Based Research. Studies in Learning, Evaluation, Innovation and Development, 5(4), 78-87

Using the WWW to teach undergraduate nurses clinical communication

On-line learning is increasingly being used in nursing education. Nevertheless, there is still insufficient evidence to demonstrate: whether students respond positively when this form of learning is used to teach relatively practical or clinical subjects; whether it is effective; and whether it is fair to students with less access to, or familiarity with, computers and the internet. In 2003, an on-line Unit on clinical communication was developed for Australian undergraduate nurses in partnership between an Australian School of Nursing and the a Department of Clinical Psychology. Students were overwhelmingly positive in their evaluation of the Unit although some regretted the lack of face-to-face contact with tutors and peers. The best aspects of the Unit included the content and structure being perceived as interesting, fun and informative, the relevance of the material for them as nurses, flexibility to work independently, promotion of critical thinking and gaining an understanding of client issues. Neither their evaluation nor their final grades were related to students’ age or whether they preferred on-line or traditional learning. Students who had readily available computer access, however, had better final grades. Also, students’ grades were correlated with how often they accessed the Unit.