Learning, evaluation, action and reflection for new technologies, empowerment and rural sustainability (The LEARNERS project)

-

Meta-evaluation to improve learning, evaluation capacity development and sustainability : Findings from a participatory evaluation project in Nepal

Recent studies suggest that meta-evaluation can be valuable in developing new approaches to evaluation, building evaluation capacities, and enhancing organizational learning. These new extensions of the concept of meta-evaluation are significant, given the growing emphasis on improving the quality and effectiveness of evaluation practices in the South Asian region. Following a review of the literature, this paper presents a case study of the use of concurrent meta-evaluation in the four-year project Assessing Communication for Social Change which developed and trialled a participatory impact assessment methodology in collaboration with a development communication Non-government organization (NGO) in Nepal. Key objectives of the meta-evaluation included to: continuously develop, adapt and improve the impact assessment methodology, Monitoring and Evaluation (M&E) systems and process and other project activities; identify impacts of the project; and build capacities in critical reflection and review. Our analysis indicates that this meta-evaluation was essential to understanding various constraints related to the organizational context that affected the success of the project and the development of improved M&E systems and capacities within the NGO. We identified several limitations of our meta-evaluation methods, which were balanced by the strengths of other methods. Our case study suggests that as well as assessing the quality, credibility and value of evaluation practices, meta-evaluations need to focus on important contextual issues that can have significant impacts on the outcomes of participatory evaluation projects. They include hierarchical organizational cultures, communication barriers, power/knowledge relations, and the time and resources available. Meta-evaluations also need to consider wider issues such as the sustainability of evaluation systems and approaches.

Livestock and learning: evaluation of a prototype 3D virtual learning environment for livestock practitioners in India and Bolivia

Livestock are a key asset for the global poor. However, access to relevant information is a critical issue for both the poor and the practitioners who serve them. Therefore, the authors describe a web-based Virtual Learning Environment to disseminate educational materials on priority animal health constraints in Bolivia and India. The aim was to explore demand for 3D among development practitioners in the South. Two wider arguments from the ICT4D literature framed the analysis: the concept of 3D as a ‘lead technology’ and the relevance of Internet skills to the adoption of a 3D format. The results illustrated that neither construct influenced demand. Rather, study participants were ready adopters but desired greater levels of interaction and thereby, a more collaborative learning environment. Therefore, 3D has a number of potential benefits to enhance knowledge sharing among community practitioners in the Global South.

Design education online : learning delivery and evaluation

Online learning has been recognised as an effective pedagogical method and tool, and is broadly integrated into various types of teaching and learning strategies in higher education. In practice, the use of Virtual Learning Environment (VLE) in higher education has become an integral strategy for quality education. The field of design education however has not been researched extensively in regard to online learning, delivery and evaluation. This paper discusses design education from an online learning perspective. It proposes an integrated framework with three key components for online learning via VLE including an interactive delivery structure, communication channels, and learning evaluation. Additionally, the paper describes and evaluates how VLE sites for two design units were built based on an integrated framework and student learning experiences. The results indicate that online design education should be integrated with various educational values and functional features in a systematic manner, and requires designing learning evaluation protocols as part of learning activities and communicative forms within online-based learning sites.

Course evaluation : reconfigurations for learning with learning management systems

The introduction of online delivery platforms such as learning management systems (LMS) in tertiary education has changed the methods and modes of curriculum delivery and communication. While course evaluation methods have also changed from paper-based in-class-administered methods to largely online-administered methods, the data collection instruments have remained unchanged. This paper reports on a small exploratory study of two tertiary-level courses. The study investigated why design of the instruments and methods to administer surveys in the courses are ineffective measures against the intrinsic characteristics of online learning. It reviewed the students' response rates of the conventional evaluations for the courses over an eight-year period. It then compared a newly developed online evaluation and the conventional methods over a two-year period. The results showed the response rates with the new evaluation method increased by more than 80% from the average of the conventional evaluations (below 30%), and the students' written feedback was more detailed and comprehensive than in the conventional evaluations. The study demonstrated the possibility that the LMS-based learning evaluation can be effective and efficient in terms of the quality of students' participation and engagement in their learning, and for an integrated pedagogical approach in an online learning environment.

Cross-cultural learning and international collaboration: evaluation of a study tour to Thailand

Purpose:
To evaluate cross-cultural learning among Thai staff and host students from the Faculty of Nursing, Mahidol University (MU) and Australian guest students from the School of Nursing and Midwifery, Deakin University (DU), who participated in a study tour.

Design:
Descriptive exploratory evaluation.

Methods:
Key stakeholders were invited to participate resulting in a convenience sample of seven MU staff, five MU and 22 DU students. Data were collected using mixed methods. Qualitative data were theme analysed and quantitative data were analysed using descriptive statistics.

Main findings:
The semi-structured interviews with MU staff, focus group with MU students and free response questions in the online survey with DU students indicated the themes of enhanced and valuable cross-cultural learning and relationship building, the challenges of different social behaviours and the importance of tolerance and acceptance. In the online survey, over 77% (n = 17) of DU students reported high satisfaction with their cross-cultural learning on the study tour. The online survey included the validated Miville-Guzman Universality-Diversity scale short form (M-GUD-S). All Australian students reported seeking diversity of contact (X ± SD = 23.1 ± 4.4), relativistic appreciation (X ± SD = 24.7 ± 3.9), and comfort with differences (X ± SD = 26.2 ± 3.0), indicating high levels of openness to cultural diversity and similarity on the M-GUD-S. 

Conclusion and recommendations:
This study provides an example of an evaluated study tour emphasising cross-cultural relationship building. Findings indicate that nursing education should include opportunities for intercultural exchange among nursing students. Nurses require excellent skills in cross-cultural nursing and relating to meet the future global challenges to health care over the next millennium.

Evaluation of online higher education: learning, interaction and technology

The work presents a theoretical framework for the evaluation of e-Teaching that aims at positioning the online activities designed and developed by the teacher as to the Learning, Interaction and Technology Dimensions. The theoretical research that underlies the study was developed reflecting current thinking on the promotion of quality of teaching and of the integration of information and communication tools into the curriculum in Higher Education (HE), i.e., bearing in mind some European guidelines and policies on this subject. This way, an answer was sought to be given to one of the aims put forward in this study, namely to contribute towards the development of a conceptual framework to support research on evaluation of e-teaching in the context of HE. Based on the theoretical research carried out, an evaluation tool (SCAI) was designed, which integrates the two questionnaires developed to collect the teachers' and the students' perceptions regarding the development of e-activities. Consequently, an empirical study was structured and carried out, allowing SCAI tool to be tested and validated in real cases. From the comparison of the theoretical framework established and the analysis of the data obtained, we found that the differences in teaching should be valued and seen as assets by HE institutions rather than annihilated in a globalizing perspective.

Ensuring student success : the role of support services in improving the quality of the student learning experience

This paper is based on the premise that universities have an obligation to provide adequate student support services, such as learning assistance (that is, assistance with academic writing and other study skills) and that in order to be effective such services must be responsive to the wider policy and social implications of student attrition and retention. The paper outlines briefly some of the factors that have influenced the development of learning assistance practices in Australia and America. This is followed by an account of experiences at one Australian metropolitan university where learning assistance service provision shifted from a decentralised, faculty-based model to a centralised model of service delivery. This shift was in response to concerns about lack of quality and consistency in a support model dependent upon faculty resources yet a follow up study identified other problems in the centralised delivery of learning assistance services. These problems, clustered under the heading contextualised versus decontextualised learning assistance, include the relevance of generic learning assistance services to students struggling with specific course related demands; the apparent tensions between challenging students and assisting students at risk of failure; and variations in the level of collaboration between learning advisers and academic staff in supporting students in the learning environment. These problems are analysed using the theoretical modelling derived from the tools made available through cultural historical activity theory and expansive visibilisation (Engeström & Miettinen, 1999).

Critical success elements for the design and implementation of organisational e-learning

Organisations are engaging in e-learning as a mechanism for delivering flexible learning to meet the needs of individuals and organisations. In light of the increasing use and organisational investment in e-learning, the need for methods to evaluate the success of its design and implementation seems more important than ever. To date, developing a standard for the evaluation of e-learning appears to have eluded both academics and practitioners. The currently accepted evaluation methods for e-learning are traditional learning and development models, such as Kirkpatrick’s model (1976). Due to the technical nature of e-learning it is important to broaden the scope and consider other evaluation models or techniques, such as the DeLone and McLean Information Success Model, that may be applicable to the e-learning domain. Research into the use of e-learning courses has largely avoided considering the applicability of information systems research. Given this observation, it is reasonable to conclude that e-learning implementation decisions and practice could be overlooking useful or additional viewpoints. This research investigated how existing evaluation models apply in the context of organisational e-learning, and resulted in an Organisational E-learning success Framework, which identifies the critical elements for success in an e-learning environment. In particular this thesis highlights the critical importance of three e-learning system creation elements; system quality, information quality, and support quality. These elements were explored in depth and the nature of each element is described in detail. In addition, two further elements were identified as factors integral to the success of an e-learning system; learner preferences and change management. Overall, this research has demonstrated the need for a holistic approach to e-learning evaluation. Furthermore, it has shown that the application of both traditional training evaluation approaches and the D&M IS Success Model are appropriate to the organisational e-learning context, and when combined can provide this holistic approach. Practically, this thesis has reported the need for organisations to consider evaluation at all stages of e-learning from design through to implementation.

REFRAME : a new approach to evaluation in higher education

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.

Evaluation of Serious Games: A Holistic Approach

Digital games constitute a major emerging technology that is expected to enter mainstream educational use within a few years. The highly engaging and motivating character of such games bears great potential to support immersive, meaningful, and situated learning experiences. To seize this potential, meaningful quality and impact measurements are indispensible. Although there is a growing body of evidence on the efficacy of games for learning, evaluation is often poorly designed, incomplete, biased, if not entirely absent. Well-designed evaluations demonstrating the educational effect as well as the return on investment of serious games may foster broader adoption by educational institutions and training providers, and support the development of the serious game industry. The European project RAGE introduces a comprehensive and multi-perspective framework for serious game evaluation, which is presented in this paper.

How training evaluation can be used to potentiate employee development in an organization

A Work Project, presented as part of the requirements for the Award of a Masters Degree in Management from the NOVA – School of Business and Economics