A combined impact-process evaluation of a program promoting active transport to school: understanding the factors that shaped program effectiveness

This mixed methods study was a comprehensive impact-process evaluation of the Ride2School program in metropolitan and regional areas in Victoria, Australia. The program aimed to promote transport to school for primary school children. Qualitative and quantitative data were collected at baseline and followup from two primary schools involved in the pilot phase of the program and two matched comparison schools, and a further 13 primary schools that participated in the implementation phase of the program. Classroom surveys, structured and unstructured observations, and interviews with Ride2School program staff were used to evaluate the pilot program. For the 13 schools in the second phase of the program, parents and students completed questionnaires at baseline (N = 889) and followup (N = 761). Based on the quantitative data, there was little evidence of an overall increase in active transport to school across participating schools, although impacts varied among individual schools. Qualitative data in the form of observations, interviews, and focus group discussions with students, school staff, and program staff provided insight into the reasons for variable program impacts. This paper highlights the benefits of undertaking a mixed methods approach to evaluating active transport to school programs that enables both measurement and understanding of program impacts.

Comparative evaluation of PBL and traditional lecturebased teaching in undergraduate engineering courses : evidence from controlled learning environment

Project and problem-based learning (PBL) has been widely recognised as an active, collaborative, cumulative and integrative learning approach that engages learners, motivates team creativity and centres on practical education. On the other hand, traditional lecture-tutorial teaching is often criticised for being a passive, surface learning and exam-focused approach. In spite of these evidence-based observations and claims over the years, the traditional lecture-tutorial teaching approach still dominates as the preferred teaching approach at Australian universities. This study sets up a control environment to compare these two teaching and learning approaches by analysing data from students' actual performance, course evaluation and expectation in two large undergraduate engineering courses in 2009 and 2010. The evidence reported in this study is broadly interesting in that both courses were taught by the same teaching staff using two entirely different learning and teaching approaches to the same cohort of students in the same semester within the same degree program. The analysis shows that there are significant differences between the students' actual performance, course evaluation and their expectation. Such conflicting differences may be some of the reasons that may negatively impact teaching staff deterring them from switching to PBL from traditional lecture-tutorial teaching.

Cross-cultural learning and international collaboration: evaluation of a study tour to Thailand

Purpose:
To evaluate cross-cultural learning among Thai staff and host students from the Faculty of Nursing, Mahidol University (MU) and Australian guest students from the School of Nursing and Midwifery, Deakin University (DU), who participated in a study tour.

Design:
Descriptive exploratory evaluation.

Methods:
Key stakeholders were invited to participate resulting in a convenience sample of seven MU staff, five MU and 22 DU students. Data were collected using mixed methods. Qualitative data were theme analysed and quantitative data were analysed using descriptive statistics.

Main findings:
The semi-structured interviews with MU staff, focus group with MU students and free response questions in the online survey with DU students indicated the themes of enhanced and valuable cross-cultural learning and relationship building, the challenges of different social behaviours and the importance of tolerance and acceptance. In the online survey, over 77% (n = 17) of DU students reported high satisfaction with their cross-cultural learning on the study tour. The online survey included the validated Miville-Guzman Universality-Diversity scale short form (M-GUD-S). All Australian students reported seeking diversity of contact (X ± SD = 23.1 ± 4.4), relativistic appreciation (X ± SD = 24.7 ± 3.9), and comfort with differences (X ± SD = 26.2 ± 3.0), indicating high levels of openness to cultural diversity and similarity on the M-GUD-S. 

Conclusion and recommendations:
This study provides an example of an evaluated study tour emphasising cross-cultural relationship building. Findings indicate that nursing education should include opportunities for intercultural exchange among nursing students. Nurses require excellent skills in cross-cultural nursing and relating to meet the future global challenges to health care over the next millennium.

More than a game: evaluation report

‘More than a Game’ is a sport-based youth mentoring program developed and implemented by Western Bulldogs in partnership with Newport Islamic Society (NIS), the Australian Federal Police, Victoria Police and Hobsons Bay City Council, with funding from the Attorney General’s Department Building Community Resilience (BCR) grant scheme. The program aimed to develop a community-based resilience model that would use team-based sports to address issues of identity, sense of belonging and cultural isolation amongst young men of Islamic faith, all of which are identified as factors that may promote forms of violent extremism. The program involved 60 young men, aged 15-25, from the Newport Islamic Society in Melbourne’s Western suburbs. The boys were engaged in numerous activities where they were mentored by staff from Western Bulldogs, Victoria Police and Australian Federal Police, who worked in conjunction with community leaders from the Newport Islamic Society.

Through sports-based training, mentoring programs, and community dialogue, ‘More Than a Game’ aimed to develop participants’ leadership, communication, and cross-cultural engagement skills; to identify and facilitate the development of young role models in the community; to enhance greater understanding of the Muslim community in Melbourne’s West, and to foster greater intercultural contact and understanding between participants and other cultural groups. A number of activities were developed and implemented as a part of the program

Evaluation of building performance for strategic facilities management in healthcare : a case study of a public hospital in Australia

Purpose - This paper evaluates the main elements of building performance namely building function, building impact and building quality in order to promote strategic facilities management in healthcare organisation to improve core (health) business activities. Design/methodology/approach - Based on current available toolkits, a questionnaire is issued to healthcare users (staff) in a public hospital about their level of agreement in relation to these elements. Statistical analysis is conducted to regroup the elements. These regrouped elements and their inter relationships are used to develop a framework for measuring building performance in healthcare buildings. Findings - The analysis helped to clarify the understanding and agreement of users in Australian healthcare organisation with regards to building performance. Based on the survey results, 11 new elements were regrouped into three groups. These new regrouped elements will be used to develop a reliable framework for measuring performance of Australian healthcare buildings. Originality/value - Currently there is no building performance toolkit available for Australian healthcare organisation. The framework developed in this paper will help healthcare organisations with a reliable performance tool for their buildings and this will promote strategic facilities management.

Formative evaluation of baby makes 3

Baby Makes 3 was originally developed by Whitehorse Community Health Service and the City of Whitehorse as a program which seeks to prevent violence by promoting respect and equality between couples who have recently become parents for the first time. This is a significant but often stressful event in the life of families and it is not uncommon for couples to adopt ways of relating which reflect gendered norms and foster inequality.
By being offered as a program to all first-time parents by maternal and child health staff, the program in Whitehorse managed to engage with couples at this critical time, many of whom found the program enabled them to adopt greater equality in their relationships. As such, the initial Baby Makes 3 program has been found to be an effective and cost-efficient violence prevention strategy (Flynn, 2011).
The introduction of Baby Makes 3 to the Great South Coast Region is the first time this program has been implemented in a non-metropolitan setting. In addition to the original program aims, a number of additional aims and activities have been added, with the new project being known as Baby Makes 3 Plus.
Programs developed in urban contexts often face different challenges when implemented in rural and regional settings. Likewise, the factors for facilitating success may vary from those in an urban setting (Maidment and Bay, 2012). Hence, this formative evaluation aims to provide feedback on the initial implementation of the original Baby Makes 3 program in the Great South Coast Region with the expectation this will enable problems with implementation and/or opportunities for further enhancement of outcomes to be identified and appropriate action taken.

An evaluation of the impact of using authentic design and build industry projects in project-based learning

BACKGROUNDChisholm’s ‘first year experience’ is a significant feature of the new industry focused Bachelor of Engineering Technology program delivered in association with the South East Melbourne Manufacturers’ Alliance (SEMMA). This conceive-design-implement-operate (CDIO Initiative) program commenced as a full time program in first semester 2012. Whereas it is common for CDIO Initiative programs to have a first year experience program containing a project typical of the type of industry project they would complete as a graduate engineer or engineering technologist, this goes further by using real industry projects provided by SEMMA members.This design-and-build industry project runs across both semesters supporting project-based learning in three first year subjects. A concern is that the industry involvement of the projects adds substantially to an already heavy student workload. This has been further increased by the addition of two additional first year initiatives: writing workshops, and training in, and substantial use of, student oral presentations. It is recognised that an excessive workload could lead students to adopt surface learning approaches in other subjects.PURPOSEThe goal of the project is to evaluate student perceptions of the value and work load impact of the industry project and the other new first year initiatives.DESIGN/METHODCentral to this project is a student survey-based evaluation of the industry project based learning that is the core of the ‘first year experience’. The participants were limited to the small group of students who, in a single year, completed all three subjects that comprise the ‘first year experience’. To avoid compromising the results the survey was administered by Chisholm Institute’s Department of Strategy and Planning with no engineering technology degree program staff present. The survey included questions to enable responses to be linked with specific student demographics without identifying any of the respondents.RESULTSThe study showed the industry project-based learning had worthwhile outcomes but placed considerable time pressures on most respondents. For some, this also impacted on their other subjects. A first year oral presentation program was also shown to have worthwhile outcomes. However no conclusions could be reliably drawn on the third initiative – writing workshops.CONCLUSIONSThe results confirm that the authentic industry project is considered a worthwhile initiative but contributes significantly to student overload. This applies also – to a lesser extent – to the first year oral presentation program. Both also require new approaches to delivery as student numbers increase. Strategies to address these issues are discussed.

The impact of increasing course enrolment on student evaluation of teaching in engineering education

Student evaluation of teaching (SET) is important, commonplace and may be used in staff performance management. The SET literature suggests that class size is a negative systematic influence on SET ratings. In this paper we investigate time-series SET data from a large first-year engineering class where a decline in SET ratings was observed over time as course enrolment increased. We observe a negative halo effect of increasing class size on mean SET ratings and conclude that increasing course enrolment leads to a significant reduction in all mean SET ratings, even when the course learning design remains essentially unchanged. We also find an additional differential effect of increasing course enrolment on mean SET ratings. We observe that the marginal reduction in mean SET ratings for each additional student in the course enrolment is greater for those aspects of the student learning experience that are likely to be most directly impacted by increasing class size. We provide implications for practice from these findings.

The systematic influences on student evaluation of teaching in engineering education

BACKGROUND Student evaluation of teaching (SET) has a long history, has grown in prevalence and importance over a period of decades, and is now common-place in many universities internationally. SET data are collected for a range of purposes, including: as diagnostic feedback to improve the quality of teaching and learning; as an input to staff performance management processes and personnel decisions such as promotion for staff; to provide information to prospective students in their selection of courses and programs; and as a source of data for research on teaching. Rovai et al. (2006) report that while SET research provides mixed results, there is evidence that, for course-related factors, smaller classes are rated more favourably than large classes, upper-year-level classes are rated more favourably than lower-year classes, and that there are rating differences between discipline areas. While additional course-related factors are also noted, other reviews of the literature on SET also identify these three factors as commonly reported systematic influences on SET ratings. The School of Engineering at Deakin University in Australia offers undergraduate and postgraduate engineering programs, and these programs are delivered in both on-campus and off-campus modes.PURPOSEThe paper presents a quantitative investigation of SET data for the School of Engineering at Deakin University to identify whether the commonly reported systematic influences on SET ratings of class size and year level are also observed here. The influence of online mode of offer is also explored.DESIGN/METHOD Deakin University’s Student Evaluation of Teaching and Units (SETU) questionnaire is administered to students enrolled in every unit of study every time that unit is offered, unless it is specially exempted. Following data collation, summary results are reported via a public website. The publicly available SETU data for all School of Engineering units of study were collected for a two year period. The collected data were subjected to analysis of variance (ANOVA) analysis to identify any significant systematic influences on mean student SETU ratings.RESULTS SETU data from 100 separate units of study over the two year period were collected, representing 3375 sets of SETU ratings, and covering unit enrolment sizes from 12 to 462 students. Although this was a modest sized investigation, significantly higher mean ratings for some SETU items were observed for units with small enrolments, for postgraduate level units compared to undergraduate level units, and for units offered in conventional mode compared to online mode of offer. The presence of the commonly observed systematic influences on SET ratings was confirmed.CONCLUSIONS While the use of SET data may have originally been primarily for formative purposes to improve teaching and learning, they are also increasingly used for summative judgements of teaching quality and teaching staff performance that have implications for personnel decision making. There may be an acceptance of the need for SET, however there remains no universal consensus as to what constitutes quality in university teaching and learning, and the increasing use of SET for high-stakes decision making puts pressure on institutions to ensure that their SET practices are sound, equitable and defensible.

Understanding feedback report uptake: process evaluation findings from a 13-month feedback intervention in long-term care settings

BACKGROUND: Long-term care settings provide care to a large proportion of predominantly older, highly disabled adults across the United States and Canada. Managing and improving quality of care is challenging, in part because staffing is highly dependent on relatively non-professional health care aides and resources are limited. Feedback interventions in these settings are relatively rare, and there has been little published information about the process of feedback intervention. Our objectives were to describe the key components of uptake of the feedback reports, as well as other indicators of participant response to the intervention. METHODS: We conducted this project in nine long-term care units in four facilities in Edmonton, Canada. We used mixed methods, including observations during a 13-month feedback report intervention with nine post-feedback survey cycles, to conduct a process evaluation of a feedback report intervention in these units. We included all facility-based direct care providers (staff) in the feedback report distribution and survey administration. We conducted descriptive analyses of the data from observations and surveys, presenting this in tabular and graphic form. We constructed a short scale to measure uptake of the feedback reports. Our analysis evaluated feedback report uptake by provider type over the 13 months of the intervention. RESULTS: We received a total of 1,080 survey responses over the period of the intervention, which varied by type of provider, facility, and survey month. Total number of reports distributed ranged from 103 in cycle 12 to 229 in cycle 3, although the method of delivery varied widely across the period, from 12% to 65% delivered directly to individuals and 15% to 84% left for later distribution. The key elements of feedback uptake, including receiving, reading, understanding, discussing, and reporting a perception that the reports were useful, varied by survey cycle and provider type, as well as by facility. Uptake, as we measured it, was consistently high overall, but varied widely by provider type and time period. CONCLUSIONS: We report detailed process data describing the aspects of uptake of a feedback report during an intensive, longitudinal feedback intervention in long-term care facilities. Uptake is a complex process for which we used multiple measures. We demonstrate the feasibility of conducting a complex longitudinal feedback intervention in relatively resource-poor long-term care facilities to a wider range of provider types than have been included in prior feedback interventions.

Health services should collect feedback from inpatients at the point of service: opinions from patients and staff in acute and subacute facilities

OBJECTIVE: Point of service feedback (POSF) enables patients to give health services feedback about their experiences during or immediately after care. Despite the increasing use of POSF, little is known regarding patients' and staffs' opinions of this practice and whether they consider it acceptable or useful. The study aimed to determine patient and staff opinions regarding POSF. DESIGN: A cross-sectional survey. SETTING: Acute and subacute healthcare facilities. PARTICIPANTS: Two hundred and forty-seven patients and 221 staff. RESULTS: Participants indicated that patients should be invited to evaluate health services when they are in hospital or subacute care and improving services was the most important reason for doing so. Staff indicated that:• collecting patients' feedback during their stay was an important part of providing care and not an interruption to it (n = 187 of 221, 85%).• collecting patients' feedback was best done with a variety of methods; talking directly with patients during their stay was the preferred option (n = 161 of 219, 74%).More patients preferred to:• give feedback during their stay (51%) than after discharge from care (15%).• give feedback by talking with someone (45%) than completing a questionnaire (31%).Some patients (14%) were concerned about reprisal from staff if they gave negative feedback. CONCLUSIONS: POSF can be acceptable and useful for evaluating health services and should be incorporated into a person-centred approach that allows patients to choose from a variety of feedback options both during and after their stay. To be most useful, feedback should be incorporated into a quality improvement system.

Evaluation of student learning outcomes in fourth year engineering mechatronics through design based learning curriculum

This paper focuses on evaluation of student learning outcomes in fourth year engineering mechatronics through design based learning curriculum. The purpose of all engineering degrees is to provide strong grounding with principles of engineering science and technology. By learning engineering methods and approaches in an academic environment, graduates can enter the world of work and tackle real world problems with innovation and creativity. In many cases, academic staff are responsible for driving and setting high expectations in their classrooms. Sometimes staff are expected to teach subjects outside their expertise. This research paper is concerned with evaluating student learning outcomes through feedback sought from students on design-based learning approach.

A text analytics evaluation of a first-year engineering project-based unit

In the undergraduate engineering program at Griffith University in Australia, the unit 1006ENG Design and Professional Skills aims to provide an introduction to engineering design and professional practice through a project-based learning (PBL) approach to problem solving. It provides students with an experience of PBL in the first-year of their programme. The unit comprises an underpinning lecture series, design work including group project activities, an individual computer-aided drawing exercise/s and an oral presentation. Griffith University employs a ‘Student Experience of Course’ (SEC) online survey as part of its student evaluation of teaching, quality improvement and staff performance management processes. As well as numerical response scale items, it includes the following two questions inviting open-ended text responses from students: i) What did you find particularly good about this course? and ii) How could this course be improved? The collection of textual data in in student surveys is commonplace, due to the rich descriptions of respondent experiences they can provide at relatively low cost. However, historically these data have been underutilised because they are time consuming to analyse manually, and there has been a lack of automated tools to exploit such data efficiently. Text analytics approaches offer analysis methods that result in visual representations of comment data that highlight key individual themes in these data and the relationships between those themes. We present a text analytics-based evaluation of the SEC open-ended comments received in the first two years of offer of the PBL unit 1006ENG. We discuss the results obtained in detail. The method developed and documented here is a practical and useful approach to analysing/visualising open-ended comment data that could be applied by others with similar comment data sets.

Applying systems theory to the evaluation of a whole school approach to violence prevention

Issue addressed: Our Watch led a complex 12-month evaluation of a whole school approach to Respectful Relationships Education (RRE) implemented in 19 schools. RRE is an emerging field aimed at preventing gender-based violence. This paper will illustrate how from an implementation science perspective, the evaluation was a critical element in the change process at both a school and policy level. Methods: Using several conceptual approaches from systems science, the evaluation sought to examine how the multiple systems layers – student, teacher, school, community and government – interacted and influenced each other. A distinguishing feature of the evaluation included ‘feedback loops’; that is, evaluation data was provided to participants as it became available. Evaluation tools included a combination of standardised surveys (with pre- and post-intervention data provided to schools via individualised reports), reflection tools, regular reflection interviews and summative focus groups. Results: Data was shared during implementation with project staff, department staff and schools to support continuous improvement at these multiple systems levels. In complex settings, implementation can vary according to context; and the impact of evaluation processes, tools and findings differed across the schools. Interviews and focus groups conducted at the end of the project illustrated which of these methods were instrumental in motivating change and engaging stakeholders at both a school and departmental level and why. Conclusion: The evaluation methods were a critical component of the pilot’s approach, helping to shape implementation through data feedback loops and reflective practice for ongoing, responsive and continuous improvement. Future health promotion research on complex interventions needs to examine how the evaluation itself is influencing implementation. So what? The pilot has demonstrated that the evaluation, including feedback loops to inform project activity, were an asset to implementation. This has implications for other health promotion activities, where evaluation tools could be utilised to enhance, rather than simply measure, an intervention. The findings are relevant to a range of health promotion research activities because they demonstrate the importance of meta-evaluation techniques that seek to understand how the evaluation itself was influencing implementation and outcomes.