The content validity of the behaviour support plan quality evaluation tool (BSP-QEII) and its potential application in accommodation and day-support services for adults with intellectual disability

Background The quality of support provided to people with disability who show challenging behaviour could be influenced by the quality of the behaviour support plans (BSPs) on which staff rely for direction. This study investigated the content validity of the Behaviour Support Plan Quality Evaluation tool (BSP-QEII), originally developed to guide the development of BSPs for children in school settings, and evaluated its application for use in accommodation and day-support services for adults with intellectual disability.

Method A three-round Delphi study involving a purposive sample of experienced behaviour support practitioners (n = 30) was conducted over an 8-week period. The analyses included deductive content analysis and descriptive statistics.

Results The 12 quality domains of the BSP-QEII were affirmed as valid for application in adult accommodation and day-support service settings. Two additional quality domains were suggested, relating to the provision of detailed background on the client and the need for plans to reflect contemporary service philosophy. Furthermore, the results suggest that some issues previously identified in the literature as being important for inclusion in BSPs might not currently be a priority for practitioners. These included: the importance of specifying replacement or alternative behaviours to be taught, descriptions of teaching strategies to be used, reinforcers, and the specification of objective goals against which to evaluate the success of the intervention programme.

Conclusions The BSP-QEII provides a potentially useful framework to guide and evaluate the development of BSPs in services for adults with intellectual disability. Further research is warranted to investigate why practitioners are potentially giving greater attention to some areas of intervention practice than others, even where research has demonstrated these others areas of practice could be important to achieving quality outcomes.

Transforming an academic library's spaces : an evaluation study of Deakin University Library at Burwood campus using TEALS discovery

Libraries worldwide are transforming their spaces to better align with the changing needs of their communities. The aim of this paper is to outline the process and outcome of an evaluation study of transformed academic library spaces at the Melbourne Burwood Campus using TEALS. In light of changing higher education practices and students learning preferences, Deakin University has been questioning the balance of informal learning spaces and more formal teaching and academic spaces across its campuses. Commissioned by Deakin University Library, TEALS (Tool for the Evaluation of Academic Library Spaces) was developed to evaluate academic library spaces. The Melbourne Burwood Campus library has undergone several phases of refurbishment to create a library environment that is centred around students’ needs and that supports their individual and group learning experiences. In addition, areas of the library yet to be improved will undergo a major redevelopment over the next year. Given this, carrying out an evaluation of the current spaces is timely to ensure that a better understanding of the impact of changes is achieved. The evaluation process involved: a review of architectural plans and space briefing documents; an observational study of spaces; focus groups with students and library staff; and an online survey of Students’ Library Experience. Use of the TEALS space evaluation tool along with an analysis of data collected during the evaluation process have provided significant insights into various dimensions of the quality of new library spaces. The areas of weakness and strength identified in the study will inform the next phase of Deakin University Library space redevelopment.

Student evaluation of teaching : keeping in touch with reality

Student evaluation of teaching is commonplace in many universities and may be the predominant input into the performance evaluation of staff and organisational units. This article used publicly available student evaluation of teaching data to present examples of where institutional responses to evaluation processes appeared to be educationally ineffective and where the pursuit of the ‘right’ student evaluation results appears to have been mistakenly equated with the aim of improved teaching and learning. If the vast resources devoted to student evaluation of teaching are to be effective, then the data produced by student evaluation systems must lead to real and sustainable improvements in teaching quality and student learning, rather than becoming an end in itself.

Development of a tool for evaluation of academic library spaces (TEALS) (0242)

The paper provides a brief description of the tool for evaluating the quality and utilisation of academic library spaces (TEALS). Supported by Deakin University Library, TEALS has been developed out of a research project in the School of Architecture and Building, Deakin University, Geelong Waterfront Campus. The tool is intended to establish the setting for evaluation of physical spaces at different phases of development of new academic library spaces and refurbishment of existing ones as well as throughout the life of buildings. The methodological framework of the tool consists of four key elements; establishing Criteria of Quality (CoQ), determining Quality Indicators, evaluating library spaces against QIs and interpreting results for future improvements. The characteristics that distinguish TEALS from existing evaluation models include adopting an approach that focus on people (students, faculty and library staff), acting as a “reflective” and “empowering” tool and being user-friendly, quick and easy to use.

A combined impact-process evaluation of a program promoting active transport to school: understanding the factors that shaped program effectiveness

This mixed methods study was a comprehensive impact-process evaluation of the Ride2School program in metropolitan and regional areas in Victoria, Australia. The program aimed to promote transport to school for primary school children. Qualitative and quantitative data were collected at baseline and followup from two primary schools involved in the pilot phase of the program and two matched comparison schools, and a further 13 primary schools that participated in the implementation phase of the program. Classroom surveys, structured and unstructured observations, and interviews with Ride2School program staff were used to evaluate the pilot program. For the 13 schools in the second phase of the program, parents and students completed questionnaires at baseline (N = 889) and followup (N = 761). Based on the quantitative data, there was little evidence of an overall increase in active transport to school across participating schools, although impacts varied among individual schools. Qualitative data in the form of observations, interviews, and focus group discussions with students, school staff, and program staff provided insight into the reasons for variable program impacts. This paper highlights the benefits of undertaking a mixed methods approach to evaluating active transport to school programs that enables both measurement and understanding of program impacts.

Comparative evaluation of PBL and traditional lecturebased teaching in undergraduate engineering courses : evidence from controlled learning environment

Project and problem-based learning (PBL) has been widely recognised as an active, collaborative, cumulative and integrative learning approach that engages learners, motivates team creativity and centres on practical education. On the other hand, traditional lecture-tutorial teaching is often criticised for being a passive, surface learning and exam-focused approach. In spite of these evidence-based observations and claims over the years, the traditional lecture-tutorial teaching approach still dominates as the preferred teaching approach at Australian universities. This study sets up a control environment to compare these two teaching and learning approaches by analysing data from students' actual performance, course evaluation and expectation in two large undergraduate engineering courses in 2009 and 2010. The evidence reported in this study is broadly interesting in that both courses were taught by the same teaching staff using two entirely different learning and teaching approaches to the same cohort of students in the same semester within the same degree program. The analysis shows that there are significant differences between the students' actual performance, course evaluation and their expectation. Such conflicting differences may be some of the reasons that may negatively impact teaching staff deterring them from switching to PBL from traditional lecture-tutorial teaching.

Cross-cultural learning and international collaboration: evaluation of a study tour to Thailand

Purpose:
To evaluate cross-cultural learning among Thai staff and host students from the Faculty of Nursing, Mahidol University (MU) and Australian guest students from the School of Nursing and Midwifery, Deakin University (DU), who participated in a study tour.

Design:
Descriptive exploratory evaluation.

Methods:
Key stakeholders were invited to participate resulting in a convenience sample of seven MU staff, five MU and 22 DU students. Data were collected using mixed methods. Qualitative data were theme analysed and quantitative data were analysed using descriptive statistics.

Main findings:
The semi-structured interviews with MU staff, focus group with MU students and free response questions in the online survey with DU students indicated the themes of enhanced and valuable cross-cultural learning and relationship building, the challenges of different social behaviours and the importance of tolerance and acceptance. In the online survey, over 77% (n = 17) of DU students reported high satisfaction with their cross-cultural learning on the study tour. The online survey included the validated Miville-Guzman Universality-Diversity scale short form (M-GUD-S). All Australian students reported seeking diversity of contact (X ± SD = 23.1 ± 4.4), relativistic appreciation (X ± SD = 24.7 ± 3.9), and comfort with differences (X ± SD = 26.2 ± 3.0), indicating high levels of openness to cultural diversity and similarity on the M-GUD-S. 

Conclusion and recommendations:
This study provides an example of an evaluated study tour emphasising cross-cultural relationship building. Findings indicate that nursing education should include opportunities for intercultural exchange among nursing students. Nurses require excellent skills in cross-cultural nursing and relating to meet the future global challenges to health care over the next millennium.

More than a game: evaluation report

‘More than a Game’ is a sport-based youth mentoring program developed and implemented by Western Bulldogs in partnership with Newport Islamic Society (NIS), the Australian Federal Police, Victoria Police and Hobsons Bay City Council, with funding from the Attorney General’s Department Building Community Resilience (BCR) grant scheme. The program aimed to develop a community-based resilience model that would use team-based sports to address issues of identity, sense of belonging and cultural isolation amongst young men of Islamic faith, all of which are identified as factors that may promote forms of violent extremism. The program involved 60 young men, aged 15-25, from the Newport Islamic Society in Melbourne’s Western suburbs. The boys were engaged in numerous activities where they were mentored by staff from Western Bulldogs, Victoria Police and Australian Federal Police, who worked in conjunction with community leaders from the Newport Islamic Society.

Through sports-based training, mentoring programs, and community dialogue, ‘More Than a Game’ aimed to develop participants’ leadership, communication, and cross-cultural engagement skills; to identify and facilitate the development of young role models in the community; to enhance greater understanding of the Muslim community in Melbourne’s West, and to foster greater intercultural contact and understanding between participants and other cultural groups. A number of activities were developed and implemented as a part of the program

Evaluation of building performance for strategic facilities management in healthcare : a case study of a public hospital in Australia

Purpose - This paper evaluates the main elements of building performance namely building function, building impact and building quality in order to promote strategic facilities management in healthcare organisation to improve core (health) business activities. Design/methodology/approach - Based on current available toolkits, a questionnaire is issued to healthcare users (staff) in a public hospital about their level of agreement in relation to these elements. Statistical analysis is conducted to regroup the elements. These regrouped elements and their inter relationships are used to develop a framework for measuring building performance in healthcare buildings. Findings - The analysis helped to clarify the understanding and agreement of users in Australian healthcare organisation with regards to building performance. Based on the survey results, 11 new elements were regrouped into three groups. These new regrouped elements will be used to develop a reliable framework for measuring performance of Australian healthcare buildings. Originality/value - Currently there is no building performance toolkit available for Australian healthcare organisation. The framework developed in this paper will help healthcare organisations with a reliable performance tool for their buildings and this will promote strategic facilities management.

Formative evaluation of baby makes 3

Baby Makes 3 was originally developed by Whitehorse Community Health Service and the City of Whitehorse as a program which seeks to prevent violence by promoting respect and equality between couples who have recently become parents for the first time. This is a significant but often stressful event in the life of families and it is not uncommon for couples to adopt ways of relating which reflect gendered norms and foster inequality.
By being offered as a program to all first-time parents by maternal and child health staff, the program in Whitehorse managed to engage with couples at this critical time, many of whom found the program enabled them to adopt greater equality in their relationships. As such, the initial Baby Makes 3 program has been found to be an effective and cost-efficient violence prevention strategy (Flynn, 2011).
The introduction of Baby Makes 3 to the Great South Coast Region is the first time this program has been implemented in a non-metropolitan setting. In addition to the original program aims, a number of additional aims and activities have been added, with the new project being known as Baby Makes 3 Plus.
Programs developed in urban contexts often face different challenges when implemented in rural and regional settings. Likewise, the factors for facilitating success may vary from those in an urban setting (Maidment and Bay, 2012). Hence, this formative evaluation aims to provide feedback on the initial implementation of the original Baby Makes 3 program in the Great South Coast Region with the expectation this will enable problems with implementation and/or opportunities for further enhancement of outcomes to be identified and appropriate action taken.

An evaluation of the impact of using authentic design and build industry projects in project-based learning

BACKGROUNDChisholm’s ‘first year experience’ is a significant feature of the new industry focused Bachelor of Engineering Technology program delivered in association with the South East Melbourne Manufacturers’ Alliance (SEMMA). This conceive-design-implement-operate (CDIO Initiative) program commenced as a full time program in first semester 2012. Whereas it is common for CDIO Initiative programs to have a first year experience program containing a project typical of the type of industry project they would complete as a graduate engineer or engineering technologist, this goes further by using real industry projects provided by SEMMA members.This design-and-build industry project runs across both semesters supporting project-based learning in three first year subjects. A concern is that the industry involvement of the projects adds substantially to an already heavy student workload. This has been further increased by the addition of two additional first year initiatives: writing workshops, and training in, and substantial use of, student oral presentations. It is recognised that an excessive workload could lead students to adopt surface learning approaches in other subjects.PURPOSEThe goal of the project is to evaluate student perceptions of the value and work load impact of the industry project and the other new first year initiatives.DESIGN/METHODCentral to this project is a student survey-based evaluation of the industry project based learning that is the core of the ‘first year experience’. The participants were limited to the small group of students who, in a single year, completed all three subjects that comprise the ‘first year experience’. To avoid compromising the results the survey was administered by Chisholm Institute’s Department of Strategy and Planning with no engineering technology degree program staff present. The survey included questions to enable responses to be linked with specific student demographics without identifying any of the respondents.RESULTSThe study showed the industry project-based learning had worthwhile outcomes but placed considerable time pressures on most respondents. For some, this also impacted on their other subjects. A first year oral presentation program was also shown to have worthwhile outcomes. However no conclusions could be reliably drawn on the third initiative – writing workshops.CONCLUSIONSThe results confirm that the authentic industry project is considered a worthwhile initiative but contributes significantly to student overload. This applies also – to a lesser extent – to the first year oral presentation program. Both also require new approaches to delivery as student numbers increase. Strategies to address these issues are discussed.

The impact of increasing course enrolment on student evaluation of teaching in engineering education

Student evaluation of teaching (SET) is important, commonplace and may be used in staff performance management. The SET literature suggests that class size is a negative systematic influence on SET ratings. In this paper we investigate time-series SET data from a large first-year engineering class where a decline in SET ratings was observed over time as course enrolment increased. We observe a negative halo effect of increasing class size on mean SET ratings and conclude that increasing course enrolment leads to a significant reduction in all mean SET ratings, even when the course learning design remains essentially unchanged. We also find an additional differential effect of increasing course enrolment on mean SET ratings. We observe that the marginal reduction in mean SET ratings for each additional student in the course enrolment is greater for those aspects of the student learning experience that are likely to be most directly impacted by increasing class size. We provide implications for practice from these findings.

The systematic influences on student evaluation of teaching in engineering education

BACKGROUND Student evaluation of teaching (SET) has a long history, has grown in prevalence and importance over a period of decades, and is now common-place in many universities internationally. SET data are collected for a range of purposes, including: as diagnostic feedback to improve the quality of teaching and learning; as an input to staff performance management processes and personnel decisions such as promotion for staff; to provide information to prospective students in their selection of courses and programs; and as a source of data for research on teaching. Rovai et al. (2006) report that while SET research provides mixed results, there is evidence that, for course-related factors, smaller classes are rated more favourably than large classes, upper-year-level classes are rated more favourably than lower-year classes, and that there are rating differences between discipline areas. While additional course-related factors are also noted, other reviews of the literature on SET also identify these three factors as commonly reported systematic influences on SET ratings. The School of Engineering at Deakin University in Australia offers undergraduate and postgraduate engineering programs, and these programs are delivered in both on-campus and off-campus modes.PURPOSEThe paper presents a quantitative investigation of SET data for the School of Engineering at Deakin University to identify whether the commonly reported systematic influences on SET ratings of class size and year level are also observed here. The influence of online mode of offer is also explored.DESIGN/METHOD Deakin University’s Student Evaluation of Teaching and Units (SETU) questionnaire is administered to students enrolled in every unit of study every time that unit is offered, unless it is specially exempted. Following data collation, summary results are reported via a public website. The publicly available SETU data for all School of Engineering units of study were collected for a two year period. The collected data were subjected to analysis of variance (ANOVA) analysis to identify any significant systematic influences on mean student SETU ratings.RESULTS SETU data from 100 separate units of study over the two year period were collected, representing 3375 sets of SETU ratings, and covering unit enrolment sizes from 12 to 462 students. Although this was a modest sized investigation, significantly higher mean ratings for some SETU items were observed for units with small enrolments, for postgraduate level units compared to undergraduate level units, and for units offered in conventional mode compared to online mode of offer. The presence of the commonly observed systematic influences on SET ratings was confirmed.CONCLUSIONS While the use of SET data may have originally been primarily for formative purposes to improve teaching and learning, they are also increasingly used for summative judgements of teaching quality and teaching staff performance that have implications for personnel decision making. There may be an acceptance of the need for SET, however there remains no universal consensus as to what constitutes quality in university teaching and learning, and the increasing use of SET for high-stakes decision making puts pressure on institutions to ensure that their SET practices are sound, equitable and defensible.

Understanding feedback report uptake: process evaluation findings from a 13-month feedback intervention in long-term care settings

BACKGROUND: Long-term care settings provide care to a large proportion of predominantly older, highly disabled adults across the United States and Canada. Managing and improving quality of care is challenging, in part because staffing is highly dependent on relatively non-professional health care aides and resources are limited. Feedback interventions in these settings are relatively rare, and there has been little published information about the process of feedback intervention. Our objectives were to describe the key components of uptake of the feedback reports, as well as other indicators of participant response to the intervention. METHODS: We conducted this project in nine long-term care units in four facilities in Edmonton, Canada. We used mixed methods, including observations during a 13-month feedback report intervention with nine post-feedback survey cycles, to conduct a process evaluation of a feedback report intervention in these units. We included all facility-based direct care providers (staff) in the feedback report distribution and survey administration. We conducted descriptive analyses of the data from observations and surveys, presenting this in tabular and graphic form. We constructed a short scale to measure uptake of the feedback reports. Our analysis evaluated feedback report uptake by provider type over the 13 months of the intervention. RESULTS: We received a total of 1,080 survey responses over the period of the intervention, which varied by type of provider, facility, and survey month. Total number of reports distributed ranged from 103 in cycle 12 to 229 in cycle 3, although the method of delivery varied widely across the period, from 12% to 65% delivered directly to individuals and 15% to 84% left for later distribution. The key elements of feedback uptake, including receiving, reading, understanding, discussing, and reporting a perception that the reports were useful, varied by survey cycle and provider type, as well as by facility. Uptake, as we measured it, was consistently high overall, but varied widely by provider type and time period. CONCLUSIONS: We report detailed process data describing the aspects of uptake of a feedback report during an intensive, longitudinal feedback intervention in long-term care facilities. Uptake is a complex process for which we used multiple measures. We demonstrate the feasibility of conducting a complex longitudinal feedback intervention in relatively resource-poor long-term care facilities to a wider range of provider types than have been included in prior feedback interventions.