The inclusion of ‘then-test’ questions in post-test questionnaires alters post-test responses: a randomized study of bias in health program evaluation

Objectives Program evaluations are frequently based on ‘then-test’ data, i.e., pre-test collected in retrospect. While the application of the then-test has practical advantages, little is known about the validity of then-test data. Because of the collection of then-test in close proximity to post-test questions, this study was aimed at exploring whether the presence of then-test questions in post-test questionnaires influenced subjects’ responses to post-test.
Patients and methods To test the influence of then-test questions, we designed a randomized three-group study in the context of chronic disease self-management programs. Interventions had comparable goals and philosophies, and all 949 study participants filled out identical Health Education Impact Questionnaires (heiQ) at pre-test. At post-test, participants were then randomized to one of the following three groups: Group A responded to post-test questions only (n = 331); Group B filled out transition questions in addition to post-test (n = 304); and Group C filled out then-test questions in addition to post-test (n = 314).
Results Significant post-test differences were found in six of eight heiQ scales, with respondents who filled out then test questions reporting significantly higher post-test scores than respondents of the other groups.
Conclusions This study provides evidence that the inclusion of then-test questions alters post-test responses,
suggesting that change scores based on then-test data be interpreted with care.

Promoting active transport to school : evaluation of the Ride2School program

This study involved an evaluation of the Ride2School program, promoting active transport to school. The study found inconsistent evidence of a program impact, and identified the importance of addressing barriers to active transport among individual schools, parents and students; and building the capacity of schools to implement active transport initiatives.

Estimating the impact of mandatory folic acid fortification on the folic acid intake of Australian women of childbearing age

Objective: The primary aim of this study was to estimate the impact of mandatory folic acid (FA) fortification of bread-making flour on the FA intake of Australian women of childbearing age (16-44 years). The secondary objective was to investigate the relationship between estimated FA intake and socio-economic status (SES) and age.

Method: Dietary modelling was used to estimate FA intake under four mandatory fortification scenarios – no supplement use, supplement use unrelated to FA intake, supplement use only among the highest consumers of bread, and increased supplement use. Data were obtained from the 1995 National Nutrition Survey for food intake patterns, the 2007 Victorian Population Health Survey for FA supplement use, and a marketplace survey.

Results: It is estimated that the National Health and Medical Research Council (NHMRC) recommendation for an additional 400 mg/day FA will be achieved by a minimum of 3.9, 25.4, 21.7 and 30% of the target population under scenarios 1-4, respectively. The FA upper level of intake is exceeded by a maximum of 0.1, 1.7, 6.1 and 4.1% of the target population for scenarios 1-4, respectively.

Conclusions: Mandatory FA fortification is not sufficient for the NHMRC recommendations for minimum and maximum intakes to be met by all of the target population under a number of plausible behaviour scenarios.

Implications: Targeted nutrition education campaigns are needed for SES and age sub-groups and research of this nature should be extended to other population groups. Monitoring and evaluation of this policy will be important to ensure appropriate FA intake.

Growing community : the impact of the Stephanie Alexander kitchen garden program on the social and learning environment in primary schools

This article presents results from a mixed-method evaluation of a structured cooking and gardening program in Australian primary schools, focusing on program impacts on the social and learning environment of the school. In particular, we address the Stephanie Alexander Kitchen Garden Program objective of providing a pleasurable experience that has a positive impact on student engagement, social connections, and confidence within and beyond the school gates. Primary evidence for the research question came from qualitative data collected from students, parents, teachers, volunteers, school principals, and specialist staff through interviews, focus groups, and participant observations. This was supported by analyses of quantitative data on child quality of life, cooperative behaviors, teacher perceptions of the school environment, and school-level educational outcome and absenteeism data. Results showed that some of the program attributes valued most highly by study participants included increased student engagement and confidence, opportunities for experiential and integrated learning, teamwork, building social skills, and connections and links between schools and their communities. In this analysis, quantitative findings failed to support findings from the primary analysis. Limitations as well as benefits of a mixed-methods approach to evaluation of complex community interventions are discussed.

The CDM-Net project : the development, implementation and evaluation of a broadband-based network for managing chronic disease

Background. In Australia most chronic disease management is funded by Medicare Australia through General Practitioner Management Plans (GPMPs) and Team Care Arrangements (TCAs). Identified barriers may be reduced effectively using a broadband-based network known as the Chronic Disease Management Service (CDMS).

Aims. To measure the uptake and adherence to CDMS, test CDMS, and assess the adherence of health providers and patients to GPMPs and TCAs generated through CDMS.

Methods. A single cohort before and after study.

Results. GPMPs and TCAs increased. There was no change to prescribed medicines or psychological quality of life. Attendance at allied health professionals increased, but decreased at pharmacies. Overall satisfaction with CDMS was high among GPs, allied health professionals, and patients.

Conclusion. This study demonstrates proof of concept, but replication or continuation of the study is desirable to enable the impact of CDMS on diabetes outcomes to be determined.

Subsequence frequency measurement and its impact on knowledge discovery in single sequences

Subsequence frequency measurement is a basic and essential problem in knowledge discovery in single sequences. Frequency based knowledge discovery in single sequences tends to be unreliable since different resulting sets may be obtained from a same sequence when different frequency metrics are adopted. In this chapter, we investigate subsequence frequency measurement and its impact on the reliability of knowledge discovery in single sequences. We analyse seven previous frequency metrics, identify their inherent inaccuracies, and explore their impacts on two kinds of knowledge discovered from single sequences, frequent episodes and episode rules. We further give three suggestions for frequency metrics and introduce a new frequency metric in order to improve the reliability. Empirical evaluation reveals the inaccuracies and verifies our findings.

An evaluation of a national program to implement the cornell scale for depression in dementia into routine practice in aged care facilities

Background: Screening tools have been recommended for use in aged care to improve the detection and treatment of depression. This study aimed to evaluate the impact of a program for the routine implementation of the Cornell Scale for Depression in Dementia in Australian facilities, to determine whether use of the instrument by nurses led to further monitoring of depressive symptoms, medical referral, and changes in treatments prescribed for depression.
Methods: A file review was completed for 412 participants out of a total of 867 older people (47.5%) who resided in ten aged care facilities. The review examined Cornell Scale assessment data, medication charts, medical history, nursing progress notes, and resident care plans. Nursing staff who administered the Cornell Scale to each participant were also interviewed, and ten facilitymanagers took part in an interview to determine barriers to the effective implementation of the instrument.
Results: The Cornell Scale had been administered to 46.8% of the sample in the previous 12 months, with 25% of these participants scoring 9–13 and 27% scoring 14 and above. Less than one third of the residents with high scores were monitored by the staff following the assessment. Only 18% of residents with high scores were referred for further assessment of depression, while 10% received a treatment change.
Conclusions: The absence of a protocol for responding to high Cornell Scale scores limited the potential of this program to result in widespread improved treatment of depressed older people. The use of the Cornell Scale by aged care nurses with limited training raised concern.

Discovering the value that the library can contribute to quality teaching and learning through student evaluation data

Large surveys of library user service quality perception are common. However, student evaluation of teaching (SET) data often show a disparity between ratings of library service quality and library resource quality. In this situation, perhaps SET data can also provide insights into what contributes to the perception of library resource quality, and hence identify leverage points for quality improvement interventions. This paper documents an analysis of available Deakin University SET data relating to student interaction with, and evaluation of, library resources. It highlights significant correlations associated with library-related SET items, and from them infers actions that the library could undertake to improve the value and perception of the quality of library resources. The following results were observed. High ratings for library resources were likely to be associated with high general ratings of teaching and unit quality. Postgraduate coursework students rated library resources significantly higher than students in the first three years of undergraduate programs. Students in one faculty (Health) rated library resources significantly higher than students in all other faculties. There was a strong correlation observed in Australasian Survey of Student Engagement data for both 2009 and 2010 between the two items “Used library resources on campus or online” and “Worked on an essay or assignment that required integrating ideas or information from various sources”. These findings suggest the following conclusions. Well-planned learning environments are likely to integrate meaningful student interaction with the library. Initiatives to improve the value and perception of the quality of library resources should be focussed on the specific characteristics and needs of particular student cohorts to have maximum impact. More sophisticated assessment tasks that require students to interact with the library have the potential to result in higher student ratings of the value of library resources.

BreastScreen Australia evaluation : a review of methodological options for evaluating the effect of BreastScreen Australia on breast cancer mortality

This report provides evidence-based recommendations for appropriate and cost-effective methods that could be used to evaluate the impact of the national BreastScreen Australia population-based mammographic screening program on mortality from female breast cancer. The report represents a significant collaboration between the Australian Government, the National Breast Cancer Centre as well as Australian and international experts in mammography research and evaluation, epidemiology and health services research.

The recommendations are based on a review of national and international evidence on approaches used to assess the impact of mammography screening programs on breast cancer mortality in other settings. The review has used a systematic approach to assessing the strategic and methodological approaches taken in each of the studies identified and their potential limitations.

The national evaluation of the BreastScreen Australia Program aims to assess the appropriateness, efficiency and effectiveness of the BreastScreen Program. The completion of this report marks an important first step in that process. In addition, the review and recommendations in this report may have broader application at an international level.

Transforming an academic library's spaces : an evaluation study of Deakin University Library at Burwood campus using TEALS discovery

Libraries worldwide are transforming their spaces to better align with the changing needs of their communities. The aim of this paper is to outline the process and outcome of an evaluation study of transformed academic library spaces at the Melbourne Burwood Campus using TEALS. In light of changing higher education practices and students learning preferences, Deakin University has been questioning the balance of informal learning spaces and more formal teaching and academic spaces across its campuses. Commissioned by Deakin University Library, TEALS (Tool for the Evaluation of Academic Library Spaces) was developed to evaluate academic library spaces. The Melbourne Burwood Campus library has undergone several phases of refurbishment to create a library environment that is centred around students’ needs and that supports their individual and group learning experiences. In addition, areas of the library yet to be improved will undergo a major redevelopment over the next year. Given this, carrying out an evaluation of the current spaces is timely to ensure that a better understanding of the impact of changes is achieved. The evaluation process involved: a review of architectural plans and space briefing documents; an observational study of spaces; focus groups with students and library staff; and an online survey of Students’ Library Experience. Use of the TEALS space evaluation tool along with an analysis of data collected during the evaluation process have provided significant insights into various dimensions of the quality of new library spaces. The areas of weakness and strength identified in the study will inform the next phase of Deakin University Library space redevelopment.

Impact of climate change on thermal comfort and energy performance in offices - a parametric study

This paper investigates the impact of climate change on comfort and energy performance in offices in relation to the influence of building design and occupants. It focuses on a typical cellular office room in the context of Athens, Greece, as input for a parametric study using the building simulation software EnergyPlus. Three different building design variations are combined with two different occupant scenarios and 4 different weather data sets for IPCC climate change scenario A2.

For naturally ventilated buildings adaptive thermal comfort is evaluated according to ASHRAE Standard 55 and EN 15251. For mixed mode context evaluation is focused on greenhouse gas emissions and peak heating / cooling loads. Results indicate significant impact of the climate change on thermal comfort, and deviations between both comfort models. Comparing climate change, building design and occupant scenarios indicates that building design is the predominant influence on thermal comfort, whereas occupants are the predominant influence on greenhouse gas emissions.

A critical evaluation and comparison of two formative measures of system quality using criterion variables

Evaluating the validity of formative variables has presented ongoing challenges for researchers. In this paper we use global criterion measures to compare and critically evaluate two alternative formative measures of System Quality. One model is based on the ISO-9126 software quality standard, and the other is based on a leading information systems research model. We find that despite both models having a strong provenance, many of the items appear to be non-significant in our study. We examine the implications of this by evaluating the quality of the criterion variables we used, and the performance of PLS when evaluating formative models with a large number of items. We find that our respondents had difficulty distinguishing between global criterion variables measuring different aspects of overall System Quality. Also, because formative indicators “compete with one another” in PLS, it may be difficult to develop a set of measures which are all significant for a complex formative construct with a broad scope and a large number of items. Overall, we suggest that there is cautious evidence that both sets of measures are valid and largely equivalent, although questions still remain about the measures, the use of criterion variables, and the use of PLS for this type of model evaluation.

A preliminary evaluation of the use of graphic novels in teaching information systems

As a result of declining student numbers, and poor satisfaction ratings, an introductory information systems unit was completely redeveloped. A number of innovative initiatives were designed and implemented and the unit has seen a significant improvement in student satisfaction rates. This paper describes the development of a series of graphic novels as a strategy to both increase student engagement with the unit content and enhance the attainment of learning outcomes. It also discusses how the graphic novels were used in lectures, workshops, and competitions to increase student engagement (with varying degrees of success). Students have generally found the graphic novels to be a valuable learning resource. There is a paucity of studies on the impact of graphic novels as a teaching tool, particularly within the context of higher education, and this research is a contribution to this evident gap in the literature.

The impact of self-efficacy and perceived system efficacy on effectiveness of virtual training systems

This study developed and tested a research model which examined the impact of user perceptions of self-efficacy (SE) and virtual environment (VE) efficacy on the effectiveness of VE training systems. The model distinguishes between the perceptions of one’s own capability to perform trained tasks effectively and the perceptions of system performance, regarding the established parameters from literature. Specifically, the model posits that user perceptions will have positive effects on task performance and memory. Seventy-six adults participated in a VE in a controlled experiment, designed to empirically test the model. Each participant performed a series of object assembly tasks. The task involved selecting, rotating, releasing, inserting and manipulating 3D objects. Initially, the results of factor analysis demonstrated dimensionality of two user perception measures and produced a set of empirical validated factors underlining the VE efficacy. The results of regression analysis revealed that SE had a significant positive effect on perceived VE efficacy. No significant effects were found of perceptions on performance and memory. Furthermore, the study provided insights into the relationships between the perception measures and performance measures for assessing the efficacy of VE training systems. The study also addressed how well users learn, perform, adapt to and perceive the VE training, which provides valuable insight into the system efficacy.
Research and practical implications are presented at the end of the paper.

The impact of the quality of behaviour support plans on the use of restraint and seclusion in disability services

Background: Behaviour support plan (BSP) quality is known to be an important aspect of the support provided to people with disability who show challenging behaviours and has been found to lead to reductions in challenging behaviour. Preliminary evidence suggests that quality of plans is associated with reductions in the use of restraint
and seclusion.
Method and materials: Quality of a sample of behaviour support plans was assessed using the Behaviour Support Plan – Quality Evaluation II (BSP-QE II) Guide (Browning Wright, Saren & Mayer, 2003). Episodes of restraint and seclusion before and during implementation of plans were examined by comparing restraint and seclusion use between individuals with high quality versus low quality plans.
Results: Compared to individuals with low quality plans, individuals with high quality plans were found to be subjected to less restraint and seclusion over time, while those with low quality plans were subjected to increases. With the exception of one quality component, all components in high quality plans were found to be associated with decreases in restraint and seclusion use.
Conclusions: The results support earlier preliminary findings that quality of behaviour support plans are associated with reductions in restrictive intervention use. The findings are important because they suggest that increasing quality of behaviour support plans may lead to decreases in restraint and seclusion over time.