Reframing evaluation of learning and teaching: An approach to change

Evaluation practices in the Higher Education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology (QUT) began a project which aims to reframe the organisation’s approach to the evaluation of learning and teaching. This paper describes the existing evaluation system; the emergence and early development of the project; and formulation of a conceptual framework identifying key dimensions of evaluation. It then compares the draft framework with other conceptualisations and models of evaluation identified in the literature, to determine its validity and suitability for supporting QUT’s plans for the future. Overall, the paper represents a structured evaluation of the REFRAME project at a particular point in its lifecycle. Given that the project follows an evidence based, practice-led process and applies an ongoing action research cycle, the findings are presented in the belief that QUT’s experience is broadly applicable to other institutions which may be contemplating change in relation to evaluation of learning and teaching.

Aerosol mass spectrometric analysis of the chemical composition of non- refractory PM1 samples from school environments in Brisbane, Australia

Long term exposure to vehicle emissions has been associated with harmful health effects. Children are amongst the most susceptible group and schools represent an environment where they can experience significant exposure to vehicle emissions. However, there are limited studies on children’s exposure to vehicle emissions in schools. The aim of this study was to quantify the concentration of organic aerosol and in particular, vehicle emissions that children are exposed to during school hours. Therefore an Aerodyne compact time-of-flight aerosol mass spectrometer (TOF-AMS) was deployed at five urban schools in Brisbane, Australia. The TOF-AMS enabled the chemical composition of the non- refractory (NR-PM1) to be analysed with a high temporal resolution to assess the concentration of vehicle emissions and other organic aerosols during school hours. At each school the organic fraction comprised the majority of NR-PM1 with secondary organic aerosols as the main constitute. At two of the schools, a significant source of the organic aerosol (OA) was slightly aged vehicle emissions from nearby highways. More aged and oxidised OA was observed at the other three schools, which also recorded strong biomass burning influences. Primary emissions were found to dominate the OA at only one school which had an O:C ratio of 0.17, due to fuel powered gardening equipment used near the TOF-AMS. The diurnal cycle of OA concentration varied between schools and was found to be at a minimum during school hours. The major organic component that school children were exposed to during school hours was secondary OA. Peak exposure of school children to HOA occurred during school drop off and pick up times. Unless a school is located near major roads, children are exposed predominately to regional secondary OA as opposed to local emissions during schools hours in urban environments.

I-vector based speaker recognition using advanced channel compensation techniques

This paper investigates advanced channel compensation techniques for the purpose of improving i-vector speaker verification performance in the presence of high intersession variability using the NIST 2008 and 2010 SRE corpora. The performance of four channel compensation techniques: (a) weighted maximum margin criterion (WMMC), (b) source-normalized WMMC (SN-WMMC), (c) weighted linear discriminant analysis (WLDA), and; (d) source-normalized WLDA (SN-WLDA) have been investigated. We show that, by extracting the discriminatory information between pairs of speakers as well as capturing the source variation information in the development i-vector space, the SN-WLDA based cosine similarity scoring (CSS) i-vector system is shown to provide over 20% improvement in EER for NIST 2008 interview and microphone verification and over 10% improvement in EER for NIST 2008 telephone verification, when compared to SN-LDA based CSS i-vector system. Further, score-level fusion techniques are analyzed to combine the best channel compensation approaches, to provide over 8% improvement in DCF over the best single approach, (SN-WLDA), for NIST 2008 interview/ telephone enrolment-verification condition. Finally, we demonstrate that the improvements found in the context of CSS also generalize to state-of-the-art GPLDA with up to 14% relative improvement in EER for NIST SRE 2010 interview and microphone verification and over 7% relative improvement in EER for NIST SRE 2010 telephone verification.

Factors associated with suboptimal adherence to antiretroviral therapy in Viet Nam: A cross-sectional study using audio computer-assisted self-interview (ACASI)

Background: Optimal adherence to antiretroviral therapy (ART) is necessary for people living with HIV/AIDS (PLHIV). There have been relatively few systematic analyses of factors that promote or inhibit adherence to antiretroviral therapy among PLHIV in Asia. This study assessed ART adherence and examined factors associated with suboptimal adherence in northern Viet Nam. Methods: Data from 615 PLHIV on ART in two urban and three rural outpatient clinics were collected by medical record extraction and from patient interviews using audio computer-assisted self-interview (ACASI). Results: The prevalence of suboptimal adherence was estimated to be 24.9% via a visual analogue scale (VAS) of past-month dose-missing and 29.1% using a modified Adult AIDS Clinical Trial Group scale for on-time dose-taking in the past 4 days. Factors significantly associated with the more conservative VAS score were: depression (p < 0.001), side-effect experiences (p < 0.001), heavy alcohol use (p = 0.001), chance health locus of control (p = 0.003), low perceived quality of information from care providers (p = 0.04) and low social connectedness (p = 0.03). Illicit drug use alone was not significantly associated with suboptimal adherence, but interacted with heavy alcohol use to reduce adherence (p < 0.001). Conclusions: This is the largest survey of ART adherence yet reported from Asia and the first in a developing country to use the ACASI method in this context. The evidence strongly indicates that ART services in Viet Nam should include screening and treatment for depression, linkage with alcohol and/or drug dependence treatment, and counselling to address the belief that chance or luck determines health outcomes.

First measurements of source apportionment of organic aerosols in the Southern Hemisphere

An Aerodyne Aerosol Mass Spectrometer was deployed at five urban schools to examine spatial and temporal variability of organic aerosols (OA) and positive matrix factorization (PMF) used for the first time in the Southern Hemisphere to apportion the sources of the OA across an urban area. The sources identified included hydrocarbon-like OA (HOA), biomass burning OA (BBOA) and oxygenated OA (OOA). At all sites, the main source was OOA, which accounted for 62–73% of the total OA mass and was generally more oxidized compared to those reported in the Northern Hemisphere. This suggests that there are differences in aging processes or regional sources in the two hemispheres. Unlike HOA and BBOA, OOA demonstrated instructive temporal variations but not spatial variation across the urban area. Application of cluster analysis to the PMF-derived sources offered a simple and effective method for qualitative comparison of PMF sources that can be used in other studies.

Paper, prices and politics : an evaluation of the Swedish support to the Bai Bang project in Vietnam : a Sida evaluation report

Sweden’s protest against the Vietnam War was given tangible form in 1969 through the decision to give economic aid to the Government of North Vietnam. The main outcome was an integrated pulp and paper mill in the Vinh Phu Province north-west of Hanoi. Known as Bai Bang after its location, the mill became the most costly, one of the longest lasting and the most controversial project in the history of Swedish development cooperation. In 1996 Bai Bang produced at its full capacity. Today the mill is exclusively managed and staffed by the Vietnamese and there are plans for future expansion. At the same time a substantial amount of money has been spent to reach these achievements. Looking back at the cumbersome history of the project the results are against many’s expectations. To learn more about the conditions for sustainable development Sida commissioned two studies of the Bai Bang project. Together they touch upon several important issues in development cooperation over a period of almost 30 years: the change of aid paradigms over time, the role of foreign policy in development cooperation, cultural obstacles, recipient responsibility versus donor led development etc. The two studies were commissioned by Sida’s Department for Evaluation and Internal Audit which is an independent department reporting directly to Sida’s Board of Directors. One study assesses the financial and economic viability of the pulp and paper mill and the broader development impact of the project in Vietnam. It has been carried out by the Centre for International Economics, an Australian private economic research agency. The other study analyses the decision-making processes that created and shaped the project over a period of two decades, and reflects on lessons from the project for development cooperation in general. This study has been carried out by the Chr. Michelsen Institute, a Norweigan independent research institution.

Higher-order aberrations and anisometropia

Purpose/aim Myopia incidence is increasing around the world. Myopisation is considered to be caused by a variety of factors. One consideration is whether higher-order aberrations (HOA) influence myopisation. More knowledge of optics in anisometropic eyes might give further insight into the development of refractive error. Materials and methods To analyse the possible influence of HOA on refractive error development, we compared HOA between anisometropes and isometropes. We analysed HOA up to the 4th order for both eyes of 20 anisometropes (mean age: 43 ± 17 years) and 20 isometropes (mean age: 33 ±17 years). HOA were measured with the Shack-Hartman i.Profiler (Carl Zeiss, Germany) and were recalculated for a 4 mm pupil. Mean spherical equivalent (MSE) was based on the subjective refraction. Anisometropia was defined as ≥1D interocular difference in MSE. The mean absolute differences between right and left eyes in spherical equivalent were 0.28 ± 0.21 D in the isometropic group and 2.81 ± 2.04 D in the anisometropic group. Interocular differences in HOA were compared with the interocular difference in MSE using correlations. Results For isometropes oblique trefoil, vertical coma, horizontal coma and spherical aberration showed significant correlations between the two eyes. In anisometropes all analysed higher-order aberrations correlated significantly between the two eyes except oblique secondary astigmatism and secondary astigmatism. When analysing anisometropes and isometropes separately, no significant correlations were found between interocular differences of higher-order aberrations and MSE. For isometropes and anisometropes combined, tetrafoil correlated significantly with MSE in left eyes. Conclusions The present study could not show that interocular differences of higher-order aberrations increase with increasing interocular difference in MSE.

New strategy toward dioxin risk reduction for local residents surrounding severe dioxin hotspots in Vietnam

Background A public health intervention program with active involvement of local related stakeholders was piloted in the Bien Hoa dioxin hot spot (2007-2009), and then expanded to the Da Nang dioxin hot spot in Vietnam (2009-2011). It aimed to reduce the risk of dioxin exposure through foods for local residents. This article presents the results of the intervention in Da Nang. Methodology To assess the results of this intervention program, pre-intervention and post-intervention knowledge-attitude-practice (KAP) surveys were implemented in 400 households, randomly selected from four wards surrounding Da Nang Airbase in 2009 and 2011, respectively. Results After the intervention, the knowledge on the existence of dioxin in food, dioxin exposure pathways, potential high risk foods and preventive measures significantly increased (p < 0.05). 98% were willing to follow advice on preventing dioxin exposure. Practices to reduce the risk of dioxin exposure also statistical significantly improved (p<0.05). After intervention, 60.4% of households undertook exposure preventive measures, significantly higher than that of the pre-intervention survey (39.6%; χ2 =40.15 , P<0.001). High risk foods had quite low rates of daily consumption (from 0% to 2.5%) and were significantly reduced (p<0.05). Conclusions This is seen as an effective intervention strategy toward reducing the risk of human exposure to dioxin at dioxin hot spots. While greater efforts are needed for remediating dioxin polluted areas inside airbases, there is also evidence to suggest that, during the past four decades, pollution has been expanding to the surrounding areas. For this reason, this model should be quickly expanded to the remaining dioxin hot spots in Vietnam to further reduce the exposure risk in these areas.

Recovering from the earthquake : early childhood teachers and children collaboratively telling stories about their experiences

Purpose – Traumatic events can cause post-traumatic stress disorder due to the severity of the often unexpected events. The purpose of this paper is to reveal how conversations around lived experiences of traumatic events, such as the Christchurch earthquake in February 2011, can work as a strategy for people to come to terms with their experiences collaboratively. By encouraging young children to recall and tell of their earthquake stories with their early childhood teachers they can begin to respond, renew, and recover (Brown, 2012), and prevent or minimise more stress being developed. Design/methodology/approach – The study involved collecting data of the participating children taking turns to wear a wireless microphone where their interactions with each other and with teachers were video recorded over one week in November 2011. A total of eight hours and 21 minutes of footage was collected; four minutes and 19 seconds of that footage are presented and analysed in this paper. The footage was watched repeatedly and transcribed using conversation analysis methods (Sacks, 1995). Findings – Through analysing the detailed turn-taking utterances between teachers and children, the orderliness of the co-production of remembering is revealed to demonstrate that each member orients to being in agreement about what actually happened. These episodes of story telling between the teachers and children demonstrate how the teachers encourage the children to tell about their experiences through actively engaging in conversations with them about the earthquake. Originality/value – The conversation analysis approach used in this research was found to be useful in investigating aspects of disasters that the participants themselves remember as important and real. This approach offers a unique insight into understanding how the earthquake event was experienced and reflected on by young children and their teachers, and so can inform future policy and provision in post-disaster situations.

Knowledge, attitudes and practices to reduce exposure among residents living near Danang dioxin hot spot : 2.5 years after a preventative intervention

Da Nang Airbase in Viet Nam served as a bulk storage and supply facility for Agent Orange and other herbicides during Operation Ranch Hand 1961-1971[1]. Studies have shown that environmental and biological samples taken around the airbase site have elevated levels of dioxin [1-3]. Residents living in the vicinity of the airbase are at risk of exposure to dioxin in soil, water and mud and particularly through the consumption of local contaminated food. In 2009, a pre-intervention cross sectional survey was undertaken. This survey examined the knowledge, attitudes and practices (KAP) of householders living near Da Nang Airbase, relevent to reducing dioxin exposure through contaminated food. The results showed that despite living near a severe dioxin hot spot, the residents had very limited knowledge of both exposure risk and measures to reduce exposure to dioxin[4]. In response, the Vietnam Public Health Association (VPHA) and Da Nang Public Health Association implemented a risk reduction program at four residential wards in the vicinities of the Da Nang Airbase in 2010. A post intervention KAP survey was under taken in 2011, and the results showed that knowledge of the existence of dioxin in food, dioxin exposure pathways, potential high risk foods, and preventive measures was significantly enhanced. This new study monitored KAP 2.5 years after the intervention through a 2013 survey of food handlers from 400 households that were randomly selected from the four intervention wards. The results show that most of the positive outcomes remained stable or had increased; some KAP indicators decreased compared to those in the post-intervention survey, but were still significantly higher than the pre-intervention levels. In 2014, these findings will be incorporated with qualitative assessments and the results of laboratory analysis of dioxin concentrations in foods in Da Nang and Bien Hoa dioxin hot spots to comprehensively assess the sustained effects of the intervention.

Talk2Me : the art of augmenting conversations

This paper describes an interactive installation work set in a large dome space. The installation is an audio and physical re-rendition of an interactive writing work. In the original work, the user interacted via keyboard and screen while online. This rendition of the work retains the online interaction, but also places the interaction within a physical space, where the main 'conversation' takes place by the participant-audience speaking through microphones and listening through headphones. The work now also includes voice and SMS input, using speech-to-text and text-to-speech conversion technologies, and audio and displayed text for output. These additions allow the participant-audience to co-author the work while they participate in audible conversation with keyword-triggering characters (bots). Communication in the space can be person-to-computer via microphone, keyboard, and phone; person-to-person via machine and within the physical space; computer-to- computer; and computer-to-person via audio and projected text.

Reframe : QUT's evaluation framework survey business rules

Reframe is changing our approach to the evaluation of courses, units, teaching and student experience at QUT. We are moving away from a single survey tool to a richer, more holistic and customisable approach. This approach will help our academics design and deliver high-quality learning experiences, and review the impact of their teaching practice on student learning. Through it, we will also be able to provide more timely access to specialised support and meet external reporting requirements.

Protocols : Reframe : QUT's evaluation framework

Reframe is changing our approach to the evaluation of courses, units, teaching and student experience at QUT. We are moving away from a single survey tool to a richer, more holistic and customisable approach. These protocols allows academic staff and administrators access to the ways in which the policy is enacted through process.

Reframe at a glance

Reframe is changing our approach to the evaluation of courses, units, teaching and student experience at QUT. This graphic image represents the evaluation framework and its purpose in a single page.

Design of a garment for data collection of toddler language and physical activity

Design process phases of development, evaluation and implementation were used to create a garment to simultaneously collect reliable data of speech production and intensity of movement of toddlers (18-36 months). A series of prototypes were developed and evaluated that housed accelerometer-based motion sensors and a digital transmitter with microphone. The approved test garment was a top constructed from loop-faced fabric with interior pockets to house devices. Extended side panels allowed for sizing. In total, 56 toddlers (28 male; 28 female; 16-36 months of age) participated in the study providing pilot and baseline data. The test garment was effective in collecting data as evaluated for accuracy and reliability using ANOVA for accelerometer data, transcription of video for type of movement, and number and length of utterances for speech production. The data collection garment has been implemented in various studies across disciplines.