Objectifying the Subjective : Assessment and Feedback in Creative Arts Studio Learning and Teaching

This paper examines the Assessment and Feedback aspects of Studio Teaching as Creative Arts pedagogy. Prompted by USQ’s newly offered Bachelor of Creative Arts (BCA), the author has developed an Assessment Matrix specifically designed to satisfy a number of imperatives, including: • ‘objectifying’ the subjective aspects of creative practice as assessable coursework/research • providing the means by which accurate, detailed, personalised and confidential feedback may be provided to students individually • providing consistent, accurate, meaningful assessment records for student, lecturer, and institution • ensuring consistency, continuity, and transparency of assessment processes and records to satisfy quality audits • minimising marking and assessment time, whilst maximising assessment integrity and depth • requiring only basic level skills and knowledge of a computer application already in common use (Microsoft Excel) • adaptability to a range of creative courses ‐ across disciplines This Assessment Matrix has been in development (and trialled) since January 2009.

Perceptions of audio feedback in higher education assessment

The aim of this paper is to present results of research investigating the effectiveness of audio feedback in a third year undergraduate unit. While there is a large and growing body of literature about providing assessment feedback, there is little focussing on the use of audio media. This study employs a mixed method approach, involving semi-structured interviews with academic staff and a survey of students. Analysis of the interview data suggests that there are a number of issues surrounding acceptance of using audio feedback by lecturers. The next stage of the study is to examine the extent to which lecturers change their perceptions as they use audio feedback and to analyse the perceptions of the students (n=120), including the perceived importance of feedback, the ways in which they used the audio feedback and the extent to which they believe they control events that affect them. Ultimately, this study seeks to provide recommendations appropriate to the implementation of audio feedback in higher education.

The automation of radiotherapy treatment planning assessment

Aim The assessment of treatment plans is an important component in the education of radiation therapists. The establishment of a grade for a plan is currently based on subjective assessment of a range of criteria. The automation of assessment could provide a number of advantages including faster feedback, reduced chance of human error, and simpler aggregation of past results. Method A collection of treatments planned by a cohort of 27 second year radiation therapy students were selected for quantitative evaluation. Treatment sites included the bladder, cervix, larynx, parotid and prostate, although only the larynx plans had been assessed in detail. The plans were designed with the Pinnacle system and exported using the DICOM framework. Assessment criteria included beam arrangement optimisation, volume contouring, target dose coverage and homogeneity, and organ-at-risk sparing. The in-house Treatment and Dose Assessor (TADA) software1 was evaluated for suitability in assisting with the quantitative assessment of these plans. Dose volume data were exported in per-student and per-structure data tables, along with beam complexity metrics, dose volume histograms, and reports on naming conventions. Results The treatment plans were exported and processed using TADA, with the processing of all 27 plans for each treatment site taking less than two minutes. Naming conventions were successfully checked against a reference protocol. Significant variations between student plans were found. Correlation with assessment feedback was established for the larynx plans. Conclusion The data generated could be used to inform the selection of future assessment criteria, monitor student development, and provide useful feedback to the students. The provision of objective, quantitative evaluations of plan quality would be a valuable addition to not only radiotherapy education programmes but also for staff development and potentially credentialing methods. New functionality within TADA developed for this work could be applied clinically to, for example, evaluate protocol compliance.

'Take away from the dry sixties style marking': Lecturer and student perceptions and experiences of audio feedback

Providing audio feedback to assessment is relatively uncommon in higher education. However, published research suggests that it is preferred over written feedback by students but lecturers were less convinced. The aim of this paper is to examine further these findings in the context of a third year business ethics unit. Data was collected from two sources. The first is a series of in-depth, semi-structured interviews conducted with three lecturers providing audio feeback for the first time in Semester One 2011. The second source of data was drawn from the university student evaluation system. A total of 363 responses were used providing 'before' and 'after' perspectives about the effectiveness of audio feedback versus written feedback. Between 2005 and 2009 the survey data provided information about student attitudes to written assessment feedback (n=261). From 2010 onwards the data relates to audio (mp3) feedback (n=102). The analysis of he interview data indicated that introducing audio feedback should be done with care. The perception of the participating lecturers was mixed, ranging from sceptism to outright enthusiasm, but over time the overall approach became positive. It was found that particular attention needs to be paid to small (but important) technical details, and lecturers need to be convinced of its effectieness, especially that it is not necessarily more time consuming than providing written feedback. For students, the analysis revealed a clear preference for audio feedback. It is concluded that there is cause for concern and reason for optimism. It is a cause for concern because there is a possibility that scepticism on the part of academic staff seems to be based on assumptions about what students prefer and a concern about using the technology. There is reason for optimism because the evidence points towards students preferring audio feedback and as academic staff become more familiar with the technology the scepticism tends to evaporate. While this study is limited in scope, questions are raised about tackling negative staff perceptions of audio feedback that are worthy of further research.

Brief motivational interviewing for depression and anxiety

Motivational interviewing (MI)can be applied as a brief, low intensity (LI) intervention of 1-4 individualised sessions (typically 45-60 minutes in duration), including screening, assessment feedback, and psycho-education. MI is a client-centred, directive therapeutic style that enhances readiness for change by helping clients explore and resolve ambivalence (Miller and Rollnick 2002). A summary of the key components of brief MI interventions is provided in Table 16.1. There is a well-established evidence base for MI in the treatment of substance misuse (particularly alcohol misuse; Moyer et al. 2002), as well as a growing evidence for the use of MI in the treatment of other mental disorders (e.g. depression, PTSD, OCD), as well as suicidality and physical health problems (Hettema et al. 2005). Brief MI intervention can be delivered as a standalone treatment or as a motivational prelude to pharmacological and/or other psychological treatments (Hettema et al. 2005). MI has been used as an accompaniment to cognitive behavioural therapy (CBT) in the treatment of both depression and anxiety for resolving ambivalence about change and developing strategies for responding to resistance (e.g. treatment attendance, homework/medication compliance; Arkowitz et al. 2008a, 2008b). This chapter will describe how to apply brief MI interventions to the treatment of depression and anxiety as applied to the case of Megan (see Box 16.1) along with some of the challenges and potential solutions to applying MI in practice.

What is the Architecture Studio?

What is the Architecture Studio? What is the studio? Where do architect’s make architecture? What happens in architecture studios at QUT? How does QUT architecture provide assessment feedback? Why is QUT architecture Experimenting with Change?

The quik fix study : a randomised controlled trial of brief interventions for young people with alcohol-related injuries and illnesses accessing emergency department and crisis support care

Background: Alcohol is a major preventable cause of injury, disability and death in young people. Large numbers of young people with alcohol-related injuries and medical conditions present to hospital emergency departments (EDs). Access to brief, efficacious, accessible and cost effective treatment is an international health priority within this age group. While there is growing evidence for the efficacy of brief motivational interviewing (MI) for reducing alcohol use in young people, there is significant scope to increase its impact, and determine if it is the most efficacious and cost effective type of brief intervention available. The efficacy of personality-targeted interventions (PIs) for alcohol misuse delivered individually to young people is yet to be determined or compared to MI, despite growing evidence for school-based PIs. This study protocol describes a randomized controlled trial comparing the efficacy and cost-effectiveness of telephone-delivered MI, PI and an Assessment Feedback/Information (AF/I) only control for reducing alcohol use and related harm in young people. Methods/design: Participants will be 390 young people aged 16 to 25 years presenting to a crisis support service or ED with alcohol-related injuries and illnesses (including severe alcohol intoxication). This single blinded superiority trial randomized young people to (i) 2 sessions of MI; (ii) 2 sessions of a new PI or (iii) a 1 session AF/I only control. Participants are reassessed at 1, 3, 6 and 12 months on the primary outcomes of alcohol use and related problems and secondary outcomes of mental health symptoms, functioning, severity of problematic alcohol use, alcohol injuries, alcohol-related knowledge, coping self-efficacy to resist using alcohol, and cost effectiveness. Discussion: This study will identify the most efficacious and cost-effective telephone-delivered brief intervention for reducing alcohol misuse and related problems in young people presenting to crisis support services or EDs. We expect efficacy will be greatest for PI, followed by MI, and then AF/I at 1, 3, 6 and 12 months on the primary and secondary outcome variables. Telephone-delivered brief interventions could provide a youth-friendly, accessible, efficacious, cost-effective and easily disseminated treatment for addressing the significant public health issue of alcohol misuse and related harm in young people. Trial registration: This trial is registered with the Australian and New Zealand Clinical Trials Registry ACTRN12613000108718.

The use of multimodal technologies to enhance reflective writing in teacher education

For preservice teachers new to the teaching profession, reflective practice can be a difficult process. Yet reflective writing, once mastered, has the capacity to support preservice teachers to make connections between teaching theory and professional practice, and to start to take control of their own professional learning journey. The reflective practice described in this chapter was scaffolded through a framework for writing, the use of annotated work samples and explicit teaching. This approach was enhanced through multimodal resources including written peer assessment, audio teacher feedback and a video recording of the class presentation. The video footage assisted the preservice teachers to reconcile the feedback that they received from multiple sources. This chapter describes and analyses the implementation of the PRT Pattern (Prompting Reflection using Technology). Results of this practice revealed that the multiple forms of feedback assisted the preservice teachers to analyse their performance in terms of their developing professional identity and practice.

Grant application: QUT Learning and Teaching scheme 2012 Cracking the Code - A checklist to complement CRAs for first year Justice students.

Widening participation brings with it increasing diversity, increased variation in the level of academic preparedness (Clarke, 2011; Nelson, Clarke, & Kift 2010). Cultural capital coupled with negotiating the academic culture creates an environment based on many assumptions about academic writing and university culture. Variations in staff and student expectations relating to the teaching and learning experience is captured in a range of national and institutional data (AUSSE, CEQ, LEX). Nationally, AUSSE data (2009) indicates that communication, writing, speaking and analytic skills, staff expectations are quite a bit higher than students. The research team noted a recognisable shift in the changing cohort of students and their understanding and engagement with feedback and CRAs, as well as variations in teaching staff and student expectations. The current reality of tutor and student roles is that: - Students self select when/how they access lectures and tutorials. - Shorter tutorial times result in reduced opportunity to develop rapport with students. - CRAs are not always used consistently by staff (different marking styles and levels of feedback). - Marking is not always undertaken by the student’s tutor/lecturer. - Student support services might be recommended to students once a poor grade has been given. Students can perceive this as remedial and a further sense of failure. - CRA sheet has a mark /grade attached to it. Stigma attached to low mark. Hard to focus on the CRA feedback with a poor mark etched next to it. - Limited opportunities for sessionals to access professional development to assist with engaging students and feedback. - FYE resources exist, however academic time is a factor in exploring and embedding these resources. Feedback is another area with differing expectations and understandings. Sadler (2009) contends that students are not equipped to decode the statements properly. For students to be able to apply feedback, they need to understand the meaning of the feedback statement. They also need to identify, the particular aspects of their work that need attention. The proposed Checklist/guide would be one page and submitted with each assessment piece thereby providing an interface to engage students and tutors in managing first year understandings and expectations around CRAs, feedback, and academic practice.

Harnessing assessment and feedback in the first year to support learning success, engagement and retention

It has been argued that intentional first year curriculum design has a critical role to play in enhancing first year student engagement, success and retention (Kift, 2008). A fundamental first year curriculum objective should be to assist students to make the successful transition to assessment in higher education. Scott (2006) has identified that ‘relevant, consistent and integrated assessment … [with] prompt and constructive feedback’ are particularly relevant to student retention generally; while Nicol (2007) suggests that ‘lack of clarity regarding expectations in the first year, low levels of teacher feedback and poor motivation’ are key issues in the first year. At the very minimum, if we expect first year students to become independent and self-managing learners, they need to be supported in their early development and acquisition of tertiary assessment literacies (Orrell, 2005). Critical to this attainment is the necessity to alleviate early anxieties around assessment information, instructions, guidance, and performance. This includes, for example:  inducting students thoroughly into the academic languages and assessment genres they will encounter as the vehicles for evidencing learning success; and  making expectations about the quality of this evidence clear. Most importantly, students should receive regular formative feedback of their work early in their program of study to aid their learning and to provide information to both students and teachers on progress and achievement. Leveraging research conducted under an ALTC Senior Fellowship that has sought to articulate a research-based 'transition pedagogy' (Kift & Nelson, 2005) – a guiding philosophy for intentional first year curriculum design and support that carefully scaffolds and mediates the first year learning experience for contemporary heterogeneous cohorts – this paper will discuss theoretical and practical strategies and examples that should be of assistance in implementing good assessment and feedback practices across a range of disciplines in the first year.

Utility of routine data sources for feedback on the quality of cancer care: an assessment based on clinical practice guidelines

Background Not all cancer patients receive state-of-the-art care and providing regular feedback to clinicians might reduce this problem. The purpose of this study was to assess the utility of various data sources in providing feedback on the quality of cancer care. Methods Published clinical practice guidelines were used to obtain a list of processes-of-care of interest to clinicians. These were assigned to one of four data categories according to their availability and the marginal cost of using them for feedback. Results Only 8 (3%) of 243 processes-of-care could be measured using population-based registry or administrative inpatient data (lowest cost). A further 119 (49%) could be measured using a core clinical registry, which contains information on important prognostic factors (e.g., clinical stage, physiological reserve, hormone-receptor status). Another 88 (36%) required an expanded clinical registry or medical record review; mainly because they concerned long-term management of disease progression (recurrences and metastases) and 28 (11.5%) required patient interview or audio-taping of consultations because they involved information sharing between clinician and patient. Conclusion The advantages of population-based cancer registries and administrative inpatient data are wide coverage and low cost. The disadvantage is that they currently contain information on only a few processes-of-care. In most jurisdictions, clinical cancer registries, which can be used to report on many more processes-of-care, do not cover smaller hospitals. If we are to provide feedback about all patients, not just those in larger academic hospitals with the most developed data systems, then we need to develop sustainable population-based data systems that capture information on prognostic factors at the time of initial diagnosis and information on management of disease progression.

Closing the loop 21st century style : providing feedback on written assessment via MP3 recordings

Feedback on student performance, whether in the classroom or on written assignments, enables them to reflect on their understandings and restructure their thinking in order to develop more powerful ideas and capabilities. Research has identified a number of broad principles of good feedback practice. These include the provision of feedback that facilitates the development of reflection in learning; helps clarify what good performance is in terms of goals, criteria and expected standards; provides opportunities to close the gap between current and desired performance; delivers high quality information to students about their learning; and encourages positive motivational beliefs and self-esteem. However, high staff–student ratios and time pressures often result in a gulf between this ideal and reality. Whilst greater use of criteria referenced assessment has enabled an improvement in the extent of feedback being provided to students, this measure alone does not go far enough to satisfy the requirements of good feedback practice. Technology offers an effective and efficient means by which personalised feedback may be provided to students. This paper presents the findings of a trial of the use of the freely available Audacity program to provide individual feedback via MP3 recordings to final year Media Law students at the Queensland University of Technology on their written assignments. The trial has yielded wide acclaim by students as an effective means of explaining the exact reasons why they received the marks they were awarded, the things they did well and the areas needing improvement. It also showed that good feedback practice can be achieved without the burden of an increase in staff workload.

Improving criteria and feedback in student assessment in law

Spurred on by both the 1987 Pearce Report1 and the general changes to higher education spawned by the “Dawkins revolution” from 1988, there has been much critical self-evaluation leading to profound improvements to the quality of teaching in Australian law schools.2 Despite the changes there are still areas of general law teaching practice which have lagged behind recent developments in our understanding of what constitutes high quality teaching. One such area is assessment criteria and feedback. The project Improving Feedback in Student Assessment in Law is an attempt to remedy this. It aims to produce a manual containing key principles for the design of assessment and the provision of feedback, with practical yet flexible ideas and illustrations which law teachers may adopt or modify. Most of the examples have been developed by teachers at the University of Melbourne Law School. The project was supported in 1996 by a Committee for the Advancement of University Teaching grant and the manual will be published late in 1997.3 This note summarises the core principles which are elaborated further in the manual.