A model for enhancing assessment and teaching practice at the coalface : Insights from a Fellow-In-Residence engagement program

Australian universities are currently engaging with new governmental policies and regulations that require them to demonstrate enhanced quality and accountability in teaching and research. The development of national academic standards for learning outcomes in higher education is one such instance of this drive for excellence. These discipline-specific standards articulate the minimum, or Threshold Learning Outcomes, to be addressed by higher education institutions so that graduating students can demonstrate their achievement to their institutions, accreditation agencies, and industry recruiters. This impacts not only on the design of Engineering courses (with particular emphasis on pedagogy and assessment), but also on the preparation of academics to engage with these standards and implement them in their day-to-day teaching practice on a micro level. This imperative for enhanced quality and accountability in teaching is also significant at a meso level, for according to the Australian Bureau of Statistics, about 25 per cent of teachers in Australian universities are aged 55 and above and more than 54 per cent are aged 45 and above (ABS, 2006). A number of institutions have undertaken recruitment drives to regenerate and enrich their academic workforce by appointing capacity-building research professors and increasing the numbers of early- and mid-career academics. This nationally driven agenda for quality and accountability in teaching permeates also the micro level of engineering education, since the demand for enhanced academic standards and learning outcomes requires both a strong advocacy for a shift to an authentic, collaborative, outcomes-focused education and the mechanisms to support academics in transforming their professional thinking and practice. Outcomes-focused education means giving greater attention to the ways in which the curriculum design, pedagogy, assessment approaches and teaching activities can most effectively make a positive, verifiable difference to students’ learning. Such education is authentic when it is couched firmly in the realities of learning environments, student and academic staff characteristics, and trustworthy educational research. That education will be richer and more efficient when staff works collaboratively, contributing their knowledge, experience and skills to achieve learning outcomes based on agreed objectives. We know that the school or departmental levels of universities are the most effective loci of changes in approaches to teaching and learning practices in higher education (Knight & Trowler, 2000). Heads of Schools are being increasingly entrusted with more responsibilities - in addition to setting strategic directions and managing the operational and sometimes financial aspects of their school, they are also expected to lead the development and delivery of the teaching, research and other academic activities. Guiding and mentoring individuals and groups of academics is one critical aspect of the Head of School’s role. Yet they do not always have the resources or support to help them mentor staff, especially the more junior academics. In summary, the international trend in undergraduate engineering course accreditation towards the demonstration of attainment of graduate attributes poses new challenges in addressing academic staff development needs and the assessment of learning. This paper will give some insights into the conceptual design, implementation and empirical effectiveness to date, of a Fellow-In-Residence Engagement (FIRE) program. The program is proposed as a model for achieving better engagement of academics with contemporary issues and effectively enhancing their teaching and assessment practices. It will also report on the program’s collaborative approach to working with Heads of Schools to better support academics, especially early-career ones, by utilizing formal and informal mentoring. Further, the paper will discuss possible factors that may assist the achievement of the intended outcomes of such a model, and will examine its contributions to engendering an outcomes-focussed thinking in engineering education.

Higher order thinking in chemistry curriculum and its assessment

Higher-order thinking has featured persistently in the reform agenda for science education. The intended curriculum in various countries sets out aspirational statements for the levels of higher-order thinking to be attained by students. This study reports the extent to which chemistry examinations from four Australian states align and facilitate the intended higher-order thinking skills stipulated in curriculum documents. Through content analysis, the curriculum goals were identified for each state and compared to the nature of question items in the corresponding examinations. Categories of higher-order thinking were adapted from the OECD’s PISA Science test to analyze question items. There was considerable variation in the extent to which the examinations from the states supported the curriculum intent of developing and assessing higher-order thinking. Generally, examinations that used a marks-based system tended to emphasize lower-order thinking, with a greater distribution of marks allocated for lower-order thinking questions. Examinations associated with a criterion-referenced examination tended to award greater credit for higher-order thinking questions. The level of complexity of chemistry was another factor that limited the extent to which examination questions supported higher-order thinking. Implications from these findings are drawn for the authorities responsible for designing curriculum and assessment procedures and for teachers.

Fresh frozen cadavers as a tool to educate student paramedics in procedural skills

Objectives – To describe the development of an educational workshop to develop procedural skills in undergraduate Paramedic students using fresh frozen cadavers and to report the student’s assessment of the program. Methods – A six-hour anatomy based workshop was developed using fresh frozen cadavers to teach a range of airway and invasive procedural skills to second year undergraduate paramedic students. Embedded QUAN (qual) methodology will be utilised to evaluate the student’s satisfaction, perception and quality of teaching as compared to other existing clinical teaching techniques such as high fidelity simulation. Students will be asked to complete an anonymous validated survey (10 questions formulated on a 5 point Likert scale) and provide a qualitative feedback pre and post the six-hour workshop. Results – This is a prospective study planned for September 2013. Low-risk human research ethics are being sought. Teaching evaluation results from the inaugural 2012 workshop (undergraduate and postgraduate Paramedic students) and interim results for 2013 will be presented. Conclusions – Clinical teaching using fresh frozen cadavers thus far has predominately been used in the education of medical and surgical trainees. A number of studies have found them to be effective and in some cases superior to traditional high fidelity simulation teaching strategies. Fresh frozen cadavers are said to provide perfect anatomy, normal tissue consistency and a realistic operative training experience (Lloyd, Maxwell-Armstrong et al. 2011). The authors believe that this study will show that the use of fresh frozen cadavers offers a safe and effective mode to teach procedural skills to student paramedics that will help bridge the skills gap and increase confidence prior to students undertaking such interventions on living patients. A modified training program may be formulated for general practitioners undertaking Emergency Medicine Advanced Rural Skills.

The effects of standardised assessment (NAPLAN) on teacher pedagogy at two Queensland schools

The purpose of this qualitative interpretative case study was to explore how the National Assessment Program – Literacy and Numeracy (NAPLAN) requirements may be affecting pedagogies of two Year 3, Year 5 and Year 7 teachers at two Queensland schools. The perceived problem was that standardised assessment NAPLAN practices and its growing status as a key measure of education quality throughout Australia has the potential to limit the everyday literacy and numeracy practices of teachers to instructional methods primarily focused on teaching to the test. The findings demonstrate how increased explicit teaching of NAPLAN content and procedural knowledge prior to testing has the potential to negatively impact on the teaching of everyday literacy and numeracy skills and knowledge that extend beyond those concerned with NAPLAN. Such teaching limited opportunity for what teachers reported as valued collaborative learning contexts aiming for long-term literacy and numeracy results.

"Ruby's Music Festival" : developing problem solving skills using online scenarios and creating opportunities to feed forward

One of the characteristics of good teaching is giving the highest quality feedback on student work but the term “feedback” is most commonly associated with summative assessment given by a teacher after work is completed. The student can often be a passive participant in the process. This article looks at the implementation of web based scenarios completed by students prior to summative assessment with the objective of improving legal problem solving skills. It examines the design process and the implementation of the problem solving activity and the approach to teaching and learning taken in the new law unit of which it is part. We argue that such activities are effective tools to feed forward and reflect on the implications for the effective teaching of law in higher education.

Could MOOCs answer the problems of teaching AQF required skills in Australian tertiary programmes?

From 2015, Australian universities will be required to demonstrate that their programmes explicitly teach, and assess achievement of, knowledge and skills and the application of both as specified by the Australian Qualifications Framework (AQF). Over the last twenty years, the sector has applied significant effort and resource to embedding the development of skills through tertiary programmes. Despite these national and institutional efforts, employer and industry concerns remain about the quality of graduate skills. The authors propose a ‘massive open online course’ (MOOC) approach to teaching and assessing AQF required skills. As an example the paper identifies the skills modules that would need to be developed by experts in each skill area for AQF level 9 master’s by coursework programmes. The proposed MOOC would include assessment tasks and rubrics allowing students to develop and demonstrate achievement of the AQF required skills. The assessment tasks could be used by institutions to provide evidence of attainment of coursework master’s standards.

Objectifying the Subjective : Assessment and Feedback in Creative Arts Studio Learning and Teaching

This paper examines the Assessment and Feedback aspects of Studio Teaching as Creative Arts pedagogy. Prompted by USQ’s newly offered Bachelor of Creative Arts (BCA), the author has developed an Assessment Matrix specifically designed to satisfy a number of imperatives, including: • ‘objectifying’ the subjective aspects of creative practice as assessable coursework/research • providing the means by which accurate, detailed, personalised and confidential feedback may be provided to students individually • providing consistent, accurate, meaningful assessment records for student, lecturer, and institution • ensuring consistency, continuity, and transparency of assessment processes and records to satisfy quality audits • minimising marking and assessment time, whilst maximising assessment integrity and depth • requiring only basic level skills and knowledge of a computer application already in common use (Microsoft Excel) • adaptability to a range of creative courses ‐ across disciplines This Assessment Matrix has been in development (and trialled) since January 2009.

A shared reading intervention with parents to enhance young children’s early literacy skills

Parental reading to children from an early age has been shown to enhance children’s emergent literacy skills. A pragmatic randomized controlled trial (RCT) was used to investigate the effects of two forms of shared reading interventions on children’s language and literacy skills. Parents of 80 preparatory year children from outer suburban schools of an Australian metropolitan city were trained to use shared reading strategies in an eight-week home intervention. Families were assigned to one of three groups: Dialogic Reading (DR), Dialogic Reading with the addition of Print Referencing (DR + PR), or an attention-matched control group. The sample comprised 42 boys and 38 girls ranging in age from 4.9 years to 6.3 years (M = 5.5, SD = 0.3). Data were collected at pre, post, and at three months follow-up. Measures assessed children’s oral language (receptive and expressive vocabulary), phonological awareness (rhyme, word completion), alphabet knowledge, and concepts about print. Analyses of change from pre to post showed significant effects for the DR and DR + PR groups compared to the control group on three of the six measures: expressive language, rhyme, and concepts about print. At 3-month follow-up assessment, the two intervention groups maintained significantly better performance on the measure of concepts of print only. At both time points, there were no group differences between the DR and DR+PR conditions. These findings illustrate the potential of a brief home focused intervention on promoting children’s emergent literacy.

Assessment Understood as Enabling

This chapter outlines a perspective of educational assessment as enabling, whereby the learner is central and assessment is focused on supporting the knowledge, skills and dispositions necessary for lifelong learning. It argues that better education for young people is achievable when educational policy and practice give priority to learning improvement, thereby making assessment for accountability a related, though secondary, concern. The chapter describes how this work of internationally recognized scholars brings together diverse perspectives and theoretical frameworks and, in so doing, provides readers with a range of ways to consider their pathway through the book. A ‘map’ and summaries of chapters suggest a reading according to a thematic approach, geographical setting, author/s profile or content purposes depending on the reader’s own priorities. A section on assessment past, present, and futures calls for a rebalancing of improvement and accountability goals, and for countries to be careful to avoid privileging large-scale testing over other forms of data about learning and achievement.

Retooling Chinese primary school teachers to use technology creatively to promote innovation and problem solving skills in science classrooms

This paper reports on the initial phase of a Professional Learning Program (PLP) undertaken by 100 primary school teachers in China that aimed to facilitate the development of adaptive expertise in using technology to facilitate innovative science teaching and learning such as that envisaged by the Chinese Ministry of Education’s (2010-2020) education reforms. Key principles derived from literature about professional learning and scaffolding of learning informed the design of the PLP. The analysis of data revealed that the participants had made substantial progress towards the development of adaptive expertise. This was manifested not only by advances in the participants’ repertoires of Subject Matter Knowledge and Pedagogical Content Knowledge but also in changes to their levels of confidence and identities as teachers. By the end of the initial phase of the PLP, the participants had coalesced into a professional learning community that readily engaged in the sharing, peer review, reuse and adaption, and collaborative design of innovative science learning and assessment activities. The findings from the study indicate that those engaged in the development of PLPs for teachers in China need to take cognizance of certain cultural factors and traditions idiosyncratic to the Chinese educational system. A set of revised principles is then presented to inform the future design and implementation of PLPs for teachers in China.

Measuring multidisciplinary team effectiveness in a ward-based healthcare setting : development of the Team Functioning Assessment Tool

Nontechnical skills relating to team functioning are vital to the effective delivery of patient care and safety. In this study, we develop a reliable behavioral marker tool for assessing nontechnical skills that are critical to the success of ward-based multidisciplinary healthcare teams. The Team Functioning Assessment Tool (TFAT) was developed and refined using a literature review, focus groups, card-sorting exercise, field observations, and final questionnaire evaluation and refinement process. Results demonstrated that Clinical Planning, Executive Tasks, and Team Relations are important facets of effective multidisciplinary healthcare team functioning. The TFAT was also shown to yield acceptable inter-rater agreement.

Measuring ward-based multidisciplinary healthcare team functioning : a validation study of the Team Functioning Assessment Tool (TFAT)

The team functioning assessment tool (TFAT) has been shown to be a reliable behavioral marker tool for assessing nontechnical skills that are critical to the success of ward-based healthcare teams. This paper aims to refine and shorten the length of the TFAT to improve usability, and establish its reliability and construct validity. Psychometric testing based on 110 multidisciplinary healthcare teams demonstrated that the TFAT is a reliable and valid tool for measuring team members’ nontechnical skills in regards to Clinical Planning, Executive Tasks, and Team Functioning. Providing support for concurrent validity, high TFAT ratings were predicted by low levels of organizational constraints and high levels of group potency. There was also partial support for the negative relationships between time pressure, leadership ambiguity, and TFAT ratings. The paper provides a discussion on the applicability of the tool for assessing multidisciplinary healthcare team functioning in the context of improving team effectiveness and patient safety for ward-based hospital teams.

Teaching and assessment of prescribing medicines : a multidisciplinary review

Background Prescribing is a complex task, requiring specific knowledge and skills, and the execution of effective, context-specific clinical reasoning. Prescribing errors can result in significant morbidity and mortality. For all professions with prescribing rights, a clear need exists to ensure students graduate with a well-defined set of prescribing skills, which will contribute to competent prescribing.

The role of radiotherapy patients in provision of student interpersonal skills feedback

Background At Queensland University of Technology, student radiation therapists receive regular feedback from clinical staff relating to clinical interpersonal skills. Although this is of great value, there is anecdotal evidence that students communicate differently with patients when under observation. Purpose The aim of this pilot was to counter this perceived observer effect by allowing patients to provide students with additional feedback. Materials and methods Radiotherapy patients from two departments were provided with anonymous feedback forms relating to aspects of student interpersonal skills. Clinical assessors, mentors and students were also provided with feedback forms, including questions about the role of patient feedback. Patient perceptions of student performance were correlated with staff feedback and assessment scores. Results Results indicated that the feedback was valued by both students and patients. Students reported that the additional dimension focused them on communication, set goals for development and increased motivation. These changes derived from both feedback and study participation, suggesting that the questionnaires could be a useful teaching tool. Patients scored more generously than mentors, although there was agreement in relative grading. Conclusions The anonymous questionnaire is a convenient and valuable method of gathering patient feedback on students. Future iterations will determine the optimum timing for this method of feedback.

Making sense of inconsistencies in psycho-educational assessment: A case report

The case is presented of an 8-year-old young boy who was referred for psycho-educational assessment because of difficulties with writing. The paper provides an example of the way in which a case unfolds as further assessment data become available, and describes a number of challenging aspects of the assessment process. In this case, dilemmas arose when test results were inconsistent at different time points, and when the results were inconsistent with clinical observations. The case report illustrates the ways in which practitioners can collaborate to make sense of such discrepancies, with each contributing a slightly different perspective or set of skills that, in combination, assist to better understand a child’s difficulties.