Sustainable assessment for large science classes : non-multiple choice, randomised assignments through a Learning Management System

This paper reports on the development of a tool that generates randomised, non-multiple choice assessment within the BlackBoard Learning Management System interface. An accepted weakness of multiple-choice assessment is that it cannot elicit learning outcomes from upper levels of Biggs’ SOLO taxonomy. However, written assessment items require extensive resources for marking, and are susceptible to copying as well as marking inconsistencies for large classes. This project developed an assessment tool which is valid, reliable and sustainable and that addresses the issues identified above. The tool provides each student with an assignment assessing the same learning outcomes, but containing different questions, with responses in the form of words or numbers. Practice questions are available, enabling students to obtain feedback on their approach before submitting their assignment. Thus, the tool incorporates automatic marking (essential for large classes), randomised tasks to each student (reducing copying), the capacity to give credit for working (feedback on the application of theory), and the capacity to target higher order learning outcomes by requiring students to derive their answers rather than choosing them. Results and feedback from students are presented, along with technical implementation details.

Peer state assessment of TxDOT maintenance program and practices : workshop and road rally findings

In an effort to evaluate and improve their practices to ensure the future excellence of the Texas highway system, the Texas Department of Transportation (TxDOT) sought a forum in which experts from other state departments of transportation could share their expertise. Thus, the Peer State Review of TxDOT Maintenance Practices project was organized and conducted for TxDOT by the Center for Transportation Research (CTR) at The University of Texas at Austin. The goal of the project was to conduct a workshop at CTR and in the Austin District that would educate the visiting peers on TxDOT’s maintenance practices and invite their feedback. CTR and TxDOT arranged the participation of the following directors of maintenance: Steve Takigawa, CA; Roy Rissky, KS; Eric Pitts, GA; Jim Carney, MO; Jennifer Brandenburg, NC; and David Bierschbach, WA. One of the means used to capture the peer reviewers’ opinions was a carefully designed booklet of 15 questions. The peers provided TxDOT with written responses to these questions, and the oral comments made during the workshop were also captured. This information was then compiled and summarized in the following report. An examination of the peers’ comments suggests that TxDOT should use a more holistic, statewide approach to funding and planning rather than funding and planning for each district separately. Additionally, the peers stressed the importance of allocating funds based on the actual conditions of the roadways instead of on inventory. The visiting directors of maintenance also recommended continuing and proliferating programs that enhance communication, such as peer review workshops.

Peer state assessment of TxDOT maintenance program and practices, part I : workshop findings and recommendations

To assess and improve their practices, and thus ensure the future excellence of the Texas highway system, the Texas Department of Transportation (TxDOT) sought a forum in which experts from other State Departments of Transportation could evaluate the TxDOT maintenance program and practices based on their expertise. To meet this need, a Peer State Review of TxDOT Maintenance Practices project was organized and conducted by the Center for Transportation Research (CTR) at The University of Texas at Austin. CTR researchers, along with TxDOT staff, conducted a workshop to present TxDOT’s maintenance practices to the visiting peer reviewers and invite their feedback. Directors of maintenance from six different states—California, Kansas, Georgia, Missouri, North Carolina, and Washington—participated in the workshop. CTR and TxDOT worked together to design a questionnaire with 15 key questions to capture the peers’ opinions on maintenance program and practices. This paper compiles and summarizes this information. The examination results suggested that TxDOT should use a more state-wide approach to funding and planning, in addition to funding and planning for each district separately. Additionally, the peers recommended that criteria such as condition and level of service of the roadways be given greater weight in the funding allocation than lane miles or vehicle miles traveled (VMT). The Peer Reviewers also determined that TxDOT maintenance employee experience and communications were strong assets. Additional strengths included the willingness of TxDOT to invite peer reviews of their practices and a willingness to consider opportunities for improvement.

How do the learning outcomes of students compare in units with different levels of practical work? A comparative study about the relation between Problem-­Based Learning and assessment approaches

The literature supporting the notion that active, student-centered learning is superior to passive, teacher-centered instruction is encyclopedic (Bonwell & Eison, 1991; Bruning, Schraw, & Ronning, 1999; Haile, 1997a, 1997b, 1998; Johnson, Johnson, & Smith, 1999). Previous action research demonstrated that introducing a learning activity in class improved the learning outcomes of students (Mejias, 2010). People acquire knowledge and skills through practice and reflection, not by watching and listening to others telling them how to do something. In this context, this project aims to find more insights about the level of interactivity in the curriculum a class should have and its alignment with assessment so the intended learning outcomes (ILOs) are achieved. In this project, interactivity is implemented in the form of problem- based learning (PBL). I present the argument that a more continuous formative feedback when implemented with the correct amount of PBL stimulates student engagement bringing enormous benefits to student learning. Different levels of practical work (PBL) were implemented together with two different assessment approaches in two subjects. The outcomes were measured using qualitative and quantitative data to evaluate the levels of student engagement and satisfaction in the terms of ILOs.

Orals ain’t orals : the role of prepared oral presentations in the classroom and beyond. How do instruction and feedback practices guide delivery choices?

Despite an ostensibly technology-driven society, the ability to communicate orally continues to feature as an essential ability for students at school and university, as it is for graduates in the workplace. Pedagogically, one rationale is that the need to develop effective oral communication skills is tied to life-long learning which includes successful participation in future work-related tasks. One tangible way that educators have assessed proficiency in the area of communication is through prepared oral presentations. While much of the literature uses the terms 'oral communication' and 'oral presentation' interchangeably, some writers question the role more formal presentations play in the overall development of oral communication skills. However, such formal speaking tasks continue to be a recognised assessment practice in both the secondary school and academy, and, therefore, worthy of further investigation. Adding to the discussion, this thesis explores the knowledge and skills students bring into the academy from previous educational experiences. It examines some of the teaching and assessment methods used in secondary schools to develop oral communication skills through the use of formal oral presentations. Specifically, it investigates criterion-referenced assessment sheets and how these tools are used as a form of instruction, as well as their role and effectiveness in the evaluation of student ability. The focus is on the student's perspective and includes 12 semi-structured interviews with school students. The purpose of this thesis is to explore key thematics underpinning oral communication and to identify tensions between expectations and practice. While acknowledging the breadth and depth of material available under the heading of 'communication theory', this study specifically draws on an expanded view of the rhetorical tradition to fully interrogate the assumptions supporting the practice of assessing oral presentations. Finally, this thesis recommends reconnecting with an updated understanding of rhetoric as a way of assisting in the development of expressive, articulate and discerning communicators.

Please take out your phones : on the spot solicitation of student feedback in class

The use of mobile devices such as smart phones and tablets in classrooms has been met with mixed sentiments. Some instructors and teachers see them as a distraction and regularly ban their usage. Others who see their potential to enhance learning have started to explore ways to integrate them into their teaching in an attempt to improve student engagement. In this paper we report on a pilot study that forms part of a university-wide project reconceptualising its approach to the student evaluation of learning and teaching. In a progressive decision to embrace mobile technology, the university decided to trial a smart phone app designed for students to check-in to class and leave feedback on the spot. Our preliminary findings from trialling the app indicate that the application establishes a more immediate feedback loop between students and teachers. However, the app’s impact depends on how feedback is shared with students and how the teaching team responds.

Sustainable infrastructure operations : a review of assessment schemes and decision support

Infrastructure forms a vital component in supporting today’s way of life and has a significant role or impact on economic, environmental and social outcomes of the region around it. The design, construction and operation of such assets are a multi-billion dollar industry in Australia alone. Another issue that will play a major role in our way life is that of climate change and the greater concept of sustainability. With limited resources and a changing natural world it is necessary for infrastructure to be developed and maintained in a manner that is sustainable. In order to achieve infrastructure sustainability in operations it is necessary for there to be: a sustainability assessment scheme that provides a scientifically sound and realistic approach to measuring an assets level of sustainability; and, systems and tools to support the making of decisions that result in sustainable outcomes by providing feedback in a timely manner. Having these in place will then help drive the consideration of sustainability during the decision making process for infrastructure operations and maintenance. In this paper we provide two main contributions; a comparison and review of sustainability assessment schemes for infrastructure and their suitability for use in the operations phase; and, a review of decision support systems/tools in the area of infrastructure sustainability in operations. For this paper, sustainability covers not just the environment, but also finance/economic and societal/community aspects as well. This is often referred to as the Triple Bottom Line and forms one of the three dimensions of corporate sustainability [Stapledon, 2004].

The enactment of formative assessment in English language classrooms in two Chinese universities : teacher and student responses

The College English Curriculum Requirements (CECR), announced by the Chinese Ministry of Education in 2007, recommended the inclusion of formative assessment into the existing summative assessment framework of College English. This policy had the potential to fundamentally change the nature of assessment and its role in the teaching and learning of English in Chinese universities. In order to document and analyse these changes, case studies involving English language teachers and learners were undertaken in two Chinese Universities: one a Key university in the national capital; the other a non-Key university in a western province. The case study design incorporated classroom observations and interviews with English language teachers and their students. The type and focus of feedback and the engagement of students in assessment were analysed in the two contexts. Fundamental to the analysis was the concept of enactment, with the focus of this study on the ways that policy ideas and principles were enacted in the practices of the Chinese university classroom. Understandings of formative assessment as applied in contexts other than the predominantly Western, Anglophone contexts from where many of its principles derive, are offered.

Using formative feedback to identify and support first year chemistry students with missing or misconceptions. A Practice Report

Students entering tertiary studies possess a diverse range of prior experiences in their academic preparation for tertiary chemistry so academics need tools to enable them to respond to issues in diversity in conceptual models possessed by entering students. Concept inventories can be used to provide formative feedback to help students identify concepts that they need to address to improve construction of subsequent understanding enabling their learning. Modular, formative learning activities that can be administered inside or outside of class in first year chemistry courses have been developed. These activities address key missing and mis-conceptions possessed by incoming student. Engagement in these learning activities by students and academics will help shift the culture of diagnostic and formative assessment within the tertiary context and address issues around the secondary/tertiary transition. This diagnostic/intervention framework is currently being trialed across five Australian tertiary institutions encompassing a large heterogeneous sample of students.

Development of the palliative care needs assessment tool (PC-NAT) for use by multi-disciplinary health professionals

Needs assessment strategies can facilitate prioritisation of resources. To develop a needs assessment tool for use with advanced cancer patients and caregivers, to prompt early intervation. A convenience sample of 103 health professionals viewed three videotaped consultations involving a simulated patient, his/her caregiver and a health professional, completed the Palliative Care Needs Assessment Tool (PC-NAT) and provided feedback on clarity, content and acceptability of the PC-NAT. Face and content validity, acceptability and feasibility of the PC-NAT were confirmed. Kappa scores indicated adequate inter-rater reliability for the majority of domains; the patient spirituality domain and the caregiver physical and family and relationship domains had low reliability. The PC-NAT can be used by health professionals with a range of clinical expertise to identify individuals' needs, thereby enabling early intervention. Further psychometric testing and an evaluation to assess the impact of the systematic use of the PC-NAT on quality of life, unmet needs and service utilisation of patients and caregivers are underway.

Development and implementation of the Australian universities radiation therapy student clinical assessment form

Purpose: Prior to 2009, one of the problems faced by radiation therapists who supervised and assessed students on placement in Australian clinical centres, was that each of the six Australian universities where Radiation Therapy (RT) programmes were conducted used different clinical assessment and reporting criteria. This paper describes the development of a unified national clinical assessment and reporting form that was implemented nationally by all six universities in 2009. Methods: A four phase methodology was used to develop the new assessment form and user guide. Phase 1 included university consensus around domains of student practice and assessment, and alignment with national competency standards; Phase 2 was a national consensus workshop attended by radiation therapists involved in student supervision and assessment; Phase 3 was an action research re-iterative Delphi technique involving two rounds of a mail-out to gain further expert consensus; and stage 4 was national piloting of the developed assessment form. Results: The new assessment form includes five main domains of practice and 19 sub-domain criteria which students are assessed against during placement. Feedback from the pilot centre participants was positive, with the new form being assessed to be comprehensive and complemented by the accompanying user guide. Conclusion: The new assessment form has improved both the formative and summative assessment of students on placement, as well as enhancing the quality of feedback to students and the universities. The new national form has high acceptance from the Australian universities and has been subject to wide review by the profession.

Work-based assessment : qualitative perspectives of novice nutrition and dietetics educators

Background The assessment of competence for health professionals including nutrition and dietetics professionals in work-based settings is challenging. The present study aimed to explore the experiences of educators involved in the assessment of nutrition and dietetics students in the practice setting and to identify barriers and enablers to effective assessment. Methods A qualitative research approach using in-depth interviews was employed with a convenience sample of inexperienced dietitian assessors. Interviews explored assessment practices and challenges. Data were analysed using a thematic approach within a phenomenological framework. Twelve relatively inexperienced practice educators were purposefully sampled to take part in the present study. Results Three themes emerged from these data. (i) Student learning and thus assessment is hindered by a number of barriers, including workload demands and case-mix. Some workplaces are challenged to provide appropriate learning opportunities and environment. Adequate support for placement educators from the university, managers and their peers and planning are enablers to effective assessment. (ii) The role of the assessor and their relationship with students impacts on competence assessment. (iii) There is a lack of clarity in the tasks and responsibilities of competency-based assessment. Conclusions The present study provides perspectives on barriers and enablers to effective assessment. It highlights the importance of reflective practice and feedback in assessment practices that are synonymous with evidence from other disciplines, which can be used to better support a work-based competency assessment of student performance.

Learner-focused evaluation cycles : facilitating learning using feedforward, concurrent and feedback evaluation

There is a growing trend to offer students learning opportunities that are flexible, innovative and engaging. As educators embrace student-centred agile teaching and learning methodologies, which require continuous reflection and adaptation, the need to evaluate students’ learning in a timely manner has become more pressing. Conventional evaluation surveys currently dominate the evaluation landscape internationally, despite recognition that they are insufficient to effectively evaluate curriculum and teaching quality. Surveys often: (1) fail to address the issues for which educators need feedback, (2) constrain student voice, (3) have low response rates and (4) occur too late to benefit current students. Consequently, this paper explores principles of effective feedback to propose a framework for learner-focused evaluation. We apply a three-stage control model, involving feedforward, concurrent and feedback evaluation, to investigate the intersection of assessment and evaluation in agile learning environments. We conclude that learner-focused evaluation cycles can be used to guide action so that evaluation is not undertaken simply for the benefit of future offerings, but rather to benefit current students by allowing ‘real-time’ learning activities to be adapted in the moment. As a result, students become co-producers of learning and evaluation becomes a meaningful, responsive dialogue between students and their instructors.

OSCE online : making assessment of clinical competencies transparent, accountable & authentic

Creating an authentic assessment which at once assesses competencies, scene management, communication and overall patient care is challenging in the competitive tertiary education market. Increasing student numbers and the cost of evaluating scenario based competencies serve to ensure the need for consistent objectivity and need for timely feedback to students on their performance. Objective structured clinical examination (OSCE) is currently the most flexible approach to competency based formative and summative assessment and widely used within paramedic degree programs. Students are understandably compelled to perform well and can be frustrated by not receiving timely and appropriate feedback. Increasingly a number of products aimed at providing a more efficient and paperless approach have begun to enter the market. These products, it is suggested are aimed at medicine programs and not at allied health professions and limited to one operating system and therefore ignore issues surrounding equity and accessibility. OSCE Online aims to address this gap in the market and is tailored to these disciplines. The application will provide a service that can be both tailored and standardised from a pre-written bank, depending upon requirement to fit around the needs of clinical competency assessment. Delivering authentic assessments to address student milestones in their training to become paramedics is the cornerstone of OSCE Online. By not being restricted to a specific device it will address issues of functionality, adaptability, accessibility, authenticity and importantly: transparency and accountability by producing contemporaneous data allowing issues to be easily identified and rectified.

A rapid assessment of the effects of extreme weather on two Great Barrier Reef industries

In coastal areas, extreme weather events, such as floods and cyclones, can have debilitating effects on the social and economic viability of marine-based industries. In March 2011, the Great Barrier Reef Marine Park Authority implemented an Extreme Weather Response Program, following a period of intense flooding and cyclonic activity between December 2010 and February 2011. In this paper, we discuss the results of a project within the Program, which aimed to: (1) assess the impacts of extreme weather events on regional tourism and commercial fishing industries; and (2) develop and road-test an impact assessment matrix to improve government and industry responses to extreme weather events. Results revealed that extreme weather events both directly and indirectly affected all five of the measured categories, i.e. ecological, personal, social, infrastructure and economic components. The severity of these impacts, combined with their location and the nature of their business, influenced how tourism operators and fishers assessed the impact of the events (low, medium, high or extreme). The impact assessment tool was revised following feedback obtained during stakeholder workshops and may prove useful for managers in responding to potential direct and indirect impacts of future extreme weather events on affected marine industries. © 2013 Planning Institute Australia.