How do the learning outcomes of students compare in units with different levels of practical work? A comparative study about the relation between Problem-­Based Learning and assessment approaches

The literature supporting the notion that active, student-centered learning is superior to passive, teacher-centered instruction is encyclopedic (Bonwell & Eison, 1991; Bruning, Schraw, & Ronning, 1999; Haile, 1997a, 1997b, 1998; Johnson, Johnson, & Smith, 1999). Previous action research demonstrated that introducing a learning activity in class improved the learning outcomes of students (Mejias, 2010). People acquire knowledge and skills through practice and reflection, not by watching and listening to others telling them how to do something. In this context, this project aims to find more insights about the level of interactivity in the curriculum a class should have and its alignment with assessment so the intended learning outcomes (ILOs) are achieved. In this project, interactivity is implemented in the form of problem- based learning (PBL). I present the argument that a more continuous formative feedback when implemented with the correct amount of PBL stimulates student engagement bringing enormous benefits to student learning. Different levels of practical work (PBL) were implemented together with two different assessment approaches in two subjects. The outcomes were measured using qualitative and quantitative data to evaluate the levels of student engagement and satisfaction in the terms of ILOs.

Development and implementation of the Australian universities radiation therapy student clinical assessment form

Purpose: Prior to 2009, one of the problems faced by radiation therapists who supervised and assessed students on placement in Australian clinical centres, was that each of the six Australian universities where Radiation Therapy (RT) programmes were conducted used different clinical assessment and reporting criteria. This paper describes the development of a unified national clinical assessment and reporting form that was implemented nationally by all six universities in 2009. Methods: A four phase methodology was used to develop the new assessment form and user guide. Phase 1 included university consensus around domains of student practice and assessment, and alignment with national competency standards; Phase 2 was a national consensus workshop attended by radiation therapists involved in student supervision and assessment; Phase 3 was an action research re-iterative Delphi technique involving two rounds of a mail-out to gain further expert consensus; and stage 4 was national piloting of the developed assessment form. Results: The new assessment form includes five main domains of practice and 19 sub-domain criteria which students are assessed against during placement. Feedback from the pilot centre participants was positive, with the new form being assessed to be comprehensive and complemented by the accompanying user guide. Conclusion: The new assessment form has improved both the formative and summative assessment of students on placement, as well as enhancing the quality of feedback to students and the universities. The new national form has high acceptance from the Australian universities and has been subject to wide review by the profession.

Towards fairer assessment

Drawing on the largest Australian collection and analysis of empirical data on multiple facets of Aboriginal and Torres Strait Islander education in state schools to date, this article critically analyses the systemic push for standardized testing and improved scores, and argues for a greater balance of assessment types by providing alternative, inclusive, participatory approaches to student assessment. The evidence for this article derives from a major evaluation of the Stronger Smarter Learning Communities. The first large-scale picture of what is occurring in classroom assessment and pedagogy for Indigenous students is reported in this evaluation yet the focus in this article remains on the issue of fairness in student assessment. The argument presented calls for “a good balance between formative and summative assessment” (OECD, Synergies for Better Learning An International Perspective on Evaluation and Assessment, Pointers for Policy Development, 2013) at a time of unrelenting high-stakes, standardized testing in Australia with a dominance of secondary as opposed to primary uses of NAPLAN data by systems, schools and principals. A case for more “intelligent accountability in education” (O’Neill, Oxford Review of Education 39(1):4–16, 2013) together with a framework for analyzing efforts toward social justice in education (Cazden, International Journal of Educational Psychology 1(3):178–198, 2012) and fairer assessment make the case for more alternative assessment practices in recognition of the need for teachers’ pedagogic practice to cater for increased diversity.

OSCE online : making assessment of clinical competencies transparent, accountable & authentic

Creating an authentic assessment which at once assesses competencies, scene management, communication and overall patient care is challenging in the competitive tertiary education market. Increasing student numbers and the cost of evaluating scenario based competencies serve to ensure the need for consistent objectivity and need for timely feedback to students on their performance. Objective structured clinical examination (OSCE) is currently the most flexible approach to competency based formative and summative assessment and widely used within paramedic degree programs. Students are understandably compelled to perform well and can be frustrated by not receiving timely and appropriate feedback. Increasingly a number of products aimed at providing a more efficient and paperless approach have begun to enter the market. These products, it is suggested are aimed at medicine programs and not at allied health professions and limited to one operating system and therefore ignore issues surrounding equity and accessibility. OSCE Online aims to address this gap in the market and is tailored to these disciplines. The application will provide a service that can be both tailored and standardised from a pre-written bank, depending upon requirement to fit around the needs of clinical competency assessment. Delivering authentic assessments to address student milestones in their training to become paramedics is the cornerstone of OSCE Online. By not being restricted to a specific device it will address issues of functionality, adaptability, accessibility, authenticity and importantly: transparency and accountability by producing contemporaneous data allowing issues to be easily identified and rectified.

The effect of feedback from pupil to teacher on assessment for leaning and visible learning: An ethnographic case study in a community school in England and the outcome in a state high school in Queensland, Australia

This study shows that there is positive regulatory effect of feedback from pupils to teachers on Assessment for Learning (AfL), classroom proactiveness, and on visible and progressive learning but not on behaviour. This research finding further articulates feedback from pupil to teacher as a paradigm shift from the classical paradigm of feedback from teacher to pupil. Here, the emphasis is geared towards pupils understanding of objectives built from previous knowledge. These are then feedback onto the teachers by the pupils in the form of discrete loops of cues and questions, where they are with their learning. This therefore enables them to move to the next level of understanding, and thus acquired independence, which in turn is reflected by their success in both formative and summative assessments. This study therefore shows that when feedback from pupil to teacher is used in combination with teacher to pupil feedback, AfL is ameliorated and hence, visible and accelerated learning occurs in a gender, nor subject non-dependent manner.

In-training assessment - Its potential in enhancing clinical teaching

Context In-training assessment (ITA) has established its place alongside formative and summative assessment at both the undergraduate and postgraduate level. In this paper the authors aimed to identify those characteristics of ITA that could enhance clinical teaching. Methods A literature review and discussions by an expert working group at the Ninth Cambridge Conference identified the aspects of ITA that could enhance clinical teaching. Results The features of ITA identified included defining the specific benefits to the learner, teacher and institution, and highlighting the patient as the context for ITA and clinical teaching. The ‘mapping’ of a learner’s progress towards the clinical teaching objectives by using multiple assessments over time, by multiple observers in both a systematic and opportunistic way correlates with the incremental nature of reaching clinical competence. Conclusions The importance of ITA based on both direct and indirect evidence of what the learner actually does in the real clinical setting is emphasized. Particular attention is given to addressing concerns in the more controversial areas of assessor training, ratings and documentation for ITA. Areas for future research are also identified.

Time Efficient Assessment and Feedback Methods for Large Computer-Aided-Design Cohorts

Timely and individualized feedback on coursework is desirable from a student perspective as it facilitates formative development and encourages reflective learning practice. Faculty however are faced with a significant and potentially time consuming challenge when teaching larger cohorts if they are to provide feedback which is timely, individualized and detailed. Additionally, for subjects which assess non-traditional submissions, such as Computer-Aided-Design (CAD), the methods for assessment and feedback tend not to be so well developed or optimized. Issues can also arise over the consistency of the feedback provided. Evaluations of Computer-Assisted feedback in other disciplines (Denton et al, 2008), (Croft et al, 2001) have shown students prefer this method of feedback to traditional “red pen” marking and also that such methods can be more time efficient for faculty.
Herein, approaches are described which make use of technology and additional software tools to speed up, simplify and automate assessment and the provision of feedback for large cohorts of first and second year engineering students studying modules where CAD files are submitted electronically. A range of automated methods are described and compared with more “manual” approaches. Specifically one method uses an application programming interface (API) to interrogate SolidWorks models and extract information into an Excel spreadsheet, which is then used to automatically send feedback emails. Another method describes the use of audio recordings made during model interrogation which reduces the amount of time while increasing the level of detail provided as feedback.
Limitations found with these methods and problems encountered are discussed along with a quantified assessment of time saving efficiencies made.

Improving E-Assessment Systems With A focus On Adaptive Feedback And Marking

Institutions involved in the provision of tertiary education across Europe are feeling the pinch. European universities, and other higher education (HE) institutions, must operate in a climate where the pressure of government spending cuts (Garben, 2012) is in stark juxtaposition to the EU’s strategy to drive forward and maintain a growth of student numbers in the sector (eurostat, 2015).

In order to remain competitive, universities and HE institutions are making ever-greater use of electronic assessment (E-Assessment) systems (Chatzigavriil et all, 2015; Ferrell, 2012). These systems are attractive primarily because they offer a cost-effect and scalable approach for assessment. In addition to scalability, they also offer reliability, consistency and impartiality; furthermore, from the perspective of a student they are most popular because they can offer instant feedback (Walet, 2012).

There are disadvantages, though.

First, feedback is often returned to a student immediately on competition of their assessment. While it is possible to disable the instant feedback option (this is often the case during an end of semester exam period when assessment scores must be can be ratified before release), however, this option tends to be a global ‘all on’ or ‘all off’ configuration option which is controlled centrally rather than configurable on a per-assessment basis.

If a formative in-term assessment is to be taken by multiple groups of
students, each at different times, this restriction means that answers to each question will be disclosed to the first group of students undertaking the assessment. As soon as the answers are released “into the wild” the academic integrity of the assessment is lost for subsequent student groups.

Second, the style of feedback provided to a student for each question is often limited to a simple ‘correct’ or ‘incorrect’ indicator. While this type of feedback has its place, it often does not provide a student with enough insight to improve their understanding of a topic that they did not answer correctly.

Most E-Assessment systems boast a wide range of question types including Multiple Choice, Multiple Response, Free Text Entry/Text Matching and Numerical questions. The design of these types of questions is often quite restrictive and formulaic, which has a knock-on effect on the quality of feedback that can be provided in each case.

Multiple Choice Questions (MCQs) are most prevalent as they are the most prescriptive and therefore most the straightforward to mark consistently. They are also the most amenable question types, which allow easy provision of meaningful, relevant feedback to each possible outcome chosen.
Text matching questions tend to be more problematic due to their free text entry nature. Common misspellings or case-sensitivity errors can often be accounted for by the software but they are by no means fool proof, as it is very difficult to predict in advance the range of possible variations on an answer that would be considered worthy of marks by a manual marker of a paper based equivalent of the same question.

Numerical questions are similarly restricted. An answer can be checked for accuracy or whether it is within a certain range of the correct answer, but unless it is a special purpose-built mathematical E-Assessment system the system is unlikely to have computational capability and so cannot, for example, account for “method marks” which are commonly awarded in paper-based marking.

From a pedagogical perspective, the importance of providing useful formative feedback to students at a point in their learning when they can benefit from the feedback and put it to use must not be understated (Grieve et all, 2015; Ferrell, 2012).

In this work, we propose a number of software-based solutions, which will overcome the limitations and inflexibilities of existing E-Assessment systems.

Assessment and learning : is it time to rethink student activities and academic roles?

Resumen basado en el de la publicación

Developing formal mathematical assessment for 4-to-8-year-olds

The assessment of children in their years before school and their first years of school has been, traditionally, informal. Further, assessment of children's mathematical skills at this level has been infrequent compared to social, emotional and physical assessments. However, there are contexts where reliable, valid, standardised data from assessment in mathematics are required. This paper outlines the development of two assessment tools for mathematics that were originally developed for such contexts. Item Response Theory (IRT) analyses enabled the construction of assessment forms that address the range of abilities of 4- to 8-year-old children, and provided the scales used for constructing formative and summative reports of achievement. A description of the development of the assessment tools and the IRT analysis that provides the reporting formats are presented together with some research uses of the tools.

Integrating errors into developmental assessment: 'time' for ages 8-13

It is widely agreed that measurement is of paramount importance to students’ overall development in mathematics. This paper describes a developmental ‘map’ of students’ understanding and skills in measurement, focussed on the topic of Time, that integrates correct and incorrect student ideas. The map is based on a Rasch analysis of data from a large-scale UK national survey for standardising assessment for children from 5 to 14 years of age. It is demonstrated how a partial credit strategy enables a developmental map to be constructed to show students’ strengths and weaknesses in a meaningful and useful summative and formative manner. This map provides evidence, of both a summative and a formative nature, which may enable teachers to craft appropriate and successful learning experiences for children.

Online self and peer assessment in large, multi-campus, multi-cohort contexts

The increasing use of team assignments within higher education is well documented. The driving forces behind this include desires to facilitate reflective and collaborative learning, to develop generic teamwork skills for graduate employment and to reduce the grading workloads of faculty staff. Students however consistently report dissatisfaction when the assessment of team assignments produces a common grade for all team members. Self-andpeer-assessment (SAPA) is presented as a fair, valid and reliable method of producing information about ongoing team processes. This information can provide ongoing feedback to team members and rich formative data to instructors attempting to assess the team process and students’ teamwork skills. This data can also enable individualised summative assessment in dysfunctional teams or situations of uneven team member contributions. Whilst manual SAPA protocols can work effectively for smaller classes, computer-assisted SAPA offers a solution to the problems of large classes. This paper reports on the early stages of an online SAPA tool, originally developed for small classes of architecture students, adapted for use by very large business communication classes comprising up to 1000 students in a semester. This large unit is delivered on four Australian campuses as well as off-campus and in off-shore mode, by up to fourteen instructors at any one time. The paper documents how three researchers from very different backgrounds worked to create their own research team, implement a pilot study, and adapt the online tool, whilst adhering to comparability of assessment constraints and maintaining integrity of research design.

Developing a scholarly approach to the evaluation of assessment practices

Purpose – The purpose of this paper is to examine and reflect on current assessment practice in a large undergraduate accounting programme delivered both in Australia and offshore, from the perspective of academics in their first semester at a “new-to-them” university.

Design/methodology/approach – The changing higher education environment and the reality of assessment in the current context are considered, as they raise a number of important issues around assessment practice. Some of the often cited literature linking teaching, learning and assessment, including student-centred learning and Confucian heritage culture, is also discussed. A reflective approach is used where Säljö's five categories of student learning are used as the basis for informed reflection of the assessment used in the “new” academics' first semester at the university. The use of empirical evidence to test these reflections would be the next step in this scholarly approach to teaching and learning.

Findings – The reflections reveal a disparity between reality and the ideal in relation to assessment practice. Issues regarding timely feedback to students and timing of assessments can result in summative assessment when it has the potential to be formative. This paper has provided an opportunity for “new” academics to engage with the higher education literature early in their careers.

Originality/value – This paper is a resource for academics beginning to engage with the higher education literature around assessment, teaching and learning and can also be used to inform and improve the teaching and learning practices of many academics in higher education.

The pragmatic portfolio : an assessment approach for distributed learning

Portfolios, especially where they involve some use of or link to online technologies, are currently a popular focus for learning innovation in universities, drawing on a tradition of using portfolios in some areas of higher education and attempting to extend and broaden this practice. In some cases this focus has led to ambitious plans for whole-of-institution approaches, often involving significant technological development. However, the term portfolio can also cover a wider variety of possible learning and assessment activities and there are ways of using portfolios which, while quite traditional in their own form and approach, enable teachers to approach other aspects of their curriculum and pedagogy in far more innovative ways. This paper explores the conceptual basis on which the Department of Internet Studies at Curtin University of Technology is utilising a pragmatic approach to portfolio assessment within individual units of study, so as to enable a more thorough implementation of distributed learning. In this form of learning, where students regularly contribute to their own and others' learning through short tasks and conversations, the evidence of achievement is widely distributed and not easily accessible for either formative or summative assessment. As explained in the paper, students are required to collate, select, and then contextualise a sample of these numerous productive moments of their ongoing study. The paper concludes that while other goals for portfolio assessment (such as encouraging reflection) can also be used with this approach, its primary value is in unleashing the potential of social media creativity in a manner that motivates students via the requirement of assessment, enables feedback to be provided to guide learning, and which promotes shared responsibility between teachers and students in determining the kind and extent of their learning activities.

Construction of the descriptive system for the assessment of quality of life AQoL-6D utility instrument

Background
Multi attribute utility (MAU) instruments are used to include the health related quality of life (HRQoL) in economic evaluations of health programs. Comparative studies suggest different MAU instruments measure related but different constructs. The objective of this paper is to describe the methods employed to achieve content validity in the descriptive system of the Assessment of Quality of Life (AQoL)-6D, MAU instrument.

Methods
The AQoL program introduced the use of psychometric methods in the construction of health related MAU instruments. To develop the AQoL-6D we selected 112 items from previous research, focus groups and expert judgment and administered them to 316 members of the public and 302 hospital patients. The search for content validity across a broad spectrum of health states required both formative and reflective modelling. We employed Exploratory Factor Analysis and Structural Equation Modelling (SEM) to meet these dual requirements.

Results and Discussion
The resulting instrument employs 20 items in a multi-tier descriptive system. Latent dimension variables achieve sensitive descriptions of 6 dimensions which, in turn, combine to form a single latent QoL variable. Diagnostic statistics from the SEM analysis are exceptionally good and confirm the hypothesised structure of the model.

Conclusions
The AQoL-6D descriptive system has good psychometric properties. They imply that the instrument has achieved construct validity and provides a sensitive description of HRQoL. This means that it may be used with confidence for measuring health related quality of life and that it is a suitable basis for modelling utilities for inclusion in the economic evaluation of health programs.