Improving E-Assessment Systems With A focus On Adaptive Feedback And Marking

Institutions involved in the provision of tertiary education across Europe are feeling the pinch. European universities, and other higher education (HE) institutions, must operate in a climate where the pressure of government spending cuts (Garben, 2012) is in stark juxtaposition to the EU’s strategy to drive forward and maintain a growth of student numbers in the sector (eurostat, 2015).

In order to remain competitive, universities and HE institutions are making ever-greater use of electronic assessment (E-Assessment) systems (Chatzigavriil et all, 2015; Ferrell, 2012). These systems are attractive primarily because they offer a cost-effect and scalable approach for assessment. In addition to scalability, they also offer reliability, consistency and impartiality; furthermore, from the perspective of a student they are most popular because they can offer instant feedback (Walet, 2012).

There are disadvantages, though.

First, feedback is often returned to a student immediately on competition of their assessment. While it is possible to disable the instant feedback option (this is often the case during an end of semester exam period when assessment scores must be can be ratified before release), however, this option tends to be a global ‘all on’ or ‘all off’ configuration option which is controlled centrally rather than configurable on a per-assessment basis.

If a formative in-term assessment is to be taken by multiple groups of
students, each at different times, this restriction means that answers to each question will be disclosed to the first group of students undertaking the assessment. As soon as the answers are released “into the wild” the academic integrity of the assessment is lost for subsequent student groups.

Second, the style of feedback provided to a student for each question is often limited to a simple ‘correct’ or ‘incorrect’ indicator. While this type of feedback has its place, it often does not provide a student with enough insight to improve their understanding of a topic that they did not answer correctly.

Most E-Assessment systems boast a wide range of question types including Multiple Choice, Multiple Response, Free Text Entry/Text Matching and Numerical questions. The design of these types of questions is often quite restrictive and formulaic, which has a knock-on effect on the quality of feedback that can be provided in each case.

Multiple Choice Questions (MCQs) are most prevalent as they are the most prescriptive and therefore most the straightforward to mark consistently. They are also the most amenable question types, which allow easy provision of meaningful, relevant feedback to each possible outcome chosen.
Text matching questions tend to be more problematic due to their free text entry nature. Common misspellings or case-sensitivity errors can often be accounted for by the software but they are by no means fool proof, as it is very difficult to predict in advance the range of possible variations on an answer that would be considered worthy of marks by a manual marker of a paper based equivalent of the same question.

Numerical questions are similarly restricted. An answer can be checked for accuracy or whether it is within a certain range of the correct answer, but unless it is a special purpose-built mathematical E-Assessment system the system is unlikely to have computational capability and so cannot, for example, account for “method marks” which are commonly awarded in paper-based marking.

From a pedagogical perspective, the importance of providing useful formative feedback to students at a point in their learning when they can benefit from the feedback and put it to use must not be understated (Grieve et all, 2015; Ferrell, 2012).

In this work, we propose a number of software-based solutions, which will overcome the limitations and inflexibilities of existing E-Assessment systems.

An evaluation of the Socrative (personal response system) app for increasing student engagement and learning in an undergraduate psychology curriculum.

Personal response systems using hardware such as 'clickers' have been around for some time, however their use is often restricted to multiple choice questions (MCQs) and they are therefore used as a summative assessment tool for the individual student. More recent innovations such as 'Socrative' have removed the need for specialist hardware, instead utilising web-based technology and devices common to students, such as smartphones, tablets and laptops. While improving the potential for use in larger classrooms, this also creates the opportunity to pose more engaging open-response questions to students who can 'text in' their thoughts on questions posed in class. This poster will present two applications of the Socrative system in an undergraduate psychology curriculum which aimed to encourage interactive engagement with course content using real-time student responses and lecturer feedback. Data is currently being collected and result will be presented at the conference.
The first application used Socrative to pose MCQs at the end of two modules (a level one Statistics module and level two Individual Differences Psychology module, class size N≈100), with the intention of helping students assess their knowledge of the course. They were asked to rate their self-perceived knowledge of the course on a five-point Likert scale before and after completing the MCQs, as well as their views on the value of the revision session and any issues that had with using the app. The online MCQs remained open between the lecture and the exam, allowing students to revisit the questions at any time during their revision.
This poster will present data regarding the usefulness of the revision MCQs, the metacognitive effect of the MCQs on student's judgements of learning (pre vs post MCQ testing), as well as student engagement with the MCQs between the revision session and the examination. Student opinions on the use of the Socrative system in class will also be discussed.
The second application used Socrative to facilitate a flipped classroom lecture on a level two 'Conceptual Issues in Psychology' module, class size N≈100). The content of this module requires students to think critically about historical and contemporary conceptual issues in psychology and the philosophy of science. Students traditionally struggle with this module due to the emphasis on critical thinking skills, rather than simply the retention of concrete knowledge. To prepare students for the written examination, a flipped classroom lecture was held at the end of the semester. Students were asked to revise their knowledge of a particular area of Psychology by assigned reading, and were told that the flipped lecture would involve them thinking critically about the conceptual issues found in this area. They were informed that questions would be posed by the lecturer in class, and that they would be asked to post their thoughts using the Socrative app for a class discussion. The level of preparation students engaged in for the flipped lecture was measured, as well as qualitative opinions on the usefulness of the session. This poster will discuss the level of student engagement with the flipped lecture, both in terms of preparation for the lecture, and engagement with questions posed during the lecture, as well as the lecturer's experience in facilitating the flipped classroom using the Socrative platform.

Consistent Feedback for a Constructivist Pedagogy : a Study of Feedback on written Assessments in Cegep English Courses.

In order to determine how consistent feedback is within the English Cégep system, this study explores three key aspects of feedback provided to students: the amount of feedback provided, the nature of the feedback according to the defined criteria, and the relative importance of the three categories. Twenty-three teachers in four English departments participated in the study. Data was collected from a detailed questionnaire to give some context in terms of teacher training, experience, and assessment practices. Respondents were then asked to provide written feedback on a sample student essay. Their comments werw analysed in terms of the nature of the feedback, how consistent that feedback was between teachers, and how closely the feedback reflected the stated instructional objectives.

Enhancing Student Learning Experiences in Sport Psychology Modules Using Practical Activities

The University of Worcester states in its most recent strategic plan (2013 – 2018) a set of enduring values that guide and direct the activities of the institution. The first listed, and perhaps the most important value is the striving to be “an outstanding university at which to be a student”. This is further supplemented by values such as “to inspire our students to reach their full potential through excellent, innovative teaching, scholarship and research” (University of Worcester 2013: p.4). One of the many ways in which the institution strives to provide this outstanding educational experience is through regular engagement, both formal and informal, with students at a number of points in each semester. Regular experiences of collating formal and informal feedback has led to the identification of a common theme amongst Higher National Diploma (HND) students in the Institute of Sport and Exercise Sciences (ISES), where they consistently request ‘more practicals’. The ISES modules however are designed to incorporate a high degree of interaction, practical activities and tasks. This is especially important for those studying at HND level as research suggests differences in learning preferences exist when compared to undergraduate students, the former preferring a more tactile style of learning (Peters et al. 2008). Using an introductory Sport Psychology HND module as an example, practical activities and tasks are fully embedded in the taught sessions to enable contextual links to be made between the learning outcomes and their subsequent use. Examples of these include: a. interviewing athletes to produce a performance profile (Butler & Hardy 1992); b. completing psychometric instruments such as the Competitive State Anxiety Inventory-2 (CSAI-2) to measure competitive anxiety in sport (Martens et al. 1990) and demonstrate data collection and construct measurement; c. performing relaxation interventions on the students to demonstrate how specific techniques (in this instance, decreasing somatic anxiety) might work in practice; d. demonstrating how observational learning facilitates skill acquisition by creating experimental conditions that the students participate in, in teaching a new skill. Nevertheless owing to the students' previously stated on-going requests for more practical activities, it became evident that assumptions about what students consider an effective means of experiential or active learning in the context of sport-related disciplines of study needed to be investigated. This is where the opportunity to undertake an action research project arose, this being a practical method commonly employed in pedagogical enquiry to aid reflection on teaching and assessment practice for the purposes of working towards continuous improvement.

An Evaluative Case Study of a Career Coaching Approach to Improving Student Placement Success in a UK Business School

Describes the innovative approach adopted in a UK business school to improve the number of student placements. A module was designed to provide students with the skills to search, apply for and gain a year-long placement as part of a degree ‘sandwich’ course. A blend of workshops, recorded lectures, online formative feedback exercises and one-to-one career coaching created a tailored, practical approach to skill enhancement. This session provides the presentation of the adopted methodology the results of the evaluative research, a live demonstration of the coaching approach and a discussion with the audience of ideas for development of the approach.

Enhancing Student Learning: Study of a Motivational Resource for Educators

This study surveyed practicing classroom teacher’s perceptions of a proposed educational resource “Avatar Academy” designed to enhance students’, particularly young boys, motivation and general attitude towards learning. The Avatar Academy resource is an instructional guide for implementing a classroom reward system based on common game mechanics. The resource emphasizes the modification of current pedagogies to exploit the use of game design to engage boys. A survey of recent literature indicated an opportunity to study teachers’ perceptions of the possible applications of game design mechanics to support the enhancement of student motivation and learning in the classroom. As a result the Avatar Academy handbook and blog resource were developed to assist teachers with the integration and administration of a program designed to enhance student motivation, especially boys, using avatars and a point based reward system. The resources were initially distributed to several practicing teachers for their review, and their feedback formed the basis for revisions of the Avatar Academy resource. After implementing changes to the resource based on initial teacher feedback, an updated Avatar Academy was redistributed and teacher opinions and perceptions of the tool’s possible impacts on classroom learning were collected.

INFO1010 Coursework 3 - Feedback

A summary document providing generic feedback on student performance encompassing the group presentation and the peer marking

COMP1205 Student Presentations I-L

Thursday 8th May Building 6 (Eustace) Room 1007, 15.00-16.40 Su & Elena Presenting: Groups: I, J, K, L Marking Groups: M, N, O, P Schedule and Topics 15.00-15.05: Introduction and protocol for the session 15.05-15.25 Group I: Sustainablity – responsiblities and legislation 15.25-15.45 Group J: Green IT – solutions and benefits 16.45-16.05 Group K: Open and linked data 16.05-16.25 Group L: What is Web Science? 16.25-16.45: Wash-up: feedback session for presentation groups

COMP1205 Student Presentations E-H

Monday 12th May Building 34 Room 3001, 12.00-13.45 Su & Rikki Presenting: Groups: E, F, G, H Marking Groups: I, J, K, L Schedule and Topics 12.00-12.05: Introduction and protocol for the session 12.05-12.25 Group E: Creative commons, open source, open movements 12.25-12.45 Group F: Trolling, Banter, Cyber Hate, Online Bullying 12.45-13.05 Group G: Personal Privacy and Security 13.05-13.25 Group H: Crime online; cyber security 13.25-13.45: Wash-up: feedback session for presentation groups

COMP1205 Student Presentations M-P

Tuesday 13th May Building 34 Room 3001, 16.15-17.45 Elena & Rikki/Jian Presenting: Groups: M, N, O, P Marking Groups: Q, R, S, T Schedule and Topics 16.15-16.20: Introduction and protocol for the session 16.20 Group M: Serious games – gaming as a driver for applications online 16.40 Group N: Open Education OERs 17.00 Group O: Big Data – the big picture 17.20 Group P: Rights and equality in the workplace 17.40-18.00: Wash-up: feedback session for presentation groups

COMP1205 Student Presentations Q-T

Monday 12th May Building 34 Room 3001, 10.00-12.45 Su & Rikki Presenting: Groups: Q, R, S, T Marking Groups: U, V, W, X Schedule and Topics 10.00-10.05: Introduction and protocol for the session 10.05-10.25 Group Q: Disablitites and rights – legal responsibilities 10.25-10.45 Group R: Computer Ethics, Professional bodies and accreditation 10.45-11.05 Group S: Digital divide 11.05-11.25 Group T: How the web is chaning the world: co-operation, co-creation, crowd funding and crowd sourcing 11.25-11.45: Wash-up: feedback session for presentation groups

COMP1205 Student Presentations U-X

Thursday 15th May Building 02A Room 2077, 15.00-16.45 Elena & Rikki Presenting: Groups: U, V, W, X Marking Groups: A, B, C, D Schedule and Topics 15.00-15.05: Introduction and protocol for the session 15.05-15.25 Group U: Digital Literacies 15.25-15.45 Group V: Will MOOCs destroy face-to-face University Education? 15.45-16.05 Group W: Groupwork and leadership skills in MMORPGs 16.05-16.25 Group X: Tools and techniques for agile project management 16.25-16.45: Wash-up: feedback session for presentation groups

Work less, teach better and get better feedback: why we tried innovative teaching methods

Speaker: Patrick McSweeney Organiser: Time: 15/10/2014 11:00-11:45 Location: B32/3077 Abstract Having started at Southampton in 2005 I have seen quite a few changes to the way courses are taught and studied. I will reflect on some of the interesting changes I have observed and suggest their causes. As a practical example I will talk about codestrom, a peer feedback tool for learning programming. We have found that this teaching method has improved the student experience and reduced the work load for the module team. Together we will discuss how this and other recent developments can enable other teaching innovations which benefit staff as well as students. Hopefully the new class of PhD students will be able to contribute from the point of view of having recently been undergraduate students here and else where.

Integrating constructive feedback in personalised e-learning

When using e-learning material some students progress readily, others have difficulties. In a traditional classroom the teacher would identify those with difficulties and direct them to additional resources. This support is not easily available within e-learning. A new approach to providing constructive feedback is developed that will enable an e-learning system to identify areas of weakness and provide guidance on further study. The approach is based on the tagging of learning material with appropriate keywords that indicate the contents. Thus if a student performs poorly on an assessment on topic X, there is a need to suggest further study of X and participation in activities related to X such as forums. As well as supporting the learner this type of constructive feedback can also inform other stakeholders. For example a tutor can monitor the progress of a cohort; an instructional designer can monitor the quality of learning objects in facilitating the appropriate knowledge across many learners.

Moving feedback forward: theory to practice

There is substantial research interest in tutor feedback and students’ perception and use of such feedback. This paper considers some of the major issues raised in relation to tutor feedback and student learning. We explore some of the current feedback drivers, most notably the need for feedback to move away from simply a monologue from a tutor to a student to a valuable tutor–student dialogue. In relation to moving feedback forward the notions of self regulation, dialogue and social learning are explored and then considered in relation to how such theory can translate into practice. The paper proposes a framework (GOALS) as a tool through which tutors can move theory into practice with the aim of improving student learning from feedback.