Longitudinal Student Evaluations of a Postgraduate Unit Using Importance-Performance Analysis

The Australian tourism tertiary education sector operates in a competitive and dynamic environment, which necessitates a market orientation to be successful. Academic staff and management in the sector must regularly assess the perceptions of prospective and current students, and monitor the satisfaction levels of current students. This study is concerned with the setting and monitoring of satisfaction levels of current students, reporting the results of three longitudinal investigations of student satisfaction in a postgraduate unit. The study also addresses a limitation of a university’s generic teaching evaluation instrument. Importance-performance analysis (IPA) has been recommended as a simple but effective tool for overcoming the deficiencies of many student evaluation studies, which have generally measured only attribute importance or importance at the end of a semester. IPA was used to compare student expectations of the unit at the beginning of semester with their perceptions of performance ten weeks later. The first stage documented key benchmarks for which amendments to the unit based on student feedback could be evaluated during subsequent teaching periods.

Longitudinal evaluations of student satisfaction with a postgraduate unit using importance-performance analysis

The Australian tourism tertiary education sector operates in a competitive and dynamic environment, which necessitates a market orientation to be successful. Academic staff and management in the sector must regularly assess the perceptions of prospective and current students and monitor the satisfaction levels of current students. This study is concerned with the setting and monitoring of satisfaction levels of current students, reporting the results of three longitudinal investigations of student satisfaction in a postgraduate unit. The study also addresses a limitation of a university’s generic teaching evaluation instrument. Importance-Performance Analysis (IPA) has been recommended as a simple but effective tool for overcoming the deficiencies of many student evaluation studies, which have generally measured only attribute performance at the end of a semester. IPA was used to compare student expectations of the unit at the beginning of a semester with their perceptions of performance 10 weeks later. The first stage documented key benchmarks for which amendments to the unit based on student feedback could be evaluated during subsequent teaching periods.

From tiers to tables – enhancing student experience through collaborative learning spaces

BACKGROUND Collaborative and active learning have been clearly identified as ways students can engage in learning with each other and the academic staff. Traditional tier based lecture theatres and the didactic style they engender are not popular with students today as evidenced by the low attendance rates for lectures. Many universities are installing spaces designed with tables for group interaction with evolutions on spaces such as the TEAL (Technology Enabled Active Learning) (Massachusetts Institute of Technology, n.d.) and SCALE-UP (Student-Centred Activities for Large-Enrolment Undergraduate Programs) (North Carolina State University, n.d.) models. Technology advances in large screen computers and applications have also aided the move to these collaborative spaces. How well have universities structured learning using these spaces and how have students engaged with the content, technology, space and each other? This paper investigates the application of collaborative learning in such spaces for a cohort of 800+ first year engineers in the context of learning about and developing professional skills representative of engineering practice. PURPOSE To determine whether moving from tiers to tables enhances the student experience. Does utilising technology rich, activity based, collaborative learning spaces lead to positive experiences and active engagement of first year undergraduate engineering students? In developing learning methodology and approach in new learning spaces, what needs to change from a more traditional lecture and tutorial configuration? DESIGN/METHOD A post delivery review and analysis of outcomes was undertaken to determine how well students and tutors engaged with learning in new collaborative learning spaces. Data was gathered via focus group and survey of tutors, students survey and attendance observations. The authors considered the unit delivery approach along with observed and surveyed outcomes then conducted further review to produce the reported results. RESULTS Results indicate high participation in the collaborative sessions while the accompanying lectures were poorly attended. Students reported a high degree of satisfaction with the learning experience; however more investigation is required to determine the degree of improvement in retained learning outcomes. Survey feedback from tutors found that students engaged well in the activities during tutorials and there was an observed improvement in the quality of professional practice modelled by students during sessions. Student feedback confirmed the positive experiences in these collaborative learning spaces with 30% improvement in satisfaction ratings from previous years. CONCLUSIONS It is concluded that the right mix of space, technology and appropriate activities does engage students, improve participation and create a rich experience to facilitate potential for improved learning outcomes. The new Collaborative Teaching Spaces, together with integrated technology and tailored activities, has transformed the delivery of this unit and improved student satisfaction in tutorials significantly.

Emerging from student to practitioner : a cumulative development tool for students

PURPOSE: This pilot project’s aim was to trial a tool and process for developing students’ ability to engage in self-assessment using reflection on their clinical experiences, including feedback from workplace learning, in order to aid them in linking theory to practice and develop strategies to improve performance. BACKGROUND: In nursing education, students can experience a mismatch in performance compared to theoretical learning, this is referred to as the ‘theory practice gap’ (Scully 2011, Chan Chan & Liu 2011). One specific contributing factor seems to be students’ inability to engage in meaningful reflection and self-correcting behaviours. A self-assessment strategy was implemented within a third year clinical unit to ameliorate this mismatch with encouraging results, as students developed self-direction in addressing learning needs. In this pilot project the above strategy was adapted for implementation between different clinical units, to create a whole of course approach to integrating workplace learning. METHOD: The methodology underpinning this project is a scaffolded, supported reflective practice process. Improved self-assessment skills is achieved by students reflecting on and engaging with feedback, then mapping this to learning outcomes to identify where performance can be improved. Evaluation of this project includes: collation of student feedback identifying successful strategies along with barriers encountered in implementation; feedback from students and teachers via above processes and tools; and comparison of the number of learning contracts issued in clinical nursing units with similar cohorts. RESULTS: Results will be complete by May 2012 and include analysis of the data collected via the above evaluation methods. Other outcomes will include the refined process and tool, plus resources that should improve cost effectiveness without reducing student support. CONCLUSION: Implementing these tools and processes over the entire student’s learning package, will assist them to demonstrate progressive development through the course. Students will have learnt to understand feedback and integrate these skills for life-long learning.

The OO files : providing online formative feedback on Contract Law learning in a narrative setting

Feedback, both formative and summative, enables students to reflect on their understandings and to restructure their thinking to develop their capabilities. It can also encourage positive motivation and help boost self-esteem. Online multiple choice questions can be an efficient and effective means of providing timely formative feedback. At the same time, locating learning in a narrative environment can facilitate engaging and effective learning experiences. Narratives can help learners to navigate through information and support cognitive and imaginative engagement. This article will discuss The 00 Files, an online suite of modules containing multiple choice questions situated in the narrative of a fictional law firm. It notes student responses to the program and discusses lessons that may be learnt from its development which may be of assistance to academics considering the development of similar programs for their courses.

How Video Lecture Capture affects Student Engagement in a University Computer Programming Course: Attendance, Video Viewing Behaviours and Student Attitudes

Introduction
The use of video capture of lectures in Higher Education is not a recent occurrence with web based learning technologies including digital recording of live lectures becoming increasing commonly offered by universities throughout the world (Holliman and Scanlon, 2004). However in the past decade the increase in technical infrastructural provision including the availability of high speed broadband has increased the potential and use of videoed lecture capture. This had led to a variety of lecture capture formats including pod casting, live streaming or delayed broadcasting of whole or part of lectures.
Additionally in the past five years there has been a significant increase in the popularity of online learning, specifically via Massive Open Online Courses (MOOCs) (Vardi, 2014). One of the key aspects of MOOCs is the simulated recording of lecture like activities. There has been and continues to be much debate on the consequences of the popularity of MOOCs, especially in relation to its potential uses within established University programmes.
There have been a number of studies dedicated to the effects of videoing lectures.
The clustered areas of research in video lecture capture have the following main themes:
• Staff perceptions including attendance, performance of students and staff workload
• Reinforcement versus replacement of lectures
• Improved flexibility of learning
• Facilitating engaging and effective learning experiences
• Student usage, perception and satisfaction
• Facilitating students learning at their own pace
Most of the body of the research has concentrated on student and faculty perceptions, including academic achievement, student attendance and engagement (Johnston et al, 2012).
Generally the research has been positive in review of the benefits of lecture capture for both students and faculty. This perception coupled with technical infrastructure improvements and student demand may well mean that the use of video lecture capture will continue to increase in frequency in the next number of years in tertiary education. However there is a relatively limited amount of research in the effects of lecture capture specifically in the area of computer programming with Watkins 2007 being one of few studies . Video delivery of programming solutions is particularly useful for enabling a lecturer to illustrate the complex decision making processes and iterative nature of the actual code development process (Watkins et al 2007). As such research in this area would appear to be particularly appropriate to help inform debate and future decisions made by policy makers.
Research questions and objectives
The purpose of the research was to investigate how a series of lecture captures (in which the audio of lectures and video of on-screen projected content were recorded) impacted on the delivery and learning of a programme of study in an MSc Software Development course in Queen’s University, Belfast, Northern Ireland. The MSc is conversion programme, intended to take graduates from non-computing primary degrees and upskill them in this area. The research specifically targeted the Java programming module within the course. It also analyses and reports on the empirical data from attendances and various video viewing statistics. In addition, qualitative data was collected from staff and student feedback to help contextualise the quantitative results.
Methodology, Methods and Research Instruments Used
The study was conducted with a cohort of 85 post graduate students taking a compulsory module in Java programming in the first semester of a one year MSc in Software Development. A pre-course survey of students found that 58% preferred to have available videos of “key moments” of lectures rather than whole lectures. A large scale study carried out by Guo concluded that “shorter videos are much more engaging” (Guo 2013). Of concern was the potential for low audience retention for videos of whole lectures.
The lecturers recorded snippets of the lecture directly before or after the actual physical delivery of the lecture, in a quiet environment and then upload the video directly to a closed YouTube channel. These snippets generally concentrated on significant parts of the theory followed by theory related coding demonstration activities and were faithful in replication of the face to face lecture. Generally each lecture was supported by two to three videos of durations ranging from 20 – 30 minutes.
Attendance
The MSc programme has several attendance based modules of which Java Programming was one element. In order to assess the consequence on attendance for the Programming module a control was established. The control used was a Database module which is taken by the same students and runs in the same semester.
Access engagement
The videos were hosted on a closed YouTube channel made available only to the students in the class. The channel had enabled analytics which reported on the following areas for all and for each individual video; views (hits), audience retention, viewing devices / operating systems used and minutes watched.
Student attitudes
Three surveys were taken in regard to investigating student attitudes towards the videoing of lectures. The first was before the start of the programming module, then at the mid-point and subsequently after the programme was complete.
The questions in the first survey were targeted at eliciting student attitudes towards lecture capture before they had experienced it in the programme. The midpoint survey gathered data in relation to how the students were individually using the system up to that point. This included feedback on how many videos an individual had watched, viewing duration, primary reasons for watching and the result on attendance, in addition to probing for comments or suggestions. The final survey on course completion contained questions similar to the midpoint survey but in summative view of the whole video programme.
Conclusions and Outcomes
The study confirmed findings of other such investigations illustrating that there is little or no effect on attendance at lectures. The use of the videos appears to help promote continual learning but they are particularly accessed by students at assessment periods. Students respond positively to the ability to access lectures digitally, as a means of reinforcing learning experiences rather than replacing them. Feedback from students was overwhelmingly positive indicating that the videos benefited their learning. Also there are significant benefits to part recording of lectures rather than recording whole lectures. The behaviour viewing trends analytics suggest that despite the increase in the popularity of online learning via MOOCs and the promotion of video learning on mobile devices in fact in this study the vast majority of students accessed the online videos at home on laptops or desktops However, in part, this is likely due to the nature of the taught subject, that being programming.
The research involved prerecording the lecture in smaller timed units and then uploading for distribution to counteract existing quality issues with recording entire live lectures. However the advancement and consequential improvement in quality of in situ lecture capture equipment may well help negate the need to record elsewhere. The research has also highlighted an area of potentially very significant use for performance analysis and improvement that could have major implications for the quality of teaching. A study of the analytics of the viewings of the videos could well provide a quick response formative feedback mechanism for the lecturer. If a videoed lecture either recorded live or later is a true reflection of the face to face lecture an analysis of the viewing patterns for the video may well reveal trends that correspond with the live delivery.

20 years on - have student evaluations made a difference?

This paper poses the question, what impact have student evaluations (SE’s) of teaching had on the improvement in teaching in Australian tertiary education. The paper proposes to assess the effectiveness of SE’s through an investigations of the ways in which have been used in Australian tertiary education over the last 20 years or so. Three approaches are discussed: a) quality assurance - student evaluations used to ensure the quality of the learning environment in which learning takes place; b) quality improvement - student evaluations used as a diagnostic tool for individual teaching staff; and c) student feedback – student evaluations used to provide prospective and current students with information to help inform their choices and expectations. The discussions reviews some of the research and known practise in Australian universities. It concludes that SE’s have had a profound impact on pedagological understandings of tertiary education by placing the student at centre stage of teaching and learning. However it suggests that we do not know how SE’s have impacted on teaching effectiveness and improvement. The paper concludes by recommending further research to define critical success factors, a better understanding of how to make the student feedback and evaluation tools and mechanisms more approachable and meaningful for students, and an assessment of the ‘survey industry’ as a sustainable enterprise.

Is it worth the effort? How feedback influences students' subsequent submission of assessable work

This paper explores the extent to which students appear to their assessors to act on feedback they have received, and questions the assumption that providing feedback alone is sufficient to effect higher standards of work by students. Feedback provided to 51 undergraduate social work students, on two consecutive assignments involving a similar task, was examined to ascertain the number of problem areas noted from seven predefined categories. While the greatest increase in marks was associated with the greatest reductions in the number of problem areas identified in the comments, overall two-thirds of all students (66.7%) were awarded marks for both assignments within four percentage points. As such, this study found only limited support for the idea that students respond to feedback by making changes which are consistent with the intent of the feedback received. Hence the assumption that providing feedback alone is sufficient to effect higher standards of work by students was not supported. These findings invite educators to critically reflect on their own practices in providing feedback to students.

Closing the loop on student evaluations

It is generally agreed that good teaching is dependent on the nature of the learning goals and the quality of the environment in which learning takes place1,2. If this is the case, then planned as opposed to unplanned variations in learning experiences are the hallmark of good teaching. This raises a complex set of relations between assuring and improving teaching within an institution. The essential question for a University is: how do we assure that learning is maximised in all given contexts?

Benchmarking the use of student feedback was identified as a priority project at the ATN annual conference in February 2002. This paper presents a number of issues arising from a study of practices across the ATN in relation to collecting, analysing and using student feedback. The project involved working with those responsible for teaching and learning improvements at the operational level to identify strategies for a systematic approach to the use of student feedback for improving communication of actions arising from results.

The framework for the inquiry involved an exploration of student feedback systems as they operate for different cohorts of students, differing measures (or constructs), and at different levels within institutional structures. The framework also explored the various tracking and reporting systems by which results generated from student feedback systems at each level were utilised to develop strategies for improving teaching and learning. Findings from the study were used to inform recommendations in relation to internal practices within each University, as well as initiatives for benchmarking student evaluation results across universities within the ATN network.

Student responses to activities designed to develop generic professional skills

This paper reports on student responses to a range of assessment activities in a final-year engineering unit. Existing assessment activities were supplemented with new activities, the overall aim being to link the assessment activities more effectively to the material being studied, and to develop a range of generic skills important in professional engineering practice. A class survey was undertaken at the beginning of the semester to establish the initial attitudes to the new assessment activities. This was followed up with an end-of-semester survey to determine the change in perceived value of the assessment activities, and to collect student feedback regarding the activities. The perceived value of the assessment activities was determined using a Likert rating scale, while student feedback was collected using open-ended questions. The assessment activities evaluated were group work, case study investigation, report writing, oral presentation, group self-assessment, industrial interviews, and written reflective journals. The responses indicate that engineering students value a range of assessment activities. They value highly visits to real engineering organizations, and—contrary to popular belief —value and enjoy oral presentation exercises.

Rethinking models of feedback for learning: the challenge of design

Student feedback is a contentious and confusing issue throughout higher education institutions. This paper develops and analyses two models of feedback: the first is based on the origins of the term in the disciplines of engineering and biology. It positions teachers as the drivers of feedback. The second draws on ideas of sustainable assessment. This positions learners as having a key role in driving learning, and thus generating and soliciting their own feedback. It suggests that the second model equips students beyond the immediate task and does not lead to false expectations that courses cannot deliver. It identifies the importance of curriculum design in creating opportunities for students to develop the capabilities to operate as judges of their own learning.

Student opportunities in materials design and manufacture: Introducing a new manufacturing with composites course

The shift towards strong and lightweight fibre reinforced polymer-matrix composites for many high performance applications has resulted in an increasing need to expose students to composite design and manufacture courses in the undergraduate curriculum. In contrast, student exposure to composite materials is often still limited to a topic within a materials or manufacturing related course (unit). This paper presents the initial offering of a composite materials elective at Griffith University in Australia. The course also addresses environmental concerns through the inclusion of natural fibre composites. An evaluation of student perceptions is considered from Griffith’s Student Experience of Course (SEC) and separate Student Experience of Teaching (SET) surveys. These evaluations demonstrate the high level of student engagement with the course, but also highlighted areas for improvement, including the need to incorporate even more hands-on practical work. Interestingly, the inclusion of natural fibre composites and the related discussion surrounding environmental and societal issues are not focused on in student feedback.

New media data to identify student training needs

The main objective is to exhibit how usage data from new media can be used to assess areas where students need more help in creating their ETDs. After attending this session, attendees will be able to use usage data from new media, in conjunction with traditional assessment data, to identify strengths and weaknesses in ETD training and resources. The burgeoning ETD program at Florida International University (FIU) has provided many opportunities to experiment with assessment strategies and new media. The usage statistics from YouTube and the ETD LibGuide revealed areas of strength and weakness in the training resources and the overall ETD training initiative. With the ability to assess these materials, they have been updated to better meet student needs. In addition to these assessment tools, there are opportunities to connect these statistics with data from a common error checklist, student feedback from ETD workshops, and final ETD submission surveys to create a full-fledged outcome based assessment program for the ETD initiative.

Is Our Students Learning - QuestionMark: Extending E-Assessment Systems for Adapative Marking and Feedback

Our key contribution is a flexible, automated marking system that adds desirable functionality to existing E-Assessment systems. In our approach, any given E-Assessment system is relegated to a data-collection mechanism, whereas marking and the generation and distribution of personalised per-student feedback is handled separately by our own system. This allows content-rich Microsoft Word feedback documents to be generated and distributed to every student simultaneously according to a per-assessment schedule.

The feedback is adaptive in that it corresponds to the answers given by the student and provides guidance on where they may have gone wrong. It is not limited to simple multiple choice which are the most prescriptive question type offered by most E-Assessment Systems and as such most straightforward to mark consistently and provide individual per-alternative feedback strings. It is also better equipped to handle the use of mathematical symbols and images within the feedback documents which is more flexible than existing E-Assessment systems, which can only handle simple text strings.

As well as MCQs the system reliably and robustly handles Multiple Response, Text Matching and Numeric style questions in a more flexible manner than Questionmark: Perception and other E-Assessment Systems. It can also reliably handle multi-part questions where the response to an earlier question influences the answer to a later one and can adjust both scoring and feedback appropriately.

New question formats can be added at any time provided a corresponding marking method conforming to certain templates can also be programmed. Indeed, any question type for which a programmatic method of marking can be devised may be supported by our system. Furthermore, since the student’s response to each is question is marked programmatically, our system can be set to allow for minor deviations from the correct answer, and if appropriate award partial marks.

Responding to student evaluation of teaching : closing the loop

Our undergraduate curriculum focuses on preparing students to be critical thinkers, problem solvers, and knowledgeable, responsible information technology (IT) professionals. Students often question the relevance of core subjects which are designed to introduce and develop these attributes; such units regularly perform poorly in student evaluation of teaching. Although these subjects are strongly grounded in meeting industry expectations regarding employability skills, students do not value the learning as the curriculum is deemed non-technical and not worthy of inclusion in an IT qualification. As these are core subjects, improving student perceptions and learning outcomes is of critical importance. Simply responding to student feedback received via student evaluation of teaching had not resulted in any improvement; so a holistic approach was adopted. A workshop was organised to explore the issues and develop a strategy to improve outcomes and perceptions of these subjects. Rather than addressing student feedback specifically, the underlying issues that led to students’ negative perceptions were identified and addressed. Recommendations were implemented in 2014 resulting in improvements in student evaluation for relevant subjects. In this paper, we will discuss the process that was adopted to respond effectively to student evaluations, and explore the impact that this had on them.