Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

Facebook for adults ? A longitudinal study of MBA student’s use of an enterprise social network for collaborative learning

The rapid pace of social media means that our understanding of the way in which it facilitates the learning process continues to lag. The findings of a longitudinal study of an executive MBA cohort over the period of eight months in their use of the social media application is presented. Over time the ownership and use of the Yammer site shifted to become student driven and facilitated. The motivations behind the site’s use, perceived advantages and disadvantages and changes in usage patterns are documented. The case provides a useful insight into the way in which students used this technology to facilitate their learning goals and how patterns of behaviour changed in response to the changing needs of the cohort.

Design thinking frameworks as transformative cross-disciplinary pedagogy

Final report for the Australian Government Office for Learning and Teaching. "This seed project ‘Design thinking frameworks as transformative cross-disciplinary pedagogy’ aimed to examine the way design thinking strategies are used across disciplines to scaffold the development of student attributes in the domain of problem solving and creativity in order to enhance the nation’s capacity for innovation. Generic graduate attributes associated with innovation, creativity and problem solving are considered to be amongst the most important of all targeted attributes (Bradley Review of Higher Education, 2009). The project also aimed to gather data on how academics across disciplines conceptualised design thinking methodologies and strategies. Insights into how design thinking strategies could be embedded at the subject level to improve student outcomes were of particular interest in this regard. A related aim was the investigation of how design thinking strategies could be used by academics when designing new and innovative subjects and courses." Case Study 3: QUT Community Engaged Learning Lab Design Thinking/Design Led Innovation Workshop by Natalie Wright Context "The author, from the discipline area of Interior Design in the QUT School of Design, Faculty of Creative Industries, is a contributing academic and tutor for The Community Engaged Learning Lab, which was initiated at Queensland University of Technology in 2012. The Lab facilitates university-wide service-learning experiences and engages students, academics, and key community organisations in interdisciplinary action research projects to support student learning and to explore complex and ongoing problems nominated by the community partners. In Week 3, Semester One 2013, with the assistance of co-lead Dr Cara Wrigley, Senior Lecturer in Design led Innovation, a Masters of Architecture research student and nine participating industry-embedded Masters of Research (Design led Innovation) facilitators, a Design Thinking/Design led Innovation workshop was conducted for the Community Engaged Learning Lab students, and action research outcomes published at 2013 Tsinghua International Design Management Symposium, December 2013 in Shenzhen, China (Morehen, Wright, & Wrigley, 2013)."

But how do we know they watched it? Adapting to the flipped classroom conundrum

Those who teach film and media need to use screen content to illustrate their subjects. For example, students want illustrations to accompany lectures on film or television genres. Our experience has been that student access to the film and television screen content underpinning a study of genres is not only desirable but is, in fact, crucial for effective teaching and learning outcomes. Not so long ago, a screening during or at the completion of a lecture was the expected method by which educators delivered screen content to illustrate their teaching. Even if student attendance fluctuated from week to week a quick head count confirmed that a certain number of students were physically present. It was assumed that this physical attendance encouraged students to reflect upon and contextualize the material post lecture. While simply attending a lecture will not translate into actual student learning, it does demonstrate a willingness by students to engage with the course content by making a commitment to attend a scheduled and recurring lecture and screening program. However, as flipped classroom models gain acceptance in educational institutions, this traditional lecture-screening model is giving way to online, off-site, and student-controlled mechanisms for screen content delivery and viewing. Nevertheless, care should be taken when assessing how online delivery translates into student engagement and learning. As Junco (2012) points out, “it’s not the technology that generates learning, but the ways in which the technology are used.” Discussed, debated, and embraced to varying degrees by educators, there remains no definitive model for the flipped classroom – although many models involve ‘flipping’ content and knowledge acquisition (including viewing films and television shows) from scheduled on-campus classes to online material viewed by students in advance of an on-campus lecture or class. The classroom or tutorial room then becomes a space to problem-solve, engage in collaborate learning, and advance and explain concepts. From an institutional perspective, the flipped classroom model could deliver an additional benefit beyond immediate pedagogical concerns. Tucker (2012) suggests through the flipped classroom model “all aspects of instruction can be rethought to best maximize the scarcest learning resource — time.” The narrative most often associated with this shift is that the move to online content delivery of lecture and cinematic / televisual material may also provide educators with more time to do other work such as engage in research, plan strategies to empower students. Experimentation with the flipped classroom model is playing out in various educational institutions. Yet several core concerns remain — one of these concerns is the crucial question of whether an online/digital flipped approach is more effective for student engagement and learning than the traditional lecture-screening mode for screen content delivery. Some urge caution in this regard, arguing that “new technology isn’t always supported by change management and professional development to ensure that digital isn’t just a goal within itself, but actually helps to transform education” (Fleming cited in Blain 2014). The most fundamental concern remains how do lecturers, instructors, and tutors know students have watched the films and television shows associated with a subject? The remainder of this discussion deals with these concerns, and possible solutions offered, through an analysis of the Film, Television and Screen Genres subject at the Queensland University of Technology (QUT) in Brisbane, Queensland.

Learning through feedback loop metaphors

The metaphor of a feedback loop underpinned a significant curriculum change in a first year teacher-education unit. Assessment for Learning (AfL) practices such as discussing examples of previous student work and giving peer feedback were embedded within the curriculum design. The metaphor of a feedback loop connected these AfL practices into a purposeful process that informed student learning as well as tutor learning about student understanding, that then informed the next teaching episode. Student teachers (n=350) in twelve tutorial groups taught by eight university tutors were able to develop a shared understanding of quality performances before completing each assessment task. As well as providing ongoing insights to improve teaching, data from this action research project enabled the participant tutor-researchers to interrogate the concept of feedback loops. The researchers theorised sociocultural feedback loops as emergent, entangled and dynamic moves in a dance of knowing during which participants negotiated meaning and identities of capability.

The QUT Training Model for Peer Leader Capacity Building

QUT has enacted a university-wide Peer Program’s Strategy which aims to improve student success and graduate outcomes. A component of this strategy is a training model providing relevant, quality-assured and timely training for all students who take on leadership roles. The training model is designed to meet the needs of the growing scale and variety of peer programs, and to recognise the multiple roles and programs in which students may be involved during their peer leader journey. The model builds peer leader capacity by offering centralised, beginning and ongoing training modules, delivered by in-house providers, covering topics which prepare students to perform their role safely, inclusively, accountably and skilfully. The model also provides efficiencies by differentiating between ‘core competency' and ‘program-specific’ modules, thus avoiding training duplication across multiple programs, and enabling training to be individually and flexibly formatted to suit the specific and unique needs of each program.

An ePortfolio Approach: Supporting Critical Reflection for Pedagogic Innovation

This chapter investigates the capacity of a well-supported holistic ePortfolio program, the QUT Student ePortfolio Program (QSeP), to support critical reflection for pedagogic innovation in higher education, by exploring practice examples. The chapter looks across faculty and discipline areas to illustrate a range of ePortfolio learning case studies, which have led pedagogical innovation across a whole institution, to enhance student learning and support academic teaching. The ePortfolio strategies discussed support innovation in learning and teaching where academics use the ePortfolio approach in different ways to develop connectedness (productive pedagogies) within learning. Students are supported to develop awareness of the connections between formal and informal learning opportunities and between their learning and personal and professional goals. Students are guided to understand what they have learned and how they have learned in terms of generic employability skills or graduate attributes and also in relation to professional standards and competencies and personal goals. In essence, the ePortfolio-supported pedagogy creates capstone events enabling students to develop a professional identity and understanding of ongoing professional development. The examples are drawn from distinct discipline areas and illustrate the capacity of ePortfolio to underpin pedagogic innovation across discipline areas: • Bachelor of Information Technology—the ePortfolio approach supports students to explore the IT industry as a means of clarifying personal expectations and goals, thereby enhancing student potential in the course c• Bachelor of Nursing and Master of Nursing Science—students develop a professional ePortfolio to show development of the nursing competencies • Master of Information Technology—Library and Information students compile a Professional Portfolio for assessment in the Professional Practice subject • Bachelor of Laws—Virtual Law Placement (VLP) is a unit of study that challenges students to critically reflect on their performance and development duringthe work placement Each case study illustrates the academic teaching goal and student ePortfolio task in context. Issues, challenges and support strategies are identified. Comments from the students and their lecturers give an indication of the effectiveness of the ePortfolio approach to meet learning and teaching goals.

Playing Monopoly™ with real estate students

This paper outlines the results from a study into the educational use of the board game Monopoly City™ in a first year real estate unit. This game play was introduced as a fun and interactive way of achieving a number of desired outcomes including: introduction of foundational threshold concepts in real estate education; introduction of problem solving and critical analysis skills; early acculturation of real estate students to enhance student retention; early team building within the student cohort; and enhanced engagement of first year students and, all in an engaging and entertaining way. Results from this two-stage research project are encouraging. The students participating in this project have demonstrated explicit linkages between their Monopoly City™ experiences and foundation urban economic and valuation theories. Students are also recognising the role strategy and chance play in the real estate sector. Findings from this project and key success factors are presented.

Employability skills: Perspectives from a knowledge-intensive industry

Purpose: While the global education debate remains focused on graduate skills and employability, the absence of a shared language between student, academic and industry stakeholder groups means that defining industry skills requirements is both essential and difficult. The aim of this study was to assess graduate skills requirements in a knowledge intensive industry from a demand perspective as distinct from a curriculum (supply) viewpoint. Design/methodology/approach: Skills items were derived from a breadth of disciplines across academic, policy and industry literature. CEOs and senior managers in the innovation and commercialisation industry were surveyed regarding perceptions of skills in graduates and skills in demand by the firm. Two rounds of exploratory factor analyses were undertaken to examine employers’ perceptions of the skills gap. Findings: First order analysis resolved 10 broad constructs that represent cognitive, interpersonal and intrapersonal skills domains as applied in this industry. Knowledge, leadership and interprofessional collaboration feature as prominent skills. Second order analysis revealed employers’ perceptions of graduate skills specifically centre on organisational fit and organisational success. An over-arching theme relates to performance of the individual in organisations. Research limitations/implications: Our findings suggest that the discourse on employability and the design of curriculum need to shift from instilling lists of skills towards enabling graduates to perform in a diversity of workplace contexts and expectations centred on organisational purpose. Originality/value: In contrast to the heterogeneous nature of industry surveys, we targeted a homogenous sector that is representative of knowledge intensive industries. This study contributes to the broader stakeholder dialogue of the value and application of graduate skills in this and other industry sectors.

Engaging nursing students in a bioscience unit using a web-based response system, GoSoapBox

Nursing students used GoSoapBox, a web-based student response system to poll responses to multiple choice questions (MCQs) presented during bioscience lectures. Participation in GoSoapBox appears to have facilitated student engagement, interaction and learning. The majority of students surveyed appreciated the immediate feedback to the student responses and being able to participate anonymously. The use of this tool facilitated collaborative group and class discussion and clarification around any misconceptions or challenging concepts. Information collected using GoSoapBox provided the academic with feedback allowing for reflection, adjustment and improvement in framing of formative and summative MCQs.

Facebook: A basis for forming a peer learning community for nursing students in a bioscience unit

First year nursing students commonly find bioscience to be challenging. A Facebook community site was established to support and engage these students. The site was facilitated by virtual peer mentors and the unit coordinator. The high participation rate and the strong recommendation to future students indicated that the site successfully enabled student interaction and engagement with their learning. The students found it to be a readily accessible network and valued the useful resources and learning strategies provided by their peers. The sharing of both learning challenges and successful learning practices can help students build a sense of belonging and an understanding of academic practices and behaviours that can contribute to their learning success at university.

Engaging nursing students in a bioscience unit using a web-based response system, GoSoapBox

Nursing students used GoSoapBox, a web-based student response system to poll responses to multiple choice questions (MCQs) presented during bioscience lectures. Participation in GoSoapBox appears to have facilitated student engagement, interaction and learning. The majority of students surveyed appreciated the immediate feedback to the student responses and being able to participate anonymously. The use of this tool facilitated collaborative group and class discussion and clarification around any misconceptions or challenging concepts. Information collected using GoSoapBox provided the academic with feedback allowing for reflection, adjustment and improvement in framing of formative and summative MCQs.

Audience response systems can facilitate communal course feedback

Background At Queensland University of Technology (QUT), the Bachelor of Radiation Therapy course evaluation has previously suffered from low online survey participation rates. A communal instantaneous feedback event using an audience response system (ARS) was evaluated as a potential solution to this problem. The aims of the project were to determine the extent to which this feedback event could be facilitated by ARS technology and to evaluate the impact the technology made on student satisfaction and engagement. Methods Students were invited to a timetabled session to provide feedback on individual study units and the course overall. They provided quantitative Likert-style responses to prompts for each unit and the course using an ARS as well as anonymous typed qualitative comments. Data collection was performed live so students were able to view collective class responses. This prompted further discussion and enabled a prospective action plan to be developed. To inform future ARS use, students were asked for their opinions on the feedback method. Results Despite technological difficulties, student evaluation indicated that all responders enjoyed the session and the opportunity to view the combined responses. All students felt that useful feedback was generated and that this method should be used in the future. The student attendance and response rates were high, and it was clear that the session had led to the development of some insightful qualitative feedback comments. Conclusions: An ARS contributed well to the collection of course feedback in a communal and interactive environment. Students found it enjoyable to use, and it helped to stimulate useful qualitative comments

Peer mentoring for radiotherapy planning skills development: A pilot study

Introduction: This study aimed to determine the potential role and guidelines for implementation of skill-based peer mentoring for radiotherapy planning education. Methods: After four weekly mentoring sessions, both Year 3 mentors (n=9) and Year 2 mentees (n=9) were invited to complete a short online questionnaire relating to the impact of the initiative. The tool contained a mixture of Likert-style questions concerning student enjoyment and perceived usefulness of the initiative as well as more qualitative open questions that gathered perceptions of the peer mentoring process, implementation methods and potential future scope. Results: Several key discussion themes related to benefits to each stakeholder group, challenges arising, improvements and potential future directions. There were high levels of enjoyment and perceived value of the mentoring from both sides with 100% of the 18 respondents enjoying the experience. The informal format encouraged further learning, while mentors reported acquisition of valuable skills and gains in knowledge. Conclusions: Peer mentoring has a valuable and enjoyable role to play in radiotherapy planning training and helps consolidate theoretical understanding for experienced students. An informal approach allows for students to adopt the most appropriate mentoring model for their needs while providing them with a free space to engender additional discussion.

A virtual radiation therapy workflow training simulation

Aim Simulation forms an increasingly vital component of clinical skills development in a wide range of professional disciplines. Simulation of clinical techniques and equipment is designed to better prepare students for placement by providing an opportunity to learn technical skills in a “safe” academic environment. In radiotherapy training over the last decade or so this has predominantly comprised treatment planning software and small ancillary equipment such as mould room apparatus. Recent virtual reality developments have dramatically changed this approach. Innovative new simulation applications and file processing and interrogation software have helped to fill in the gaps to provide a streamlined virtual workflow solution. This paper outlines the innovations that have enabled this, along with an evaluation of the impact on students and educators. Method Virtual reality software and workflow applications have been developed to enable the following steps of radiation therapy to be simulated in an academic environment: CT scanning using a 3D virtual CT scanner simulation; batch CT duplication; treatment planning; 3D plan evaluation using a virtual linear accelerator; quantitative plan assessment, patient setup with lasers; and image guided radiotherapy software. Results Evaluation of the impact of the virtual reality workflow system highlighted substantial time saving for academic staff as well as positive feedback from students relating to preparation for clinical placements. Students valued practice in the “safe” environment and the opportunity to understand the clinical workflow ahead of clinical department experience. Conclusion Simulation of most of the radiation therapy workflow and tasks is feasible using a raft of virtual reality simulation applications and supporting software. Benefits of this approach include time-saving, embedding of a case-study based approach, increased student confidence, and optimal use of the clinical environment. Ongoing work seeks to determine the impact of simulation on clinical skills.