Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

Who are our repeating students? Profiling students persisting after failure

Research on attrition has focused on the economic significance of low graduation rates in terms of costs to students (fees that do not culminate in a credential) and impact on future income. For a student who fails a unit and repeats the unit multiple times, the financial impact is significant and lasting (Bexley, Daroesman, Arkoudis & James 2013). There are obvious advantages for the timely completion of a degree, both for the student and the institution. Advantages to students include fee minimisation, enhanced engagement opportunities, effectual pathway to employment and a sense of worth, morale and cohort-identity benefits. Work undertaken by the QUT Analytics Project in 2013 and 2014 explored student engagement patterns capturing a variety of data sources and specifically, the use of LMS amongst students in 804 undergraduate units in one semester. Units with high failure rates were given further attention and it was found that students who were repeating a unit were less likely to pass the unit than students attempting it for the first time. In this repeating cohort, academic and behavioural variables were consistently more significant in the modelling than were any demographic variables, indicating that a student’s performance at university is far more impacted by what they do once they arrive than it is by where they come from. The aim of this poster session is to examine the findings and commonalities of a number of case studies that articulated the engagement activities of repeating students (which included collating data from Individual Unit Reports, academic and peer advising programs and engagement with virtual learning resources). Understanding the profile of the repeating student cohort is therefore as important as considering the characteristics of successful students so that the institution might be better placed to target the repeating students and make proactive interventions as early as possible.

Education and the relation to the outside: A little real reality

One of the aims of Deleuze. Guattari. Schizoanalysis. Education. is to focus on the radical reconfiguration that education is undergoing, impacting educator, administrator, institution and ‘sector’ alike. More to the point, it is the responses to that process of reconfiguration - this newly emerging assemblage - that are a key focal point in this issue. Essential to these responses, we propose, is Deleuze and Guattari’s method of schizonalysis, which offers a way to not only understand the rules of this new game, but also, hopefully, some escape from the promise of a brave new world of continuous education and motivation. A brave new world of digitised courses, impersonal and corporate expertise, updatable performance metrics, Massive Open Online Courses (MOOCs), learning analytics, transformative teaching and learning, online high-stakes testing in the name of transforming and augmenting human capital overlays the corporeal practices of institutional surveillance, examination and categorical sorting. A brave new world, importantly, where people’s continuous education is instituted less, or not simply, through disciplinary practices, and increasingly through a constant and continuous sampling and profiling of not simply performance but their activity, measured against the profiled activity of a ‘like’ age group, person, or an institution. This continuous education, including the sampling that accompanies it, we are all informed through various information and marketing campaigns, is in our best interest. An interest that is driven and governed by an ever-increasing corporatisation and monetisation of ‘the knowledge sector’, as well as an interest that is sustained through an ever-increasing, as well as continuous, debt.

Towards automated content analysis of discussion transcripts: A cognitive presence case

In this paper, we present the results of an exploratory study that examined the problem of automating content analysis of student online discussion transcripts. We looked at the problem of coding discussion transcripts for the levels of cognitive presence, one of the three main constructs in the Community of Inquiry (CoI) model of distance education. Using Coh-Metrix and LIWC features, together with a set of custom features developed to capture discussion context, we developed a random forest classification system that achieved 70.3% classification accuracy and 0.63 Cohen's kappa, which is significantly higher than values reported in the previous studies. Besides improvement in classification accuracy, the developed system is also less sensitive to overfitting as it uses only 205 classification features, which is around 100 times less features than in similar systems based on bag-of-words features. We also provide an overview of the classification features most indicative of the different phases of cognitive presence that gives an additional insights into the nature of cognitive presence learning cycle. Overall, our results show great potential of the proposed approach, with an added benefit of providing further characterization of the cognitive presence coding scheme.

An Integrative framework of the factors affecting process model understanding : a learning perspective

Process models are used by information professionals to convey semantics about the business operations in a real world domain intended to be supported by an information system. The understandability of these models is vital to them actually being used. After all, what is not understood cannot be acted upon. Yet until now, understandability has primarily been defined as an intrinsic quality of the models themselves. Moreover, those studies that looked at understandability from a user perspective have mainly conceptualized users through rather arbitrary sets of variables. In this paper we advance an integrative framework to understand the role of the user in the process of understanding process models. Building on cognitive psychology, goal-setting theory and multimedia learning theory, we identify three stages of learning required to realize model understanding, these being Presage, Process, and Product. We define eight relevant user characteristics in the Presage stage of learning, three knowledge construction variables in the Process stage and three potential learning outcomes in the Product stage. To illustrate the benefits of the framework, we review existing process modeling work to identify where our framework can complement and extend existing studies.

Reflections on the national trial of the draft Early Years Learning Framework and Educator's Guide

To promote and support informed discussion, we look here at the experiences of some services in the national trial of the draft Early Years Learning Framework, and the more recent trial of the supporting draft Educator's Guide. Reflecting on these experiences, the paper offers some examples of how a service can 'get started' with the EYLF.

Departures from tradition : the early years learning framework for Australia

In 2009, the Commonwealth Government of Australia published the first national learning framework for use with children aged birth to five years. The framework marks a departure from tradition in that it emphasizes intentional teaching, learning as well as child development, a particular type of play-based learning, outcomes, and equity. This article analyzes aspects of the document that depart from well established approaches to early childhood education in Australia and identifies challenges for educators who are required to use the document. It concludes that ongoing and supportive professional learning opportunities must accompany the introduction and enactment of the document.

Wellness education : an integrated theoretical framework for fostering transformative learning

Optimum Wellness involves the development, refinement and practice of lifestyle choices which resonate with personally meaningful frames of reference. Personal transformations are the means by which our frames of reference are refined across the lifespan. It is through critical reflection, supportive relationships and meaning making of our experiences that we construct and reconstruct our life paths. When individuals are able to be what they are destined to be or reach their higher purpose, then they are able to contribute to the world in positive and meaningful ways. Transformative education facilitates the changes in perspective that enable one to contemplate and travel a path in life that leads to self-actualisation. This thesis argues for an integrated theoretical framework for optimum Wellness Education. It establishes a learner centred approach to Wellness education in the form of an integrated instructional design framework derived from both Wellness and Transformative education constructs. Students’ approaches to learning and their study strategies in a Wellness education context serve to highlight convergences in the manner in which students can experience perspective transformation. As they learn to critically reflect, pursue relationships and adapt their frames of reference to sustain their pursuit of both learning and Wellness goals, strengthening the nexus between instrumental and transformative learning is a strategically important goal for educators. The aim of this exploratory research study was to examine those facets that serve to optimise the learning experiences of students in a Wellness course. This was accomplished through three research issues: 1) What are the relationships between Wellness, approaches to learning and academic success? 2) How are students approaching learning in an undergraduate Wellness subject? Why are students approaching their learning in the ways they do? 3) What sorts of transformations are students experiencing in their Wellness? How can transformative education be formulated in the context of an undergraduate Wellness subject? Subsequent to a thorough review of the literature pertaining to Wellness education, a mixed method embedded case study design was formulated to explore the research issues. This thesis examines the interrelationships between student, content and context in a one semester university undergraduate unit (a coherent set of learning activities which is assigned a unit code and a credit point value). The experiences of a cohort of 285 undergraduate students in a Wellness course formed the unit of study and seven individual students from a total of sixteen volunteers whose profiles could be constructed from complete data sets were selected for analysis as embedded cases. The introductory level course required participants to engage in a personal project involving a behaviour modification plan for a self-selected, single dimension of Wellness. Students were given access to the Standard Edition Testwell Survey to assess and report their Wellness as a part of their personal projects. To identify relationships among the constructs of Self-Regulated Learning (SRL), Wellness and Student Approaches to Learning (SAL) a blend of quantitative and qualitative methods to collect and analyse data was formulated. Surveys were the primary instruments for acquiring quantitative data. Sources included the Wellness data from Testwell surveys, SAL data from R-SPQ surveys, SRL data from MSLQ surveys and student self-evaluation data from an end of semester survey. Students’ final grades and GPA scores were used as indicators of academic performance. The sources of qualitative data included subject documentation, structured interview transcripts and open-ended responses to survey items. Subsequent to a pilot study in which survey reliability and validity were tested in context, amendments to processes for and instruments of data collection were made. Students who adopted meaning oriented (deep/achieving) approaches tended to assess their Wellness at a higher level, seek effective learning strategies and perform better in formal study. Posttest data in the main study revealed that there were significant positive statistical relationships between academic performance and total wellness scores (rs=.297, n=205, p<.01). Deep (rs=.343, n=137, p<.01) and achieving (rs=.286, n=123, p<.01) approaches to learning also significantly correlated with Wellness whilst surface approaches had negative correlations that were not significant. SRL strategies including metacognitive selfregulation, effort, help-seeking and critical thinking were increasingly correlated with Wellness. Qualitative findings suggest that while all students adopt similar patterns of day to day activities for example attending classes, taking notes, working on assignments the level of care with which these activities is undertaken varies considerably. The dominant motivational trigger for students in this cohort was the personal relevance and associated benefits of the material being learned and practiced. Students were inclined to set goals that had a positive impact on affect and used “sense of happiness” to evaluate their achievement status. Students who had a higher drive to succeed and/or understand tended to have or seek a wider range of strategies. Their goal orientations were generally learning rather than performance based and barriers presented a challenge which could be overcome as opposed to a blockage which prevented progress. Findings from an empirical analysis of the Testwell data suggest that a single third order Wellness construct exists. A revision of the instrument is necessary in order to juxtapose it with the chosen six dimensional Wellness model that forms the foundation construct in the course. Further, redevelopment should be sensitive to the Australian context and culture including choice of language, examples and scenarios used in item construction. This study concludes with an heuristic for use in Wellness education. Guided by principles of Transformative education theory and behaviour change theory, and informed by this representative case study the “CARING” heuristic is proposed as an instructional design tool for Wellness educators seeking to foster transformative learning. Based upon this study, recommendations were made for university educators to provide authentic and personal experiences in Wellness curricula. Emphasis must focus on involving students and teachers in a partnership for implementing Wellness programs both in the curriculum and co-curricularly. The implications of this research for practice are predicated on the willingness of academics to embrace transformative learning at a personal level and a professional one. To explore students’ profiles in detail is not practical however teaching students how to guide us in supporting them through the “pain” of learning is a skill which would benefit them and optimise the learning and teaching process. At a theoretical level, this research contributes to an ecological theory of Wellness education as transformational change. By signposting the wider contexts in which learning takes place, it seeks to encourage changing paradigms to ones which harness the energy of each successive contextual layer in which students live. Future research which amplifies the qualities of individuals and groups who are “Well” and seeks the refinement and development of instruments to measure Wellness constructs would be desirable for both theoretical and applied knowledge bases. Mixed method Wellness research derived and conducted by teams that incorporate expertise from multiple disciplines such as psychology, anthropology, education, and medicine would enable creative and multi-perspective programs of investigation to be designed and implemented. Congruences and inconsistencies in health promotion and education would provide valuable material for strengthening the nexus between transformational learning and behaviour change theories. Future development of and research on the effectiveness of the CARING heuristic would be valuable in advancing the understanding of pedagogies which advance rather than impede learning as a transformative process. Exploring pedagogical models that marry with transformative education may render solutions to the vexing challenge of teaching and learning in diverse contexts.

Applying a social justice framework to ensure good practice in monitoring student learning engagement

A current Australian Learning and Teaching Council (ALTC) funded action research project aims to provide a set of practical resources founded on a social justice framework, to guide good practice for monitoring student learning engagement (MSLE) in higher education. The project involves ten Australasian institutions, eight of which are engaged in various MSLE type projects. A draft framework, consisting of six social justice principles which emerged from the literature has been examined with reference to the eight institutional approaches for MSLE in conjunction with the personnel working on these initiatives during the first action research cycle. The cycle will examine the strategic and operational implications of the framework in each of the participating institutions. Cycle 2 will also build capacity to embed the principles within the institutional MSLE program and will identify and collect examples and resources that exemplify the principles in practice. The final cycle will seek to pilot the framework to guide new MSLE initiatives. In its entirety, the project will deliver significant resources to the sector in the form of a social justice framework for MSLE, guidelines and sector exemplars for MSLE. As well as increasing the awareness amongst staff around the criticality of transition to university (thereby preventing attrition) and the significance of the learning and teaching agenda in enhancing student engagement, the project will build leadership capacity within the participating institutions and provide a knowledge base and institutional capacity for the Australasian HE sector to deploy the deliverables that will safeguard student learning engagement At this early stage of the project the workshop session provides an opportunity to discuss and examine the draft set of social justice principles and to discuss their potential value for the participants’ institutional contexts. Specifically, the workshop will explore critical questions associated with the principles.

The pedagogical, linguistic and content features of popular English language learning websites in China : a framework for analysis and design

As increasing numbers of Chinese language learners choose to learn English online (CNNIC, 2012), there is a need to investigate popular websites and their language learning designs. This paper reports on the first stage of a study that analysed the pedagogical, linguistic and content features of 25 Chinese English Language Learning (ELL) websites ranked according to their value and importance to users. The website ranking was undertaken using a system known as PageRank. The aim of the study was to identify the features characterising popular sites as opposed to those of less popular sites for the purpose of producing a framework for ELL website design in the Chinese context. The study found that a pedagogical focus with developmental instructional materials accommodating diverse proficiency levels was a major contributor to website popularity. Chinese language use for translations and teaching directives and intermediate level English for learning materials were also significant features. Content topics included Anglophone/Western and non-Anglophone/Eastern contexts. Overall, popular websites were distinguished by their mediation of access to and scaffolded support for ELL.

Pursuing better childhoods and futures through curriculum : utopian visions in the development of Australia’s early years learning framework

In recent years, globalised curriculum discourses have given rise to local curriculum texts that convey and produce particularised imaginings and narratives, as well as hopes for, and expectations of, young children, their childhoods and their futures. In this article, the authors employ concepts from utopian studies and Deleuzeguattarian concepts of assemblage, rhizomes and lines (supple, rigid and lines of flight) to undertake a preliminary and partial rhizomatic mapping of utopian visions of better childhoods and futures evident in the development of the Early Years Learning Framework, Australia’s first national curriculum for early childhood settings. Drawing on the perspective of policy makers, News Corporation, the public, politicians, academics and practitioners who shaped the development of the Framework, the authors seek alternatives to the well-rehearsed dichotomies that so often characterise and confine curriculum politics and debates, and ways of exploring spaces between the possible and not (yet) possible.

A social justice framework for safeguarding student learning engagement

The set of social justice principles and the Social Justice Framework (SJF), developed as resources for the sector as part of an Australian Government Office for Learning and Teaching project, adopt a recognitive approach to social justice and emphasise full participation and contribution within democratic society (Gale, 2000; Gale & Densmore, 2000). The SJF is contained within the major deliverable of the project, which is A Good Practice Guide for Safeguarding Student Learning Engagement (Nelson & Creagh, 2013) and is focused on good practice for activities that monitor student learning engagement and identify students at risk of disengaging in their first year. Examination of the social justice literature and its application to the higher education sector produced a set of five principles: Self-determination, Rights, Access, Equity and Participation. Each principle was defined and elucidated by a rationale and implications for practice, thus completing the SJF. The framework: reflects the notions of equity and social justice; provides a strategic approach for safeguarding engagement activities; and is supported by a suite of resources for practice and practitioners. The aim of this poster session is to engage in conversations about the SJF and how it might be applied to other types of student engagement activities critical to the first year of university life, such as orientation and transition programs, teamwork activities, peer programs and other academic support initiatives.

Valuing student and community voices in the university : action research as a framework for community service-learning

The Community Service-learning Lab (the Lab) was initiated as a university-wide service-learning experience at an Australian university. The Lab engages students, academics, and key community organisations in interdisciplinary action research projects to support student learning and to explore complex and ongoing problems nominated by the community partners. The current study uses feedback from the first offering of the Lab and focuses on exploring student experiences of the service learning project using an action research framework. Student reflections on this experience have revealed some positive outcomes of the Lab such as an appreciation for positive and strengths-based change. These outcomes are corroborated by collected reflections from community partners and academics. The students also identified challenges balancing the requirements for assessment and their goals to serve the community partner’s needs. This feedback has provided vital information for the academic team, highlighting the difficulties in balancing the agenda of the academic framework and the desire to give students authentic experiences.

Collaborative learning and critical thinking in technology-enhanced environments : an instructional design framework

The nature and characteristics of how learners learn today are changing. As technology use in learning and teaching continues to grow, its integration to facilitate deep learning and critical thinking becomes a primary consideration. The implications for learner use, implementation strategies, design of integration frameworks and evaluation of their effectiveness in learning environments cannot be overlooked. This study specifically looked at the impact that technology-enhanced learning environments have on different learners’ critical thinking in relation to eductive ability, technological self-efficacy, and approaches to learning and motivation in collaborative groups. These were explored within an instructional design framework called CoLeCTTE (collaborative learning and critical thinking in technology-enhanced environments) which was proposed, revised and used across three cases. The field of investigation was restricted to three key questions: 1) Do learner skill bases (learning approach and eductive ability) influence critical thinking within the proposed CoLeCTTE framework? If so, how?; 2) Do learning technologies influence the facilitation of deep learning and critical thinking within the proposed CoLeCTTE framework? If so, how?; and 3) How might learning be designed to facilitate the acquisition of deep learning and critical thinking within a technology-enabled collaborative environment? The rationale, assumptions and method of research for using a mixed method and naturalistic case study approach are discussed; and three cases are explored and analysed. The study was conducted at the tertiary level (undergraduate and postgraduate) where participants were engaged in critical technical discourse within their own disciplines. Group behaviour was observed and coded, attributes or skill bases were measured, and participants interviewed to acquire deeper insights into their experiences. A progressive case study approach was used, allowing case investigation to be implemented in a "ladder-like" manner. Cases 1 and 2 used the proposed CoLeCTTE framework with more in-depth analysis conducted for Case 2 resulting in a revision of the CoLeCTTE framework. Case 3 used the revised CoLeCTTE framework and in-depth analysis was conducted. The findings led to the final version of the framework. In Cases 1, 2 and 3, content analysis of group work was conducted to determine critical thinking performance. Thus, the researcher used three small groups where learner skill bases of eductive ability, technological self-efficacy, and approaches to learning and motivation were measured. Cases 2 and 3 participants were interviewed and observations provided more in-depth analysis. The main outcome of this study is analysis of the nature of critical thinking within collaborative groups and technology-enhanced environments positioned in a theoretical instructional design framework called CoLeCTTE. The findings of the study revealed the importance of the Achieving Motive dimension of a student’s learning approach and how direct intervention and strategies can positively influence critical thinking performance. The findings also identified factors that can adversely affect critical thinking performance and include poor learning skills, frustration, stress and poor self-confidence, prioritisations over learning; and inadequate appropriation of group role and tasks. These findings are set out as instructional design guidelines for the judicious integration of learning technologies into learning and teaching practice for higher education that will support deep learning and critical thinking in collaborative groups. These guidelines are presented in two key areas: technology and tools; and activity design, monitoring, control and feedback.