Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

A conceptual approach to track design changes within a multi-disciplinary BIM environment

Successful management of design changes is critical for the efficient delivery of construction projects. Building Information Modeling (BIM) is envisioned to play an important role in integrating design, construction and facility management processes through coordinated changes throughout the project life-cycle. BIM currently provides significant benefits in coordinating changes across different views in a single model, and identifying conflicts between different discipline-specific models. However, current BIM tools provide limited support in managing changes across several discipline-specific models. This paper describes an approach to represent, coordinate, and track changes within a collaborative multi-disciplinary BIM environment. This approach was informed by a detailed case study of a large, complex, fast-tracked BIM project where we investigated numerous design changes, analyzed change management processes, and evaluated existing BIM tools. Our approach characterises design changes in an ontology to represent changed component attributes, dependencies between components, and change impacts. It explores different types of dependencies amongst different design changes and describes how a graph based approach and dependency matrix could assist with automating the propagation and impact of changes in a BIM-based project delivery process.

Research on conceptual modelling : less known knowns and more unknown unknowns, please

In this invited paper I describe some personal views on the research field of conceptual modelling. I argue that the field has become entrenched in some “bad habits” that usually emerge in evolved paradigms and that we need to proactively pursue a dual research strategy incorporating new and different avenues that lead us to novel and impactful research contexts of conceptual modelling. I provide a framework that can guide this exploration and finish with some recommendations about how conceptual modelling research programs could proceed.

Conceptual beliefs about human values and their implications: Human nature beliefs predict value importance, value trade-offs, and responses to value-laden rhetoric

Beliefs that may underlie the importance of human values were investigated in 4 studies, drawing on research that distinguishes natural-kind (natural), nominal-kind (conventional), and artifact (functional) beliefs. Values were best characterized by artifact and nominal-kind beliefs, as well as a natural-kind belief specific to the social domain, “human nature” (Studies 1 and 2). The extent to which values were considered central to human nature was associated with value importance in both Australia and Japan (Study 2), and experimentally manipulating human nature beliefs influenced value importance (Study 3). Beyond their association with importance, human nature beliefs predicted participants’ reactions to value trade-offs (Study 1) and to value-laden rhetorical statements (Study 4). Human nature beliefs therefore play a central role in the psychology of values.

Unlearning leadership in the conceptual age

"We live in times in which unlearning has become as important as learning. Dan Pink has called these times the Conceptual Age,i to distinguish them from the Knowledge/Information Age in which many of us were born and educated. Before the current Conceptual Age, the core business of learning was the routine accessing of information to solve routine problems, so there was real value in retaining and reusing the templates taught to us at schools and universities. What is different about the Conceptual Age is that it is characterised by new cultural forms and modes of consumption that require us to unlearn our Knowledge/Information Age habits to live well in our less predictable social world. The ‘correct’ way to write, for example, is no longer ‘correct’ if communicating by hypertext rather than by essay or letter. And who would bother with an essay or a letter or indeed a pen these days? Whether or not we agree that the Conceptual Age, amounts to the first real generation gap since rock and roll, as Ken Robinson claims,ii it certainly makes unique demands of educators, just as it makes unique demands of the systems, strategies and sustainability of organisations. Foremost among these demands, according to innovation analyst Charlie Leadbeater,iii is to unlearn the idea that we are becoming a more knowledgeable society with each new generation. If knowing means being intimately familiar with the knowledge embedded in the technologies we use in our daily lives, then, Leadbeater says, we have never been more ignorant.iv He reminds us that our great grandparents had an intimate knowledge of the technologies around them, and had no problem with getting the butter churn to work or preventing the lamp from smoking. Few of us would know what to do if our mobile phones stopped functioning, just as few of us know what is ‘underneath’ or ‘behind’ the keys of our laptops. Nor, indeed, do many of us want to know. But this means that we are all very quickly reduced to the quill and the lamp if we lose our power sources or if our machines cease to function. This makes us much more vulnerable – as well as much more ignorant in relative terms – than our predecessors."

Establishing the conceptual and practical foundations of a new form of theatre for development: Theatre in conversations

This research was developed between Australia and Papua New Guinea (PNG) over two years investigating ways in which theatre for development could be held accountable for the claims it makes especially in PNG. The motivation to improve theatre for development (TfD) practice was triggered by the desire to enhance the democratic processes of collaboration and co–creativity often lacking in TfD activity in Papua New Guinea. Through creative practice as research and reflective processes, working with established and experienced local community theatre practitioners, a new form of theatre for development, Theatre in Conversations evolved. This form integrated three related genres of TfD including process drama, community theatre and community conversations. The suitability and impact of Theatre in Conversations was tested in three remote villages in PNG. Findings and outputs from the study have the potential to be used by theatre for development practitioners in other countries.

Made by motion: A conceptual framework for abstracted animation derived from motion-captured movements

Despite an increasing number of acclaimed abstract animations being created through the application of motion capture technologies there has been little detailed documentation and analysis of this approach for abstract animation production. More specifically, it is unclear what the key considerations are, and what issues practitioners might face, when integrating motion capture movement data into their practice. In response to this issue this study explored and documented the practice of generating abstract visual and temporal artefacts from motion captured dance movements that compose abstract animated short films. The study has resulted in a possible framework for this form of practice and outlines five key considerations which should be taken into account by practitioners who use motion capture in the production of abstract animated short films.

The role of design in the future of digital channels: Conceptual insights and future research directions

Purpose In the past channel literature has looked to other disciplines in developing and refining their theories, models and methods in order to evolve the field. This paper traces such history and highlights the substantial changes caused by the digital age. In light of this, the inclusion of design theory into future channel management is presented to overcome existing concerns. Design/methodology/approach A comprehensive review of literature on the history of channels, the emotional experience (people), limitations of digital innovation (technology) and the role of design (business) has been conducted to create a new approach, built upon the theory of the techno-economic innovation model. Findings The findings of this study propose design-led channel management as a new research area, providing novel research questions and future research directions. The inclusion of design and emotion theories indicates that the future of digital channel design requires a deeper understanding of customers and needs to go beyond technological advances. Theoretical implications The findings provide an opportunity to explore dynamic theories and methodologies within the field of design that will broaden the horizons and challenge existing notions in channel literature. Originality/value This paper is the first paper that introduces the theory of Emotionate, as the next evolution of channel literature. The value of Emotionate lies in providing a new design-led process of integrating emotion to provide advice to practitioners as well as identifies research areas for academia, thereby extending the reach and richness of this emerging research field.

A probabilistic framework for analysing the compositionality of conceptual combinations

Conceptual combination performs a fundamental role in creating the broad range of compound phrases utilised in everyday language. While the systematicity and productivity of language provide a strong argument in favour of assuming compositionality, this very assumption is still regularly questioned in both cognitive science and philosophy. This article provides a novel probabilistic framework for assessing whether the semantics of conceptual combinations are compositional, and so can be considered as a function of the semantics of the constituent concepts, or not. Rather than adjudicating between different grades of compositionality, the framework presented here contributes formal methods for determining a clear dividing line between compositional and non-compositional semantics. Compositionality is equated with a joint probability distribution modelling how the constituent concepts in the combination are interpreted. Marginal selectivity is emphasised as a pivotal probabilistic constraint for the application of the Bell/CH and CHSH systems of inequalities (referred to collectively as Bell-type). Non-compositionality is then equated with either a failure of marginal selectivity, or, in the presence of marginal selectivity, with a violation of Bell-type inequalities. In both non-compositional scenarios, the conceptual combination cannot be modelled using a joint probability distribution with variables corresponding to the interpretation of the individual concepts. The framework is demonstrated by applying it to an empirical scenario of twenty-four non-lexicalised conceptual combinations.

Differential processing of thematic and categorical conceptual relations in spoken word production

Studies of semantic context effects in spoken word production have typically distinguished between categorical (or taxonomic) and associative relations. However, associates tend to confound semantic features or morphological representations, such as whole-part relations and compounds (e.g., BOAT-anchor, BEE-hive). Using a picture-word interference paradigm and functional magnetic resonance imaging (fMRI), we manipulated categorical (COW-rat) and thematic (COW-pasture) TARGET-distractor relations in a balanced design, finding interference and facilitation effects on naming latencies, respectively, as well as differential patterns of brain activation compared with an unrelated distractor condition. While both types of distractor relation activated the middle portion of the left middle temporal gyrus (MTG) consistent with retrieval of conceptual or lexical representations, categorical relations involved additional activation of posterior left MTG, consistent with retrieval of a lexical cohort. Thematic relations involved additional activation of the left angular gyrus. These results converge with recent lesion evidence implicating the left inferior parietal lobe in processing thematic relations and may indicate a potential role for this region during spoken word production.

The roles of shared vs. Distinctive conceptual features in lexical access

Contemporary models of spoken word production assume conceptual feature sharing determines the speed with which objects are named in categorically-related contexts. However, statistical models of concept representation have also identified a role for feature distinctiveness, i.e., features that identify a single concept and serve to distinguish it quickly from other similar concepts. In three experiments we investigated whether distinctive features might explain reports of counter-intuitive semantic facilitation effects in the picture word interference (PWI) paradigm. In Experiment 1, categorically-related distractors matched in terms of semantic similarity ratings (e.g., zebra and pony) and manipulated with respect to feature distinctiveness (e.g., a zebra has stripes unlike other equine species) elicited interference effects of comparable magnitude. Experiments 2 and 3 investigated the role of feature distinctiveness with respect to reports of facilitated naming with part-whole distractor-target relations (e.g., a hump is a distinguishing part of a CAMEL, whereas knee is not, vs. an unrelated part such as plug). Related part distractors did not influence target picture naming latencies significantly when the part denoted by the related distractor was not visible in the target picture (whether distinctive or not; Experiment 2). When the part denoted by the related distractor was visible in the target picture, non-distinctive part distractors slowed target naming significantly at SOA of -150 ms (Experiment 3). Thus, our results show that semantic interference does occur for part-whole distractor-target relations in PWI, but only when distractors denote features shared with the target and other category exemplars. We discuss the implications of these results for some recently developed, novel accounts of lexical access in spoken word production.

The Step approach to Message Design and Testing (SatMDT): A conceptual framework to guide the development and evaluation of persuasive health messages

This paper provides an important and timely overview of a conceptual framework designed to assist with the development of message content, as well as the evaluation, of persuasive health messages. While an earlier version of this framework was presented in a prior publication by the authors in 2009, important refinements to the framework have seen it evolve in recent years, warranting the need for an updated review. This paper outlines the Step approach to Message Design and Testing (or SatMDT) in accordance with the theoretical evidence which underpins, as well as empirical evidence which demonstrates the relevance and feasibility, of each of the framework’s steps. The development and testing of the framework have thus far been based exclusively within the road safety advertising context; however, the view expressed herein is that the framework may have broader appeal and application to the health persuasion context.

The development and refinement of the practice domain framework as a conceptual tool for understanding and guiding social care practice

This study identified the common factors that influence social care practice across disciplines (such as social work and psychology), practice fields, and geographical contexts and further developed the Practice Domain Framework as an empirically-based conceptual framework to assist practitioners in understanding practice complexities. The framework has application in critical reflection, professional supervision, interdisciplinary understanding, teamwork, management, teaching and research. A mixed-methods design was used to identify the components and structure of the refined framework. Eighteen influential factors were identified and organised into eight domains: the Societal, Structural, Organisational, Practice Field, Professional Practice, Accountable Practice, Community of Place, and Personal.

Perceived quality of e‐health services: A conceptual scale development of e‐health service quality based on the C‐OAR‐SE approach

Purpose Health service quality is an important determinant for health service satisfaction and behavioral intentions. The purpose of this paper is to investigate requirements of e‐health services and to develop a measurement model to analyze the construct of “perceived e‐health service quality.” Design/methodology/approach The paper adapts the C‐OAR‐SE procedure for scale development by Rossiter. The focal aspect is the “physician‐patient relationship” which forms the core dyad in the healthcare service provision. Several in‐depth interviews were conducted in Switzerland; first with six patients (as raters), followed by two experts of the healthcare system (as judges). Based on the results and an extensive literature research, the classification of object and attributes is developed for this model. Findings The construct e‐health service quality can be described as an abstract formative object and is operationalized with 13 items: accessibility, competence, information, usability/user friendliness, security, system integration, trust, individualization, empathy, ethical conduct, degree of performance, reliability, and ability to respond. Research limitations/implications Limitations include the number of interviews with patients and experts as well as critical issues associated with C‐OAR‐SE. More empirical research is needed to confirm the quality indicators of e‐health services. Practical implications Health care providers can utilize the results for the evaluation of their service quality. Practitioners can use the hierarchical structure to measure service quality at different levels. The model provides a diagnostic tool to identify poor and/or excellent performance with regard to the e‐service delivery. Originality/value The paper contributes to knowledge with regard to the measurement of e‐health quality and improves the understanding of how customers evaluate the quality of e‐health services.