Using automated text analysis to evaluate students’ conceptual understanding

Background: A major challenge for assessing students’ conceptual understanding of STEM subjects is the capacity of assessment tools to reliably and robustly evaluate student thinking and reasoning. Multiple-choice tests are typically used to assess student learning and are designed to include distractors that can indicate students’ incomplete understanding of a topic or concept based on which distractor the student selects. However, these tests fail to provide the critical information uncovering the how and why of students’ reasoning for their multiple-choice selections. Open-ended or structured response questions are one method for capturing higher level thinking, but are often costly in terms of time and attention to properly assess student responses. Purpose: The goal of this study is to evaluate methods for automatically assessing open-ended responses, e.g. students’ written explanations and reasoning for multiple-choice selections. Design/Method: We incorporated an open response component for an online signals and systems multiple-choice test to capture written explanations of students’ selections. The effectiveness of an automated approach for identifying and assessing student conceptual understanding was evaluated by comparing results of lexical analysis software packages (Leximancer and NVivo) to expert human analysis of student responses. In order to understand and delineate the process for effectively analysing text provided by students, the researchers evaluated strengths and weakness for both the human and automated approaches. Results: Human and automated analyses revealed both correct and incorrect associations for certain conceptual areas. For some questions, that were not anticipated or included in the distractor selections, showing how multiple-choice questions alone fail to capture the comprehensive picture of student understanding. The comparison of textual analysis methods revealed the capability of automated lexical analysis software to assist in the identification of concepts and their relationships for large textual data sets. We also identified several challenges to using automated analysis as well as the manual and computer-assisted analysis. Conclusions: This study highlighted the usefulness incorporating and analysing students’ reasoning or explanations in understanding how students think about certain conceptual ideas. The ultimate value of automating the evaluation of written explanations is that it can be applied more frequently and at various stages of instruction to formatively evaluate conceptual understanding and engage students in reflective

A systematic review of patient-reported outcome instruments of dermatologic adverse events associated with targeted cancer therapies

Purpose Dermatologic adverse events (dAEs) in cancer treatment are frequent with the use of targeted therapies. These dAEs have been shown to have significant impact on health-related quality of life (HRQoL). While standardized assessment tools have been developed for physicians to assess severity of dAEs, there is a discord between objective and subjective measures. The identification of patient-reported outcome (PRO) instruments useful in the context of targeted cancer therapies is therefore important in both the clinical and research settings for the overall evaluation of dAEs and their impact on HRQoL. Methods A comprehensive, systematic literature search of published articles was conducted by two independent reviewers in order to identify PRO instruments previously utilized in patient populations with dAEs from targeted cancer therapies. The identified PRO instruments were studied to determine which HRQoL issues relevant to dAEs were addressed, as well as the process of development and validation of these instruments. Results Thirteen articles identifying six PRO instruments met the inclusion criteria. Four instruments were general dermatology (Skindex-16©, Skindex-29©, Dermatology Life Quality Index (DLQI), and DIELH-24) and two were symptom-specific (functional assessment of cancer therapy-epidermal growth factor receptor inhibitor-18 (FACT-EGFRI-18) and hand-foot syndrome-14 (HFS-14)). Conclusions While there are several PRO instruments that have been tested in the context of targeted cancer therapy, additional work is needed to develop new instruments and to further validate the instruments identified in this study in patients receiving targeted therapies.

Informing methods for preparing public health overviews of reviews: a comparison of public health overviews with Cochrane Overviews published between 1999 and 2014

Background: Overviews of systematic reviews (SRs) are useful for public health policy; however there is an absence of Cochrane Overviews covering public health (PH) topics. Objectives: We sought to analyze the methodological approaches used in existing Cochrane Overviews and Protocols for overviews (primarily clinical in nature), and compare these to the methods and approaches used in PH overviews (non-Cochrane). The intent was to identify issues that would be relevant for undertaking Cochrane overviews. Methods: We conducted a descriptive analysis of overviews published between 1999 and 2014. We searched the Cochrane Database of Systematic Reviews for Cochrane Protocols for overviews and Cochrane Overviews, and the HealthEvidence.org for PH overviews. The primary characteristics of the overviews and elements of the methodology were extracted and compared. Results: A total of 61 overviews of SRs were included in our analysis; specifically, this included 21 Cochrane Protocols for overviews, 15 Cochrane Overviews, and 27 non-Cochrane PH overviews. Amongst the overviews, the most significant differences are that PH overviews (non-Cochrane) tend to: include earlier and more reviews, greater number of participants, allow lower levels of evidence, use assessment tools other than AMSTAR (A Measurement Tool to Assess Systematic Reviews, i.e. a tool for assessing quality of SRs), not assess quality of evidence in reviews, search more databases overall, specify search limits including English-only reviews, and not consider recent primary studies for inclusion. Some of these differences clearly related to quality, however many relate to the nuances of PH interventions. Conclusions: The methodology in Cochrane overviews and PH overviews varies widely. Future PH overviews may benefit from the Cochrane methodology but the Cochrane approach requires modification to accommodate PH research methodology. Additionally, the use of databases that pre-screen and quality assess relevant PH systematic reviews may help expedite the search process.

Spectrum of ankylosing spondylitis in Portugal. Development of BASDAI, BASFI, BASMI and mSASSS reference centile charts

The availability of population-specific normative data regarding disease severity measures is essential for patient assessment. The goals of the current study were to characterize the pattern of ankylosing spondylitis (AS) in Portuguese patients and to develop reference centile charts for BASDAI, BASFI, BASMI and mSASSS, the most widely used assessment tools in AS. AS cases were recruited from hospital outpatient clinics, with AS defined according to the modified New York criteria. Demographic and clinical data were recorded. All radiographs were evaluated by two independent experienced readers. Centile charts for BASDAI, BASFI, BASMI and mSASSS were constructed for both genders, using generalized linear models and regression models with duration of disease as independent variable. A total of 369 patients (62.3% male, mean ± (SD) age 45.4 ± 13.2 years, mean ± (SD) disease duration 11.4 ± 10.5 years, 70.7% B27-positive) were included. Family history of AS in a first-degree relative was reported in 17.6% of the cases. Regarding clinical disease pattern, at the time of assessment 42.3% had axial disease, 2.4% peripheral disease, 40.9% mixed disease and 7.1% isolated enthesopatic disease. Anterior uveitis (33.6%) was the most common extra-articular manifestation. The centile charts suggest that females reported greater disease activity and more functional impairment than males but had lower BASMI and mSASSS scores. Data collected through this study provided a demographic and clinical profile of patients with AS in Portugal. The development of centile charts constitutes a useful tool to assess the change of disease pattern over time and in response to therapeutic interventions.

Management of persistent pain in the older people

Persistent pain is a commonly experienced symptom. It affects 25% of community-dwelling older adults and up to 80% of nursing home residents, and can have a major impact on quality of life and functional capacity. Unfortunately pain in older patients is often undertreated and misunderstood. Assessment of pain type and severity is important. Most older people, even with moderately impaired cognition, are able to self-report pain. Validated assessment tools using non-verbal pain cues are available for people with more advanced cognitive impairment. Management of pain in older people can be challenging. Physiological changes may impact on pain perception and the pharmacodynamics and pharmacokinetics of medications. Older people are often more sensitive to the adverse effects of analgesic medications and are at risk of drug–drug interactions due to the presence of co-morbidities and polypharmacy. In general, analgesic medications should be commenced at low doses, titrated based on effect and tolerability, and regularly reviewed. Contemporary pain management often utilises multiple analgesics in lower doses to optimise efficacy and avoid dose-related toxicity. A bio-psycho-social approach to the management of persistent pain, utilising a multidisciplinary team and including non-drug strategies, may produce the best results. The goal of pain management is not always to eliminate pain, since this may not be attainable, but rather to enhance function and improve quality of life. This article discusses persistent non-cancer pain in older people, its assessment and management, and the risks and benefits of pharmacological treatment in this population.

Responding to an ageing workforce in an Australian state

Many industrialised nations have changing demographic profiles, as increased longevity and decreased birth rates lead to an ageing population. This presents significant challenges for workforces, as older employees retire and there are insufficient numbers of younger employees to take their place. This leads to skills shortages, and strong competition for those who are available. This paper considers these issues in the context of Queensland, the third largest state of Australia. The Queensland Government is addressing the issues for all industries in the state, primarily through a Skills Plan and an Experience Pays Awareness Strategy. As the largest employer in the state, the Queensland Government has commenced implementing the Experience Pays Awareness Strategy within its own workforce. The approach touches on many facets of HRM. The HRM policy framework and tools are examined for their potential to support increased participation of older employees. A range of issues are addressed for older workers, including their competence and health and safety issues. Issues for managers include addressing myths and subtle discrimination against older workers, as well as managing cross-generational workforce. Other strategies and methods are targeted at cultural factors, such as the expectations of older workers, and the myths and discrimination against older workers. Yet other strategies are aimed at organisational issues such retention of knowledge and succession planning.

Genesis and role of standards : theoretical foundations and socio-economical model for the construction and use of standards

A key issue in the economic development and performance of organizations is the existence of standards. Their definition and control are sources of power and it is important to understand their concept, as it gives standards their direction and their legitimacy, and to explore how they are represented and applied. The difficulties posed by classical micro-economics in establishing a theory of standardization that is compatible with its fundamental axiomatic are acknowledged. We propose to reconsider the problem by taking the opposite perspective in questioning its theoretical base and by reformulating assumptions about the independent and autonomous decisions taken by actors. The Theory of Conventions will offer us a theoretical framework and tools enabling us to understand the systemic dimension and dynamic structure of standards. These will be seen as a special case of conventions. This work aims to provide a sound basis and promote a better consciousness in the development of global project management standards. It aims also to emphasize that social construction is not a matter of copyright but a matter of open minds, collective cognitive process and freedom for the common wealth.

Genesis and role of standards : theoretical foundations and socio-economical model for the construction and use of standards

A key issue for the economic development and for performance of organizations is the existence of standards. As their definitions and control are source of power, it seems to be important to understand the concept and to wonder about the representations authorized by the concept which give their direction and their legitimacy. The difficulties of classical microeconomics of establishing a theory of standardisation compatible with its fundamental axiomatic are underlined. We propose to reconsider the problem by carrying out the opposite way: to question the theoretical base, by reformulating assumptions on the autonomy of the choice of the actors. The theory of conventions will offer us both a theoretical framework and tools, enabling us to understand the systemic dimension and dynamic structure of standards seen as special case of conventions. This work aims thus to provide a sound basis and promote a better consciousness in the development of global project management standards, aiming also to underline that social construction is not a matter of copyright but a matter of open minds, collective cognitive process and freedom for the common wealth.

Design led innovation : shifting from smart follower to digital strategy leader in the Australian airport sector

This paper presents and discusses organisational barriers and opportunities arising from the dissemination of design led innovation within a leading Australian airport corporation. This research is part of a greater action research program which aims to integrate design as a strategic capability through design led innovation within Australian businesses. Findings reveal that there is an opportunity to employ the theoretical framework and tools of design led innovation in practice to build collaborative idea generation by involving customers and stakeholders within the proposal of new to world propositions. The iterative gathering of deep customer insights also provided an opportunity to leverage a greater understanding of stakeholders and customers in strengthening continuing business partnerships through co-design. Challenges to the design led approach include resistance to the exploratory nature of gathering deep customer insights, the testing of long held assumptions and market data, and the disruption of an organisational mindset geared toward risk aversion instilled within the aviation industry. The implication from these findings is that design led innovation can provide the critical platform to allow for a business to grow and sustain internal design capabilities necessary to challenge prevailing assumptions about how its business model operates to deliver value to customers and stakeholders alike. The platform of design led innovation also provides an avenue to support a cultural transformation towards anticipating future needs necessary for establishing a position of leadership within the broader economic environment.

Decision making guidelines for sustainable construction of industrialised building systems

The construction industry has an obligation to respond to sustainability expectations of our society. Solutions that integrate innovative, intelligent and sustainability deliverables are vital for us to meet new and emerging challenges. Industrialised Building Systems (IBS), or known otherwise as prefabrication, employs a combination of ready-made components in the construction of buildings. They promote quality of production, enhance simplification of construction processes and minimise waste. The unique characteristics of this construction method respond well to sustainability. Despite the promises however, IBS has yet to be effectively implemented in Malaysia. There are often misconceptions among key stakeholders about IBS applications. The existing rating schemes fail to assess IBS against sustainability measures. To ensure the capture of full sustainability potential in buildings developed, the critical factors and action plans agreeable to all participants in the development processes need to be identified. Through questionnaire survey, eighteen critical factors relevant to IBS sustainability were identified and encapsulated into a conceptual framework to coordinate a systematic IBS decision making approach. Five categories were used to separate the critical factors into: ecological performance; economic value; social equity and culture; technical quality; and implementation and enforcement. This categorisation extends the "Triple Bottom Lines" to include social, economic, environmental and institutional dimensions. Semi-structured interviews help identify strategies of actions and solutions of potential problems through a SWOT analysis framework. These tools help the decision-makers maximise the opportunities by using available strengths, avoid weaknesses, and diagnose possible threats in the examined issues. The recommendations formed an integrated action plan to present information on what and how to improve sustainability through tackling each critical factor during IBS development. It can be used as part of the project briefing documents for IBS designers. For validation and finalisation the research deliverables, three case studies were conducted. The research fills a current gap by responding to IBS project scenarios in developing countries. It also provides a balanced view for designers to better understand sustainability potential and prioritize attentions to manage sustainability issues in IBS applications.

Course quality assurance at a glance

From 2014, QUT will be adopting a life-cycle approach to Course Quality Assurance informed by a wider and richer range of historic, ‘live’ and ‘predictive’ course data. Key data elements continue to be grouped according to the three broad categories – Viability, Quality of Learning Environment and Outcomes – and are further supported with analytic data presented within tables and charts. Course Quality Assurance and this Consolidated Courses Performance Report illuminate aspects of courses from a data evidence base highlighting the strengths and weaknesses of our courses. It provides the framework and tools to achieve QUT's commitment to excellent graduate outcomes by drawing attention and focus to the quality of our courses and providing a structured approach for bringing about change. Our portfolio of courses forms a vital part of QUT, generating almost $600 million in 2013 alone. Real world courses are fundamental to the strength of the Institution; they are what our many thousands of current and future students are drawn to and invest their time and aspirations in. As we move through a period of some regulatory and deregulatory uncertainty, there is a greater need for QUT to monitor and respond to the needs and expectations of our students. The life-cycle approach, with its rich and predicative data, provides the best source of evidence we have had, to date, to assure the quality of our courses and their relevance in a rapidly changing higher education context.

Towards a building sustainability assessment framework

This paper provides an overview of a new framework for a design stage Building Environmental Assessment (BEA) tool and a discussion of strategic responses to existing tool issues and relative stakeholder requirements that lead to the development of this tool founded on new information and communication technology (ICT) related to developments in 3D CAD technology. After introducing the context of the BEA and some of their team’s new work the authors • Critique current BEA tool theory; • Review previous assessments of stakeholder needs; • Introduce a new framework applied to analyse such tools • Highlight and key results considering illustrative ICT capabilities and • Discuss their potential significance upon BEA tool stakeholders.

Redefining life cycle for a building sustainability assessment framework

Understanding the differences between the temporal and physical aspects of the building life cycle is an essential ingredient in the development of Building Environmental Assessment (BEA) tools. This paper illustrates a theoretical Life Cycle Assessment (LCA) framework aligning temporal decision-making with that of material flows over building development phases. It was derived during development of a prototype commercial building design tool that was based on a 3-D CAD information and communications technology (ICT) platform and LCA software. The framework aligns stakeholder BEA needs and the decision-making process against characteristics of leading green building tools. The paper explores related integration of BEA tool development applications on such ICT platforms. Key framework modules are depicted and practical examples for BEA are provided for: • Definition of investment and service goals at project initiation; • Design integrated to avoid overlaps/confusion over the project life cycle; • Detailing the supply chain considering building life cycle impacts; • Delivery of quality metrics for occupancy post-construction/handover; • Deconstruction profiling at end of life to facilitate recovery.

Formative and summative assessment and the notion of constructive alignment

This paper discusses the different perceptions of first year accounting students about their tutorial activities and their engagements in assessment. As the literature suggests, unless participation in learning activities forms part of graded assessment, it is often difficult to engage students in these activities. Using an action research model, this paper reports the study of first year accounting students' responses to action-oriented learning tasks in tutorials. The paper focuses on the importance of aligning curriculum objectives, learning and teaching activities and assessment, i.e. the notion of constructive alignment. However, as the research findings indicate, without support at institutional level, applying constructive alignment to facilitate quality student learning outcomes is a difficult task. Thus, the impacts of policy constraints on curriculum issues are also discussed, focusing on the limitations faced by tutors and their lack of involvement in curriculum development.

Formative and Summative Assessment and the Notion of Constructive Allignment

This paper discusses perceptions of first year accounting students about their tutorial activities and their engagements in assessment. As the literature suggests, unless participation in learning activities forms part of graded assessment it is often difficult to engage students in these activities. Using an action research model, this paper reports the study of first year accounting students' responses to action-orientated learning tasks in tutorials. The paper focuses on the importance of aligning curriculum objectives, learning and teaching activities and assessment,i.e. the notion of constructive alignment. However, as the research findings indicate, without support at institutional level, applying constructive alignment to facilitate quality student learning outcomes is a difficult task. Thus, the impacts of policy constraints on curriculum issues are also discussed, focusing on the limitations faced by tutors and their lack of involvement in curriculum development.