Factors influencing robustness and effectiveness of conditional random fields in active learning frameworks

Active learning approaches reduce the annotation cost required by traditional supervised approaches to reach the same effectiveness by actively selecting informative instances during the learning phase. However, effectiveness and robustness of the learnt models are influenced by a number of factors. In this paper we investigate the factors that affect the effectiveness, more specifically in terms of stability and robustness, of active learning models built using conditional random fields (CRFs) for information extraction applications. Stability, defined as a small variation of performance when small variation of the training data or a small variation of the parameters occur, is a major issue for machine learning models, but even more so in the active learning framework which aims to minimise the amount of training data required. The factors we investigate are a) the choice of incremental vs. standard active learning, b) the feature set used as a representation of the text (i.e., morphological features, syntactic features, or semantic features) and c) Gaussian prior variance as one of the important CRFs parameters. Our empirical findings show that incremental learning and the Gaussian prior variance lead to more stable and robust models across iterations. Our study also demonstrates that orthographical, morphological and contextual features as a group of basic features play an important role in learning effective models across all iterations.

Active learning: A step towards automating medical concept extraction

Objective This paper presents an automatic active learning-based system for the extraction of medical concepts from clinical free-text reports. Specifically, (1) the contribution of active learning in reducing the annotation effort, and (2) the robustness of incremental active learning framework across different selection criteria and datasets is determined. Materials and methods The comparative performance of an active learning framework and a fully supervised approach were investigated to study how active learning reduces the annotation effort while achieving the same effectiveness as a supervised approach. Conditional Random Fields as the supervised method, and least confidence and information density as two selection criteria for active learning framework were used. The effect of incremental learning vs. standard learning on the robustness of the models within the active learning framework with different selection criteria was also investigated. Two clinical datasets were used for evaluation: the i2b2/VA 2010 NLP challenge and the ShARe/CLEF 2013 eHealth Evaluation Lab. Results The annotation effort saved by active learning to achieve the same effectiveness as supervised learning is up to 77%, 57%, and 46% of the total number of sequences, tokens, and concepts, respectively. Compared to the Random sampling baseline, the saving is at least doubled. Discussion Incremental active learning guarantees robustness across all selection criteria and datasets. The reduction of annotation effort is always above random sampling and longest sequence baselines. Conclusion Incremental active learning is a promising approach for building effective and robust medical concept extraction models, while significantly reducing the burden of manual annotation.

External knowledge and query strategies in active learning: A study in clinical information extraction

This paper presents a new active learning query strategy for information extraction, called Domain Knowledge Informativeness (DKI). Active learning is often used to reduce the amount of annotation effort required to obtain training data for machine learning algorithms. A key component of an active learning approach is the query strategy, which is used to iteratively select samples for annotation. Knowledge resources have been used in information extraction as a means to derive additional features for sample representation. DKI is, however, the first query strategy that exploits such resources to inform sample selection. To evaluate the merits of DKI, in particular with respect to the reduction in annotation effort that the new query strategy allows to achieve, we conduct a comprehensive empirical comparison of active learning query strategies for information extraction within the clinical domain. The clinical domain was chosen for this work because of the availability of extensive structured knowledge resources which have often been exploited for feature generation. In addition, the clinical domain offers a compelling use case for active learning because of the necessary high costs and hurdles associated with obtaining annotations in this domain. Our experimental findings demonstrated that 1) amongst existing query strategies, the ones based on the classification model’s confidence are a better choice for clinical data as they perform equally well with a much lighter computational load, and 2) significant reductions in annotation effort are achievable by exploiting knowledge resources within active learning query strategies, with up to 14% less tokens and concepts to manually annotate than with state-of-the-art query strategies.

Pedagogical tools designed to promote engagement and active learning in the biomedical sciences and allied health disciplines

Perceiving students, science students especially, as mere consumers of facts and information belies the importance of a need to engage them with the principles underlying those facts and is counter-intuitive to the facilitation of knowledge and understanding. Traditional didactic lecture approaches need a re-think if student classroom engagement and active learning are to be valued over fact memorisation and fact recall. In our undergraduate biomedical science programs across Years 1, 2 and 3 in the Faculty of Health at QUT, we have developed an authentic learning model with an embedded suite of pedagogical strategies that foster classroom engagement and allow for active learning in the sub-discipline area of medical bacteriology. The suite of pedagogical tools we have developed have been designed to enable their translation, with appropriate fine-tuning, to most biomedical and allied health discipline teaching and learning contexts. Indeed, aspects of the pedagogy have been successfully translated to the nursing microbiology study stream at QUT. The aims underpinning the pedagogy are for our students to: (1) Connect scientific theory with scientific practice in a more direct and authentic way, (2) Construct factual knowledge and facilitate a deeper understanding, and (3) Develop and refine their higher order flexible thinking and problem solving skills, both semi-independently and independently. The mindset and role of the teaching staff is critical to this approach since for the strategy to be successful tertiary teachers need to abandon traditional instructional modalities based on one-way information delivery. Face-to-face classroom interactions between students and lecturer enable realisation of pedagogical aims (1), (2) and (3). The strategy we have adopted encourages teachers to view themselves more as expert guides in what is very much a student-focused process of scientific exploration and learning. Specific pedagogical strategies embedded in the authentic learning model we have developed include: (i) interactive lecture-tutorial hybrids or lectorials featuring teacher role-plays as well as class-level question-and-answer sessions, (ii) inclusion of “dry” laboratory activities during lectorials to prepare students for the wet laboratory to follow, (iii) real-world problem-solving exercises conducted during both lectorials and wet laboratory sessions, and (iv) designing class activities and formative assessments that probe a student’s higher order flexible thinking skills. Flexible thinking in this context encompasses analytical, critical, deductive, scientific and professional thinking modes. The strategic approach outlined above is designed to provide multiple opportunities for students to apply principles flexibly according to a given situation or context, to adapt methods of inquiry strategically, to go beyond mechanical application of formulaic approaches, and to as much as possible self-appraise their own thinking and problem solving. The pedagogical tools have been developed within both workplace (real world) and theoretical frameworks. The philosophical core of the pedagogy is a coherent pathway of teaching and learning which we, and many of our students, believe is more conducive to student engagement and active learning in the classroom. Qualitative and quantitative data derived from online and hardcopy evaluations, solicited and unsolicited student and graduate feedback, anecdotal evidence as well as peer review indicate that: (i) our students are engaging with the pedagogy, (ii) a constructivist, authentic-learning approach promotes active learning, and (iii) students are better prepared for workplace transition.

Web-based active learning and frequent feedback: Engaging first-year university students

Web-based technology is particularly well-suited to promoting active student involvement in the processes of learning. All students enrolled in a first-year educational psychology unit were required to complete ten weekly online quizzes, ten weekly student-generated questions and ten weekly student answers to those questions. Results of an online survey of participating students strongly support the viability and perceived benefits of such an instructional approach. Although students reported that the 30 assessments were useful and reasonable, the most common theme to emerge from the professional reflections of participating lecturers was that the marking of questions and answers was unmanageable.

Can active learning principles be applied to the bioscience assessments of nursing students? A review of the literature

- Objectives To explore if active learning principles be applied to nursing bioscience assessments and will this influence student perception of confidence in applying theory to practice? - Design and Data Sources A review of the literature utilising searches of various databases including CINAHL, PUBMED, Google Scholar and Mosby's Journal Index. - Methods The literature search identified research from twenty-six original articles, two electronic books, one published book and one conference proceedings paper. - Results Bioscience has been identified as an area that nurses struggle to learn in tertiary institutions and then apply to clinical practice. A number of problems have been identified and explored that may contribute to this poor understanding and retention. University academics need to be knowledgeable of innovative teaching and assessing modalities that focus on enhancing student learning and address the integration issues associated with the theory practice gap. Increased bioscience education is associated with improved patient outcomes therefore by addressing this “bioscience problem” and improving the integration of bioscience in clinical practice there will subsequently be an improvement in health care outcomes. - Conclusion From the literature several themes were identified. First there are many problems with teaching nursing students bioscience education. These include class sizes, motivation, concentration, delivery mode, lecturer perspectives, student's previous knowledge, anxiety, and a lack of confidence. Among these influences the type of assessment employed by the educator has not been explored or identified as a contributor to student learning specifically in nursing bioscience instruction. Second that educating could be achieved more effectively if active learning principles were applied and the needs and expectations of the student were met. Lastly, assessment influences student retention and the student experience and as such assessment should be congruent with the subject content, align with the learning objectives and be used as a stimulus tool for learning.

Dichotomy in the design studio : adapting to new blended learning environments

In a study aimed at better understanding how staff and students adapt to new blended studio learning environments (BSLE’s), a group of 165 second year architecture students at a large school of architecture in Australia were separated into two different design studio learning environments. 70% of students were allocated to a traditional studio design learning environment (TSLE) and 30% to a new, high technology embedded, prototype digital learning laboratory. The digital learning laboratory was purpose designed for the case-study users, adapted Student-Centred Active Learning Environment for Undergraduate Programs (SCALE-UP) principles, and built as part of a larger university research project. The architecture students attended the same lectures, followed the same studio curriculum and completed the same pieces of assessment; the only major differences were the teaching staff and physical environment within which the studios were conducted. At the end of the semester, the staff and students were asked to complete a questionnaire about their experiences and preferences within the two respective learning environments. Following this, participants were invited to participate in focus groups, where a synergistic approach was effected. Using a dual method qualitative approach, the questionnaire and survey data were coded and extrapolated using both thematic analysis and grounded theory methodology. The results from these two different approaches were compared, contrasted and finally merged, to reveal six distinct emerging themes, which were instrumental in offering resistance or influencing adaptation to, the new BLSE. This paper reports on the study, discusses the major contributors to negative resistance and proposes points for consideration, when transitioning from a TSLE to a BLSE.

Good Practice Report : Blended Learning

In early 2011, the Australian Learning and Teaching Council Ltd (ALTC) commissioned a series of Good Practice Reports on completed ALTC projects and fellowships. This report will: • Provide a summative evaluation of the good practices and key outcomes for teaching and learning from completed ALTC projects and fellowships relating to blended learning • Include a literature review of the good practices and key outcomes for teaching and learning from national and international research • Identify areas in which further work or development are appropriate. The literature abounds with definitions; it can be argued that the various definitions incorporate different perspectives, but there is no single, collectively accepted definition. Blended learning courses in higher education can be placed somewhere on a continuum, between fully online and fully face-to-face courses. Consideration must therefore be given to the different definitions for blended learning presented in the literature and by users and stakeholders. The application of this term in these various projects and fellowships is dependent on the particular focus of the team and the conditions and situations under investigation. One of the key challenges for projects wishing to develop good practice in blended learning is the lack of a universally accepted definition. The findings from these projects and fellowships reveal the potential of blended learning programs to improve both student outcomes and levels of satisfaction. It is clear that this environment can help teaching and learning engage students more effectively and allow greater participation than traditional models. Just as there are many definitions, there are many models and frameworks that can be successfully applied to the design and implementation of such courses. Each academic discipline has different learning objectives and in consequence there can’t be only one correct approach. This is illustrated by the diversity of definitions and applications in the ALTC funded projects and fellowships. A review of the literature found no universally accepted guidelines for good practice in higher education. To inform this evaluation and literature review, the Seven Principles for Good Practice in Undergraduate Education, as outlined by Chickering and Gamson (1987), were adopted: 1. encourages contacts between students and faculty 2. develops reciprocity and cooperation among students 3. uses active learning techniques 4. gives prompt feedback 5. emphasises time on task 6. communicates high expectations 7. respects diverse talents and ways of learning. These blended learning projects have produced a wide range of resources that can be used in many and varied settings. These resources include: books, DVDs, online repositories, pedagogical frameworks, teaching modules. In addition there is valuable information contained in the published research data and literature reviews that inform good practice and can assist in the development of courses that can enrich and improve teaching and learning.

‘Please leave your mobile phone on’ : Social educational networking in a social society : Encouraging in-class engagement at QUT across physical and virtual learning environments

Though stadium style seating in large lecture theatres may suggest otherwise, effective teaching and learning is a not a spectator sport. A challenge in creating effective learning environments in both physical and virtual spaces is to provide optimal opportunity for student engagement in active learning. Queensland University of Technology (QUT) has developed the Open Web Lecture (OWL), a new web-based student response application, which seamlessly integrates a virtual learning environment within the physical learning space. The result is a blended learning experience; a fluid collaboration between academic and students connected to OWL via the University’s Wi-Fi using their own laptop or mobile web device. QUT is currently piloting the OWL application to encourage student engagement. OWL offers opportunities for participants to: • Post comments and questions • Reply to comments
 • "Like" comments
 • Poll students and review data • Review archived sessions. Many of these features instinctively appeal to student users of social networking media, yet avail the academic of control within the University network. Student privacy is respected through a system of preserving peer-peer anonymity, a functionality that seeks to address a traditional reluctance to speak up in large classes. The pilot is establishing OWL as an opportunity for engaging students in active learning opportunities by enabling • virtual learning in physical spaces for large group lectures, seminar groups, workshops and conferences • live collaborative technology connecting students and the academic via the wireless network using their own laptop or mobile device • an non- intimidating environment in which to ask questions • promotion of a sense of community • instant feedback • problem based learning. The student and academic response to OWL has been overwhelmingly positive, crediting OWL as an easy to use application, which creates effective learning opportunities though interactivity and immediate feedback. This poster and accompanying online presentation of the technology will demonstrate how OWL offers new possibilities for active learning in physical spaces by: • providing increased opportunity for student engagement • supporting a range of learners and learning activities • fostering blended learning experiences. The presentation will feature visual displays of the technology, its various interfaces and feedback including clips from interviews with students and academics participating in the early stages of the pilot.

Active ageing : developing a quantitative multi-dimensional measure

With estimates that two billion of the world’s population will be 65 years or older by 2050, ensuring that older people ‘age well’ is an international priority. To date, however, there is significant disagreement and debate about how to define and measure ‘ageing well’, with no consensus on either terminology or measurement. Thus, this chapter describes the research rationale, methodology and findings of the Australian Active Ageing Study (Triple A Study), which surveyed 2620 older Australians to identify significant contributions to quality of life for older people: work, learning, social participation, spirituality, emotional wellbeing, health, and life events. Exploratory factor analyses identified eight distinct elements (grouped into four key concepts) which appear to define ‘active ageing’ and explained 55% of the variance: social and life participation (25%), emotional health (22%), physical health and functioning (4%) and security (4%). These findings highlight the importance of understanding and supporting the social and emotional dimensions of ageing, as issues of social relationships, life engagement and emotional health dominated the factor structure. Our intension is that this paper will prompt informed debate and discussion on defining and measuring active ageing, facilitating exploration and understanding of the complexity of issues that intertwine, converge and enhance the ageing experience.

Helping undergraduate students across disciplines and cultures actively engage and collaborate as equal members of a community of scholars : peer review within an e-learning environment

At the international level, the higher education sector is currently being subjected to increased calls for public accountability and the current move by the OECD to rank universities based on the quality of their teaching and learning outcomes. At the national level, Australian universities and their teaching staff face numerous challenges including financial restrictions, increasing student numbers and the reality of an increasingly diverse student population. The Australian higher education response to these competing policy and accreditation demands focuses on precise explicit systems and procedures which are inflexible and conservative and which ignore the fact that assessment is the single biggest influence on how students approach their learning. By seriously neglecting the quality of student learning outcomes, assessment tasks are often failing to engage students or reflect the tasks students will face in the world of practice. Innovative assessment design, which includes new paradigms of student engagement and learning and pedagogically based technologies have the capacity to provide some measure of relief from these internal and external tensions by significantly enhancing the learning experience for an increasingly time-poor population of students. That is, the assessment process has the ability to deliver program objectives and active learning through a knowledge transfer process which increases student participation and engagement. This social constructivist view highlights the importance of peer review in assisting students to participate and collaborate as equal members of a community of scholars with both their peers and academic staff members. As a result of increasing the student’s desire to learn, peer review leads to more confident, independent and reflective learners who also become more skilled at making independent judgements of their own and others' work. Within this context, in Case Study One of this project, a summative, peer-assessed, weekly, assessment task was introduced in the first “serious” accounting subject offered as part of an undergraduate degree. The positive outcomes achieved included: student failure rates declined 15%; tutorial participation increased fourfold; tutorial engagement increased six-fold; and there was a 100% student-based approval rating for the retention of the assessment task. However, in stark contrast to the positive student response, staff issues related to the loss of research time associated with the administration of the peer-review process threatened its survival. This paper contributes to the core conference topics of new trends and experiences in undergraduate assessment education and in terms of innovative, on-line, learning and teaching practices, by elaborating the Case Study Two “solution” generated to this dilemma. At the heart of the resolution is an e-Learning, peer-review process conducted in conjunction with the University of Melbourne which seeks to both create a virtual sense of belonging and to efficiently and effectively meet academic learning objectives with minimum staff involvement. In outlining the significant level of success achieved, student-based qualitative and quantitative data will be highlighted along with staff views in a comparative analysis of the advantages and disadvantages to both students and staff of the staff-led, peer review process versus its on-line counterpart.

How do the learning outcomes of students compare in units with different levels of practical work? A comparative study about the relation between Problem-­Based Learning and assessment approaches

The literature supporting the notion that active, student-centered learning is superior to passive, teacher-centered instruction is encyclopedic (Bonwell & Eison, 1991; Bruning, Schraw, & Ronning, 1999; Haile, 1997a, 1997b, 1998; Johnson, Johnson, & Smith, 1999). Previous action research demonstrated that introducing a learning activity in class improved the learning outcomes of students (Mejias, 2010). People acquire knowledge and skills through practice and reflection, not by watching and listening to others telling them how to do something. In this context, this project aims to find more insights about the level of interactivity in the curriculum a class should have and its alignment with assessment so the intended learning outcomes (ILOs) are achieved. In this project, interactivity is implemented in the form of problem- based learning (PBL). I present the argument that a more continuous formative feedback when implemented with the correct amount of PBL stimulates student engagement bringing enormous benefits to student learning. Different levels of practical work (PBL) were implemented together with two different assessment approaches in two subjects. The outcomes were measured using qualitative and quantitative data to evaluate the levels of student engagement and satisfaction in the terms of ILOs.

The introduction of an interactive teaching and learning activity (TLA) in class and its relationship to students’ learning-outcomes

In the university education arena, it is becoming apparent that traditional methods of conducting classes are not the most effective ways to achieve desired learning outcomes. The traditional class/method involves the instructor verbalizing information for passive, note-taking students who are assumed to be empty receptacles waiting to be filled with knowledge. This method is limited in its effectiveness, as the flow of information is usually only in one direction. Furthermore, “It has been demonstrated that students in many cases can recite and apply formulas in numerical problems, but the actual meaning and understanding of the concept behind the formula is not acquired (Crouch & Mazur)”. It is apparent that memorization is the main technique present in this approach. A more effective method of teaching involves increasing the students’ level of activity during, and hence their involvement in the learning process. This technique stimulates self- learning and assists in keeping these students’ levels of concentration more uniform. In this work, I am therefore interested in studying the influence of a particular TLA on students’ learning-outcomes. I want to foster high-level understanding and critical thinking skills using active learning (Silberman, 1996) techniques. The TLA in question aims to promote self-study by students and to expose them to a situation where their learning-outcomes can be tested. The motivation behind this activity is based on studies which suggest that some sensory modalities are more effective than others. Using various instruments for data collection and by means of a thorough analysis I present evidence of the effectiveness of this action research project which aims to improve my own teaching practices, with the ultimate goal of enhancing student’s learning.

Second Life machinima enhancing the learning of law : Lessons from successful endeavours

A traditional approach centred on weekly lectures, perhaps supported by a tutorial programme, still predominates in modern legal education in Australia. This approach tends to focus on the transmission of knowledge about legal rules and doctrine to students who adopt a largely passive role. Criticisms of the traditional approach have led to law schools expanding their curricula to include the teaching of skills, including the skill of negotiation and an appreciation of legal ethics and professional responsibility. However, in a climate of limited government funding for law schools in Australia, innovation in legal education remains a challenge. This paper considers the successful use of Second Life machinima in two programs, Air Gondwana and Entry into Valhalla and their part in the creation of engaging, effective learning environments. These programs not only engage students in active learning but also facilitate flexibility in their studies and other benefits. The programs yield important lessons concerning the use of machinima innovations in curricula, not only for academics involved in legal education but also those in other disciplines, especially those that rely on traditional passive lectures in their teaching and learning approaches.