Learner-focused evaluation cycles : facilitating learning using feedforward, concurrent and feedback evaluation

There is a growing trend to offer students learning opportunities that are flexible, innovative and engaging. As educators embrace student-centred agile teaching and learning methodologies, which require continuous reflection and adaptation, the need to evaluate students’ learning in a timely manner has become more pressing. Conventional evaluation surveys currently dominate the evaluation landscape internationally, despite recognition that they are insufficient to effectively evaluate curriculum and teaching quality. Surveys often: (1) fail to address the issues for which educators need feedback, (2) constrain student voice, (3) have low response rates and (4) occur too late to benefit current students. Consequently, this paper explores principles of effective feedback to propose a framework for learner-focused evaluation. We apply a three-stage control model, involving feedforward, concurrent and feedback evaluation, to investigate the intersection of assessment and evaluation in agile learning environments. We conclude that learner-focused evaluation cycles can be used to guide action so that evaluation is not undertaken simply for the benefit of future offerings, but rather to benefit current students by allowing ‘real-time’ learning activities to be adapted in the moment. As a result, students become co-producers of learning and evaluation becomes a meaningful, responsive dialogue between students and their instructors.

Evaluating Many Treatments and Biomarkers in Oncology: A New Design

There is a pressing need for more-efficient trial designs for biomarker-stratified clinical trials. We suggest a new approach to trial design that links novel treatment evaluation with the concurrent evaluation of a biomarker within a confirmatory phase II/III trial setting. We describe a new protocol using this approach in advanced colorectal cancer called FOCUS4. The protocol will ultimately answer three research questions for a number of treatments and biomarkers: (1) After a period of first-line chemotherapy, do targeted novel therapies provide signals of activity in different biomarker-defined populations? (2) If so, do these definitively improve outcomes? (3) Is evidence of activity restricted to the biomarker-defined groups? The protocol randomizes novel agents against placebo concurrently across a number of different biomarker-defined population-enriched cohorts: BRAF mutation; activated AKT pathway: PI3K mutation/absolute PTEN loss tumors; KRAS and NRAS mutations; and wild type at all the mentioned genes. Within each biomarker-defined population, the trial uses a multistaged approach with flexibility to adapt in response to planned interim analyses for lack of activity. FOCUS4 is the first test of a protocol that assigns all patients with metastatic colorectal cancer to one of a number of parallel population-enriched, biomarker-stratified randomized trials. Using this approach allows questions regarding efficacy and safety of multiple novel therapies to be answered in a relatively quick and efficient manner, while also allowing for the assessment of biomarkers to help target treatment.

A Research Roadmap for Complementary and Alternative Medicine – What We Need to Know by 2020

Background: The CAMbrella coordination action was funded within the Framework Programme 7. Its aim is to provide a research roadmap for clinical and epidemiological research for complementary and alternative medicine (CAM) that is appropriate for the health needs of European citizens and acceptable to their national research institutes and healthcare providers in both public and private sectors. One major issue in the European research agenda is the demographic change and its impact on health care. Our vision for 2020 is that there is an evidence base that enables European citizens to make informed decisions about CAM, both positive and negative. This roadmap proposes a strategic research agenda for the field of CAM designed to address future European health care challenges. This roadmap is based on the results of CAMbrella’s several work packages, literature reviews and expert discussions including a consensus meeting. Methods: We first conducted a systematic literature review on key issues in clinical and epidemiological research in CAM to identify the general concepts, methods and the strengths and weaknesses of current CAM research. These findings were discussed in a workshop (Castellaro, Italy, September 7–9th 2011) with international CAM experts and strategic and methodological recommendations were defined in order to improve the rigor and relevance of CAM research. These recommendations provide the basis for the research roadmap, which was subsequently discussed in a consensus conference (Järna, Sweden, May 9–11th 2012) with all CAMbrella members and the CAMbrella advisory board. The roadmap was revised after this discussion in CAMbrella Work Package (WP) 7 and finally approved by CAMbrella’s scientific steering committee on September 26th 2012. Results: Our main findings show that CAM is very heterogenous in terms of definitions and legal regulations between the European countries. In addition, citizens’ needs and attitudes towards CAM as well as the use and provision of CAM differ significantly between countries. In terms of research methodology, there was consensus that CAM researchers should make use of all the commonly accepted scientific research methods and employ those with utmost diligence combined in a mixed methods framework. Conclusions: We propose 6 core areas of research that should be investigated to achieve a robust knowledge base and to allow stakeholders to make informed decisions. These are: Research into the prevalence of CAM in Europe: Reviews show that we do not know enough about the circumstances in which CAM is used by Europeans. To enable a common European strategic approach, a clear picture of current use is of the utmost importance. Research into differences regarding citizens’ attitudes and needs towards CAM: Citizens are the driver for CAM utilization. Their needs and views on CAM are a key priority, and their interests must be investigated and addressed in future CAM research. Research into safety of CAM: Safety is a key issue for European citizens. CAM is considered safe, but reliable data is scarce although urgently needed in order to assess the risk and cost-benefit ratio of CAM. Research into the comparative effectiveness of CAM: Everybody needs to know in what situation CAM is a reasonable choice. Therefore, we recommend a clear emphasis on concurrent evaluation of the overall effectiveness of CAM as an additional or alternative treatment strategy in real-world settings. Research into effects of context and meaning: The impact of effects of context and meaning on the outcome of CAM treatments must be investigated; it is likely that they are significant. Research into different models of CAM health care integration: There are different models of CAM being integrated into conventional medicine throughout Europe, each with their respective strengths and limitations. These models should be described and concurrently evaluated; innovative models of CAM provision in health care systems should be one focus for CAM research. We also propose a methodological framework for CAM research. We consider that a framework of mixed methodological approaches is likely to yield the most useful information. In this model, all available research strategies including comparative effectiveness research utilising quantitative and qualitative methods should be considered to enable us to secure the greatest density of knowledge possible. Stakeholders, such as citizens, patients and providers, should be involved in every stage of developing the specific and relevant research questions, study design and the assurance of real-world relevance for the research. Furthermore, structural and sufficient financial support for research into CAM is needed to strengthen CAM research capacity if we wish to understand why it remains so popular within the EU. In order to consider employing CAM as part of the solution to the health care, health creation and self-care challenges we face by 2020, it is vital to obtain a robust picture of CAM use and reliable information about its cost, safety and effectiveness in real-world settings. We need to consider the availability, accessibility and affordability of CAM. We need to engage in research excellence and utilise comparative effectiveness approaches and mixed methods to obtain this data. Our recommendations are both strategic and methodological. They are presented for the consideration of researchers and funders while being designed to answer the important and implicit questions posed by EU citizens currently using CAM in apparently increasing numbers. We propose that the EU actively supports an EUwide strategic approach that facilitates the development of CAM research. This could be achieved in the first instance through funding a European CAM coordinating research office dedicated to foster systematic communication between EU governments, public, charitable and industry funders as well as researchers, citizens and other stakeholders. The aim of this office would be to coordinate research strategy developments and research funding opportunities, as well as to document and disseminate international research activities in this field. With the aim to develop sustainability as second step, a European Centre for CAM should be established that takes over the monitoring and further development of a coordinated research strategy for CAM, as well as it should have funds that can be awarded to foster high quality and robust independent research with a focus on citizens health needs and pan-European collaboration. We wish to establish a solid funding for CAM research to adequately inform health care and health creation decision-making throughout the EU. This centre would ensure that our vision of a common, strategic and scientifically rigorous approach to CAM research becomes our legacy and Europe’s reality. We are confident that our recommendations will serve these essential goals for EU citizens.

Concurrent multi-scale modeling of civil infrastructures for analyses on structural deterioration—Part I : Modeling methodology and strategy

This paper aims to develop the methodology and strategy for concurrent finite element modeling of civil infrastructures at the different scale levels for the purposes of analyses of structural deteriorating. The modeling strategy and method were investigated to develop the concurrent multi-scale model of structural behavior (CMSM-of-SB) in which the global structural behavior and nonlinear damage features of local details in a large complicated structure could be concurrently analyzed in order to meet the needs of structural-state evaluation as well as structural deteriorating. In the proposed method, the “large-scale” modeling is adopted for the global structure with linear responses between stress and strain and the “small-scale” modeling is available for nonlinear damage analyses of the local welded details. A longitudinal truss in steel bridge decks was selected as a case to study how a CMSM-of-SB was developed. The reduced-scale specimen of the longitudinal truss was studied in the laboratory to measure its dynamic and static behavior in global truss and local welded details, while the multi-scale models using constraint equations and substructuring were developed for numerical simulation. The comparison of dynamic and static response between the calculated results by different models indicated that the proposed multi-scale model was found to be the most efficient and accurate. The verification of the model with results from the tested truss under the specific loading showed that, responses at the material scale in the vicinity of local details as well as structural global behaviors could be obtained and fit well with the measured results. The proposed concurrent multi-scale modeling strategy and implementation procedures were applied to Runyang cable-stayed bridge (RYCB) and the CMSM-of-SB of the bridge deck system was accordingly constructed as a practical application.

A three year outcome evaluation of a theory based drink driving education program

This study reports on the impact of a "drink driving education program" taught to grade ten high school students. The program which involves twelve lessons uses strategies based on the Ajzen and Madden theory of planned behavior. Students were trained to use alternatives to drink driving and passenger behaviors. One thousand seven hundred and seventy-four students who had been taught the program in randomly assigned control and intervention schools were followed up three years later. There had been a major reduction in drink driving behaviors in both intervention and control students. In addition to this cohort change there was a trend toward reduced drink driving in the intervention group and a significant reduction in passenger behavior in this group. Readiness to use alternatives suggested that the major impact of the program was on students who were experimenting with the behavior at the time the program was taught. The program seems to have optimized concurrent social attitude and behavior change.

Strategic alliances in building construction : a tender evaluation tool for the public sector

Building construction is a highly competitive and risky business. This competitiveness is compounded where conflicting objectives amongst contracting and subcontracting firms set the stage for an adversarial and potentially destructive business relationship. Clients, especially those from the public sector, need broader tender evaluation criteria to complement the traditional focus on bid price. There is also a need for change in the construction industry—not only to a more cooperative approach between the constructing parties—but also from a confrontationist attitude to a more harmonious relationship between all stakeholders in providing constructed facilities. A strategic alliance is a cooperative relationship between two or more organisations that forms part of their overall strategies, and contributes to achieving their major goals and objectives. Strategic alliances in building construction may provide a useful tool to assist public sector construction managers evaluate tenders and concurrently encourage more cooperative relationships amongst construction stakeholders. This paper begins with an overview of the Australian building construction industry, then reviews the existing strategic alliance literature and describes an analysis framework comprising six attributes of strategic alliances for application to construction organisations—trust, commitment, interdependence, cooperation, communication, and joint problem solving. These attributes are currently being used to collect data from 70 building construction firms in Queensland to assess their respective levels of strategic alliance. Given the trend towards broader indicators of construction firm performance, these attributes are proposed as a tool for use in the tender evaluation process for public works.

Recognising the capacities of dynamic reconfiguration for the QoS assurance of running systems in concurrent and parallel environments

Recognizing the impact of reconfiguration on the QoS of running systems is especially necessary for choosing an appropriate approach to dealing with dynamic evolution of mission-critical or non-stop business systems. The rationale is that the impaired QoS caused by inappropriate use of dynamic approaches is unacceptable for such running systems. To predict in advance the impact, the challenge is two-fold. First, a unified benchmark is necessary to expose QoS problems of existing dynamic approaches. Second, an abstract representation is necessary to provide a basis for modeling and comparing the QoS of existing and new dynamic reconfiguration approaches. Our previous work [8] has successfully evaluated the QoS assurance capabilities of existing dynamic approaches and provided guidance of appropriate use of particular approaches. This paper reinvestigates our evaluations, extending them into concurrent and parallel environments by abstracting hardware and software conditions to design an evaluation context. We report the new evaluation results and conclude with updated impact analysis and guidance.

Meta-evaluation to improve learning, evaluation capacity development and sustainability : Findings from a participatory evaluation project in Nepal

Recent studies suggest that meta-evaluation can be valuable in developing new approaches to evaluation, building evaluation capacities, and enhancing organizational learning. These new extensions of the concept of meta-evaluation are significant, given the growing emphasis on improving the quality and effectiveness of evaluation practices in the South Asian region. Following a review of the literature, this paper presents a case study of the use of concurrent meta-evaluation in the four-year project Assessing Communication for Social Change which developed and trialled a participatory impact assessment methodology in collaboration with a development communication Non-government organization (NGO) in Nepal. Key objectives of the meta-evaluation included to: continuously develop, adapt and improve the impact assessment methodology, Monitoring and Evaluation (M&E) systems and process and other project activities; identify impacts of the project; and build capacities in critical reflection and review. Our analysis indicates that this meta-evaluation was essential to understanding various constraints related to the organizational context that affected the success of the project and the development of improved M&E systems and capacities within the NGO. We identified several limitations of our meta-evaluation methods, which were balanced by the strengths of other methods. Our case study suggests that as well as assessing the quality, credibility and value of evaluation practices, meta-evaluations need to focus on important contextual issues that can have significant impacts on the outcomes of participatory evaluation projects. They include hierarchical organizational cultures, communication barriers, power/knowledge relations, and the time and resources available. Meta-evaluations also need to consider wider issues such as the sustainability of evaluation systems and approaches.

Evaluation of implementation of a healthy food and drink supply strategy throughout the whole school environment in Queensland state schools, Australia

BACKGROUND/OBJECTIVES: This paper reports on the evaluation of the Smart Choices healthy food and drink supply strategy for Queensland schools (Smart Choices) implementation across the whole school environment in state government primary and secondary schools in Queensland, Australia. SUBJECTS/METHODS: Three concurrent surveys using different methods for each group of stakeholders that targeted all 1275 school Principals, all 1258 Parent and Citizens’ Associations (P&Cs) and a random sample of 526 tuckshop convenors throughout Queensland. Nine hundred and seventy-three Principals, 598 P&Cs and 513 tuckshop convenors participated with response rates of 78%, 48% and 98%, respectively. RESULTS: Nearly all Principals (97%), P&Cs (99%) and tuckshop convenors (97%) reported that their school tuckshop had implemented Smart Choices. The majority of Principals and P&Cs reported implementation, respectively, in: school breakfast programs (98 and 92%); vending machine stock (94 and 83%); vending machine advertising (85 and 84%); school events (87 and 88%); school sporting events (81 and 80%); sponsorship and advertising (93 and 84%); fundraising events (80 and 84%); and sporting clubs (73 and 75%). Implementation in curriculum activities, classroom rewards and class parties was reported, respectively, by 97%, 86% and 75% of Principals. Respondents also reported very high levels of understanding of Smart Choices and engagement of the school community. CONCLUSIONS: The results demonstrated that food supply interventions to promote nutrition across all domains of the school environment can be implemented successfully.

The development and evaluation of a medical imaging training immersive environment

Introduction A novel realistic 3D virtual reality (VR) application has been developed to allow medical imaging students at Queensland University of Technology to practice radiographic techniques independently outside the usual radiography laboratory. Methods A flexible agile development methodology was used to create the software rapidly and effectively. A 3D gaming environment and realistic models were used to engender presence in the software while tutor-determined gold standards enabled students to compare their performance and learn in a problem-based learning pedagogy. Results Students reported high levels of satisfaction and perceived value and the software enabled up to 40 concurrent users to prepare for clinical practice. Student feedback also indicated that they found 3D to be of limited value in the desktop version compared to the usual 2D approach. A randomised comparison between groups receiving software-based and traditional practice measured performance in a formative role play with real equipment. The results of this work indicated superior performance with the equipment for the VR trained students (P = 0.0366) and confirmed the value of VR for enhancing 3D equipment-based problem-solving skills. Conclusions Students practising projection techniques virtually performed better at role play assessments than students practising in a traditional radiography laboratory only. The application particularly helped with 3D equipment configuration, suggesting that teaching 3D problem solving is an ideal use of such medical equipment simulators. Ongoing development work aims to establish the role of VR software in preparing students for clinical practice with a range of medical imaging equipment.

Development, reliability and validity of the queensland evaluation of wheelchair skills (QEWS)

- Objectives To develop and test a valid and reliable assessment of wheelchair skills for individuals with spinal cord injuries (SCI); the Queensland Evaluation of Wheelchair Skills (QEWS). - Setting Hospital, Australia. - Methods Phase 1: Four Delphi panel rounds with clinical experts were used to develop the QEWS. Phase 2: Intra-rater and inter-rater reliability of the QEWS items were examined in 100 people with SCI. Phase 3a: Concurrent validity was investigated by examining the association between QEWS total scores and physiotherapists’ global ratings of wheelchair skill performance. Phase 3b: Construct validity was tested in 20 people with recent SCI by examining change in QEWS total scores between when they first mobilised in a wheelchair and scores obtained 10 weeks later. - Results Phase 1: The QEWS was developed. Phase 2: The intra-class correlation coefficients reflecting the intra-rater reliability and the inter-rater reliability for the QEWS total score were 1.00 and 0.98, with scores being within one point of each other 96 and 91% of the time, respectively. Phase 3a: The QEWS total scores were comparable with the global rating of wheelchair skill performance (r2=0.93). Phase 3b: The QEWS scores changed by a median (interquartile range (IQR)) of 4 (1 to 6) points over the 10-week period following first wheelchair mobilisation. - Conclusion The QEWS is a valid and reliable tool for measuring wheelchair skills in individuals with SCI. The QEWS is efficient and practical to administer and does not require specialised equipment.

An evaluation of a new product data model

Engineering companies face many challenges today such as increased competition, higher expectations from consumers and decreasing product lifecycle times. This means that product development times must be reduced to meet these challenges. Concurrent engineering, reuse of engineering knowledge and the use of advanced methods and tools are among the ways of reducing product development times. Concurrent engineering is crucial in making sure that the products are designed with all issues considered simultaneously. The reuse of engineering knowledge allows existing solutions to be reused. It can also help to avoid the mistakes made in previous designs. Computer-based tools are used to store information, automate tasks, distribute work, perform simulation and so forth. This research concerns the evaluation of tools that can be used to support the design process. These tools are evaluated in terms of the capture of information generated during the design process. This information is vital to allow the reuse of knowledge. Present CAD systems store only information on the final definition of the product such as geometry, materials and manufacturing processes. Product Data Management (PDM) systems can manage all this CAD information along with other product related information. The research includes the evaluation of two PDM systems, Windchill and Metaphase, using the design of a single-handed water tap as a case study. The two PDMs were then compared to PROSUS/DDM. PROSUS is the Process-Based Support System proposed by [Blessing 94] using the same case study. The Design Data Model is the product data model that includes PROSUS. The results look promising. PROSUS/DDM is able to capture most design information and structure and present it logically. The design process and product information is related and stored within the DDM structure. The PDMs can capture most design information, but information from early stages of design is stored only as unstructured documentation. Some problems were found with PROSUS/DDM. A proposal is made that may make it possible to resolve these problems, but this will require further research.

Hypothesis Formation and Evaluation in Medical Diagnosis

This thesis describes some aspects of a computer system for doing medical diagnosis in the specialized field of kidney disease. Because such a system faces the spectre of combinatorial explosion, this discussion concentrates on heuristics which control the number of concurrent hypotheses and efficient "compiled" representations of medical knowledge. In particular, the differential diagnosis of hematuria (blood in the urine) is discussed in detail. A protocol of a simulated doctor/patient interaction is presented and analyzed to determine the crucial structures and processes involved in the diagnosis procedure. The data structure proposed for representing medical information revolves around elementary hypotheses which are activated when certain disposing of findings, activating hypotheses, evaluating hypotheses locally and combining hypotheses globally is examined for its heuristic implications. The thesis attempts to fit the problem of medical diagnosis into the framework of other Artifcial Intelligence problems and paradigms and in particular explores the notions of pure search vs. heuristic methods, linearity and interaction, local vs. global knowledge and the structure of hypotheses within the world of kidney disease.

Above-water reflectance for the evaluation of adjacency effects in Earth observation data: initial results and methods comparison for near-coastal waters in the Western Channel, UK

Un-supervised hyperspectral remote-sensing reflectance data (<15 km from the shore) were collected from a moving research vessel. Two different processing methods were compared. The results were similar to concurrent Aqua-MODIS and Suomi-NPP-VIIRS satellite data.