The evaluation of qualitative methods selection in the field of design and emotion

The aim of this paper is to aid researchers in selecting appropriate qualitative methods in order to develop and improve future studies in the field of emotional design. These include observations, think-aloud protocols, questionnaires, diaries and interviews. Based on the authors’ experiences, it is proposed that the methods under review can be successfully used for collecting data on emotional responses to evaluate user product relationships. This paper reviews the methods; discusses the suitability, advantages and challenges in relation to design and emotion studies. Furthermore, the paper outlines the potential impact of technology on the application of these methods, discusses the implications of these methods for emotion research and concludes with recommendations for future work in this area.

Evidence of disagreement between patient-perceived change and conventional longitudinal evaluation of change in health-related quality of life among older adults

Objective: To identify agreement levels between conventional longitudinal evaluation of change (post–pre) and patient-perceived change (post–then test) in health-related quality of life. Design: A prospective cohort investigation with two assessment points (baseline and six-month follow-up) was implemented. Setting: Community rehabilitation setting. Subjects: Frail older adults accessing community-based rehabilitation services. Intervention: Nil as part of this investigation. Main measures: Conventional longitudinal change in health-related quality of life was considered the difference between standard EQ-5D assessments completed at baseline and follow-up. To evaluate patient-perceived change a ‘then test’ was also completed at the follow-up assessment. This required participants to report (from their current perspective) how they believe their health-related quality of life was at baseline (using the EQ-5D). Patient-perceived change was considered the difference between ‘then test’ and standard follow-up EQ-5D assessments. Results: The mean (SD) age of participants was 78.8 (7.3). Of the 70 participants 62 (89%) of data sets were complete and included in analysis. Agreement between conventional (post–pre) and patient-perceived (post–then test) change was low to moderate (EQ-5D utility intraclass correlation coefficient (ICC)¼0.41, EQ-5D visual analogue scale (VAS) ICC¼0.21). Neither approach inferred greater change than the other (utility P¼0.925, VAS P¼0.506). Mean (95% confidence interval (CI)) conventional change in EQ-5D utility and VAS were 0.140 (0.045,0.236) and 8.8 (3.3,14.3) respectively, while patient-perceived change was 0.147 (0.055,0.238) and 6.4 (1.7,11.1) respectively. Conclusions: Substantial disagreement exists between conventional longitudinal evaluation of change in health-related quality of life and patient-perceived change in health-related quality of life (as measured using a then test) within individuals.

Pattern-Based Evaluation of Scientific Workflow Management Systems

In recent years several scientific Workflow Management Systems (WfMSs) have been developed with the aim to automate large scale scientific experiments. As yet, many offerings have been developed, but none of them has been promoted as an accepted standard. In this paper we propose a pattern-based evaluation of three among the most widely used scientific WfMSs: Kepler, Taverna and Triana. The aim is to compare them with traditional business WfMSs, emphasizing the strengths and deficiencies of both systems. Moreover, a set of new patterns is defined from the analysis of the three considered systems.

Last shall be first : a field study of biases in sequential performance evaluation on the Idol series

When performances are evaluated they are very often presented in a sequential order. Previous research suggests that the sequential presentation of alternatives may induce systematic biases in the way performances are evaluated. Such a phenomenon has been scarcely studied in economics. Using a large dataset of performance evaluation in the Idol series (N=1522), this paper presents new evidence about the systematic biases in sequential evaluation of performances and the psychological phenomena at the origin of these biases.

Information dependencies between architects and services engineers for early design evaluation : a framework for an energy design tool for architects

Effective strategies for the design of effi cient and environmentally sensitive buildings require a close collaboration between architects and engineers in the design of the building shell and environmental control systems at the outset of projects. However, it is often not practical for engineers to be involved early on in the design process. It is therefore essential that architects be able to perform preliminary energy analyses to evaluate their proposed designs prior to the major building characteristics becoming fi xed. Subsequently, a need exists for a simplifi ed energy design tool for architects. This paper discusses the limitations of existing analysis software in supporting early design explorations and proposes a framework for the development of a tool that provides decision support by permitting architects to quickly assess the performance of design alternatives.

Concomitant or sequential administration of live attenuated Japanese encephalitis chimeric virus vaccine and yellow fever 17D vaccine : randomized double-blind phase II evaluation of safety and immunogenicity

A randomized, double-blind, study was conducted to evaluate the safety, tolerability and immunogenicity of a live attenuated Japanese encephalitis chimeric virus vaccine (JE-CV) co-administered with live attenuated yellow fever (YF) vaccine (YF-17D strain; Stamaril(®), Sanofi Pasteur) or administered successively. Participants (n = 108) were randomized to receive: YF followed by JE-CV 30 days later, JE followed by YF 30 days later, or the co-administration of JE and YF followed or preceded by placebo 30 days later or earlier. Placebo was used in a double-dummy fashion to ensure masking. Neutralizing antibody titers against JE-CV, YF-17D and selected wild-type JE virus strains was determined using a 50% serum-dilution plaque reduction neutralization test. Seroconversion was defined as the appearance of a neutralizing antibody titer above the assay cut-off post-immunization when not present pre-injection at day 0, or a least a four-fold rise in neutralizing antibody titer measured before the pre-injection day 0 and later post vaccination samples. There were no serious adverse events. Most adverse events (AEs) after JE vaccination were mild to moderate in intensity, and similar to those reported following YF vaccination. Seroconversion to JE-CV was 100% and 91% in the JE/YF and YF/JE sequential vaccination groups, respectively, compared with 96% in the co-administration group. All participants seroconverted to YF vaccine and retained neutralizing titers above the assay cut-off at month six. Neutralizing antibodies against JE vaccine were detected in 82-100% of participants at month six. These results suggest that both vaccines may be successfully co-administered simultaneously or 30 days apart.

Evaluation of the year 1 literacy and numeracy checkpoints assessments trial - 2010

The draft Year 1 Literacy and Numeracy Checkpoints Assessments were in open and supported trial during Semester 2, 2010. The purpose of these trials was to evaluate the Year 1 Literacy and Numeracy Checkpoints Assessments (hereafter the Year 1 Checkpoints) that were designed in 2009 as a way to incorporate the use of the Year 1 Literacy and Numeracy Indicators as formative assessment in Year 1 in Queensland Schools. In these trials there were no mandated reporting requirements. The processes of assessment were related to future teaching decisions. As such the trials were trials of materials and the processes of using those materials to assess students, plan and teach in year 1 classrooms. In their current form the Year 1 Checkpoints provide assessment resources for teachers to use in February, June and October. They aim to support teachers in monitoring children's progress and making judgments about their achievement of the targeted P‐3 Literacy and Numeracy Indicators by the end of Year 1 (Queensland Studies Authority, 2010 p. 1). The Year 1 Checkpoints include support materials for teachers and administrators, an introductory statement on assessment, work samples, and a Data Analysis Assessment Record (DAAR) to record student performance. The Supported Trial participants were also supported with face‐to‐face and on‐line training sessions, involvement in a moderation process after the October Assessments, opportunities to participate in discussion forums as well as additional readings and materials. The assessment resources aim to use effective early years assessment practices in that the evidence is gathered from hands‐on teaching and learning experiences, rather than more formal assessment methods. They are based in a model of assessment for learning, and aim to support teachers in the “on‐going process of determining future learning directions” (Queensland Studies Authority, 2010 p. 1) for all students. Their aim is to focus teachers on interpreting and analysing evidence to make informed judgments about the achievement of all students, as a way to support subsequent planning for learning and teaching. The Evaluation of the Year 1 Literacy and Numeracy Checkpoints Assessments Supported Trial (hereafter the Evaluation) aimed to gather information about the appropriateness, effectiveness and utility of the Year 1 Checkpoints Assessments from early years’ teachers and leaders in up to one hundred Education Queensland schools who had volunteered to be part of the Supported Trial. These sample schools represent schools across a variety of Education Queensland regions and include schools with:  - A high Indigenous student population; - Urban, rural and remote school locations; - Single and multi‐age early phase classes; - A high proportion of students from low SES backgrounds. The purpose of the Evaluation was to: Evaluate the materials and report on the views of school‐based staff involved in the trial on the process, materials, and assessment practices utilised. The Evaluation has reviewed the materials, and used surveys, interviews, and observations of processes and procedures to collect relevant data to help present an informed opinion on the Year 1 Checkpoints as assessment for the early years of schooling. Student work samples and teacher planning and assessment documents were also collected. The evaluation has not evaluated the Year 1 Checkpoints in any other capacity than as a resource for Year 1 teachers and relevant support staff.

An evaluation framework for sustaining the impact of educational development

Notwithstanding significant efforts by international aid agencies, aid ineffectiveness became apparent in 1990s as the impact of continued development intervention did not endure the expected outcomes. Conventional monitoring and evaluation by those agencies is critiqued for focusing on measuring project outcomes and giving little attention to aspects of sustainability. As a result, devising a rigorous evaluation framework for educational development has been sought in light of recent paradigm shifts in international development. This paper reports on a case study of an Egyptian educational development project highlighting the importance of transforming the evaluation procedures to process evaluation so as to enhance project impact and longevity. This requires building evaluation capacity of the aid recipient country.

Early Intervention Program Final Evaluation Report

The Early Intervention Program aims to facilitate contact between children and their non-residential parent as soon as practicable after breakdown and separation of the immediate family unit. The Program is auspiced by the Sunshine Coast Family Contact Centre Association. The Program has been offered since late 2007 following receipt of a grant from the National Community Crime Prevention Program [NCCP] for Domestic Violence Prevention. This external evaluator report summarised main achievements with respect to meeting program objectives and also makes recommentations in view the of the continuation of this innovative program.

An assessment system for evaluation of driving competencies

Automobiles have deeply impacted the way in which we travel but they have also contributed to many deaths and injury due to crashes. A number of reasons for these crashes have been pointed out by researchers. Inexperience has been identified as a contributing factor to road crashes. Driver’s driving abilities also play a vital role in judging the road environment and reacting in-time to avoid any possible collision. Therefore driver’s perceptual and motor skills remain the key factors impacting on road safety. Our failure to understand what is really important for learners, in terms of competent driving, is one of the many challenges for building better training programs. Driver training is one of the interventions aimed at decreasing the number of crashes that involve young drivers. Currently, there is a need to develop comprehensive driver evaluation system that benefits from the advances in Driver Assistance Systems. A multidisciplinary approach is necessary to explain how driving abilities evolves with on-road driving experience. To our knowledge, driver assistance systems have never been comprehensively used in a driver training context to assess the safety aspect of driving. The aim and novelty of this thesis is to develop and evaluate an Intelligent Driver Training System (IDTS) as an automated assessment tool that will help drivers and their trainers to comprehensively view complex driving manoeuvres and potentially provide effective feedback by post processing the data recorded during driving. This system is designed to help driver trainers to accurately evaluate driver performance and has the potential to provide valuable feedback to the drivers. Since driving is dependent on fuzzy inputs from the driver (i.e. approximate distance calculation from the other vehicles, approximate assumption of the other vehicle speed), it is necessary that the evaluation system is based on criteria and rules that handles uncertain and fuzzy characteristics of the driving tasks. Therefore, the proposed IDTS utilizes fuzzy set theory for the assessment of driver performance. The proposed research program focuses on integrating the multi-sensory information acquired from the vehicle, driver and environment to assess driving competencies. After information acquisition, the current research focuses on automated segmentation of the selected manoeuvres from the driving scenario. This leads to the creation of a model that determines a “competency” criterion through the driving performance protocol used by driver trainers (i.e. expert knowledge) to assess drivers. This is achieved by comprehensively evaluating and assessing the data stream acquired from multiple in-vehicle sensors using fuzzy rules and classifying the driving manoeuvres (i.e. overtake, lane change, T-crossing and turn) between low and high competency. The fuzzy rules use parameters such as following distance, gaze depth and scan area, distance with respect to lanes and excessive acceleration or braking during the manoeuvres to assess competency. These rules that identify driving competency were initially designed with the help of expert’s knowledge (i.e. driver trainers). In-order to fine tune these rules and the parameters that define these rules, a driving experiment was conducted to identify the empirical differences between novice and experienced drivers. The results from the driving experiment indicated that significant differences existed between novice and experienced driver, in terms of their gaze pattern and duration, speed, stop time at the T-crossing, lane keeping and the time spent in lanes while performing the selected manoeuvres. These differences were used to refine the fuzzy membership functions and rules that govern the assessments of the driving tasks. Next, this research focused on providing an integrated visual assessment interface to both driver trainers and their trainees. By providing a rich set of interactive graphical interfaces, displaying information about the driving tasks, Intelligent Driver Training System (IDTS) visualisation module has the potential to give empirical feedback to its users. Lastly, the validation of the IDTS system’s assessment was conducted by comparing IDTS objective assessments, for the driving experiment, with the subjective assessments of the driver trainers for particular manoeuvres. Results show that not only IDTS was able to match the subjective assessments made by driver trainers during the driving experiment but also identified some additional driving manoeuvres performed in low competency that were not identified by the driver trainers due to increased mental workload of trainers when assessing multiple variables that constitute driving. The validation of IDTS emphasized the need for an automated assessment tool that can segment the manoeuvres from the driving scenario, further investigate the variables within that manoeuvre to determine the manoeuvre’s competency and provide integrated visualisation regarding the manoeuvre to its users (i.e. trainers and trainees). Through analysis and validation it was shown that IDTS is a useful assistance tool for driver trainers to empirically assess and potentially provide feedback regarding the manoeuvres undertaken by the drivers.

Test and evaluation of non conventional instrument transformers and sampled value process bus on Powerlink's transmission network

IEC 61850 Process Bus technology has the potential to improve cost, performance and reliability of substation design. Substantial costs associated with copper wiring (designing, documentation, construction, commissioning and troubleshooting) can be reduced with the application of digital Process Bus technology, especially those based upon international standards. An IEC 61850-9-2 based sampled value Process Bus is an enabling technology for the application of Non-Conventional Instrument Transformers (NCIT). Retaining the output of the NCIT in its native digital form, rather than conversion to an analogue output, allows for improved transient performance, dynamic range, safety, reliability and reduced cost. In this paper we report on a pilot installation using NCITs communicating across a switched Ethernet network using the UCAIug Implementation Guideline for IEC 61850-9-2 (9-2 Light Edition or 9-2LE). This system was commissioned in a 275 kV Line Reactor bay at Powerlink Queensland’s Braemar substation in 2009, with sampled value protection IEDs 'shadowing' the existing protection system. The results of commissioning tests and twelve months of service experience using a Fibre Optic Current Transformer (FOCT) from Smart Digital Optics (SDO) are presented, including the response of the system to fault conditions. A number of remaining issues to be resolved to enable wide-scale deployment of NCITs and IEC 61850-9-2 Process Bus technology are also discussed.

Development of new evaluation methods for qualitative alternatives, using fuzzy calculations

Evaluation, selection and finally decision making are all among important issues, which engineers face in long run of projects. Engineers implement mathematical and nonmathematical methods to make accurate and correct decisions, whenever needed. As extensive as these methods are, effects of any selected method on outputs achieved and decisions made are still suspicious. This is more controversial and challengeable, where evaluation is made among non-quantitative alternatives. In civil engineering and construction management problems, criteria include both quantitative and qualitative ones, such as aesthetic, construction duration, building and operation costs, and environmental considerations. As the result, decision making frequently takes place among non-quantitative alternatives. It should be noted that traditional comparison methods, including clear-cut and inflexible mathematics, have always been criticized. This paper demonstrates a brief review of traditional methods of evaluating alternatives. It also offers a new decision making method using, fuzzy calculations. The main focus of this research is some engineering issues, which have flexible nature and vague borders. Suggested method provides analyzability of evaluation for decision makers. It is also capable to overcome multi criteria and multi-referees problems. In order to ease calculations, a program named DeMA is introduced.

Strategic alliances in building construction : a tender evaluation tool for the public sector

Building construction is a highly competitive and risky business. This competitiveness is compounded where conflicting objectives amongst contracting and subcontracting firms set the stage for an adversarial and potentially destructive business relationship. Clients, especially those from the public sector, need broader tender evaluation criteria to complement the traditional focus on bid price. There is also a need for change in the construction industry—not only to a more cooperative approach between the constructing parties—but also from a confrontationist attitude to a more harmonious relationship between all stakeholders in providing constructed facilities. A strategic alliance is a cooperative relationship between two or more organisations that forms part of their overall strategies, and contributes to achieving their major goals and objectives. Strategic alliances in building construction may provide a useful tool to assist public sector construction managers evaluate tenders and concurrently encourage more cooperative relationships amongst construction stakeholders. This paper begins with an overview of the Australian building construction industry, then reviews the existing strategic alliance literature and describes an analysis framework comprising six attributes of strategic alliances for application to construction organisations—trust, commitment, interdependence, cooperation, communication, and joint problem solving. These attributes are currently being used to collect data from 70 building construction firms in Queensland to assess their respective levels of strategic alliance. Given the trend towards broader indicators of construction firm performance, these attributes are proposed as a tool for use in the tender evaluation process for public works.

The development of the extended adolescent injury checklist (E-AIC) : a measure for injury prevention program evaluation

The Extended Adolescent Injury Checklist (E-AIC), a self-report measure of injury based on the model of the Adolescent Injury Checklist (AIC), was developed for use in the evaluation of school-based interventions. The three stages of this development involved focus groups with adolescents and consultations with medical staff, pilot testing of the revised AIC in a high school context, and use of the finalised checklist in pre- and post-questionnaires to examine its utility. Results revealed that responses to the final version of the E-AIC were meaningful and remained consistent over time. The E-AIC appears to be a promising measure of adolescent injury that is simple, time-efficient and appropriate for use in the evaluation of school-based injury prevention programs.