REFRAME : a new approach to evaluation in higher education

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.

Understanding IT value : a multilevel, complex, and adaptive system perspective

One remaining difficulty in the Information Technology (IT) business value evaluation domain is the direct linkage between IT value and the underlying determinants of IT value or surrogates of IT value. This paper proposes a research that examines the interacting effects of the determinants of IT value, and their influences on IT value. The overarching research question is how those determinants interact with each other and affect the IT value at organizational value. To achieve this, this research embraces a multilevel, complex, and adaptive system view, where the IT value emerges from the interacting of underlying determinants. This research is theoretically grounded on three organizational theories – multilevel theory, complex adaptive systems theory, and adaptive structuration theory. By integrating those theoretical paradigms, this research proposes a conceptual model that focuses on the process where IT value is created from interactions of those determinants. To answer the research question, agent-based modeling technique is used in this research to build a computational representation based on the conceptual model. Computational experimentation will be conducted based on the computational representation. Validation procedures will be applied to consolidate the validity of this model. In the end, hypotheses will be tested using computational experimentation data.

Quantitative evaluation of polymer gel dosimeters by broadband ultrasound attenuation

Ultrasound has been examined previously as an alternative readout method for irradiated polymer gel dosimeters, with authors reporting varying dose response to ultrasound transmission measurements. In this current work we extend previous work to measure the broadband ultrasound attenuation (BUA) response of irradiated PAGAT gel dosimeters, using a novel ultrasound computed tomography system.

The PHP intelligent tutoring system

Teaching introductory programming has challenged educators through the years. Although Intelligent Tutoring Systems that teach programming have been developed to try to reduce the problem, none have been developed to teach web programming. This paper describes the design and evaluation of the PHP Intelligent Tutoring System (PHP ITS) which addresses this problem. The evaluation process showed that students who used the PHP ITS showed a significant improvement in test scores

The design science research roadmap : in progress evaluation

Although Design Science Research (DSR) is now an accepted approach to research in the Information Systems (IS) discipline, consensus on the methodology of DSR has yet to be achieved. Lack of a comprehensive and detailed methodology for Design Science Research (DSR) in the Information System (IS) discipline is a main issue. Prior research (the parent-study) aimed to remedy this situation and resulted in the DSR-Roadmap (Alturki et al., 2011a). Continuing empirical validation and revision of the DSR-Roadmap strives towards a methodology with appropriate levels of detail, integration, and completeness for novice researchers to efficiently and effectively conduct and report DSR in IS. The sub-study reported herein contributes to this larger, ongoing effort. This paper reports results from a formative evaluation effort of the DSR-Roadmap conducted using focus group analysis. Generally, participants endorsed the utility and intuitiveness of the DSR-Roadmap, while also suggesting valuable refinements. Both parent-study and sub-study make methodological contributions. The parent-study is the first attempt of utilizing DSR to develop a research methodology showing an example of how to use DSR in research methodology construction. The sub-study demonstrates the value of the focus group method in DSR for formative product evaluation.

Exploring empathy as a variable in the evaluation of professional development programmes for palliative care nurses

Research indicates that empathy, a quality regarded as fundamentally important to nursing practice, is a teachable skill. Because empathic nurse-patient relationships are particularly important in the care of the terminally ill, this has direct relevance to the professional development of palliative care nurses. This article discusses the place of empathy as a criterion variable in the evaluation of a professional development program for palliative care nurses introduced at the Centre for Mental Health Nursing Research at Queensland University of Technology, Brisbane, Australia. A modified version of the Staff-Patient Interaction Response Scale (SPIRS) was used as a pre- and postintervention measure to assess the expressed empathy of the participating nurses. The modifications to SPIR and its coding system to make it suitable for palliative care nursing, and the mechanisms for improving and evaluating the reliability of this instrument will be discussed. The full description of this particular modification of SPIRS for palliative care research is provided as an example of how this instrument could be used in projects for which nurses undertake the difficult task of providing compassionate care to the terminally ill.

Retrieval of health advice on the web: AEHRC at ShARe/CLEF eHealth Evaluation Lab Task 3

This paper details the participation of the Australian e- Health Research Centre (AEHRC) in the ShARe/CLEF 2013 eHealth Evaluation Lab { Task 3. This task aims to evaluate the use of information retrieval (IR) systems to aid consumers (e.g. patients and their relatives) in seeking health advice on the Web. Our submissions to the ShARe/CLEF challenge are based on language models generated from the web corpus provided by the organisers. Our baseline system is a standard Dirichlet smoothed language model. We enhance the baseline by identifying and correcting spelling mistakes in queries, as well as expanding acronyms using AEHRC's Medtex medical text analysis platform. We then consider the readability and the authoritativeness of web pages to further enhance the quality of the document ranking. Measures of readability are integrated in the language models used for retrieval via prior probabilities. Prior probabilities are also used to encode authoritativeness information derived from a list of top-100 consumer health websites. Empirical results show that correcting spelling mistakes and expanding acronyms found in queries signi cantly improves the e ectiveness of the language model baseline. Readability priors seem to increase retrieval e ectiveness for graded relevance at early ranks (nDCG@5, but not precision), but no improvements are found at later ranks and when considering binary relevance. The authoritativeness prior does not appear to provide retrieval gains over the baseline: this is likely to be because of the small overlap between websites in the corpus and those in the top-100 consumer-health websites we acquired.

Identify disorders in health records using Conditional Random Fields and Metamap: AEHRC at ShARe/CLEF 2013 eHealth Evaluation Lab Task 1

The Australian e-Health Research Centre (AEHRC) recently participated in the ShARe/CLEF eHealth Evaluation Lab Task 1. The goal of this task is to individuate mentions of disorders in free-text electronic health records and map disorders to SNOMED CT concepts in the UMLS metathesaurus. This paper details our participation to this ShARe/CLEF task. Our approaches are based on using the clinical natural language processing tool Metamap and Conditional Random Fields (CRF) to individuate mentions of disorders and then to map those to SNOMED CT concepts. Empirical results obtained on the 2013 ShARe/CLEF task highlight that our instance of Metamap (after ltering irrelevant semantic types), although achieving a high level of precision, is only able to identify a small amount of disorders (about 21% to 28%) from free-text health records. On the other hand, the addition of the CRF models allows for a much higher recall (57% to 79%) of disorders from free-text, without sensible detriment in precision. When evaluating the accuracy of the mapping of disorders to SNOMED CT concepts in the UMLS, we observe that the mapping obtained by our ltered instance of Metamap delivers state-of-the-art e ectiveness if only spans individuated by our system are considered (`relaxed' accuracy).

Multi-converter topology evaluation for connection of low voltage DC sources

A design for a cascaded multilevel DC-DC converter is proposed. The applications of a multilevel converter and the design issues involved in changing from a single converter to multiple converters are discussed. Implementation of the multilevel system using multiple Cuk converters is suggested and explanations of design decisions are given. The merits of the proposed design are discussed.

Development and formative evaluation of a motorcycle rider training intervention to address risk taking

The need to address on-road motorcycle safety in Australia is important due to the disproportionately high percentage of riders and pillions killed and injured each year. One approach to preventing motorcycle-related injury is through training and education. However, motorcycle rider training lacks empirical support as an effective road safety countermeasure to reduce crash involvement. Previous reviews have highlighted that risk-taking is a contributing factor in many motorcycle crashes, rather than merely a lack of vehicle-control skills (Haworth & Mulvihill, 2005; Jonah, Dawson & Bragg, 1982; Watson et al, 1996). Hence, though the basic vehicle-handling skills and knowledge of road rules that are taught in most traditional motorcycle licence training programs may be seen as an essential condition of safe riding, they do not appear to be sufficient in terms of crash reduction. With this in mind there is considerable scope for the improvement of program focus and content for rider training and education. This program of research examined an existing traditional pre-licence motorcycle rider training program and formatively evaluated the addition of a new classroom-based module to address risky riding; the Three Steps to Safer Riding program. The pilot program was delivered in the real world context of the Q-Ride motorcycle licensing system in the state of Queensland, Australia. Three studies were conducted as part of the program of research: Study 1, a qualitative investigation of delivery practices and student learning needs in an existing rider training course; Study 2, an investigation of the extent to which an existing motorcycle rider training course addressed risky riding attitudes and motives; and Study 3, a formative evaluation of the new program. A literature review as well as the investigation of learning needs for motorcyclists in Study 1 aimed to inform the initial planning and development of the Three Steps to Safer Riding program. Findings from Study 1 suggested that the training delivery protocols used by the industry partner training organisation were consistent with a learner-centred approach and largely met the learning needs of trainee riders. However, it also found that information from the course needs to be reinforced by on-road experiences for some riders once licensed and that personal meaning for training information was not fully gained until some riding experience had been obtained. While this research informed the planning and development of the new program, a project team of academics and industry experts were responsible for the formulation of the final program. Study 2 and Study 3 were conducted for the purpose of formative evaluation and program refinement. Study 2 served primarily as a trial to test research protocols and data collection methods with the industry partner organisation and, importantly, also served to gather comparison data for the pilot program which was implemented with the same rider training organisation. Findings from Study 2 suggested that the existing training program of the partner organisation generally had a positive (albeit small) effect on safety in terms of influencing attitudes to risk taking, the propensity for thrill seeking, and intentions to engage in future risky riding. However, maintenance of these effects over time and the effects on riding behaviour remain unclear due to a low response rate upon follow-up 24 months after licensing. Study 3 was a formative evaluation of the new pilot program to establish program effects and possible areas for improvement. Study 3a examined the short term effects of the intervention pilot on psychosocial factors underpinning risky riding compared to the effects of the standard traditional training program (examined in Study 2). It showed that the course which included the Three Steps to Safer Riding program elicited significantly greater positive attitude change towards road safety than the existing standard licensing course. This effect was found immediately following training, and mean scores for attitudes towards safety were also maintained at the 12 month follow-up. The pilot program also had an immediate effect on other key variables such as risky riding intentions and the propensity for thrill seeking, although not significantly greater than the traditional standard training. A low response rate at the 12 month follow-up unfortunately prevented any firm conclusions being drawn regarding the impact of the pilot program on self-reported risky riding once licensed. Study 3a further showed that the use of intermediate outcomes such as self-reported attitudes and intentions for evaluation purposes provides insights into the mechanisms underpinning risky riding that can be changed by education and training. A multifaceted process evaluation conducted in Study 3b confirmed that the intervention pilot was largely delivered as designed, with course participants also rating most aspects of training delivery highly. The complete program of research contributed to the overall body of knowledge relating to motorcycle rider training, with some potential implications for policy in the area of motorcycle rider licensing. A key finding of the research was that psychosocial influences on risky riding can be shaped by structured education that focuses on awareness raising at a personal level and provides strategies to manage future riding situations. However, the formative evaluation was mainly designed to identify areas of improvement for the Three Steps to Safer Riding program and found several areas of potential refinement to improve future efficacy of the program. This included aspects of program content, program delivery, resource development, and measurement tools. The planned future follow-up of program participants' official crash and traffic offence records over time may lend further support for the application of the program within licensing systems. The findings reported in this thesis offer an initial indication that the Three Steps to Safer Riding is a useful resource to accompany skills-based training programs.

The structured and practical approach in development of decision support system for consultant selection in public sector infrastructure project

Nowadays, most of the infrastructure development projects undertaken are complex in nature. Practically, public clients who do not have a good understanding of the design and management may suffer severe losses, especially for infrastructure projects. There is a need for luring the right consultant to secure client's investment in infrastructure developments. Throughout the project life cycle, consultants play vital role from the inception to completion stage of a project. A few studies in Malaysia show that infrastructure projects involving irrigation and drainage have experience problems such as poor workmanship, delay and cost overrun due to the consultant's inability or the client incompetence of recruiting consultants in time. This highlights the need of aided decision making and an efficient system to select the best consultant by using Decision Support System (DSS). On the other hand, recent trends reveal that most DSS in construction only concentrate on decision model development. These models are impractical and unused as they are complicated or difficult for laymen such as project managers to utilize. Thus, this research attempts to develop an efficient DSS for consultant selection namely consultDeSS. Driven by the motivation and research aims, this study deployed Design Science Research Methodology (DSRM) dominant with a combination of case studies at the Malaysian Department of Irrigation and Drainage (DID). Two real projects involving irrigation and drainage infrastructure were used to design, implement and evaluate the artefact. The 3-tier consultDeSS was revised after the evaluation and the design was significantly improved based on user feedback. By developing desirable tools that fit client's needs will enhance the productivity and minimize conflict within groups and organisations. The tool is more usable and efficient compared to previous studies in construction. Thus, this research has demonstrated a purposeful artefact with a practical and valid structured development approach that is applicable in a variety of problems in construction discipline.

System level tests of transformer differential protection using an IEC 61850 process bus

The IEC 61850 family of standards for substation communication systems were released in the early 2000s, and include IEC 61850-8-1 and IEC 61850-9-2 that enable Ethernet to be used for process-level connections between transmission substation switchyards and control rooms. This paper presents an investigation of process bus protection performance, as the in-service behavior of multi-function process buses is largely unknown. An experimental approach was adopted that used a Real Time Digital Simulator and 'live' substation automation devices. The effect of sampling synchronization error and network traffic on transformer differential protection performance was assessed and compared to conventional hard-wired connections. Ethernet was used for all sampled value measurements, circuit breaker tripping, transformer tap-changer position reports and Precision Time Protocol synchronization of sampled value merging unit sampling. Test results showed that the protection relay under investigation operated correctly with process bus network traffic approaching 100% capacity. The protection system was not adversely affected by synchronizing errors significantly larger than the standards permit, suggesting these requirements may be overly conservative. This 'closed loop' approach, using substation automation hardware, validated the operation of protection relays under extreme conditions. Digital connections using a single shared Ethernet network outperformed conventional hard-wired solutions.

REFRAME: a new approach to evaluation in higher education

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.

Evaluation of brief dietary questions to estimate vegetable and fruit consumption – using serum carotenoids and red-cell folate

Objective: To evaluate responses to self-administered brief questions regarding consumption of vegetables and fruit by comparison with blood levels of serum carotenoids and red-cell folate. Design: A cross-sectional study in which participants reported their usual intake of fruit and vegetables in servings per day, and serum levels of five carotenoids (α-carotene, β-carotene, β-cryptoxanthin, lutein/zeaxanthin and lycopene) and red-cell folate were measured. Serum carotenoid levels were determined by high-performance liquid chromatography, and red-cell folate by an automated immunoassay system. Settings and subjects: Between October and December 2000, a sample of 1598 adults aged 25 years and over, from six randomly selected urban centres in Queensland, Australia, were examined as part of a national study conducted to determine the prevalence of diabetes and associated cardiovascular risk factors. Results: Statistically significant (P<0.01) associations with vegetable and fruit intake (categorised into groups: ≤1 serving, 2–3 servings and ≥4 servings per day) were observed for α-carotene, β-carotene, β-cryptoxanthin, lutein/zeaxanthin and red-cell folate. The mean level of these carotenoids and of red-cell folate increased with increasing frequency of reported servings of vegetables and fruit, both before and after adjusting for potential confounding factors. A significant association with lycopene was observed only for vegetable intake before adjusting for confounders. Conclusions: These data indicate that brief questions may be a simple and valuable tool for monitoring vegetable and fruit intake in this population.

An evaluation of different features and learning models for anomalous event detection

The huge amount of CCTV footage available makes it very burdensome to process these videos manually through human operators. This has made automated processing of video footage through computer vision technologies necessary. During the past several years, there has been a large effort to detect abnormal activities through computer vision techniques. Typically, the problem is formulated as a novelty detection task where the system is trained on normal data and is required to detect events which do not fit the learned ‘normal’ model. There is no precise and exact definition for an abnormal activity; it is dependent on the context of the scene. Hence there is a requirement for different feature sets to detect different kinds of abnormal activities. In this work we evaluate the performance of different state of the art features to detect the presence of the abnormal objects in the scene. These include optical flow vectors to detect motion related anomalies, textures of optical flow and image textures to detect the presence of abnormal objects. These extracted features in different combinations are modeled using different state of the art models such as Gaussian mixture model(GMM) and Semi- 2D Hidden Markov model(HMM) to analyse the performances. Further we apply perspective normalization to the extracted features to compensate for perspective distortion due to the distance between the camera and objects of consideration. The proposed approach is evaluated using the publicly available UCSD datasets and we demonstrate improved performance compared to other state of the art methods.