Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol

BACKGROUND The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

The development and field test of a Mealtime Interaction Clinical Observation Tool: a pilot study and clinical research portfolio

Objective: The purpose of this study was to develop and test psychometric properties of a Mealtime Interaction Clinical Observation Tool (MICOT) that could be used to facilitate assessment and behavioural intervention in childhood feeding difficulties. Methods: Thematic analysis of four focus groups with feeding and behaviour experts identified the content and structure of the MICOT. Following refinement, inter-rater reliability was tested between three healthcare professionals. Results: Six themes were identified for the MICOT, which utilises a traffic-light system to identify areas of strength and areas for intervention. Despite poor inter-rater reliability, for which a number of reasons are postulated, some correlation between psychologists’ ratings was evident. Healthcare professionals liked the tool and reported that it could have good clinical utility. Conclusion: The study provides a promising first version of a clinical observation tool that facilitates assessment and behavioural intervention in childhood feeding difficulties.

Crash risk estimation and assessment tool

Currently in Australia, there are no decision support tools for traffic and transport engineers to assess the crash risk potential of proposed road projects at design level. A selection of equivalent tools already exists for traffic performance assessment, e.g. aaSIDRA or VISSIM. The Urban Crash Risk Assessment Tool (UCRAT) was developed for VicRoads by ARRB Group to promote methodical identification of future crash risks arising from proposed road infrastructure, where safety cannot be evaluated based on past crash history. The tool will assist practitioners with key design decisions to arrive at the safest and the most cost -optimal design options. This paper details the development and application of UCRAT software. This professional tool may be used to calculate an expected mean number of casualty crashes for an intersection, a road link or defined road network consisting of a number of such elements. The mean number of crashes provides a measure of risk associated with the proposed functional design and allows evaluation of alternative options. The tool is based on historical data for existing road infrastructure in metropolitan Melbourne and takes into account the influence of key design features, traffic volumes, road function and the speed environment. Crash prediction modelling and risk assessment approaches were combined to develop its unique algorithms. The tool has application in such projects as road access proposals associated with land use developments, public transport integration projects and new road corridor upgrade proposals.

A virtual test assessment tool for second year signal processing students

The development and use of a virtual assessment tool for a signal processing unit is described. It allows students to take a test from anywhere using a web browser to connect to the university server that hosts the test. While student responses are of the multiple choice type, they have to work out problems to arrive at the answer to be entered. CGI programming is used to verify student identification information and record their scores as well as provide immediate feedback after the test is complete. The tool has been used at QUT for the past 3 years and student feedback is discussed. The virtual assessment tool is an efficient alternative to marking written assignment reports that can often take more hours than actual lecture hall contact from a lecturer or tutor. It is especially attractive for very large classes that are now the norm at many universities in the first two years.

Construction safety : development of an assessment tool to reduce risk on building sites

Australian construction and building workers are exposed to serious workplace risks - including injury, illness and death - and although there have been improvements in occupational health and safety (OHS) performance over the past 20 years, the injury and fatality rate in the Australian construction industry remains a matter of concern. The concept of safety culture is rapidly being adopted in the industry, including recognising the critical role that organisational leaders play in overall safety performance. This paper reviews recent research in construction safety leadership and provides some examples and applications relevant to risk reduction in the workforce. By focusing on developing safety competency in those that fulfil safety critical roles, and clearly articulating the relevant safety management tasks, leaders can positively influence the organisation’s safety culture. Finally, some promising research on Safety Effectiveness Indicators (SEIs) may be an industry-friendly solution to reducing workplace risks across the industry, by providing a credible, accurate, and timely measure of safety performance.

Development of a sewerage infrastructure buffer assessment tool for local authorities

Urban expansion continues to encroach on existing or newly implemented sewerage infrastructure. In this context, legislation and guidelines, both national and international, provide limited direction to the amenity allocation of appropriate buffering distances for land use planners and infrastructure providers. A review of published literature suggests the dominant influences include topography, wind speed and direction, temperature, humidity, existing land uses and vegetation profiles. A statistical criteria review of these factors against six years of sewerage odour complaint data was undertaken to ascertain their influence and a complaint severity hierarchy was established. These hierarchical results suggested the main criteria were: topographical location, elevation relative to the odour source and wind speed. Establishing a justifiable criterion for buffer zone allocations will assist in analytically determining a basis for buffer separations and will assist planners and infrastructure designers in assessing lower impact sewerage infrastructure locations.

Preliminary development of a sewerage infrastructure buffer assessment tool for engineering risk and strategic land use planning

Urban expansion continues to encroach on once isolated sewerage infrastructure. In this context,legislation and guidelines provide limited direction to the amenity allocation of appropriate buffer distances for land use planners and infrastructure providers. Topography, wind speed and direction,temperature, humidity, existing land uses and vegetation profiles are some of the factors that require investigation in analytically determining a basis for buffer separations. This paper discusses the compilation and analysis of six years of Logan sewerage odour complaint data. Graphically,relationships between the complaints, topographical features and meteorological data are presented. Application of a buffer sizing process could assist planners and infrastructure designers alike, whilst automatically providing extra green spaces. Establishing a justifiable criterion for buffer zone allocations can only assist in promoting manageable growth for healthier and more sustainable communities.

When non-significance maybe significant: lessons learned from a study into the development, implementation and evaluation of a risk assessment tool for fleet settings

Introduction This study reports on the development of a self report assessment tool to increase the efficacy of crash prediction within Australian Fleet settings Over last 20 years an array of measures have been produced (Driver anger scale, Driving Skill Inventory, Manchester Driver Behaviour Questionnaire, Driver Attitude Questionnaire, Driver Stress Inventory, Safety Climate Questionnaire) While these tools are useful, research has demonstrated limited ability to accurately identify individuals most likely to be involved in a crash. Reasons cited include; - Crashes are relatively rare - Other competing factors may influence crash event - Ongoing questions regarding the validity of self report measures (common method variance etc) - Lack of contemporary issues relating to fleet driving performance

A survey to evaluate the implementation of a national clinical assessment form

Purpose: The Australian Universities Radiation Therapy Student Clinical Assessment Form (AURTSCAF) was designed to assess the clinical skills of radiation therapy (RT) students from the six universities that offer entry level RT programs. Given the AURTSCAF has now been in use for over two years, the Radiation Therapy Program Coordinators (RTPC) group initiated a post implementation evaluation survey. This formed the final phase of the AURTSCAF project and was funded by the Radiation Oncology Division of the Department of Health and Ageing. Methods: A cross-sectional designed survey using purposive sampling was distributed via email to all RT clinical sites. The survey asked questions about the requirements of a pass grade for students at different stages of their program, and the addition of a new category of assessment related to fitness to practise. Response types included both forced choice closed ended responses and open ended responses. There was also a section for open comments about the AURTSCAF. Results: There were 100 responses (55%) from clinicians who had utilised the assessment form over the previous 12 month period. Responses highlighted several positives with regard to the utility and implementation of the form. Comments regarding areas for improvement with the standardisation of the grading of students and consensus for the addition of a new domain in fitness for practise have informed the recommended changes proposed for 2012. Conclusion: This evaluation has provided a representative sample of the views of clinicians involved in assessing students on clinical placement. Recommendations include the addition of the sixth domain of assessment: Fitness for practise, the addition of descriptors and prompts for this domain in the user guide, the addition of a consensus statement about the use of the rating scale and dissemination of the proposed changes nationally.

When non-significance may be significant : lessons learned from a study of the development, implementation and evaluation of a fleet risk assessment tool

This paper reports on the development and implementation of a self-report risk assessment tool that was developed in an attempt to increase the efficacy of crash prediction within Australian fleet settings. This study forms a part of a broader program of research into work related road safety and identification of driving risk. The first phase of the study involved a series of focus groups being conducted with 217 professional drivers which revealed that the following factors were proposed to influence driving performance: Fatigue, Knowledge of risk, Mood, Impatience and frustration, Speed limits, Experience, Other road users, Passengers, Health, and Culture. The second phase of the study involved piloting the newly developed 38 item Driving Risk Assessment Scale - Work Version (DRAS-WV) with 546 professional drivers. Factor analytic techniques identified a 9 factor solution that was comprised of speeding, aggression, time pressure, distraction, casualness, awareness, maintenance, fatigue and minor damage. Speeding and aggressive driving manoeuvres were identified to be the most frequent aberrant driving behaviours engaged in by the sample. However, a series of logistic regression analyses undertaken to determine the DRAS-WV scale’s ability to predict self-reported crashes revealed limited predictive efficacy e.g., 10% of crashes. This paper outlines proposed reasons for this limited predictive ability of the DRAS-WV as well as provides suggestions regarding the future of research that aims to develop methods to identify “at risk” drivers.

An assessment tool for selection of appropriate daylighting solutions for buildings in tropical and subtropical regions : Validation using radiance simulation

A new decision-making tool that will assist designers in the selection of appropriate daylighting solutions for buildings in tropical locations has been previously proposed by the authors. Through an evaluation matrix that prioritizes the parameters that best respond to the needs of tropical climates (e.g. reducing solar gain and protection from glare) the tool determines the most appropriate devices for specific climate and building inputs. The tool is effective in demonstrating the broad benefits and limitations of the different daylight strategies for buildings in the tropics. However for thorough analysis and calibration of the tool, validation is necessary. This paper presents a first step in the validation process. RADIANCE simulations were conducted to compare simulation performance with the performance predicted by the tool. To this end, an office building case study in subtropical Brisbane, Australia, and five different daylighting devices including openings, light guiding systems and light transport systems were simulated. Illuminance, light uniformity, daylight penetration and glare analysis were assessed for each device. The results indicate the tool can appropriately rank and recommend daylighting strategies based on specific building inputs for tropical and subtropical regions, making it a useful resource for designers.

The development of a supportive care needs assessment tool for Indigenous people with cancer

Background: Little is known about the supportive care needs of Indigenous people with cancer and to date, existing needs assessment tools have not considered cultural issues for this population. We aimed to adapt an existing supportive care needs assessment tool for use with Indigenous Australians with cancer. Methods: Face-to-face interviews with Indigenous cancer patients (n = 29) and five focus groups with Indigenous key-informants (n = 23) were conducted to assess the face and content validity, cultural acceptability, utility and relevance of the Supportive Care Needs Survey - Short Form 34 (SCNS-SF34) for use with Indigenous patients with cancer. Results: All items from the SCNS-SF34 were shortened and changed to use more appropriate language (e.g. the word 'anxiety' was substituted with 'worry'). Seven questions were omitted (e.g. items on death and future considerations) as they were deemed culturally inappropriate or irrelevant and 12 items were added (e.g. accessible transport). Optional instructions were added before the sexual items. The design and response format of the SCNS-SF34 was modified to make it easier to use for Indigenous cancer patients. Given the extensive modifications to the SCNS-SF34 and the liklihood of a different factor structure we consider this tool to be a new tool rather than a modification. The Supportive care needs assessment tool for Indigenous people (SCNAT-IP) shows promising face and content validity and will be useful in informing services where they need to direct their attention for these patients. Conclusions: Indigenous people with cancer have language, customs and specific needs that are not accommodated within the standard SCNS-SF34. Our SCNAT-IP improves acceptability, relevance and face validity for Indigenous-specific concerns. Our SCNAT-IP will allow screening for supportive care needs that are specific to Indigenous cancer patients' and greatly inform targeted policy development and practice.