Development and interrater reliability testing of a telephone interview training programme for Australian nurse interviewers

Summary Background The final phase of a three phase study analysing the implementation and impact of the nurse practitioner role in Australia (the Australian Nurse Practitioner Project or AUSPRAC) was undertaken in 2009, requiring nurse telephone interviewers to gather information about health outcomes directly from patients and their treating nurse practitioners. A team of several registered nurses was recruited and trained as telephone interviewers. The aim of this paper is to report on development and evaluation of the training process for telephone interviewers. Methods The training process involved planning the content and methods to be used in the training session; delivering the session; testing skills and understanding of interviewers post-training; collecting and analysing data to determine the degree to which the training process was successful in meeting objectives and post-training follow-up. All aspects of the training process were informed by established educational principles. Results Interrater reliability between interviewers was high for well-validated sections of the survey instrument resulting in 100% agreement between interviewers. Other sections with unvalidated questions showed lower agreement (between 75% and 90%). Overall the agreement between interviewers was 92%. Each interviewer was also measured against a specifically developed master script or gold standard and for this each interviewer achieved a percentage of correct answers of 94.7% or better. This equated to a Kappa value of 0.92 or better. Conclusion The telephone interviewer training process was very effective and achieved high interrater reliability. We argue that the high reliability was due to the use of well validated instruments and the carefully planned programme based on established educational principles. There is limited published literature on how to successfully operationalise educational principles and tailor them for specific research studies; this report addresses this knowledge gap.

Measuring factors that influence the utilisation of preventive care services provided by general practitioners in Australia

Background: Relatively little research attention has been given to the development of standardised and psychometrically sound scales for measuring influences relevant to the utilisation of health services. This study aims to describe the development, validation and internal reliability of some existing and new scales to measure factors that are likely to influence utilisation of preventive care services provided by general practitioners in Australia.----- Methods: Relevant domains of influence were first identified from a literature review and formative research. Items were then generated by using and adapting previously developed scales and published findings from these. The new items and scales were pre-tested and qualitative feedback was obtained from a convenience sample of citizens from the community and a panel of experts. Principal Components Analyses (PCA) and internal reliability testing (Cronbach's alpha) were then conducted for all of the newly adapted or developed scales utilising data collected from a self-administered mailed survey sent to a randomly selected population-based sample of 381 individuals (response rate 65.6 per cent).----- Results: The PCA identified five scales with acceptable levels of internal consistency were: (1) social support (ten items), alpha 0.86; (2) perceived interpersonal care (five items), alpha 0.87, (3) concerns about availability of health care and accessibility to health care (eight items), alpha 0.80, (4) value of good health (five items), alpha 0.79, and (5) attitudes towards health care (three items), alpha 0.75.----- Conclusion The five scales are suitable for further development and more widespread use in research aimed at understanding the determinants of preventive health services utilisation among adults in the general population.

Validation of the impact of Health Information Technology (I-HIT) scale : an international collaborative

In 2005, the Healthcare Information Management Systems Society (HIMSS) Nursing Informatics Community developed a survey to measure the impact of health information technology (HIT), the IHIT Scale, on the role of nurses and interdisciplinary communication in hospital settings. In 2007, nursing informatics colleagues from Australia, England, Finland, Ireland, New Zealand, Scotland and the United States formed a research collaborative to validate the IHIT across countries. All teams have completed construct and face validation in their countries. Five out of six teams have initiated reliability testing by practicing nurses. This paper reports the international collaborative’s validation of the IHIT Scale completed to date.

A validated self-administered female pelvic floor questionnaire

Introduction and hypothesis: The aim of this study was to validate a self-administered version of the already validated interviewer-administered Australian pelvic floor questionnaire. Methods: The questionnaire was completed by 163 women attending an urogynecological clinic. Face and convergent validity was assessed. Reliability testing and comparison with the interviewer-administered version was performed in a subset of 105 patients. Responsiveness was evaluated in a subset of 73 women. Results: Missing data did not exceed 4% for any question. Cronbach’s alpha coefficients were acceptable in all domains. Kappa coefficients for the test–retest analyses varied from 0.64–1.0. Prolapse symptoms correlated significantly with the pelvic organ prolapse quantification. Urodynamics confirmed the reported symptom stress incontinence in 70%. The self and interviewer administered questionnaires demonstrated equivalence. Effect sizes ranged from 0.6 to 1.4. Conclusions: This self-administered pelvic floor questionnaire assessed pelvic floor function in a reproducible and valid fashion and due to its responsiveness, can be used for routine clinical assessment and outcome research.

What factors influence nurses’ assessment practices? Development of the Barriers to Nurses’ use of Physical Assessment Scale

Aim To develop and psychometrically test the Barriers to Nurses’ use of Physical Assessment Scale. Background There is growing evidence of failure to recognise hospitalised patients at risk of clinical deterioration, in part due to inadequate physical assessment by nurses. Yet, little is known about the barriers to nurses’ use of physical assessment in the acute hospital setting and no validated scales have been published. Design Instrument development study. Method Scale development was based on a comprehensive literature review, focus groups, expert review and psychometric evaluation. The scale was administered to 434 acute care registered nurses working at a large Australian teaching hospital between June and July 2013. Psychometric analysis included factor analysis, model fit statistics and reliability testing. Results The final scale was reduced to 38 items representing seven factors, together accounting for 57.7% of the variance: (1) reliance on others and technology, (2) lack of time and interruptions, (3) ward culture, (4) lack of confidence, (5) lack of nursing role models, (6) lack of influence on patient care, and; (7) specialty area. Internal reliability ranged from .70 to .86. Conclusion Findings provide initial evidence for the validity and reliability of the Barriers to Nurses’ use of Physical Assessment Scale and point to the importance of understanding the organisational determinants of nurses’ assessment practices. The new scale has potential clinical and research applications to support nursing assessment in acute care settings.

Validation of the dietetic confidence scale

Dietitians have reported a lack of confidence in counselling clients with mental health issues. Standardised tools are needed to evaluate programs aiming to improve confidence. The Dietetic Confidence Scale (DCS) was developed to assess dietitians’perception of their capability about working with clients experiencing depression. Exploratory research revealed a 13-item, two-factor model. Dietetic confidence was associated with: 1) Confidence using the Nutrition Care Process; and 2) Confidence in Advocacy for Self-care and Client-care. This study aimed to validate the DCS using this two-factor model.The DCS was administered to 458 dietitians. Confirmatory factor analysis (CFA) assessed the scale’s psychometric validity. Reliability was measured using Cronbach’s alpha (α) co-efficient. CFA results supported the hypothesised two-factor, 13-item model. The Good Fit Index (GFI = 0.95) indicated a strong fit. Item-factor correlations ranged from r = 0.50 to 0.89. The overall scale and subscales showed good reliability (α = 0.93 to 0.76). This is the first study to validate an instrument that measures dietetic confidence about working with clients experiencing depression. The DCS can be used to measure changes in perceived confidence and identify where further training, mentoring or experience is needed. The findings also suggest that initiatives aimed at building dietitians' confidence about working with clients experiencing depression, should focus on improving client-focused nutrition care, foster advocacy, reflective practice, mentoring and encourage professional support networks. Avenues for future research include further validity and reliability testing to expand the generalisability of results; and modifying the scale for other disease or client populations.

A real-time software architecture for robotics and automation

This paper describes a software architecture for real-world robotic applications. We discuss issues of software reliability, testing and realistic off-line simulation that allows the majority of the automation system to be tested off-line in the laboratory before deployment in the field. A recent project, the automation of a very large mining machine is used to illustrate the discussion.

Use of objective psychomotor tests in health professionals

Evaluation of psychomotor skills is undertaken in a number of broad contexts. This includes testing of health professional populations as a measure of innate ability, to evaluate skill acquisition, or to compare professions. However, the use of psychomotor tests is frequently confounded by a lack of understanding of a particular tool's psychometric properties, strengths, and weaknesses. To identify and appraise the most commonly used tests on health professional populations, 86 articles were reviewed and the top nine tests identified. Few tests have had sufficient validity or reliability testing on health professionals. Based on the evidence available, use of the Grooved Pegboard Test, the Purdue Pegboard Test, or the Finger Tapping Test is recommended for the evaluation of dexterity in a health professional population; however, this choice may be dependent on the task(s) to which findings are generalised. More rigorous evaluation of validity and other psychometric properties is required.

Preliminary reliability and validity testing of a self-efficacy questionnaire for Chinese family caregivers

Objectives: To develop and test preliminary reliability and validity of a Self-Efficacy Questionnaire for Chinese Family Caregivers (SEQCFC). Methods: A cross-sectional survey of 196 family caregivers (CGs) of people with dementia (CGs) was conducted to determine the factor structure of a SEQCFC of people with dementia. Following factor analyses, preliminary testing was performed, including internal consistency, 4-week test retest reliability, and construct and convergent validity. Results: Factor analyses with direct oblimin rotation were performed. Eight items were removed and five subscales(selfefficacy for gathering information about treatment, symptoms and health care; obtaining support; responding to behaviour disturbances; managing household, personal and medical care; and managing distress associated with caregiving) were identified. The Cronbach’s alpha coefficients for the whole scale and for each subscale were all over 0.80. The 4-week testretest reliabilities for the whole scale and for each subscale ranged from 0.64 to 0.85. The convergent validity was acceptable. Conclusions: Evidence for the preliminary testing of the SEQCFC was encouraging. A future follow-up study using confirmatory factor analysis with a new sample from different recruitment centres in Shanghai will be conducted. Future psychometric property testings of the questionnaire will be required for CGs from other regions of mainland China.

Reliability of point-of-care testing of INR in acute stroke

Background In the emergency department, portable point-of-care testing (POCT) coagulation devices may facilitate stroke patient care by providing rapid International Normalized Ratio (INR) measurement. The objective of this study was to evaluate the reliability, validity, and impact on clinical decision-making of a POCT device for INR testing in the setting of acute ischemic stroke (AIS). Methods A total of 150 patients (50 healthy volunteers, 51 anticoagulated patients, 49 AIS patients) were assessed in a tertiary care facility. The INR's were measured using the Roche Coaguchek S and the standard laboratory technique. Results The interclass correlation coefficient and 95% confidence interval between overall POCT device and standard laboratory value INRs was high (0.932 (0.69 - 0.78). In the AIS group alone, the correlation coefficient and 95% CI was also high 0.937 (0.59 - 0.74) and diagnostic accuracy of the POCT device was 94%. Conclusions When used by a trained health professional in the emergency department to assess INR in acute ischemic stroke patients, the CoaguChek S is reliable and provides rapid results. However, as concordance with laboratory INR values decreases with higher INR values, it is recommended that with CoaguChek S INRs in the > 1.5 range, a standard laboratory measurement be used to confirm the results.

Testing a tri-partite contingent model of engineering cultures : a pilot study

For some time there has been a growing awareness of organizational culture and its impact on the functioning of engineering and maintenance departments. Those wishing to implement contemporary maintenance regimes (e.g. condition based maintenance) are often encouraged to develop “appropriate cultures” to support a new method’s introduction. Unfortunately these same publications often fail to specifically articulate the cultural values required to support those efforts. In the broader literature, only a limited number of case examples document the cultural values held by engineering asset intensive firms and how they contribute to their success (or failure). Consequently a gap exists in our knowledge of what engineering cultures currently might look like, or what might constitute a best practice engineering asset culture. The findings of a pilot study investigating the perceived ideal characteristics of engineering asset cultures are reported. Engineering managers, consultants and academics (n=47), were surveyed as to what they saw were essential attributes of both engineering cultures and engineering asset personnel. Valued cultural elements included those orientated around continuous improvement, safety and quality. Valued individual attributes included openness to change, interpersonal skills and conscientiousness. The paper concludes with a discussion regarding the development of a best practice cultural framework for practitioners and engineering managers.

A Reliability test system for educational purposes-basic data

The IEEE Subcommittee on the Application of Probability Methods (APM) published the IEEE Reliability Test System (RTS) [1] in 1979. This system provides a consistent and generally acceptable set of data that can be used both in generation capacity and in composite system reliability evaluation [2,3]. The test system provides a basis for the comparison of results obtained by different people using different methods. Prior to its publication, there was no general agreement on either the system or the data that should be used to demonstrate or test various techniques developed to conduct reliability studies. Development of reliability assessment techniques and programs are very dependent on the intent behind the development as the experience of one power utility with their system may be quite different from that of another utility. The development and the utilization of a reliability program are, therefore, greatly influenced by the experience of a utlity and the intent of the system manager, planner and designer conducting the reliability studies. The IEEE-RTS has proved to be extremely valuable in highlighting and comparing the capabilities (or incapabilities) of programs used in reliability studies, the differences in the perception of various power utilities and the differences in the solution techniques. The IEEE-RTS contains a reasonably large power network which can be difficult to use for initial studies in an educational environment.

A Reliability test system for educational purposes-basic data

The IEEE Reliability Test System (RTS) developed by the Application of Probability Method Subcommittee has been used to compare and test a wide range of generating capacity and composite system evaluation techniques and subsequent digital computer programs. A basic reliability test system is presented which has evolved from the reliability education and research programs conducted by the Power System Research Group at the University of Saskatchewan. The basic system data necessary for adequacy evaluation at the generation and composite generation and transmission system levels are presented together with the fundamental data required to conduct reliability-cost/reliability-worth evaluation

A taxonomy of model-based testing approaches

Model-based testing (MBT) relies on models of a system under test and/or its environment to derive test cases for the system. This paper discusses the process of MBT and defines a taxonomy that covers the key aspects of MBT approaches. It is intended to help with understanding the characteristics, similarities and differences of those approaches, and with classifying the approach used in a particular MBT tool. To illustrate the taxonomy, a description of how three different examples of MBT tools fit into the taxonomy is provided.

Testing the psychometric properties of the Brisbane Practice Environment Measure using Exploratory Factor Analysis and Confirmatory Factor Analysis in an Australian registered nurse population

A cross-sectional survey was conducted, and the construct validity and reliability of the Brisbane Practice Environment Measure in an Australian sample of registered nurses were examined. Nurses were randomly selected from the database of an Australian nursing organization. The original 33 items of the Brisbane Practice Environment Measure were utilized to inform the psychometric properties using confirmatory factor analysis. The Cronbach's alpha was 0.938 for the total scale and ranged 0.657–0.887 for the subscales. A five-factor structure of the measure was confirmed, χ2 = 944.622, (P < 0.01), χ2/d.f. ratio = 2.845, Tucker Lewis Index 0.929, Root Mean Square Error = 0.061 and Comparative Fit Index = 0.906. The selected 28 items of the measure proved reliable and valid in measuring effects of the practice environment upon Australian nurses. The implications are that regular measurement of the practice environment using these 28 items might assist in the development of strategies which might improve job satisfaction and retention of registered nurses in Australia.