Appraising the quality of randomized controlled trials: inter-rater reliability for the OTseeker evidence database

Rationale and aims 'OTseeker' is an online database of randomized controlled trials (RCTs) and systematic reviews relevant to occupational therapy. RCTs are critically appraised and rated for quality using the 'PEDro' scale. We aimed to investigate the inter-rater reliability of the PEDro scale before and after revising rating guidelines. Methods In study 1, five raters scored 100 RCTs using the original PEDro scale guidelines. In study 2, two raters scored 40 different RCTs using revised guidelines. All RCTs were randomly selected from the OTseeker database. Reliability was calculated using Kappa and intraclass correlation coefficients [ICC (model 2,1)]. Results Inter-rater reliability was 'good to excellent' in the first study (Kappas >= 0.53; ICCs >= 0.71). After revising the rating guidelines, the reliability levels were equivalent or higher to those previously obtained (Kappas >= 0.53; ICCs >= 0.89), except for the item, 'groups similar at baseline', which still had moderate reliability (Kappa = 0.53). In study 2, two PEDro scale items, which had their definitions revised, 'less than 15% dropout' and 'point measures and variability', showed higher reliability. In both studies, the PEDro items with the lowest reliability were 'groups similar at baseline' (Kappas = 0.53), 'less than 15% dropout' (Kappas

Reliability and validity of a brief physical activity assessment for use by family doctors

Objective: To evaluate the reliability and validity of a brief physical activity assessment tool suitable for doctors to use to identify inactive patients in the primary care setting. Methods: Volunteer family doctors (n = 8) screened consenting patients (n = 75) for physical activity participation using a brief physical activity assessment tool. Inter-rater reliability was assessed within one week (n = 71). Validity was assessed against an objective physical activity monitor (computer science and applications accelerometer; n = 42). Results: The brief physical activity assessment tool produced repeatable estimates of sufficient total physical activity, correctly classifying over 76% of cases (kappa 0.53, 95% confidence interval (CI) 0.33 to 0.72). The validity coefficient was reasonable (kappa 0.40, 95% CI 0.12 to 0.69), with good percentage agreement (71%). Conclusions: The brief physical activity assessment tool is a reliable instrument, with validity similar to that of more detailed self report measures of physical activity. It is a tool that can be used efficiently in routine primary healthcare services to identify insufficiently active patients who may need physical activity advice.

The diagnostic interview for psychoses (DIP): development, reliability and applications

Background. We describe the development, reliability and applications of the Diagnostic Interview for Psychoses (DIP), a comprehensive interview schedule for psychotic disorders. Method. The DIP is intended for use by interviewers with a clinical background and was designed to occupy the middle ground between fully structured, lay-administered schedules, and semi-structured., psychiatrist-administered interviews. It encompasses four main domains: (a) demographic data; (b) social functioning and disability; (c) a diagnostic module comprising symptoms, signs and past history ratings; and (d) patterns of service utilization Lind patient-perceived need for services. It generates diagnoses according to several sets of criteria using the OPCRIT computerized diagnostic algorithm and can be administered either on-screen or in a hard-copy format. Results. The DIP proved easy to use and was well accepted in the field. For the diagnostic module, inter-rater reliability was assessed on 20 cases rated by 24 clinicians: good reliability was demonstrated for both ICD-10 and DSM-III-R diagnoses. Seven cases were interviewed 2-11 weeks apart to determine test-retest reliability, with pairwise agreement of 0.8-1.0 for most items. Diagnostic validity was assessed in 10 cases, interviewed with the DIP and using the SCAN as 'gold standard': in nine cases clinical diagnoses were in agreement. Conclusions. The DIP is suitable for use in large-scale epidemiological studies of psychotic disorders. as well as in smaller Studies where time is at a premium. While the diagnostic module stands on its own, the full DIP schedule, covering demography, social functioning and service utilization makes it a versatile multi-purpose tool.

Assessing perceived need for mental health care in a community survey: development of the Perceived Need for Care Questionnaire (PNCQ)

Background: The Perceived Need for Care Questionnaire (PNCQ) was designed for the Australian National Survey of Mental Health and Wellbeing. The PNCQ complemented collection of data on diagnosis and disability with the survey participants' perceptions of their needs for mental health care and the meeting of those needs. The four-stage design of the PNCQ mimics a conversational exploration of the topic of perceived needs. Five categories of perceived need are each assigned to one of four levels of perceived need (no need, unmet need, partially met need and met need). For unmet need and partially met need, information on barriers to care is collected, Methods: Inter-rater reliabilities of perceived needs assessed by the PNCQ were examined in a study of 145 anxiety clinic attenders. Construct validity of these items was tested, using a multi-trait multi-method approach and hypotheses regarding extreme groups, in a study with a sample of 51 general practice and community psychiatric service patients. Results: The instrument is brief to administer and has proved feasible for use in various settings. Inter-rater reliabilities for major categories, measured by the kappa statistic, exceeded 0.60 in most cases; for the summary category of all perceived needs, inter-rater reliability was 0.62. The multi-trait multi-method approach lent support to the construct validity of the instrument, as did findings in extreme groups. Conclusions: The PNCQ shows acceptable feasibility, reliability and validity, adding to the range of assessment tools available for epidemiological and health services research.

Developing a reliable audit instrument to measure the physical environment for physical activity

Background: The physical environment plays an important role in influencing participation in physical activity, although which factors of the physical environment have the greatest effect on patterns of activity remain to be determined. We describe the development of a comprehensive instrument to measure the physical environmental factors that may influence walking and cycling in local neighborhoods and report on its reliability. Methods: Following consultation with experts from a variety of fields and a literature search, we developed a Systematic Pedestrian and Cycling Environmental Scan (SPACES) instrument and used it to collect data over a total of 1987 kilometers of roads in metropolitan Perth, Western Australia. The audit instrument is available from the first author on request. Additional environmental information was collected using desktop methods and geographic information systems (GIS) technology. We assessed inter- and intra-rater reliability of the instrument among the 16 observers who collected the data. Results: The observers reported that the audit instrument was easy to use. Both inter- and intra-rater reliability of the environmental scan instrument were generally high. Conclusions: Our instrument provides a reliable, practical, and easy to-use method for collecting detailed street-level data on physical environmental factors that are potential influences on walking in local neighborhoods.

Consensus neuropathological diagnosis of common dementia syndromes: testing and standardising the use of multiple diagnostic criteria

The aim of this study was to assess the variation between neuropathologists in the diagnosis of common dementia syndromes when multiple published protocols are applied. Fourteen out of 18 Australian neuropathologists participated in diagnosing 20 cases (16 cases of dementia, 4 age-matched controls) using consensus diagnostic methods. Diagnostic criteria, clinical synopses and slides from multiple brain regions were sent to participants who were asked for case diagnoses. Diagnostic sensitivity, specificity, predictive value, accuracy and variability were determined using percentage agreement and kappa statistics. Using CERAD criteria, there was a high inter-rater agreement for cases with probable and definite Alzheimer's disease but low agreement for cases with possible Alzheimer's disease. Braak staging and the application of criteria for dementia with Lewy bodies also resulted in high inter-rater agreement. There was poor agreement for the diagnosis of frontotemporal dementia and for identifying small vessel disease. Participants rarely diagnosed more than one disease in any case. To improve efficiency when applying multiple diagnostic criteria, several simplifications were proposed and tested on 5 of the original 210 cases. Inter-rater reliability for the diagnosis of Alzheimer's disease and dementia with Lewy bodies significantly improved. Further development of simple and accurate methods to identify small vessel lesions and diagnose frontotemporal dementia is warranted.

Relationship between cranio-cervical flexion range of motion and pressure change during the cranio-cervical flexion test

The purpose of this study was to quantify the sagittal angular displacement of the head (cranio-cervical flexion) for the five incremental stages of the cranio-cervical flexion test (CCFT). Range of cranio-cervical flexion during the CCFT was measured using a digital imaging method in 20 healthy volunteer subjects. The intra- and inter-rater reliability of the digital imaging technique for the assessment of this movement were also examined. The results of this study demonstrated a linear relationship between the incremental pressure targets of the CCFT and the percentages of full range cranio-cervical flexion range of motion (ROM) measured in the supine lying position of the test using a digital imaging technique. A mean of 22.9% full range cranio-cervical flexion was used to reach the first pressure target of the CCFT followed by linear increments up to 76.6% for the last stage of the test. An increasing amount of cranio-cervical flexion ROM was used to achieve the five successive stages of the CCFT reflecting an increasing contractile demand on the deep cervical flexor muscles. Excellent inter-rater (ICC = 0.994) and intra-rater reliability (ICC = 0.988-0.998) were demonstrated for the angular measurements using this digital imaging technique. (C) 2003 Elsevier Science Ltd. All rights reserved.

Can the Internet be used as a medium to evaluate knee angle?

Telemedicine promises to revolutionize medical care delivery in rural and remote areas. The ability to accurately evaluate physical impairment via the Internet is important to the possible future provision of Internet-based physiotherapy. This study evaluated the reliability and validity of assessing knee range of motion via the Internet. Two therapists evaluated knee angle on a single subject via two methods of assessment: the Internet and the traditional method (face-to-face). Nine random positions of the knee were chosen with the principal examiner performing 20 face-to-face and two sets of 20 Internet measures in each position (n=540). The secondary therapist performed Internet assessments only. The Internet connection was established at a readily available speed of 17 kbit/s. The Internet-based goniometer was found to be a valid tool for measuring both knee flexion and extension angles. It was shown to possess both high intra and inter-rater reliability. Difference average plots of the scores verified the consistency of measurement between both modes of assessment. The successful evaluation of the physical outcome measure of knee range of motion via the Internet assists the further development of Internet-based physiotherapy applications. (C) 2003 Elsevier Science Ltd. All rights reserved.

An organizational perspective on goal setting in community-based brain injury rehabilitation

Purpose. To use a taxonomy of goal content, developed in community-based brain injury rehabilitation to examine and compare the content of goals set within two different service settings; and to further examine the potential of the taxonomy to be a reliable and comprehensive framework for classifying goals. Method. Qualitative analysis and categorization of 1492 goal statements extracted from a community-based brain injury rehabilitation service over two time periods (1996-97, 1998-99), and cross-organizational comparison of ratings of goal classifications using a random sample of 100 goal statements drawn from this data set and the original 1765 goal statements used in developing the taxonomy. Results. Application of the taxonomy beyond the original service setting in which it was developed indicated a strong inter-rater reliability, with a high test-retest agreement reported over time. For both services, a small number of categories accounted for a substantial proportion of goals set within the two time periods, while considerable change was evident in goals between the two periods for one service. Further, both placed emphasis on individually focused goals rather than relationship or family-related goals. Conclusion. The taxonomy provides a reliable means for classifying goals and is a useful tool for exploration of the multiple influences on goal setting. Further application of the taxonomy to examine the relative influence on goal setting of client factors versus a range of organizational factors would be beneficial.

The family attitude scale: Reliability and validity of a new scale for measuring the emotional climate of families

Research on outcomes from psychiatric disorders has highlighted the importance of expressed emotion (EE), but its cost-effective measurement remains a challenge. This article describes development of the Family Attitude Scale (FAS), a 30-item instrument that can be completed by any informant. Its psychometric characteristics are reported in parents of undergraduate students and in 70 families with a schizophrenic member. The total FAS had high internal consistency in all samples, and reports of angry behaviour in FAS items showed acceptable inter-rater agreement. The FAS was associated with the reported anger, anger expression and anxiety of respondents. Substantial associations between the parents' FAS and the anger and anger expression of students was also observed. Parents of schizophrenic patients had higher FAS scores than parents of students, and the FAS was higher if disorder duration was longer or patient functioning was poorer. Hostility, high criticism and low warmth on the Camberwell Family Interview (CFI) were associated with a more negative FAS. The highest FAS in the family was a good predictor of a highly critical environment on the CFI. The FAS is a reliable and valid indicator of relationship stress and expressed anger that has wide applicability. (C) 1997 Elsevier Science Ireland Ltd.

Measurement of functional ability following traumatic brain injury using the Clinical Outcomes Variable Scale: A reliability study

This study determined the inter-tester and intra-tester reliability of physiotherapists measuring functional motor ability of traumatic brain injury clients using the Clinical Outcomes Variable Scale (COVS). To test inter-tester reliability, 14 physiotherapists scored the ability of 16 videotaped patients to execute the items that comprise the COVS. Intra-tester reliability was determined by four physiotherapists repeating their assessments after one week, and three months later. The intra-class correlation coefficients (ICC) were very high for both inter-tester reliability (ICC > 0.97 for total COVS scores, ICC > 0.93 for individual COVS items) and intra-tester reliability (ICC > 0.97). This study demonstrates that physiotherapists are reliable in the administration of the COVS.

Reliability and validity of physical fitness field tests for adults aged 55 to 70 years

The aim of this study was to examine the reliability and validity of field tests for assessing physical function in mid-aged and young-old people (55–70 y). Tests were selected that required minimal space and equipment and could be implemented in multiple field settings such as a general practitioner's office. Nineteen participants completed 2 field and 1 laboratory testing sessions. Intra-class correlations showed good reliability for the tests of upper body strength (lift and reach, R= .66), lower body strength (sit to stand, R= .80) and functional capacity (Canadian Step Test, R= .92), but not for leg power (single timed chair rise, R= .28). There was also good reliability for the balance test during 3 stances: parallel (94.7% agreement), semi-tandem (73.7%), and tandem (52.6%). Comparison of field test results with objective laboratory measures found good validity for the sit to stand (cf 1RM leg press, Pearson r= .68, p< .05), and for the step test (cf PWC140, r= −.60, p< .001), but not for the lift and reach (cf 1RM bench press, r= .43, p> .05), balance (r= −.13, −.18, .23) and rate of force development tests (r= −.28). It was concluded that the lower body strength and cardiovascular function tests were appropriate for use in field settings with mid-aged and young-old adults.