Multiplicity of data in trial reports and the reliability of meta-analyses: empirical study

Objectives To examine the extent of multiplicity of data in trial reports and to assess the impact of multiplicity on meta-analysis results. Design Empirical study on a cohort of Cochrane systematic reviews. Data sources All Cochrane systematic reviews published from issue 3 in 2006 to issue 2 in 2007 that presented a result as a standardised mean difference (SMD). We retrieved trial reports contributing to the first SMD result in each review, and downloaded review protocols. We used these SMDs to identify a specific outcome for each meta-analysis from its protocol. Review methods Reviews were eligible if SMD results were based on two to ten randomised trials and if protocols described the outcome. We excluded reviews if they only presented results of subgroup analyses. Based on review protocols and index outcomes, two observers independently extracted the data necessary to calculate SMDs from the original trial reports for any intervention group, time point, or outcome measure compatible with the protocol. From the extracted data, we used Monte Carlo simulations to calculate all possible SMDs for every meta-analysis. Results We identified 19 eligible meta-analyses (including 83 trials). Published review protocols often lacked information about which data to choose. Twenty-four (29%) trials reported data for multiple intervention groups, 30 (36%) reported data for multiple time points, and 29 (35%) reported the index outcome measured on multiple scales. In 18 meta-analyses, we found multiplicity of data in at least one trial report; the median difference between the smallest and largest SMD results within a meta-analysis was 0.40 standard deviation units (range 0.04 to 0.91). Conclusions Multiplicity of data can affect the findings of systematic reviews and meta-analyses. To reduce the risk of bias, reviews and meta-analyses should comply with prespecified protocols that clearly identify time points, intervention groups, and scales of interest.

Hip Osteoarthritis MRI Scoring System (HOAMS): reliability and associations with radiographic and clinical findings

To develop a semiquantitative MRI-based scoring system (HOAMS) of hip osteoarthritis (OA) and test its reliability and validity.

Photographs of dental casts or digital models: rating dental arch relationships in bilateral cleft lip and palate

Yardsticks have been developed to measure dental arch relations in cleft lip and palate (CLP) patients as diagnostic proxies for the underlying skeletal relationship. Travelling with plaster casts to compare results between CLP centres is inefficient so the aim of this study was to investigate the reliability of using digital models or photographs of dental casts instead of plaster casts for rating dental arch relationships in children with complete bilateral cleft lip and palate (CBCLP). Dental casts of children with CBCLP (n=20) were included. Plaster casts, digital models and photographs of the plaster casts were available for all the children at 6, 9, and 12 years of age. All three record formats were scored using the bilateral cleft lip and palate (BCLP) yardstick by four observers in random order. No significant differences were found for the BCLP yardstick scores among the three formats. The interobserver weighted kappa scores were between 0.672 and 0.934. Comparison between the formats per observer resulted in weighted kappa scores between 0.692 and 0.885. It is concluded that digital models and photographs of dental casts can be used for rating dental arch relationships in patients with CBCLP. These formats are a reliable alternative for BCLP yardstick assessments on conventional plaster casts.

Femoral morphology and epiphyseal growth plate changes of the hip during maturation: MR assessments in a 1-year follow-up on a cross-sectional asymptomatic cohort in the age range of 9-17 years

The goal of this prospective study was to characterize the morphology and physeal changes of the femoral head during maturation using MRI in a population-based group of asymptomatic volunteers.

Clinical application and reliability of a post abdominal surgery pain assessment scale (PASPAS) in horses

The aim of this study was to refine a multi-dimensional scale based on physiological and behavioural parameters, known as the post abdominal surgery pain assessment scale (PASPAS), to quantify pain after laparotomy in horses. After a short introduction, eight observers used the scale to assess eight horses at multiple time points after laparotomy. In addition, a single observer was used to test the correlation of each parameter with the total pain index in 34 patients, and the effect of general anaesthesia on PASPAS was investigated in a control group of eight horses. Inter-observer variability was low (coefficient of variation 0.3), which indicated good reliability of PASPAS. The correlation of individual parameters with the total pain index differed between parameters. PASPAS, which was not influenced by general anaesthesia, was a useful tool to evaluate pain in horses after abdominal surgery and may also be useful to investigate analgesic protocols or for teaching purposes.

The postoperative COFAS end-stage ankle arthritis classification system: interobserver and intraobserver reliability

End-stage ankle arthritis is operatively treated with numerous designs of total ankle replacement and different techniques for ankle fusion. For superior comparison of these procedures, outcome research requires a classification system to stratify patients appropriately. A postoperative 4-type classification system was designed by 6 fellowship-trained foot and ankle surgeons. Four surgeons reviewed blinded patient profiles and radiographs on 2 occasions to determine the interobserver and intraobserver reliability of the classification. Excellent interobserver reliability (κ = .89) and intraobserver reproducibility (κ = .87) were demonstrated for the postoperative classification system. In conclusion, the postoperative Canadian Orthopaedic Foot and Ankle Society (COFAS) end-stage ankle arthritis classification system appears to be a valid tool to evaluate the outcome of patients operated for end-stage ankle arthritis.

German validation of the Conners Adult ADHD Rating Scales (CAARS) II: reliability, validity, diagnostic sensitivity and specificity

The German version of the Conners Adult ADHD Rating Scales (CAARS) has proven to show very high model fit in confirmative factor analyses with the established factors inattention/memory problems, hyperactivity/restlessness, impulsivity/emotional lability, and problems with self-concept in both large healthy control and ADHD patient samples. This study now presents data on the psychometric properties of the German CAARS-self-report (CAARS-S) and observer-report (CAARS-O) questionnaires.

The Development, Reliability, and Validity of the OARS Multidimensional Functional Assessment Questionnaire

This report outlines the development, validity, and reliability of Part A of the OARS Multidimensional Functional Assessment Questionnaire. Part A permits assessment of individuals' functioning on each of five dimensions (social, economic, mental health, physical health and self-care capacity), the detailed information in each area being summarized on a 6-point rating scale by a rater. Content and consensual validity were ensured by the manner of construction. Information on criterion validity was obtained for all dimensions except social. The criterion used and their associated Kendall's Tau values were: an objective economic scale (.62); ratings based on personal interviews by geropsychiatrists (.60); physician's associates (.82); and physical therapists (.89). For 11 geographically dispersed raters from research and clinic settings, intraclass correlational coefficients, based on 30 subjects, ranged from .66 on physical health to .87 in self-care capacity; 74% of the ratings were in complete agreement, 24% differed by one point.

Validity evidence and reliability of a Simulated Patient Feedback Instrument

In the training of healthcare professionals, one of the advantages of communication training with simulated patients (SPs) is the SP's ability to provide direct feedback to students after a simulated clinical encounter. The quality of SP feedback must be monitored, especially because it is well known that feedback can have a profound effect on student performance. Due to the current lack of valid and reliable instruments to assess the quality of SP feedback, our study examined the validity and reliability of one potential instrument, the 'modified Quality of Simulated Patient Feedback Form' (mQSF). Methods Content validity of the mQSF was assessed by inviting experts in the area of simulated clinical encounters to rate the importance of the mQSF items. Moreover, generalizability theory was used to examine the reliability of the mQSF. Our data came from videotapes of clinical encounters between six simulated patients and six students and the ensuing feedback from the SPs to the students. Ten faculty members judged the SP feedback according to the items on the mQSF. Three weeks later, this procedure was repeated with the same faculty members and recordings. Results All but two items of the mQSF received importance ratings of > 2.5 on a four-point rating scale. A generalizability coefficient of 0.77 was established with two judges observing one encounter. Conclusions The findings for content validity and reliability with two judges suggest that the mQSF is a valid and reliable instrument to assess the quality of feedback provided by simulated patients.