Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.

Reference photographs for nasolabial appearance rating in unilateral cleft lip and palate

A popular method for nasolabial rating in unilateral cleft lip and palate (UCLP) is the Asher-McDade system consisting of a 5-point ordinal scale assessing nasal form, nasal symmetry, nasal profile, and vermilion border. The aim of the current study was to identify reference photographs illustrating this scale to facilitate its use.Four observers assessed nasolabial appearance on frontal and profile photographs of the nasolabial area of 42 children of Caucasian origin with a repaired UCLP at age 9 years. Cronbachs alpha, based on the individual scores of the 4 observers, ranged from 0.73 to 0.82 for the 4 nasolabial ratings, indicating a good reliability. The reliability of the overall score (mean of the 4 component scores) was also high (Cronbachs alpha, 0.83). Both for the nasolabial component ratings and for the overall score, duplicate measurement errors were small. The reliability for the mean of the 4 observers' scores was good, Spearman rank correlation coefficients ranging from 0.56 to 0.96.Subsequently, photographs were selected that showed the highest agreement among observers. For each of the 4 components (eg, nasal form, nasal deviation, nasal profile, and shape of the vermilion border), 5 photographs were selected to illustrate the whole range of the scale (score, 1-5), resulting in the selection of 20 pictures.It was concluded that nasolabial appearance rating can be performed reliably using a panel of judges and averaging the scores of all observers. Reference photographs, as developed from this study, may facilitate the rating task.

Learning to drive: learners' self-reported cognitive failure level predicts driving instructor's observation rating of driving performance

OBJECTIVES Evidence increases that cognitive failure may be used to screen for drivers at risk. Until now, most studies have relied on driving learners. This exploratory pilot study examines self-report of cognitive failure in driving beginners and error during real driving as observed by driving instructors. METHODS Forty-two driving learners of 14 driving instructors filled out a work-related cognitive failure questionnaire. Driving instructors observed driving errors during the next driving lesson. In multiple linear regression analysis, driving errors were regressed on cognitive failure with the number of driving lessons as an estimator of driving experience controlled. RESULTS Higher cognitive failure predicted more driving errors (p < .01) when age, gender and driving experience were controlled in analysis. CONCLUSIONS Cognitive failure was significantly associated with observed driving errors. Systematic research on cognitive failure in driving beginners is recommended.