Appraising the quality of randomized controlled trials: inter-rater reliability for the OTseeker evidence database

Rationale and aims 'OTseeker' is an online database of randomized controlled trials (RCTs) and systematic reviews relevant to occupational therapy. RCTs are critically appraised and rated for quality using the 'PEDro' scale. We aimed to investigate the inter-rater reliability of the PEDro scale before and after revising rating guidelines. Methods In study 1, five raters scored 100 RCTs using the original PEDro scale guidelines. In study 2, two raters scored 40 different RCTs using revised guidelines. All RCTs were randomly selected from the OTseeker database. Reliability was calculated using Kappa and intraclass correlation coefficients [ICC (model 2,1)]. Results Inter-rater reliability was 'good to excellent' in the first study (Kappas >= 0.53; ICCs >= 0.71). After revising the rating guidelines, the reliability levels were equivalent or higher to those previously obtained (Kappas >= 0.53; ICCs >= 0.89), except for the item, 'groups similar at baseline', which still had moderate reliability (Kappa = 0.53). In study 2, two PEDro scale items, which had their definitions revised, 'less than 15% dropout' and 'point measures and variability', showed higher reliability. In both studies, the PEDro items with the lowest reliability were 'groups similar at baseline' (Kappas = 0.53), 'less than 15% dropout' (Kappas

The effect of personal characteristics on the validity of nutrient intake estimates using a food-frequency questionnaire

Objective: To assess validity of the Nambour food-frequency questionnaire (FFQ) relative to weighed food records (WFRs), and the extent to which selected demographic, anthropometric and social characteristics explain differences between the two dietary methods. Design: Inter-method validity study; 129-item FFQ vs. 12 days of WFR over 12 months. Setting: Community-based Nambour Skin Cancer Prevention Trial. Subjects: One hundred and fifteen of 168 randomly selected participants in the trial (68% acceptance rate) aged 25-75 years. Results: Spearman correlations between intakes from the two methods ranged from 0.18 to 0.71 for energy-adjusted values. Differences between FFQ and WFR regressed on personal characteristics were significantly associated with at least one characteristic for 16 of the 21 nutrients. Sex was significantly associated with differences for nine nutrients; body mass index (BMI), presence of any medical condition and age were each significantly associated with differences for three to six nutrients; use of dietary supplements and occupation were associated with differences for one nutrient each. There was no consistency in the direction of the significant associations. Regression models explained from 7% (riboflavin) to 27% (saturated fat) of variation in differences in intakes. Conclusions: The relative validity of FFQ estimates for many nutrients is quite different for males than for females. Age, BMI, medical condition and level of intake were also associated with relative validity for some nutrients, resulting in the need to adjust intakes estimates for these in modelling diet-disease relationships. Estimates for cholesterol, beta-carotene equivalents, retinol equivalents, thiamine, riboflavin and calcium would not benefit from this.