Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study

Objective To quantify and compare the treatment effect and risk of bias of trials reporting biomarkers or intermediate outcomes (surrogate outcomes) versus trials using final patient relevant primary outcomes. Design Meta-epidemiological study. Data sources All randomised clinical trials published in 2005 and 2006 in six high impact medical journals: Annals of Internal Medicine, BMJ, Journal of the American Medical Association, Lancet, New England Journal of Medicine, and PLoS Medicine. Study selection Two independent reviewers selected trials. Data extraction Trial characteristics, risk of bias, and outcomes were recorded according to a predefined form. Two reviewers independently checked data extraction. The ratio of odds ratios was used to quantify the degree of difference in treatment effects between the trials using surrogate outcomes and those using patient relevant outcomes, also adjusted for trial characteristics. A ratio of odds ratios >1.0 implies that trials with surrogate outcomes report larger intervention effects than trials with patient relevant outcomes. Results 84 trials using surrogate outcomes and 101 using patient relevant outcomes were considered for analyses. Study characteristics of trials using surrogate outcomes and those using patient relevant outcomes were well balanced, except for median sample size (371 v 741) and single centre status (23% v 9%). Their risk of bias did not differ. Primary analysis showed trials reporting surrogate endpoints to have larger treatment effects (odds ratio 0.51, 95% confidence interval 0.42 to 0.60) than trials reporting patient relevant outcomes (0.76, 0.70 to 0.82), with an unadjusted ratio of odds ratios of 1.47 (1.07 to 2.01) and adjusted ratio of odds ratios of 1.46 (1.05 to 2.04). This result was consistent across sensitivity and secondary analyses. Conclusions Trials reporting surrogate primary outcomes are more likely to report larger treatment effects than trials reporting final patient relevant primary outcomes. This finding was not explained by differences in the risk of bias or characteristics of the two groups of trials.

Sample size requirements for interval estimation of the strengthof association effect sizes in multiple regression analysis

Resumen tomado de la publicación

Meta-analysis to integrate effect sizes within a paper: Possible misuse and Type-1 error inflation

In recent years an increasing number of papers have employed meta-analysis to integrate effect sizes of researchers’ own series of studies within a single paper (“internal meta-analysis”). Although this approach has the obvious advantage of obtaining narrower confidence intervals, we show that it could inadvertently inflate false-positive rates if researchers are motivated to use internal meta-analysis in order to obtain a significant overall effect. Specifically, if one decides whether to stop or continue a further replication experiment depending on the significance of the results in an internal meta-analysis, false-positive rates would increase beyond the nominal level. We conducted a set of Monte-Carlo simulations to demonstrate our argument, and provided a literature review to gauge awareness and prevalence of this issue. Furthermore, we made several recommendations when using internal meta-analysis to make a judgment on statistical significance.

Effect sizes and standardization in neighbourhood models of forest stands: potential biases and misinterpretations

Effects of conspecific neighbours on survival and growth of trees have been found to be related to species abundance. Both positive and negative relationships may explain observed abundance patterns. Surprisingly, it is rarely tested whether such relationships could be biased or even spurious due to transforming neighbourhood variables or influences of spatial aggregation, distance decay of neighbour effects and standardization of effect sizes. To investigate potential biases, communities of 20 identical species were simulated with log-series abundances but without species-specific interactions. No relationship of conspecific neighbour effects on survival or growth with species abundance was expected. Survival and growth of individuals was simulated in random and aggregated spatial patterns using no, linear, or squared distance decay of neighbour effects. Regression coefficients of statistical neighbourhood models were unbiased and unrelated to species abundance. However, variation in the number of conspecific neighbours was positively or negatively related to species abundance depending on transformations of neighbourhood variables, spatial pattern and distance decay. Consequently, effect sizes and standardized regression coefficients, often used in model fitting across large numbers of species, were also positively or negatively related to species abundance depending on transformation of neighbourhood variables, spatial pattern and distance decay. Tests using randomized tree positions and identities provide the best benchmarks by which to critically evaluate relationships of effect sizes or standardized regression coefficients with tree species abundance. This will better guard against potential misinterpretations.

Robust effect sizes and their confidence intervals for group difference between trajectories in hierarchical linear growth model

Hierarchical linear growth model (HLGM), as a flexible and powerful analytic method, has played an increased important role in psychology, public health and medical sciences in recent decades. Mostly, researchers who conduct HLGM are interested in the treatment effect on individual trajectories, which can be indicated by the cross-level interaction effects. However, the statistical hypothesis test for the effect of cross-level interaction in HLGM only show us whether there is a significant group difference in the average rate of change, rate of acceleration or higher polynomial effect; it fails to convey information about the magnitude of the difference between the group trajectories at specific time point. Thus, reporting and interpreting effect sizes have been increased emphases in HLGM in recent years, due to the limitations and increased criticisms for statistical hypothesis testing. However, most researchers fail to report these model-implied effect sizes for group trajectories comparison and their corresponding confidence intervals in HLGM analysis, since lack of appropriate and standard functions to estimate effect sizes associated with the model-implied difference between grouping trajectories in HLGM, and also lack of computing packages in the popular statistical software to automatically calculate them. ^ The present project is the first to establish the appropriate computing functions to assess the standard difference between grouping trajectories in HLGM. We proposed the two functions to estimate effect sizes on model-based grouping trajectories difference at specific time, we also suggested the robust effect sizes to reduce the bias of estimated effect sizes. Then, we applied the proposed functions to estimate the population effect sizes (d ) and robust effect sizes (du) on the cross-level interaction in HLGM by using the three simulated datasets, and also we compared the three methods of constructing confidence intervals around d and du recommended the best one for application. At the end, we constructed 95% confidence intervals with the suitable method for the effect sizes what we obtained with the three simulated datasets. ^ The effect sizes between grouping trajectories for the three simulated longitudinal datasets indicated that even though the statistical hypothesis test shows no significant difference between grouping trajectories, effect sizes between these grouping trajectories can still be large at some time points. Therefore, effect sizes between grouping trajectories in HLGM analysis provide us additional and meaningful information to assess group effect on individual trajectories. In addition, we also compared the three methods to construct 95% confident intervals around corresponding effect sizes in this project, which handled with the uncertainty of effect sizes to population parameter. We suggested the noncentral t-distribution based method when the assumptions held, and the bootstrap bias-corrected and accelerated method when the assumptions are not met.^

The effect of Amblyopia on motor and psychosocial skills in children

Background/Aims: In an investigation of the functional impact of amblyopia on children, the fine motor skills, perceived self-esteem and eye movements of amblyopic children were compared with that of age-matched controls. The influence of amblyogenic condition or treatment factors that might predict any decrement in outcome measures was investigated. The relationship between indirect measures of eye movements that are used clinically and eye movement characteristics recorded during reading was examined and the relevance of proficiency in fine motor skills to performance on standardised educational tests was explored in a sub-group of the control children. Methods: Children with amblyopia (n=82; age 8.2 ± 1.3 years) from differing causes (infantile esotropia n=17, acquired strabismus n=28, anisometropia n=15, mixed n=13 and deprivation n=9), and a control group of children (n=106; age 9.5 ± 1.2 years) participated in this study. Measures of visual function included monocular logMAR visual acuity (VA) and stereopsis assessed with the Randot Preschool Stereoacuity test, while fine motor skills were measured using the Visual-Motor Control (VMC) and Upper Limb Speed and Dexterity (ULSD) subtests of the Brunicks-Oseretsky Test of Motor Proficiency. Perceived self esteem was assessed for those children from grade 3 school level with the Harter Self Perception Profile for Children and for those in younger grades (preschool to grade 2) with the Pictorial Scale of Perceived Competence and Acceptance for Young Children. A clinical measure of eye movements was made with the Developmental Eye Movement (DEM) test for those children aged eight years and above. For appropriate case-control comparison of data, the results from amblyopic children were compared with age-matched sub-samples drawn from the group of children with normal vision who completed the tests. Eye movements during reading for comprehension were recorded by the Visagraph infra-red recording system and results of standardised tests of educational performance were also obtained for a sub-set of the control group. Results Amblyopic children (n=82; age 8.2 ± 1.7 years) performed significantly poorer than age-matched control children (n=37; age 8.3 ± 1.3 years) on 9 of 16 fine motor skills sub-items and for the overall age-standardised scores for both VMC and ULSD items (p<0.05); differences were most evident on timed manual dexterity tasks. The underlying aetiology of amblyopia and level of stereoacuity significantly affected fine motor skill performance on both items. However, when examined in a multiple regression model that took into account the inter-correlation between visual characteristics, poorer fine motor skills performance was only associated with strabismus (F1,75 = 5.428; p =0. 022), and not with the level of stereoacuity, refractive error or visual acuity in either eye. Amblyopic children from grade 3 school level and above (n=47; age 9.2 ± 1.3 years), particularly those with acquired strabismus, had significantly lower social acceptance scores than age-matched control children (n=52; age 9.4 ± 0.5 years) (F(5,93) = 3.14; p = 0.012). However, the scores of the amblyopic children were not significantly different to controls for other areas related to self-esteem, including scholastic competence, physical appearance, athletic competence, behavioural conduct and global self worth. A lower social acceptance score was independently associated with a history of treatment with patching but not with a history of strabismus or wearing glasses. Amblyopic children from pre-school to grade 2 school level (n=29; age = 6.6 ± 0.6 years) had similar self-perception scores to their age-matched peers (n=20; age = 6.4 ± 0.5 years). There were no significant differences between the amblyopic (n=39; age 9.1 ± 0.9 years) and age-matched control (n = 42; age = 9.3 ± 0.38 years) groups for any of the DEM outcome measures (Vertical Time, Horizontal Time, Number of Errors and Ratio (Horizontal time/Vertical time)). Performance on the DEM did not significantly relate to measures of VA in either eye, level of binocular function, history of strabismus or refractive error. Developmental Eye Movement test outcome measures Horizontal Time and Vertical Time were significantly correlated with reading rates measured by the Visagraph for both reading for comprehension and naming numbers (r>0.5). Some moderate correlations were also seen between the DEM Ratio and word reading rates as recorded by Visagraph (r=0.37). In children with normal vision, academic scores in mathematics, spelling and reading were associated with measures of fine motor skills. Strongest effect sizes were seen with the timed manual dexterity domain, Upper Limb Speed and Dexterity. Conclusions Amblyopia may have a negative impact on a child’s fine motor skills and an older child’s sense of acceptance by their peers may be influenced by treatment that includes eye patching. Clinical measures of eye movements were not affected in amblyopic children. A number of the outcome measures of the DEM are associated with objective recordings of reading rates, supporting its clinical use for identification of children with slower reading rates. In children with normal vision, proficiency on clinical measures of fine motor skill are associated with outcomes on standardised measures of educational performance. Scores on timed manual dexterity tasks had the strongest association with educational performance. Collectively, the results of this study indicate that, in addition to the reduction in visual acuity and binocular function that define the condition, amblyopes have functional impairment in childhood development skills that underlie proficiency in everyday activities. The study provides support for strategies aimed at early identification and remediation of amblyopia and the co-morbidities that arise from abnormal visual neurodevelopment.

Should athletes return to sport after applying Ice? A systematic review of the effect of local 3 cooling on functional performance

Applying ice or other forms of topical cooling is a popular method of treating sports injuries. It is commonplace for athletes to return to competitive activity, shortly or immediately after the application of a cold treatment. In this article, we examine the effect of local tissue cooling on outcomes relating to functional performance and to discuss their relevance to the sporting environment. A computerized literature search, citation tracking and hand search was performed up to April, 2011. Eligible studies were trials involving healthy human participants, describing the effects of cooling on outcomes relating to functional performance. Two reviewers independently assessed the validity of included trials and calculated effect sizes. Thirty five trials met the inclusion criteria; all had a high risk of bias. The mean sample size was 19. Meta-analyses were not undertaken due to clinical heterogeneity. The majority of studies used cooling durations >20 minutes. Strength (peak torque/force) was reported by 25 studies with approximately 75% recording a decrease in strength immediately following cooling. There was evidence from six studies that cooling adversely affected speed, power and agility-based running tasks; two studies found this was negated with a short rewarming period. There was conflicting evidence on the effect of cooling on isolated muscular endurance. A small number of studies found that cooling decreased upper limb dexterity and accuracy. The current evidence base suggests that athletes will probably be at a performance disadvantage if they return to activity immediately after cooling. This is based on cooling for longer than 20 minutes, which may exceed the durations employed in some sporting environments. In addition, some of the reported changes were clinically small and may only be relevant in elite sport. Until better evidence is available, practitioners should use short cooling applications and/or undertake a progressive warm up prior to returning to play.

Comprehensive treatments for social cognitive deficits in schizophrenia: A critical review and effect-size analysis of controlled studies

Recent advances in psychosocial treatments for schizophrenia have targeted social cognitive deficits. A critical literature review and effect-size (ES) analysis was conducted to investigate the efficacy of comprehensive programs of social cognitive training in schizophrenia. Results revealed 16 controlled studies consisting of seven models of comprehensive treatment with only three of these treatment models investigated in more than one study. The effects of social cognitive training were reported in 11/15 studies that included facial affect recognition skills (ES=.84) and 10/13 studies that included theory-of-mind (ES=.70) as outcomes. Less than half (4/9) of studies that measured attributional style as an outcome reported effects of treatment, but effect sizes across studies were significant (ESs=.30-.52). The effect sizes for symptoms were modest, but, with the exception of positive symptoms, significant (ESs=.32-.40). The majority of trials were randomized (13/16), selected active control conditions (11/16) and included at least 30 participants (12/16). Concerns for this area of research include the absence of blinded outcome raters in more than 50% of trials and low rates of utilization of procedures for maintaining treatment fidelity. These findings provide preliminary support for the broader use of comprehensive social cognitive training procedures as a psychosocial intervention for schizophrenia.

The Effect of a Stickhandling and Puck Control (SPC) Training Intervention on SPC Skills and Wrist Shot Performance Variables in Female Collegiate Ice Hockey Players

The purpose of the study was to investigate the effect of a 16 session stickhandling and puck control (SPC) off-ice training intervention on SPC skills and wrist shot performance variables. Eighteen female collegiate ice hockey players participated in a crossover design training intervention, whereby players were randomly assigned to two groups. Each group completed 16 SPC training sessions in two conditions [normal vision (NV) and restricted vision (RV)]. Measures obtained after the training intervention revealed significant improvements in SPC skills and wrist shot accuracy. Order of training condition did not reach significance, meaning that SPC improvement occurred as a result of total training volume as opposed to order of training condition. However, overall changes in the RV-NV condition revealed consistently higher effect sizes, meaning a greater improvement in performance. Therefore, support can be provided for this technical approach to SPC training and an alternative method of challenging SPC skills.

Is the allegiance effect an epiphenomenon of true efficacy differences between treatments? a meta-analysis

Many meta-analyses of comparative outcome studies found a substantial association of researcher allegiance (RA) and relative treatment effects. Therefore, RA is regarded as a biasing factor in comparative outcome research (RA bias hypothesis). However, the RA bias hypothesis has been criticized as causality might be reversed. That is, RA might be a reflection of true efficacy differences between treatments (true efficacy hypothesis). Consequently, the RA-outcome association would not be indicative of bias but an epiphenomenon of true efficacy differences. This meta-analysis tested the validity of the true efficacy hypothesis. This was done by controlling the RA-outcome association for true efficacy differences by restricting analysis to direct comparisons of treatments with equivalent efficacy. We included direct comparisons of different versions of trauma-focused therapy (TFT) in the treatment of posttraumatic stress disorder (PTSD). RA was measured from the research reports. Relative effect sizes for symptoms of PTSD were calculated. Random effects meta-regression was conducted. Twenty-nine comparisons of TFTs from 20 studies were identified. Initial heterogeneity among relative effect sizes was low. RA was a significant predictor of outcome and explained 12% of the variance in outcomes. The true efficacy hypothesis predicted the RA-outcome association to be zero; however, a substantial association was found. Thus, this study does not support the true efficacy hypothesis. Given findings from psychotherapy research and other fields that support a biasing influence of researcher preferences, RA should be regarded as a causal factor and conceptualized as a threat to the validity of conclusions from comparative outcome studies.

Effect of therapeutic exercise for hip osteoarthritis pain: results of a meta-analysis

OBJECTIVE: Recommendations for lower extremity osteoarthritis (OA) and exercise have been primarily based on knee studies. To provide more targeted recommendations for the hip, we gathered evidence for the efficacy of exercise for hip OA from randomized controlled trials. METHODS: A bibliographic search identified trials that were randomized, controlled, completed by >or=60% of subjects, and involved an exercise group (strengthening and/or aerobic) versus a non exercise control group for pain relief in hip OA. Two reviewers independently performed the data extraction and contacted the authors when necessary. Effect sizes (ES) of treatment versus control and the I(2) statistic to assess heterogeneity across trials were calculated. Trial data were combined using a random-effects meta-analysis. RESULTS: Nine trials met the inclusion criteria (1,234 subjects), 7 of which combined hip and knee OA; therefore, we contacted the authors who provided the data on hip OA patients. In comparing exercise treatment versus control, we found a beneficial effect of exercise with an ES of -0.38 (95% confidence interval [95% CI] -0.68, -0.08; P = 0.01), but with high heterogeneity (I(2) = 75%) among trials. Heterogeneity was caused by 1 trial consisting of an exercise intervention that was not administered in person. Removing this study left 8 trials (n = 493) with similar exercise strategy (specialized hands-on exercise training, all of which included at least some element of muscle strengthening), and demonstrated exercise benefit with an ES of -0.46 (95% CI -0.64, -0.28; P < 0.0001). CONCLUSION: Therapeutic exercise, especially with an element of strengthening, is an efficacious treatment for hip OA.

Effect of psychological interventions on depressive symptoms in long-term rehabilitation after an acquired brain injury: a systematic review and meta-analysis

OBJECTIVE To summarize empirical studies on the effectiveness of psychological interventions in long-term rehabilitation after an acquired brain injury (ABI) in reducing depressive symptoms. DATA SOURCES A systematic literature search was conducted on MEDLINE, PsycINFO, Embase, and CINAHL to identify articles published between January 1990 and October 2011. Search terms included the 3 concepts (1) "brain injur*" or "stroke," (2) "psychotherap*" or "therapy" or "intervention" or "rehabilitation," and (3) "depress*." STUDY SELECTION Studies evaluating psychological interventions in patients after ABI were included. Time since injury was on average more than 1 year. Trials reported data on validated depression questionnaires before and after the psychological intervention. DATA EXTRACTION Two independent reviewers extracted information from the sample, the intervention, and the outcome of the included studies and calculated effect sizes (ESs) from depression questionnaires. Thirteen studies were included in a pre-post analysis. Seven studies were eligible for a meta-analysis of ESs in active interventions and control conditions. DATA SYNTHESIS Pre-post ESs were significant in 4 of 13 studies. The overall ES of .69 (95% confidence interval [CI], .29-1.09) suggests a medium effectiveness of psychological interventions on depressive symptoms compared with control conditions. Moderator analysis of the number of sessions and adequate randomization procedure did not show significant ES differences between strata. Studies with adequate randomization did not, however, suggest the effectiveness of psychological interventions on depressive symptoms after ABI. CONCLUSIONS Psychological interventions are a promising treatment option for depressive symptoms in long-term rehabilitation after ABI. Since only a few adequately randomized controlled trials (RCTs) exist, more RCTs are required to confirm this initial finding.