Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.

Subordinate removal affects parental investment, but not offspring survival in a cooperative cichlid

Summary Subordinates in cooperative breeding systems may provide help to dominant pairs, who can benefit by either an increased total investment in their current brood or a reduced personal contribution to this investment. In the social cichlid Julidochromis ornatus, one large male subordinate generally spends 90% of his time in close proximity to the breeding shelter, whereas the dominants only spend 50% of their time close to the shelter. We experimentally removed the large subordinate for 30 days (approximating one breeding cycle) to study the investment strategies of dominants and the effects on offspring survival, while accounting for subordinate immigration. Experimental groups were compared with control groups, from which subordinates were also caught but not removed. On day one following removal, we tested whether dominants overcompensated, fully compensated or undercompensated for absence of the subordinate on several parental behaviours. Moreover, we tested whether the pairs' potential compensatory behaviour remained high seven days following large subordinate removal. One day following removal, dominants increased their time spent in the territory and their frequency of breeding shelter visits and defence, compared with the pre-removal phase and control groups. The dominant pair overcompensated for the loss of subordinate help in their breeding shelter visits, fully compensated in defence and undercompensated their time spent in the territory. Seven days after large subordinate removal, behavioural differences between treatments had disappeared. However, when distinguishing between groups with or without a new immigrant subordinate, dominant pairs only diminished investment in the presence of an immigrant, suggesting a compensatory role of the large subordinate. Finally, survival of juvenile group members was not affected by the treatment. Our experiments indicate that the presence of a large subordinate does not increase the dominant pairs' current reproductive success, but instead allows them to reduce their personal contribution to investment in the current brood. In addition, we illustrate that dominants may show strikingly different compensatory responses depending on the type of behaviour and emphasize the importance of immigrant subordinates to relieve dominants from costly compensatory responses in cooperative breeding systems.

Group size adjustment to ecological demand in a cooperative breeder

Environmental factors can determine which group size will maximize the fitness of group members. This is particularly important in cooperative breeders, where group members often serve different purposes. Experimental studies are yet lacking to check whether ecologically mediated need for help will change the propensity of dominant group members to accept immigrants. Here, we manipulated the perceived risk of predation for dominant breeders of the cooperatively breeding cichlid fish Neolamprologus pulcher to test their response to unrelated and previously unknown immigrants. Potential immigrants were more readily accepted if groups were exposed to fish predators or egg predators than to herbivorous fish or control situations lacking predation risk. Our data are consistent with both risk dilution and helping effects. Egg predators were presented before spawning, which might suggest that the fish adjust acceptance rates also to a potential future threat. Dominant group members of N. pulcher apparently consider both present and future need of help based on ecological demand. This suggests that acceptance of immigrants and, more generally, tolerance of group members on demand could be a widespread response to ecological conditions in cooperatively breeding animals.