Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.

A Primer on ACTA What Europeans Should Fear about the Anti-Counterfeiting Trade Agreement

The European Commission recently published the first official draft of the Anti-Counterfeiting Trade Agreement (ACTA). The article describes the institutional background of the negotiations on ACTA and its relationship to the existing legal framework. The civil enforcement provisions and the Internet chapter are compared with the international and European instruments in the field. For the most part, ACTA will not oblige EU member states to enact rules that go beyond the already established European standards. But stricter rules could be implemented regarding injunctions against non-infringing intermediaries, strict liability rules for damages, and ex parte measures in preliminary proceedings. According to the published draft, the termination of user accounts in the case of repeated intellectual property infringement will not be mandatory for member ACTA states.

EU Commission: Comments on the “Opinion of European Academics on Anti-Counterfeiting Trade Agreement”

The “Opinion of European Academics on Anti-Counterfeiting Trade Agreement” (ACTA) of February 11, 2011, was published in 2 JIPITEC 65 (2011). Signed by more than 25 law professors and academics from across Europe who specialize in the field, this opinion addressed the following concern: Although it is uncontested that the infringement of intellectual property rights, especially in the Internet, prejudices the legitimate interests of right holders, it is still very controversial in Europe and abroad whether the enforcement standards of ACTA are balanced. The European Commission, DG Trade, has now published a document with detailed comments on the Opinion. The comments, which are also available on the website of the European Commission [http://trade.ec.europa.eu/doclib/ html/147853.htm], are republished here with the kind permission of the European Commission.

Assessing collective defensive performances in football: A Qualitative Comparative Analysis of central back pairs

Ahead of the World Cup in Brazil the crucial question for the Swiss national coach is the nomination of the starting eleven central back pair. A fuzzy set Qualitative Comparative Analysis assesses the defensive performances of different Swiss central back pairs during the World Cup campaign (2011 – 2014). This analysis advises Ottmar Hitzfeld to nominate Steve von Bergen and Johan Djourou as the starting eleven central back pair. The alternative with a substantially weaker empirical validity would be Johan Djourou together with Phillippe Senderos. Furthermore, this paper aims to be a step forward in mainstream football analytics. It analyses the undervalued and understudied defense (Anderson and Sally 2012, Statsbomb 2013) by explaining collective defensive performances instead of assessments of individual player or team performances. However, a qualitatively (better defensive metrics) and quantitatively (more games) improved and extended data set would allow for a more sophisticated analysis of collective defensive performances.