Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.

Biactantial agreement in the Gongduk transitive verb in the broader Tibeto-Burman context

The Gongduk language is spoken in an enclave in south central Bhutan comprising several villages and hamlets in the mountains west of the Kurichu. The language occupies a distinct phylogenetic position within the Tibeto-Burman language family. The intransitive verb agrees for person and number with the subject, and the transitive shows biactantial agreement for person and number with both agent and patient. A morphological analysis has identified the individual agreement morphemes, their precise grammatical meaning and their patterns of allomorphy. The cognacy of the greater part of the desinences of the Gongduk verb with morphemes identifiable in the biactantial agreement systems of other Tibeto-Burman languages supports the view that at least a portion of such conjugational morphology must be reconstructed to the common ancestral language.

Rezension zu: Inge Govaere and Hanns Ulrich (eds.): Intellectual Property, Public Policy and International Trade

The contemporary intellectual property rights (IPR) system is not a simple, smoothly working block of rules but is complex and full of ambiguities, and as many argue, imperfections. Some deficits relate on the one hand to the inherent centrality of authorship, originality and mercantilism to the ‘Western’ IP model, which leaves numerous non-Western, collaborative or folkloric modes of production outside the scope of protection. On the other hand, some imperfections stem from the way IPR are granted, whereby creators acquire a temporary monopoly over their works and thus exclude the public from having access to them. In this sense, it is often uncertain whether the existent IPR model appropriately reflects the precarious balance between private and public interests, and whether the best incentives to promote creativity and innovation - the initially stated objectives of intellectual property protection - are offered. The matter becomes still more complicated when one considers that the IPR system is not domestically contained but is globalised and strongly affected by rules at the regional and international levels. The question of whether the balance between private interests and public values is sustained within the international legal framework, epitomised by the Agreement on Trade-Related Aspects of Intellectual Property Rights (TRIPS) of the World Trade Organization (WTO), is precisely the topic of the book reviewed here. Review of Intellectual Property, Public Policy, and International Trade, edited by Inge Govaere and Hanns Ullrich, P.I.E. Peter Lang, 2007.

Inter-observer agreement for diagnostic classification of esophageal motility disorders defined in high-resolution manometry

High-resolution esophageal manometry (HRM) is a recent development used in the evaluation of esophageal function. Our aim was to assess the inter-observer agreement for diagnosis of esophageal motility disorders using this technology. Practitioners registered on the HRM Working Group website were invited to review and classify (i) 147 individual water swallows and (ii) 40 diagnostic studies comprising 10 swallows using a drop-down menu that followed the Chicago Classification system. Data were presented using a standardized format with pressure contours without a summary of HRM metrics. The sequence of swallows was fixed for each user but randomized between users to avoid sequence bias. Participants were blinded to other entries. (i) Individual swallows were assessed by 18 practitioners (13 institutions). Consensus agreement (≤2/18 dissenters) was present for most cases of normal peristalsis and achalasia but not for cases of peristaltic dysmotility. (ii) Diagnostic studies were assessed by 36 practitioners (28 institutions). Overall inter-observer agreement was 'moderate' (kappa 0.51) being 'substantial' (kappa > 0.7) for achalasia type I/II and no lower than 'fair-moderate' (kappa >0.34) for any diagnosis. Overall agreement was somewhat higher among those that had performed >400 studies (n = 9; kappa 0.55) and 'substantial' among experts involved in development of the Chicago Classification system (n = 4; kappa 0.66). This prospective, randomized, and blinded study reports an acceptable level of inter-observer agreement for HRM diagnoses across the full spectrum of esophageal motility disorders for a large group of clinicians working in a range of medical institutions. Suboptimal agreement for diagnosis of peristaltic motility disorders highlights contribution of objective HRM metrics.

Law and science: risk assessment and risk management in the WTO agreement on the application of sanitary and phytosanitary measures

Assessing and managing risks relating to the consumption of food stuffs for humans and to the environment has been one of the most complex legal issues in WTO law, ever since the Agreement on Sanitary and Phytosanitary Measures was adopted at the end of the Uruguay Round and entered into force in 1995. The problem was expounded in a number of cases. Panels and the Appellate Body adopted different philosophies in interpreting the agreement and the basic concept of risk assessment as defined in Annex A para. 4 of the Agreement. Risk assessment entails fundamental question on law and science. Different interpretations reflect different underlying perceptions of science and its relationship to the law. The present thesis supported by the Swiss National Research Foundation undertakes an in-depth analysis of these underlying perceptions. The author expounds the essence and differences of positivism and relativism in philosophy and natural sciences. He clarifies the relationship of fundamental concepts such as risk, hazards and probability. This investigation is a remarkable effort on the part of lawyer keen to learn more about the fundamentals based upon which the law – often unconsciously – is operated by the legal profession and the trade community. Based upon these insights, he turns to a critical assessment of jurisprudence both of panels and the Appellate Body. Extensively referring and discussing the literature, he deconstructs findings and decisions in light of implied and assumed underlying philosophies and perceptions as to the relationship of law and science, in particular in the field of food standards. Finding that both positivism and relativism does not provide adequate answers, the author turns critical rationalism and applies the methodologies of falsification developed by Karl R. Popper. Critical rationalism allows combining discourse in science and law and helps preparing the ground for a new approach to risk assessment and risk management. Linking the problem to the doctrine of multilevel governance the author develops a theory allocating risk assessment to international for a while leaving the matter of risk management to national and democratically accountable government. While the author throughout the thesis questions the possibility of separating risk assessment and risk management, the thesis offers new avenues which may assist in structuring a complex and difficult problem

A Transatlantic Trade and Investment Partnership Agreement: Implications for Swiss Agriculture

To say that regionalism is gaining momentum has become an understatement. To mourn the lack of progress in multilateral trade rule-making is a commonplace in the discourse of politicians regretting the WTO negotiation standstill, and of “know-what-to-do” academics. The real problem is the uneven level-playing field resulting from increasing differences of rules and obligations. The Transatlantic Trade and Investment Partnership Agreement (TTIP) is a very ambitious project. WTI studies in 2014 have shown that the implications for Switzerland could be enormous. But even the combined market power of the two TTIP participants – the EU and the USA – will not level the playing field impairing the regulatory framework, and the market access barriers for trade in agriculture. Such differences will remain in three areas which, incidentally, are also vital for a global response to the food security challenge to feed 9 billion people before the year 2050: market access, non-tariff barriers, and trade-distorting domestic support programmes. This means that without multilateral progress the TTIP and other so-called mega-regionals, if successfully concluded, will exacerbate rather than lessen trade distortions. While this makes farmers in rich countries safer from competition, competitive production in all countries will be hampered. Consequently, and notwithstanding the many affirmations to the contrary, farm policies worldwide will continue to only address farmer security without increasing global food security. What are the implications of the TTIP for Swiss agriculture? This article, commissioned by Waseda University in Tokyo, finds that the failure to achieve further reforms – including a number of areas where earlier reforms have been reversed – is presenting Switzerland and Swiss agriculture with a terrible dilemma in the eventuality of a successful conclusion of the TTIP. If Swiss farm production is to survive for more than another generation, continuous reform efforts are required, and over-reliance on the traditional instruments of border protection and product support is to be avoided. Without a substantial TTIP obliging Switzerland to follow suit, autonomous reforms will remain extremely fragile.

Intervals for the assessment of measurement agreement: Similarities, differences, and consequences of incorrect interpretations

Several intervals have been proposed to quantify the agreement of two methods intended to measure the same quantity in the situation where only one measurement per method and subject is available. The limits of agreement are probably the most well-known among these intervals, which are all based on the differences between the two measurement methods. The different meanings of the intervals are not always properly recognized in applications. However, at least for small-to-moderate sample sizes, the differences will be substantial. This is illustrated both using the width of the intervals and on probabilistic scales related to the definitions of the intervals. In particular, for small-to-moderate sample sizes, it is shown that limits of agreement and prediction intervals should not be used to make statements about the distribution of the differences between the two measurement methods or about a plausible range for all future differences. Care should therefore be taken to ensure the correct choice of the interval for the intended interpretation.

Analysis of the prospects for updating the trade pillar of the European Union-Chile Association Agreement

The perception of the present state of trade relations with Chile is obscured by a lack of adequate understanding of its legal framework as well as of the policy behind it. This study attempts to clarify the present state of and future prospects for trade between the EU and Chile through an examination of previous agreements and the EU’s new approach to trade liberalisation. The authors agree with the large consensus existing on both the EU and Chilean sides regarding the efficacy of the Association Agreement, but note that any extension of an agreement with Chile should capture the spirit of older EU agreements rather than simply following the ‘NAFTA route’. The study also includes a comparative analysis between the EU-Chile agreement and current trade agreements being negotiated by the EU and Chile with third countries.

Analysis of the upcoming Modernisation of the Trade Pillar of the European Union-Mexico Global Agreement

Study for the EU Parliament co-authored by Rodrigo Polanco: The 1997 Global Agreement between the EC and its Member States and Mexico, together with the set of decisions taken in its framework, has been effective, and thus modifications of the agreement are mainly motivated by changes in the global landscape since it was first enacted. Therefore, broad considerations on how the European Union (EU) trade policy is shaped are extremely relevant for the upcoming negotiations with Mexico. In this context, the needs and expectations, both from the EU and Mexico, regarding any further agreements are examined, focusing in particular on areas beyond trade in goods and services such as procurement, investment, and regulatory cooperation. It is argued that the 'old' Association Agreements should be taken as models for any modifications, given their emphasis on EU-specific issues and their ability to accommodate the needs of Mexico in any deepened agreement.