Pre-operative tractography of the facial nerve in vestibular schwannomas: inter-observer agreement with surgical findings

Pre-operative diffusion tensor (DT) tractography is currently employed in our institutions. We use it to predict the course of the facial nerve (FN) in the vicinity of vestibular schwannomas (VS) of the cerebellopontine angle (CPA). In this study we were interested to assess the inter-observer reproducibility of this method. Two Neuroradiologists (PMGP and TT) determined independently the location of the FN by tractography and compared the results with in-vivo findings of microsurgery of VS.

Inter-observer agreement for spectral- and time-domain optical coherence tomography image grading: a prospective study

The purpose of this study was to compare inter-observer agreement of Stratus™ OCT versus Spectralis™ OCT image grading in patients with neovascular age-related macular degeneration (AMD). Thirty eyes with neovascular AMD were examined with Stratus™ OCT and Spectralis™ OCT. Four different scan protocols were used for imaging. Three observers graded the images for the presence of various pathologies. Inter-observer agreement between OCT models was assessed by calculating intra-class correlation coefficients (ICC). In Stratus™ OCT highest interobserver agreement was found for subretinal fluid (ICC: 0.79), and in Spectralis™ OCT for intraretinal cysts (IRC) (ICC: 0.93). Spectralis™ OCT showed superior interobserver agreement for IRC and epiretinal membranes (ERM) (ICC(Stratus™): for IRC 0.61; for ERM 0.56; ICC(Spectralis™): for IRC 0.93; for ERM 0.84). Increased image resolution of Spectralis™ OCT did improve the inter-observer agreement for grading intraretinal cysts and epiretinal membranes but not for other retinal changes.

Inter-observer agreement for diagnostic classification of esophageal motility disorders defined in high-resolution manometry

High-resolution esophageal manometry (HRM) is a recent development used in the evaluation of esophageal function. Our aim was to assess the inter-observer agreement for diagnosis of esophageal motility disorders using this technology. Practitioners registered on the HRM Working Group website were invited to review and classify (i) 147 individual water swallows and (ii) 40 diagnostic studies comprising 10 swallows using a drop-down menu that followed the Chicago Classification system. Data were presented using a standardized format with pressure contours without a summary of HRM metrics. The sequence of swallows was fixed for each user but randomized between users to avoid sequence bias. Participants were blinded to other entries. (i) Individual swallows were assessed by 18 practitioners (13 institutions). Consensus agreement (≤2/18 dissenters) was present for most cases of normal peristalsis and achalasia but not for cases of peristaltic dysmotility. (ii) Diagnostic studies were assessed by 36 practitioners (28 institutions). Overall inter-observer agreement was 'moderate' (kappa 0.51) being 'substantial' (kappa > 0.7) for achalasia type I/II and no lower than 'fair-moderate' (kappa >0.34) for any diagnosis. Overall agreement was somewhat higher among those that had performed >400 studies (n = 9; kappa 0.55) and 'substantial' among experts involved in development of the Chicago Classification system (n = 4; kappa 0.66). This prospective, randomized, and blinded study reports an acceptable level of inter-observer agreement for HRM diagnoses across the full spectrum of esophageal motility disorders for a large group of clinicians working in a range of medical institutions. Suboptimal agreement for diagnosis of peristaltic motility disorders highlights contribution of objective HRM metrics.

Intra- and inter-observer reliability of MRI examination of intervertebral disc abnormalities in patients with cervical myelopathy

Purpose: Intervertebral cervical disc herniation (CDH) is a relatively common disorder that can coexist with degenerative changes to worsen cervicogenic myelopathy. Despite the frequent disc abnormalities found in asymptomatic populations, magnetic resonance imaging (MRI) is considered excellent at detecting cervical spine myelopathy (CSM) associated with disc abnormality. The objective of this study was to investigate the intra- and inter-observer reliability of MRI detection of CSM in subjects who also had co-existing intervertebral disc abnormalities. Materials and methods: Seven experienced radiologists reviewed twice the MRI of 10 patients with clinically and/or imaging determined myelopathy. MRI assessment was performed individually, with and without operational guidelines. A Fleiss Kappa statistic was used to evaluate the intra- and inter-observer agreement. Results: The study found high intra-observer percent agreement but relatively low Kappa values on selected variables. Inter-observer reliability was also low and neither observation was improved with operational guidelines. We believe that those low values may be associated with the base rate problem of Kappa. Conclusion: In conclusion, this study demonstrated high intra-observer percent agreement in MR examination for intervertebral disc abnormalities in patients with underlying cervical myelopathy, but differing levels of intra- and inter-observer Kappa agreement among seven radiologists. (c) 2007 Elsevier Ireland Ltd. All rights reserved.

Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study

OBJECTIVE: To study the inter-observer variation related to extraction of continuous and numerical rating scale data from trial reports for use in meta-analyses. DESIGN: Observer agreement study. DATA SOURCES: A random sample of 10 Cochrane reviews that presented a result as a standardised mean difference (SMD), the protocols for the reviews and the trial reports (n=45) were retrieved. DATA EXTRACTION: Five experienced methodologists and five PhD students independently extracted data from the trial reports for calculation of the first SMD result in each review. The observers did not have access to the reviews but to the protocols, where the relevant outcome was highlighted. The agreement was analysed at both trial and meta-analysis level, pairing the observers in all possible ways (45 pairs, yielding 2025 pairs of trials and 450 pairs of meta-analyses). Agreement was defined as SMDs that differed less than 0.1 in their point estimates or confidence intervals. RESULTS: The agreement was 53% at trial level and 31% at meta-analysis level. Including all pairs, the median disagreement was SMD=0.22 (interquartile range 0.07-0.61). The experts agreed somewhat more than the PhD students at trial level (61% v 46%), but not at meta-analysis level. Important reasons for disagreement were differences in selection of time points, scales, control groups, and type of calculations; whether to include a trial in the meta-analysis; and data extraction errors made by the observers. In 14 out of the 100 SMDs calculated at the meta-analysis level, individual observers reached different conclusions than the originally published review. CONCLUSIONS: Disagreements were common and often larger than the effect of commonly used treatments. Meta-analyses using SMDs are prone to observer variation and should be interpreted with caution. The reliability of meta-analyses might be improved by having more detailed review protocols, more than one observer, and statistical expertise.

Observer agreement of spine stenosis on magnetic resonance imaging analysis of patients with cervical spine myelopathy

Objectives: The purpose of this study was to measure the intraobserver and interobserver reliability of magnetic resonance detection of cervical spondylotic myelopathy with and without operational guidelines. Methods: Seven radiologists examined images from 10 patients with cord signal abnormalities and clinical signs of myelopathy. Radiologist examined films twice, with and without operational guidelines designed to define stenotic changes, while blinded to the clinical findings of the patients. Analyses included a Fleiss kappa assessment of intraobserver and interobserver reliability. Results: Results demonstrated high percentage of agreement and strong intraobserver reliability and variable Fleiss kappa, values for interobserver assessment. Operational guidelines did not improve the intraobserver or interobserver agreement. Conclusion: Although the percentage of agreement was high in some cases, the kappa agreement was low-most likely a result of the base rate problem of a kappa analysis. Sample bias toward severe degenerative changes resulted in highly prevalent selections and kappa adjusted values. Nonetheless, the results do suggest that substantial intraobserver kappa agreement and a wide range of interobserver kappa agreement exists among trained radiologists during detection of stenotic changes associated with cervical spondylotic myelopathy.

Refinements in stress echocardiographic techniques improve inter-institutional agreement in interpretation of dobutamine stress echocardiograms

Aims To determine the degree of inter-institutional agreement in the assessment of dobutamine stress echocardiograms using modern stress echo cardiographic technology in combination with standardized data acquisition and assessment criteria. Method and Results Among six experienced institutions, 150 dobutamine stress echocardiograms (dobutamine up to 40 mug.kg(-1) min(-1) and atropine up to I mg) were performed on patients with suspected coronary artery disease using fundamental and harmonic imaging following a consistent digital acquisition protocol. Each dobutamine stress echocardiogram was assessed at every institution regarding endocardial visibility and left ventricular wall motion without knowledge of any other data using standardized reading criteria. No patients were excluded due to poor image quality or inadequate stress level. Coronary angiography was performed within 4 weeks. Coronary angiography demonstrated significant coronary artery disease (less than or equal to50% diameter stenosis) in 87 patients. Using harmonic imaging an average of 5.2+/-0.9 institutions agreed on dobutamine stress echocardiogram results as being normal or abnormal (mean kappa 0.55; 95% CI 0.50-0.60). Agreement was higher in patients with no (equal assessment of dobutamine stress echocardiogram results by 5.5 +/- 0.8 institutions) or three-vessel coronary artery disease (5.4 +/- 0.8 institutions) and lower in one- or two- vessel disease (5.0 +/- 0.9 and 5.2 +/- 1.0 institutions, respectively-, P=0.041). Disagreement on test results was greater in only minor wall motion abnormalities. Agreement on dobutamine stress echocardiogram results was lower using fundamental imaging (mean kappa 0.49; 95% CI 0.44-0.54; P

Inter-informant agreement and prevalence estimates for substance use disorders: direct interview versus family history method.

OBJECTIVES: Family studies typically use multiple sources of information on each individual including direct interviews and family history information. The aims of the present study were to: (1) assess agreement for diagnoses of specific substance use disorders between direct interviews and the family history method; (2) compare prevalence estimates according to the two methods; (3) test strategies to approximate prevalence estimates according to family history reports to those based on direct interviews; (4) determine covariates of inter-informant agreement; and (5) identify covariates that affect the likelihood of reporting disorders by informants. METHODS: Analyses were based on family study data which included 1621 distinct informant (first-degree relatives and spouses) - index subject pairs. RESULTS: Our main findings were: (1) inter-informant agreement was fair to good for all substance disorders, except for alcohol abuse; (2) the family history method underestimated the prevalence of drug but not alcohol use disorders; (3) lowering diagnostic thresholds for drug disorders and combining multiple family histories increased the accuracy of prevalence estimates for these disorders according to the family history method; (4) female sex of index subjects was associated with higher agreement for nearly all disorders; and (5) informants who themselves had a history of the same substance use disorder were more likely to report this disorder in their relatives, which entails the risk of overestimation of the size of familial aggregation. CONCLUSION: Our findings have important implications for the best-estimate procedure applied in family studies.

Inter-informant agreement and prevalence estimates for mood syndromes: Direct interview vs. family history method.

BACKGROUND: The use of the family history method is recommended in family studies as a type of proxy interview of non-participating relatives. However, using different sources of information can result in bias as direct interviews may provide a higher likelihood of assigning diagnoses than family history reports. The aims of the present study were to: 1) compare diagnoses for threshold and subthreshold mood syndromes from interviews to those relying on information from relatives; 2) test the appropriateness of lowering the diagnostic threshold and combining multiple reports from the family history method to obtain comparable prevalence estimates to the interviews; 3) identify factors that influence the likelihood of agreement and reporting of disorders by informants. METHODS: Within a family study, 1621 informant-index subject pairs were identified. DSM-5 diagnoses from direct interviews of index subjects were compared to those derived from family history information provided by their first-degree relatives. RESULTS: 1) Inter-informant agreement was acceptable for Mania, but low for all other mood syndromes. 2) Except for Mania and subthreshold depression, the family history method provided significantly lower prevalence estimates. The gap improved for all other syndromes after lowering the threshold of the family history method. 3) Individuals who had a history of depression themselves were more likely to report depression in their relatives. LIMITATIONS: Low proportion of affected individuals for manic syndromes and lack of independence of data. CONCLUSIONS: The higher likelihood of reporting disorders by affected informants entails the risk of overestimation of the size of familial aggregation of depression.

Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist

Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). Methods: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. Results: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions.Conclusions: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.

Assessment of inter-examiner agreement and variability in the manual classification of auditory brainstem response

Abstract Background: The analysis of the Auditory Brainstem Response (ABR) is of fundamental importance to the investigation of the auditory system behaviour, though its interpretation has a subjective nature because of the manual process employed in its study and the clinical experience required for its analysis. When analysing the ABR, clinicians are often interested in the identification of ABR signal components referred to as Jewett waves. In particular, the detection and study of the time when these waves occur (i.e., the wave latency) is a practical tool for the diagnosis of disorders affecting the auditory system. Significant differences in inter-examiner results may lead to completely distinct clinical interpretations of the state of the auditory system. In this context, the aim of this research was to evaluate the inter-examiner agreement and variability in the manual classification of ABR. Methods: A total of 160 ABR data samples were collected, for four different stimulus intensity (80dBHL, 60dBHL, 40dBHL and 20dBHL), from 10 normal-hearing subjects (5 men and 5 women, from 20 to 52 years). Four examiners with expertise in the manual classification of ABR components participated in the study. The Bland-Altman statistical method was employed for the assessment of inter-examiner agreement and variability. The mean, standard deviation and error for the bias, which is the difference between examiners’ annotations, were estimated for each pair of examiners. Scatter plots and histograms were employed for data visualization and analysis. Results: In most comparisons the differences between examiner’s annotations were below 0.1 ms, which is clinically acceptable. In four cases, it was found a large error and standard deviation (>0.1 ms) that indicate the presence of outliers and thus, discrepancies between examiners. Conclusions: Our results quantify the inter-examiner agreement and variability of the manual analysis of ABR data, and they also allows for the determination of different patterns of manual ABR analysis.

Towards criterion validity in classroom language analysis: methodological constraints of metadiscourse and inter-rater agreement

This paper reports on a process to validate a revised version of a system for coding classroom discourse in foreign language lessons, a context in which the dual role of language (as content and means of communication) and the speakers' specific pedagogical aims lead to a certain degree of ambiguity in language analysis. The language used by teachers and students has been extensively studied, and a framework of concepts concerning classroom discourse well-established. Models for coding classroom language need, however, to be revised when they are applied to specific research contexts. The application and revision of an initial framework can lead to the development of earlier models, and to the re-definition of previously established categories of analysis that have to be validated. The procedures followed to validate a coding system are related here as guidelines for conducting research under similar circumstances. The advantages of using instruments that incorporate two types of data, that is, quantitative measures and qualitative information from raters' metadiscourse, are discussed, and it is suggested that such procedure can contribute to the process of validation itself, towards attaining reliability of research results, as well as indicate some constraints of the adopted research methodology.

Impact of inter-judge agreement on perceptual judgment of nasality

Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)