From text to codings: intercoder reliability assessment in qualitative content analysis

BACKGROUND: High intercoder reliability (ICR) is required in qualitative content analysis for assuring quality when more than one coder is involved in data analysis. The literature is short of standardized procedures for ICR procedures in qualitative content analysis. OBJECTIVE: To illustrate how ICR assessment can be used to improve codings in qualitative content analysis. METHODS: Key steps of the procedure are presented, drawing on data from a qualitative study on patients' perspectives on low back pain. RESULTS: First, a coding scheme was developed using a comprehensive inductive and deductive approach. Second, 10 transcripts were coded independently by two researchers, and ICR was calculated. A resulting kappa value of .67 can be regarded as satisfactory to solid. Moreover, varying agreement rates helped to identify problems in the coding scheme. Low agreement rates, for instance, indicated that respective codes were defined too broadly and would need clarification. In a third step, the results of the analysis were used to improve the coding scheme, leading to consistent and high-quality results. DISCUSSION: The quantitative approach of ICR assessment is a viable instrument for quality assurance in qualitative content analysis. Kappa values and close inspection of agreement rates help to estimate and increase quality of codings. This approach facilitates good practice in coding and enhances credibility of analysis, especially when large samples are interviewed, different coders are involved, and quantitative results are presented.

Clinical application and reliability of a post abdominal surgery pain assessment scale (PASPAS) in horses

The aim of this study was to refine a multi-dimensional scale based on physiological and behavioural parameters, known as the post abdominal surgery pain assessment scale (PASPAS), to quantify pain after laparotomy in horses. After a short introduction, eight observers used the scale to assess eight horses at multiple time points after laparotomy. In addition, a single observer was used to test the correlation of each parameter with the total pain index in 34 patients, and the effect of general anaesthesia on PASPAS was investigated in a control group of eight horses. Inter-observer variability was low (coefficient of variation 0.3), which indicated good reliability of PASPAS. The correlation of individual parameters with the total pain index differed between parameters. PASPAS, which was not influenced by general anaesthesia, was a useful tool to evaluate pain in horses after abdominal surgery and may also be useful to investigate analgesic protocols or for teaching purposes.

Reliability of radiologic assessment of the fracture anatomy at the posterior tibial plafond in malleolar fractures

The aim of this study was to assess the ability to extract surgically relevant information from plain radiographs in trimalleolar fractures and to compare this with the information gathered from computed tomography (CT).

Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol

BACKGROUND The Cochrane risk of bias (RoB) tool has been widely embraced by the systematic review community, but several studies have reported that its reliability is low. We aim to investigate whether training of raters, including objective and standardized instructions on how to assess risk of bias, can improve the reliability of this tool. We describe the methods that will be used in this investigation and present an intensive standardized training package for risk of bias assessment that could be used by contributors to the Cochrane Collaboration and other reviewers. METHODS/DESIGN This is a pilot study. We will first perform a systematic literature review to identify randomized clinical trials (RCTs) that will be used for risk of bias assessment. Using the identified RCTs, we will then do a randomized experiment, where raters will be allocated to two different training schemes: minimal training and intensive standardized training. We will calculate the chance-corrected weighted Kappa with 95% confidence intervals to quantify within- and between-group Kappa agreement for each of the domains of the risk of bias tool. To calculate between-group Kappa agreement, we will use risk of bias assessments from pairs of raters after resolution of disagreements. Between-group Kappa agreement will quantify the agreement between the risk of bias assessment of raters in the training groups and the risk of bias assessment of experienced raters. To compare agreement of raters under different training conditions, we will calculate differences between Kappa values with 95% confidence intervals. DISCUSSION This study will investigate whether the reliability of the risk of bias tool can be improved by training raters using standardized instructions for risk of bias assessment. One group of inexperienced raters will receive intensive training on risk of bias assessment and the other will receive minimal training. By including a control group with minimal training, we will attempt to mimic what many review authors commonly have to do, that is-conduct risk of bias assessment in RCTs without much formal training or standardized instructions. If our results indicate that an intense standardized training does improve the reliability of the RoB tool, our study is likely to help improve the quality of risk of bias assessments, which is a central component of evidence synthesis.

Comprehensive assessment of gesture production: a new test of upper limb apraxia (TULIA)

BACKGROUND: Only few standardized apraxia scales are available and they do not cover all domains and semantic features of gesture production. Therefore, the objective of the present study was to evaluate the reliability and validity of a newly developed test of upper limb apraxia (TULIA), which is comprehensive and still short to administer. METHODS: The TULIA consists of 48 items including imitation and pantomime domain of non-symbolic (meaningless), intransitive (communicative) and transitive (tool related) gestures corresponding to 6 subtests. A 6-point scoring method (0-5) was used (score range 0-240). Performance was assessed by blinded raters based on videos in 133 stroke patients, 84 with left hemisphere damage (LHD) and 49 with right hemisphere damage (RHD), as well as 50 healthy subjects (HS). RESULTS: The clinimetric findings demonstrated mostly good to excellent internal consistency, inter- and intra-rater (test-retest) reliability, both at the level of the six subtests and at individual item level. Criterion validity was evaluated by confirming hypotheses based on the literature. Construct validity was demonstrated by a high correlation (r = 0.82) with the De Renzi-test. CONCLUSION: These results show that the TULIA is both a reliable and valid test to systematically assess gesture production. The test can be easily applied and is therefore useful for both research purposes and clinical practice.

Test-retest reliability of the nociceptive withdrawal reflex and electrical pain thresholds after single and repeated stimulation in patients with chronic low back pain

Recent studies have shown that the nociceptive withdrawal reflex threshold (NWR-T) and the electrical pain threshold (EP-T) are reliable measures in pain-free populations. However, it is necessary to investigate the reliability of these measures in patients with chronic pain in order to translate these techniques from laboratory to clinic. The aims of this study were to determine the test-retest reliability of the NWR-T and EP-T after single and repeated (temporal summation) electrical stimulation in a group of patients with chronic low back pain, and to investigate the association between the NWR-T and the EP-T. To this end, 25 patients with chronic pain participated in three identical sessions, separated by 1 week in average, in which the NWR-T and the EP-T to single and repeated stimulation were measured. Test-retest reliability was assessed using intra-class correlation coefficient (ICC), coefficient of variation (CV), and Bland-Altman analysis. The association between the thresholds was assessed using the coefficient of determination (r (2)). The results showed good-to-excellent reliability for both NWR-T and EP-T in all cases, with average ICC values ranging 0.76-0.90 and average CV values ranging 12.0-17.7%. The association between thresholds was better after repeated stimulation than after single stimulation, with average r (2) values of 0.83 and 0.56, respectively. In conclusion, the NWR-T and the EP-T are reliable assessment tools for assessing the sensitivity of spinal nociceptive pathways in patients with chronic pain.

The Bern psychopathology scale for the assessment of system-specific psychotic symptoms

The translation from psychiatric core symptoms to brain functions and vice versa is a largely unresolved issue. In particular, the search for disorders of single brain regions explaining classical symptoms has not yielded the expected results. Based on the assumption that the psychopathology of psychosis is related to a functional imbalance of higher-order brain systems, the authors focused on three specific candidate brain circuitries, namely the language, and limbic and motor systems. These domains are of particular interest for understanding the disastrous communication breakdown during psychotic disorders. Core symptoms of psychosis were mapped on these domains by shaping their definitions in order to match the related brain functions. The resulting psychopathological assessment scale was tested for interrater reliability and internal consistency in a group of 168 psychotic patients. The items of the scale were reliable and a principal component analysis (PCA) was best explained by a solution resembling the three candidate systems. Based on the results, the scale was optimized as an instrument to identify patient subgroups characterized by a prevailing dysfunction of one or more of these systems. In conclusion, the scale is apt to distinguish symptom domains related to the activity of defined brain systems. PCA showed a certain degree of independence of the system-specific symptom clusters within the patient group, indicating relative subgroups of psychosis. The scale is understood as a research instrument to investigate psychoses based on a system-oriented approach. Possible immediate advantages in the clinical application of the understanding of psychoses related to system-specific symptom domains are also discussed.

Evaluation of perpendicular reflection intensity for assessment of caries lesion activity/inactivity

The aim of this study was to evaluate, using visual assessment, an experimental optical sensor measuring perpendicular reflection intensity (PRI) as an indicator of enamel caries lesion activity/inactivity. Forty teeth with either an active or an inactive enamel lesion were selected from a pool of extracted teeth. Each tooth was cut into halves, with a clinically sound half and a half with a non-cavitated enamel lesion. After gentle plaque removal, the teeth were kept moistened. The lesions were then photographed and a defined measuring site per lesion was chosen and indicated with an arrow on a printout. Independently, the chosen site was visually assessed for lesion activity, and its glossiness was measured with PRI assessment. Surface roughness (SR) was assessed with optical profilometry using a confocal microscope. Visual assessment and PRI were repeated after several weeks and a reliability analysis was performed. For enamel lesions visually scored as active versus inactive, significantly different values were obtained with both PRI and SR. PRI values of the clinically sound control surfaces were significantly different only from active lesions. Generally, inactive lesions had the same glossiness and the same roughness as the sound control surfaces. The reliabilities for visual assessment (? = 0.89) and for PRI (ICC = 0.86) were high. It is concluded that, within the limits of this study, PRI can be regarded as a promising tool for quantitative enamel lesion activity assessment. There is scope and potential for the PRI device to be considerably improved for in vivo use.

Assessment of total uncertainty in cocaine and benzoylecgonine wastewater load measurements

To check the effectiveness of campaigns preventing drug abuse or indicating local effects of efforts against drug trafficking, it is beneficial to know consumed amounts of substances in a high spatial and temporal resolution. The analysis of drugs of abuse in wastewater (WW) has the potential to provide this information. In this study, the reliability of WW drug consumption estimates is assessed and a novel method presented to calculate the total uncertainty in observed WW cocaine (COC) and benzoylecgonine (BE) loads. Specifically, uncertainties resulting from discharge measurements, chemical analysis and the applied sampling scheme were addressed and three approaches presented. These consist of (i) a generic model-based procedure to investigate the influence of the sampling scheme on the uncertainty of observed or expected drug loads, (ii) a comparative analysis of two analytical methods (high performance liquid chromatography-tandem mass spectrometry and gas chromatography-mass spectrometry), including an extended cross-validation by influent profiling over several days, and (iii) monitoring COC and BE concentrations in WW of the largest Swiss sewage treatment plants. In addition, the COC and BE loads observed in the sewage treatment plant of the city of Berne were used to back-calculate the COC consumption. The estimated mean daily consumed amount was 107 ± 21 g of pure COC, corresponding to 321 g of street-grade COC.

Assessment of three frequently used blood glucose monitoring devices in clinical routine

Self-monitoring of blood glucose plays an important role in the management of diabetes and has been shown to improve metabolic control. The use of blood glucose meters in clinical practice requires sufficient reliability to allow adequate treatment. Direct comparison of different blood glucose meters in clinical practice, independent of the manufactures is scarce. We, therefore, aimed to evaluate three frequently used blood glucose meters in daily clinical practice.

Back related outcome assessment instruments

A literature review of the most widely used condition specific, self administered assessment questionnaires for low back pain had been undertaken. General and historic aspects, reliability, responsiveness and minimum clinically important difference, external validity, floor and ceiling effects, and available languages were analysed. These criteria, however, are only part of the consideration. Of similar importance are the content, wording of questions and answers in each of the six questionnaires and an analysis of the different score results. The issue of score bias is discussed and suggestions are given in order to increase the construct validity in the practical use of the individual questionnaires.

Assessing the intra- and inter-reader reliability of dynamic ultrasound images in power Doppler ultrasonography

OBJECTIVE: To assess the intra-reader and inter-reader reliabilities of interpreting ultrasonography by several experts using video clips. METHOD: 99 video clips of healthy and rheumatic joints were recorded and delivered to 17 physician sonographers in two rounds. The intra-reader and inter-reader reliabilities of interpreting the ultrasound results were calculated using a dichotomous system (normal/abnormal) and a graded semiquantitative scoring system. RESULTS: The video reading method worked well. 70% of the readers could classify at least 70% of the cases correctly as normal or abnormal. The distribution of readers answering correctly was wide. The most difficult joints to assess were the elbow, wrist, metacarpophalangeal (MCP) and knee joints. The intra-reader and inter-reader agreements on interpreting dynamic ultrasound images as normal or abnormal, as well as detecting and scoring a Doppler signal were moderate to good (kappa = 0.52-0.82). CONCLUSIONS: Dynamic image assessment (video clips) can be used as an alternative method in ultrasonography reliability studies. The intra-reader and inter-reader reliabilities of ultrasonography in dynamic image reading are acceptable, but more definitions and training are needed to improve sonographic reproducibility.

Whole-body MRI of multiple myeloma: comparison of different MRI sequences in assessment of different growth patterns

PURPOSE: To determine sensitivity, specificity and inter-observer variability of different whole-body MRI (WB-MRI) sequences in patients with multiple myeloma (MM). METHODS AND MATERIALS: WB-MRI using a 1.5T MRI scanner was performed in 23 consecutive patients (13 males, 10 females; mean age 63+/-12 years) with histologically proven MM. All patients were clinically classified according to infiltration (low-grade, n=7; intermediate-grade, n=7; high-grade, n=9) and to the staging system of Durie and Salmon PLUS (stage I, n=12; stage II, n=4; stage III, n=7). The control group consisted of 36 individuals without malignancy (25 males, 11 females; mean age 57+/-13 years). Two observers independently evaluated the following WB-MRI sequences: T1w-TSE (T1), T2w-TIRM (T2), and the combination of both sequences, including a contrast-enhanced T1w-TSE with fat-saturation (T1+/-CE/T2). They had to determine growth patterns (focal and/or diffuse) and the MRI sequence that provided the highest confidence level in depicting the MM lesions. Results were calculated on a per-patient basis. RESULTS: Visual detection of MM was as follows: T1, 65% (sensitivity)/85% (specificity); T2, 76%/81%; T1+/-CE/T2, 67%/88%. Inter-observer variability was as follows: T1, 0.3; T2, 0.55; T1+/-CE/T2, 0.55. Sensitivity improved depending on infiltration grade (T1: 1=60%; 2=36%; 3=83%; T2: 1=70%; 2=71%; 3=89%; T1+/-CE/T2: 1=50%; 2=50%; 3=89%) and clinical stage (T1: 1=58%; 2=63%; 3=79%; T2: 1=58%; 2=88%; 3=100%; T1+/-CE/T2: 1=50%; 2=63%; 3=100%). T2w-TIRM sequences achieved the best reliability in depicting the MM lesions (65% in the mean of both readers). CONCLUSIONS: T2w-TIRM sequences achieved the highest level of sensitivity and best reliability, and thus might be valuable for initial assessment of MM. For an exact staging and grading the examination protocol should encompass unenhanced and enhanced T1w-MRI sequences, in addition to T2w-TIRM.

A pain assessment scale for population-based studies: Development and validation of the Pain Module of the Standard Evaluation Questionnaire

The objectives of this study were to develop and validate a tool for assessing pain in population-based observational studies and to develop three subscales for back/neck, upper extremity and lower extremity pain. Based on a literature review, items were extracted from validated questionnaires and reviewed by an expert panel. The initial questionnaire consisted of a pain manikin and 34 items relating to (i) intensity of pain in different body regions (7 items), (ii) pain during activities of daily living (18 items) and (iii) various pain modalities (9 items). Psychometric validation of the initial questionnaire was performed in a random sample of the German-speaking Swiss population. Analyses included tests for reliability, correlation analysis, principal components factor analysis, tests for internal consistency and validity. Overall, 16,634 of 23,763 eligible individuals participated (70%). Test-retest reliability coefficients ranged from 0.32 to 0.97, but only three coefficients were below 0.60. Subscales were constructed combining four items for each of the subscales. Item-total coefficients ranged from 0.76 to 0.86 and Cronbach's alpha were 0.75 or higher for all subscales. Correlation coefficients between subscales and three validated instruments (WOMAC, SPADI and Oswestry) ranged from 0.62 to 0.79. The final Pain Standard Evaluation Questionnaire (SEQ Pain) included 28 items and the pain manikin and accounted for the multidimensionality of pain by assessing pain location and intensity, pain during activity, triggers and time of onset of pain and frequency of pain medication. It was found to be reliable and valid for the assessment of pain in population-based observational studies.