Intra-Rater and Inter-Rater Reliability of a Medical Record Abstraction Study on Transition of Care after Childhood Cancer.

BACKGROUND The abstraction of data from medical records is a widespread practice in epidemiological research. However, studies using this means of data collection rarely report reliability. Within the Transition after Childhood Cancer Study (TaCC) which is based on a medical record abstraction, we conducted a second independent abstraction of data with the aim to assess a) intra-rater reliability of one rater at two time points; b) the possible learning effects between these two time points compared to a gold-standard; and c) inter-rater reliability. METHOD Within the TaCC study we conducted a systematic medical record abstraction in the 9 Swiss clinics with pediatric oncology wards. In a second phase we selected a subsample of medical records in 3 clinics to conduct a second independent abstraction. We then assessed intra-rater reliability at two time points, the learning effect over time (comparing each rater at two time-points with a gold-standard) and the inter-rater reliability of a selected number of variables. We calculated percentage agreement and Cohen's kappa. FINDINGS For the assessment of the intra-rater reliability we included 154 records (80 for rater 1; 74 for rater 2). For the inter-rater reliability we could include 70 records. Intra-rater reliability was substantial to excellent (Cohen's kappa 0-6-0.8) with an observed percentage agreement of 75%-95%. In all variables learning effects were observed. Inter-rater reliability was substantial to excellent (Cohen's kappa 0.70-0.83) with high agreement ranging from 86% to 100%. CONCLUSIONS Our study showed that data abstracted from medical records are reliable. Investigating intra-rater and inter-rater reliability can give confidence to draw conclusions from the abstracted data and increase data quality by minimizing systematic errors.

Intra-examiner reproducibility of the agar plate method for estmation of phospholipase activity in candida albicans

Although measurement errors can impair statistical analysis, reliability analysis has been neglected in applied microbiology. This study assessed the intra-rater reproducibility of the Agar-based method for estimation of phospholipase activity (Pz). Pz readings were performed twice by two examiners (E1, E2), either directly on plates or in photos, and both black and white backgrounds were used. Pz values were taken from one or triplicate colonies from each sample (n=30). Intra-examiner reproducibility was estimated using Intraclass Correlation Coefficient (ICC). For both examiners, reading triplicate (ICCE1=0.91, ICCE2=0.86) was better than one colony (ICCE1=0.86, ICCE2=0.80). E1 had an excellent concordance when measurements were performed on photos using a white background (ICC=0.95) and a good concordance in the other conditions

The S.T.A.B. Trial - Standardised Testing of Artificial Blood. A comparative study of various products that may be used as artificial blood for high fidelity simulation training in the critical care setting

Aim: In the current climate of medical education, there is an ever-increasing demand for and emphasis on simulation as both a teaching and training tool. The objective of our study was to compare the realism and practicality of a number of artificial blood products that could be used for high-fidelity simulation. Method: A literature and internet search was performed and 15 artificial blood products were identified from a variety of sources. One product was excluded due to its potential toxicity risks. Five observers, blinded to the products, performed two assessments on each product using an evaluation tool with 14 predefined criteria including color, consistency, clotting, and staining potential to manikin skin and clothing. Each criterion was rated using a five-point Likert scale. The products were left for 24 hours, both refrigerated and at room temperature, and then reassessed. Statistical analysis was performed to identify the most suitable products, and both inter- and intra-rater variability were examined. Results: Three products scored consistently well with all five assessors, with one product in particular scoring well in almost every criterion. This highest-rated product had a mean rating of 3.6 of 5.0 (95% posterior Interval 3.4-3.7). Inter-rater variability was minor with average ratings varying from 3.0 to 3.4 between the highest and lowest scorer. Intrarater variability was negligible with good agreement between first and second rating as per weighted kappa scores (K = 0.67). Conclusion: The most realistic and practical form of artificial blood identified was a commercial product called KD151 Flowing Blood Syrup. It was found to be not only realistic in appearance but practical in terms of storage and stain removal.

The reliability, accuracy and minimal detectable difference of a multi-segment kinematic model of the foot-shoe complex

Kinematic models are commonly used to quantify foot and ankle kinematics, yet no marker sets or models have been proven reliable or accurate when wearing shoes. Further, the minimal detectable difference of a developed model is often not reported. We present a kinematic model that is reliable, accurate and sensitive to describe the kinematics of the foot–shoe complex and lower leg during walking gait. In order to achieve this, a new marker set was established, consisting of 25 markers applied on the shoe and skin surface, which informed a four segment kinematic model of the foot–shoe complex and lower leg. Three independent experiments were conducted to determine the reliability, accuracy and minimal detectable difference of the marker set and model. Inter-rater reliability of marker placement on the shoe was proven to be good to excellent (ICC = 0.75–0.98) indicating that markers could be applied reliably between raters. Intra-rater reliability was better for the experienced rater (ICC = 0.68–0.99) than the inexperienced rater (ICC = 0.38–0.97). The accuracy of marker placement along each axis was <6.7 mm for all markers studied. Minimal detectable difference (MDD90) thresholds were defined for each joint; tibiocalcaneal joint – MDD90 = 2.17–9.36°, tarsometatarsal joint – MDD90 = 1.03–9.29° and the metatarsophalangeal joint – MDD90 = 1.75–9.12°. These thresholds proposed are specific for the description of shod motion, and can be used in future research designed at comparing between different footwear.

A radiological method to determine the accuracy of motion capture marker placement on palpable anatomical landmarks through a shoe

The accuracy of marker placement on palpable surface anatomical landmarks is an important consideration in biomechanics. Although marker placement reliability has been studied in some depth, it remains unclear whether or not the markers are accurately positioned over the intended landmark in order to define the static position and orientation of the segment. A novel method using commonly available X-ray imaging was developed to identify the accuracy of markers placed on the shoe surface by palpating landmarks through the shoe. An anterior–posterior and lateral–medial X-ray was taken on 24 participants with a newly developed marker set applied to both the skin and shoe. The vector magnitude of both skin- and shoe-mounted markers from the anatomical landmark was calculated, as well as the mean marker offset between skin- and shoe-mounted markers. The accuracy of placing markers on the shoe relative to the skin-mounted markers, accounting for shoe thickness, was less than 5mm for all markers studied. Further, when using the developed guidelines provided in this study, the method was deemed reliable (Intra-rater ICCs¼0.50–0.92). In conclusion, the method proposed here can reliably assess marker placement accuracy on the shoe surface relative to chosen anatomical landmarks beneath the skin.

Intratherapist reliability in the rating of scapula posture in multiple planes of reference

Background: Evaluation of scapular posture is a fundamental component in the clinical evaluation of the upper quadrant. This study examined the intrarater reliability of scapular posture ratings. Methods: A test-retest reliability investigation was undertaken with one week between assessment sessions. At each session physical therapists conducted visual assessments of scapula posture (relative to the thorax) in five different scapula postural planes (plane of scapula, sagittal plane, transverse plane, horizontal plane, and vertical plane). These five plane ratings were performed for four different scapular posture perturbating conditions (rest, isometric shoulder; flexion, abduction, and external rotation). Results. A total of 100 complete scapular posture ratings (50 left, 50 right) were undertaken at each assessment. The observed agreement between the test and retest postural plane ratings ranged from 59% to 87%; 16 of the 20 plane-condition combinations exceeded 75% observed agreement. Kappa (and prevalence adjusted bias adjusted kappa) values were inconsistent across the postural planes and perturbating conditions. Conclusions: This investigation generally revealed fair to moderate intrarater reliability in the rating of scapular posture by visual inspection. However, enough disagreement between assessments was present to warrant caution when interpreting perceived changes in scapula position between longitudinal assessments using visual inspection alone.

Inter-vertebral rotational deformity after endoscopic anterior scoliosis correction may contribute to rib hump recurrence after two years

Introduction. Endoscopic anterior scoliosis correction has been employed recently as a less invasive and level-sparing approach compared with open surgical techniques. We have previously demonstrated that during the two-year post-operative period, there was a mean loss of rib hump correction by 1.4 degrees. The purpose of this study was to determine whether intra- or inter-vertebral rotational deformity during the post-operative period could account for the loss of rib hump correction. Materials and Methods. Ten consecutive patients diagnosed with adolescent idiopathic scoliosis were treated with an endoscopic anterior scoliosis correction. Low-dose computed tomography scans of the instrumented segment were obtained post-operatively at 6 and 24 months following institutional ethical approval and patient consent. Three-dimensional multi-planar reconstruction software (Osirix Imaging Software, Pixmeo, Switzerland) was used to create axial slices of each vertebral level, corrected in both coronal and sagittal planes. Vertebral rotation was measured using Ho’s method for every available superior and inferior endplate at 6 and 24 months. Positive changes in rotation indicate a reduction and improvement in vertebral rotation. Intra-observer variability analysis was performed on a subgroup of images. Results. Mean change in rotation for vertebral endplates between 6 and 24 months post-operatively was -0.26˚ (range -3.5 to 4.9˚) within the fused segment and +1.26˚ (range -7.2 to 15.1˚) for the un-instrumented vertebrae above and below the fusion. Mean change in clinically measured rib hump for the 10 patients was -1.6˚ (range -3 to 0˚). The small change in rotation within the fused segment accounts for only 16.5% of the change in rib hump measured clinically whereas the change in rotation between the un-instrumented vertebrae above and below the construct accounts for 78.8%. There was no clear association between rib hump recurrence and intra- or inter-vertebral rotation in individual patients. Intra-rater variability was ± 3˚. Conclusions. Intra- and inter-vertebral rotation continues post-operatively both within the instrumented and un-instrumented segments of the immature spine. Rotation between the un-instrumented vertebrae above and below the fusion was +1.26˚, suggesting that the un-instrumented vertebrae improved and de-rotated slightly after surgery. This may play a role in rib hump recurrence, however this remains clinically insignificant.

Is the clinical Queensland High Risk Foot Form valid or reliable for research?

Background High-risk foot complications such as neuropathy, ischaemia, deformity, infections, ulcers and amputations consume considerable health care resources and typically result from chronic diseases. This study aimed to develop and test the validity and reliability of a Queensland High Risk Foot Form (QHRFF) tool. Methods Phase one involved developing a QHRFF using an existing diabetes high-risk foot tool, literature search, expert panel and several state-wide stakeholder groups. Phase two tested the criterion-related validity along with inter- and intra-rater reliability of the final QHRFF. Three cohorts of patients (n = 94) and four clinicians, representing different levels of expertise, were recruited. Validity was determined by calculating sensitivity, specificity and positive predictive values (PPV). Kappa and intra-class correlation (ICC) statistics were used to establish reliability. Results A QHRFF tool containing 46-items across seven domains was developed and endorsed. The majority of QHRFF items achieved moderate-to-perfect validity (PPV = 0.71 – 1) and reliability (Kappa/ICC = 0.41 – 1). Items with weak validity and/or reliability included those identifying health professionals previously attending the patient, other (non-listed) co-morbidity, previous foot ulcer, foot deformity, optimum offloading and optimum footwear. Conclusions The QHRFF had moderate-to-perfect validity and reliability across the majority of items, particularly identifying individual co-morbidities and foot complications. Items with weak validity or reliability need to be re-defined or removed. Overall, the QHRFF appears to be a valid and reliable tool to assess, collect and measure clinical data pertaining to high-risk foot complications for clinical or research purposes.

The evaluation of a clinical scar scale for porcine burn scars

This study describes the evaluation of a clinical scar scale for our porcine burn scars, which includes scar cosmetic outcome, colour, height and hair, supplemented with reference porcine scar photographs representing each scar outcome and scar colour scores. A total of 72 porcine burn scars at week 6 after burn were rated in vivo and/or on photographs. Good agreements were achieved for both intra-rater reliability (correlation is 0.86-0.98) and inter-rater reliability (ICC=80-85%). The results showed statistically significant correlations for each pair in this clinical scar scale (p<0.01), with the best correlation found between scar cosmetic outcome and scar colour. A multivariate principle components analysis revealed that this clinical scar assessment was highly correlated with scar histology, wound size, and re-epithelialisation data (p<0.001). More severe scars are clinically characterised by darker purple colouration, more elevation, no presence of hair, histologically by thicker scar tissue, thinner remaining normal dermis, are more likely to have worse contraction, and slower re-epithelialisation. This study demonstrates that our clinical scar scale is a reliable, independent and valuable tool for assessing porcine burn outcome and truthfully reflects scar appearance and function. To our knowledge, this is the first study demonstrating a high correlation between clinical scar assessment and scar histology, wound contraction and re-epithelialisation data on porcine burn scars. We believe that the successful use of porcine scar scales is invaluable for assessing potential human burn treatments.

Reliability of a wellness inventory for use among adolescent females aged 12-14 years

Background The wellness construct has application in a number of fields including education, healthcare and counseling, particularly with regard to female adolescents. The effective measurement of wellness in adolescents can assist researchers and practitioners in determining lifestyle behaviors in which they are lacking. Behavior change interventions can then be designed which directly aid in the promotion of these areas. Methods The 5-Factor Wellness Inventory (designed to measure the Indivisible Self model of wellness) is a popular instrument for measuring the broad aspects of wellness amongst adolescents. The instrument comprises 97 items contributing to 17 subscales, five dimension scores, four context scores, total wellness score, and a life satisfaction index. This investigation evaluated the test-retest (intra-rater) reliability of the 5 F-Wel instrument in repeated assessments (seven days apart) among adolescent females aged 12-14 years. Percentages of exact agreement for individual items, and the number of respondents who scored within +/-5, +/-7.5 and +/-10 points for total wellness and the five summary dimension scores were calculated. Results Overall, 46 (95.8%) participants responded with complete data and were included in the analysis. Item agreement ranged from 47.8% to 100% across the 97 items (median 69.9%, interquartile range 60.9%-73.9%). The percentage of respondents who scored within +/-5, +/-7.5 and +/-10 points for total wellness at the re-assessment was 87.0%, 97.8% and 97.8% respectively. The percentage of respondents who scored within +/-5, +/-7.5 and +/-10 for the domain scores at the reassessment ranged between 54.3-76.1%, 78.3-95.7% and 89.1-95.7% respectively across the five dimensions. Conclusions These findings suggest there was considerable variation in agreement between the two assessments on some individual items. However, the total wellness score and the five dimension summary scores remained comparatively stable between assessments.

The Queensland high risk foot form (QHRFF): Is it a reliable and valid clinical research tool for foot disease?

Background Foot disease complications, such as foot ulcers and infection, contribute to considerable morbidity and mortality. These complications are typically precipitated by “high-risk factors”, such as peripheral neuropathy and peripheral arterial disease. High-risk factors are more prevalent in specific “at risk” populations such as diabetes, kidney disease and cardiovascular disease. To the best of the authors’ knowledge a tool capturing multiple high-risk factors and foot disease complications in multiple at risk populations has yet to be tested. This study aimed to develop and test the validity and reliability of a Queensland High Risk Foot Form (QHRFF) tool. Methods The study was conducted in two phases. Phase one developed a QHRFF using an existing diabetes foot disease tool, literature searches, stakeholder groups and expert panel. Phase two tested the QHRFF for validity and reliability. Four clinicians, representing different levels of expertise, were recruited to test validity and reliability. Three cohorts of patients were recruited; one tested criterion measure reliability (n = 32), another tested criterion validity and inter-rater reliability (n = 43), and another tested intra-rater reliability (n = 19). Validity was determined using sensitivity, specificity and positive predictive values (PPV). Reliability was determined using Kappa, weighted Kappa and intra-class correlation (ICC) statistics. Results A QHRFF tool containing 46 items across seven domains was developed. Criterion measure reliability of at least moderate categories of agreement (Kappa > 0.4; ICC > 0.75) was seen in 91% (29 of 32) tested items. Criterion validity of at least moderate categories (PPV > 0.7) was seen in 83% (60 of 72) tested items. Inter- and intra-rater reliability of at least moderate categories (Kappa > 0.4; ICC > 0.75) was seen in 88% (84 of 96) and 87% (20 of 23) tested items respectively. Conclusions The QHRFF had acceptable validity and reliability across the majority of items; particularly items identifying relevant co-morbidities, high-risk factors and foot disease complications. Recommendations have been made to improve or remove identified weaker items for future QHRFF versions. Overall, the QHRFF possesses suitable practicality, validity and reliability to assess and capture relevant foot disease items across multiple at risk populations.

Development, reliability and validity of the queensland evaluation of wheelchair skills (QEWS)

- Objectives To develop and test a valid and reliable assessment of wheelchair skills for individuals with spinal cord injuries (SCI); the Queensland Evaluation of Wheelchair Skills (QEWS). - Setting Hospital, Australia. - Methods Phase 1: Four Delphi panel rounds with clinical experts were used to develop the QEWS. Phase 2: Intra-rater and inter-rater reliability of the QEWS items were examined in 100 people with SCI. Phase 3a: Concurrent validity was investigated by examining the association between QEWS total scores and physiotherapists’ global ratings of wheelchair skill performance. Phase 3b: Construct validity was tested in 20 people with recent SCI by examining change in QEWS total scores between when they first mobilised in a wheelchair and scores obtained 10 weeks later. - Results Phase 1: The QEWS was developed. Phase 2: The intra-class correlation coefficients reflecting the intra-rater reliability and the inter-rater reliability for the QEWS total score were 1.00 and 0.98, with scores being within one point of each other 96 and 91% of the time, respectively. Phase 3a: The QEWS total scores were comparable with the global rating of wheelchair skill performance (r2=0.93). Phase 3b: The QEWS scores changed by a median (interquartile range (IQR)) of 4 (1 to 6) points over the 10-week period following first wheelchair mobilisation. - Conclusion The QEWS is a valid and reliable tool for measuring wheelchair skills in individuals with SCI. The QEWS is efficient and practical to administer and does not require specialised equipment.

A influência da espessura óssea na medição da altura da borda alveolar em imagens de tomografia computadorizada de feixe cônico e espiral

Imagens de tomografia computadorizada (TC) permitem a visualização, sem distorções ou sobreposições, do complexo maxilo-facial, principalmente do osso alveolar. Estudos demonstraram boa reprodutibilidade e precisão da mensuração da altura da borda alveolar, todavia a influência da espessura óssea ainda é pouco descrita. Através da comparação com a mensuração direta, o objetivo deste estudo foi avaliar a precisão, reprodutibilidade e a influência da espessura óssea, na mensuração da altura da borda alveolar em imagens volumétricas e imagens bidimensionais multiplanares em TC de feixe cônico (TCFC) e em TC espiral (TCE). Utilizando 10 mandíbulas secas de humanos, 57 dentes anteriores foram tomografados em equipamentos iCAT (Imaging Science International, Hatfield, PA, EUA) e Brilliance 64 canais (Philips Eletronics, Eindhoven, Holanda), ambos utilizando voxels de 0,25 mm. Através de imagens volumétricas (3D) e imagens bidimensionais (2D) de cortes multiplanares, foi comparada a mensuração da altura da borda alveolar dessas imagens com a mensuração direta nas mandíbulas, feita por vestibular e lingual, por três avaliadores, com o auxílio de um paquímetro, totalizando 114 bordas alveolares medidas. Alta reprodutibilidade intra-avaliador (0,999 a 0,902) e interavaliador (0,998 e 0,868) foi observada através do índice de correlação intraclasse (ICC). Observou-se alta correlação entre a mensuração direta e indireta da altura da borda alveolar em imagens 2D, sendo r=0,923** e 0,916**, e em imagens 3D, com r=0,929** e 0,954*, em TCFC e TCE, respectivamente. Imagens 2D superestimam a altura da borda alveolar em 0,32 e 0,49 mm e imagens 3D em 0,34 e 0,30 mm, em TCFC e TCE respectivamente. Quando o osso alveolar apresenta espessura de no mínimo 0,6 mm a média da diferença entre medidas diretas e indiretas é de 0,16 e 0,28 mm em imagens 2D e de 0,12 e 0,03 mm em imagens 3D para TCFC e TCE respectivamente, sendo que 95% do limite de concordância varia de -0,46 a 0,79 mm e -0,32 a 0,88 mm em imagens 2D, e de -0,64 a 0,67 mm e -0,57 a 0,62 mm em imagens 3D, para TCFC e TCE respectivamente. Quando o osso alveolar é mais fino do que 0,6 mm a TC é imprecisa, pois 95% do limite de concordância variou de -1,74 a 5,42 mm e -1,64 a 5,42 mm em imagens 2D, e de -3,70 a 4,28 mm e -3,49 a 4,25 mm em imagens 3D, para TCFC e TCE respectivamente. Conclui-se que a mensuração da altura da borda alveolar através de imagens tomográficas apresenta alta reprodutibilidade, sendo que quando a borda alveolar apresenta pelo menos 0,6 mm, a precisão da mensuração é alta, todavia quando esta espessura é menor do que 0,6 mm a técnica é imprecisa.

Entorhinal cortex volume in older adults: reliability and validity considerations for three published measurement protocols.

Measuring the entorhinal cortex (ERC) is challenging due to lateral border discrimination from the perirhinal cortex. From a sample of 39 nondemented older adults who completed volumetric image scans and verbal memory indices, we examined reliability and validity concerns for three ERC protocols with different lateral boundary guidelines (i.e., Goncharova, Dickerson, Stoub, & deToledo-Morrell, 2001; Honeycutt et al., 1998; Insausti et al., 1998). We used three novice raters to assess inter-rater reliability on a subset of scans (216 total ERCs), with the entire dataset measured by one rater with strong intra-rater reliability on each technique (234 total ERCs). We found moderate to strong inter-rater reliability for two techniques with consistent ERC lateral boundary endpoints (Goncharova, Honeycutt), with negligible to moderate reliability for the technique requiring consideration of collateral sulcal depth (Insausti). Left ERC and story memory associations were moderate and positive for two techniques designed to exclude the perirhinal cortex (Insausti, Goncharova), with the Insausti technique continuing to explain 10% of memory score variance after additionally controlling for depression symptom severity. Right ERC-story memory associations were nonexistent after excluding an outlier. Researchers are encouraged to consider challenges of rater training for ERC techniques and how lateral boundary endpoints may impact structure-function associations.

Emotional intelligence, personality and decoding non-verbal expressions of emotion

Previous research has highlighted theoretical and empirical links between measures of both personality and trait emotional intelligence (EI), and the ability to decode facial expressions of emotion. Research has also found that the posed, static characteristics of the photographic stimuli used to explore these links affects the decoding process and differentiates them from the natural expressions they represent. This undermines the ecological validity of established trait-emotion decoding relationships. This study addresses these methodological shortcomings by testing relationships between the reliability of participant ratings of dynamic, spontaneously elicited expressions of emotion with personality and trait EI. Fifty participants completed personality and self-report EI questionnaires, and used a computer-logging program to continuously rate change in emotional intensity expressed in video clips. Each clip was rated twice to obtain an intra-rater reliability score. The results provide limited support for links between both trait EI and personality variables and how reliably we decode natural expressions of emotion. Limitations and future directions are discussed.