Development of a quality use of medicines coding system to rate clinical pharmacists' medication review recommendations

Objective: To develop a 'quality use of medicines' coding system for the assessment of pharmacists' medication reviews and to apply it to an appropriate cohort. Method: A 'quality use of medicines' coding system was developed based on findings in the literature. These codes were then applied to 216 (111 intervention, 105 control) veterans' medication profiles by an independent clinical pharmacist who was supported by a clinical pharmacologist with the aim to assess the appropriateness of pharmacy interventions. The profiles were provided for veterans participating in a randomised, controlled trial in private hospitals evaluating the effect of medication review and discharge counselling. The reliability of the coding was tested by two independent clinical pharmacists in a random sample of 23 veterans from the study population. Main outcome measure: Interrater reliability was assessed by applying Cohen's kappa score on aggregated codes. Results: The coding system based on the literature consisted of 19 codes. The results from the three clinical pharmacists suggested that the original coding system had two major problems: (a) a lack of discrimination for certain recommendations e. g. adverse drug reactions, toxicity and mortality may be seen as variations in degree of a single effect and (b) certain codes e. g. essential therapy were in low prevalence. The interrater reliability for an aggregation of all codes into positive, negative and clinically non-significant codes ranged from 0.49-0.58 (good to fair). The interrater reliability increased to 0.72-0.79 (excellent) when all negative codes were excluded. Analysis of the sample of 216 profiles showed that the most prevalent recommendations from the clinical pharmacists were a positive impact in reducing adverse responses (31.9%), an improvement in good clinical pharmacy practice (25.5%) and a positive impact in reducing drug toxicity (11.1%). Most medications were assigned the clinically non-significant code (96.6%). In fact, the interventions led to a statistically significant difference in pharmacist recommendations in the categories; adverse response, toxicity and good clinical pharmacy practice measured by the quality use of medicine coding system. Conclusion: It was possible to use the quality use of medicine coding system to rate the quality and potential health impact of pharmacists' medication reviews, and the system did pick up differences between intervention and control patients. The interrater reliability for the summarised coding system was fair, but a larger sample of medication regimens is needed to assess the non-summarised quality use of medicines coding system.

Reliability of scores between stroke patients and significant others on the Reintegration to Normal Living (RNL) Index

Purpose: This study measured reliability between stroke patients' and significant others' scores on items on the Reintegration to Normal Living (RNL) Index and whether there were any scoring biases. Method The 11-item RNL Index was administered to 57 pairs of patients and significants six months after stroke rehabilitation. The index was scored using a 10-point visual analogue scale. Patient and significant other demographic information and data on patients' clinical, functional and cognitive status were collected. Reliability was measured using the intra-class correlation coefficient (ICC) and percent agreement. Results: Overall poor reliability was found for the RNL Index total score (ICC=.36, 95% CI. 07 to .59) and the daily functioning subscale (ICC=.24, 95% CI -.003 to .46) and moderate reliability was found for the perception of self subscale (ICC=.55, 95 % CI .28 to .73). There was a moderate bias for patients to rate themselves as achieving better reintegration than was indicated by significant others, although no demographic or clinical factors were associated with this bias. Exact match agreement was best for the subjective items and worse for items reflecting mobility around the community and participation in a work activity. Conclusions: Caution is needed when interpreting patient information reported by significant others on the RNL Index. The use of a shorter scale to rate the RNL Index requires investigation.

On measuring quality of a probabilistic commodity forecast for a system that incorporates seasonal climate forecasts

Regional commodity forecasts are being used increasingly in agricultural industries to enhance their risk management and decision-making processes. These commodity forecasts are probabilistic in nature and are often integrated with a seasonal climate forecast system. The climate forecast system is based on a subset of analogue years drawn from the full climatological distribution. In this study we sought to measure forecast quality for such an integrated system. We investigated the quality of a commodity (i.e. wheat and sugar) forecast based on a subset of analogue years in relation to a standard reference forecast based on the full climatological set. We derived three key dimensions of forecast quality for such probabilistic forecasts: reliability, distribution shift, and change in dispersion. A measure of reliability was required to ensure no bias in the forecast distribution. This was assessed via the slope of the reliability plot, which was derived from examination of probability levels of forecasts and associated frequencies of realizations. The other two dimensions related to changes in features of the forecast distribution relative to the reference distribution. The relationship of 13 published accuracy/skill measures to these dimensions of forecast quality was assessed using principal component analysis in case studies of commodity forecasting using seasonal climate forecasting for the wheat and sugar industries in Australia. There were two orthogonal dimensions of forecast quality: one associated with distribution shift relative to the reference distribution and the other associated with relative distribution dispersion. Although the conventional quality measures aligned with these dimensions, none measured both adequately. We conclude that a multi-dimensional approach to assessment of forecast quality is required and that simple measures of reliability, distribution shift, and change in dispersion provide a means for such assessment. The analysis presented was also relevant to measuring quality of probabilistic seasonal climate forecasting systems. The importance of retaining a focus on the probabilistic nature of the forecast and avoiding simplifying, but erroneous, distortions was discussed in relation to applying this new forecast quality assessment paradigm to seasonal climate forecasts. Copyright (K) 2003 Royal Meteorological Society.

Evaluation of maternal and neonatal hospital care: quality index of completeness

OBJECTIVE Develop an index to evaluate the maternal and neonatal hospital care of the Brazilian Unified Health System.METHODS This descriptive cross-sectional study of national scope was based on the structure-process-outcome framework proposed by Donabedian and on comprehensive health care. Data from the Hospital Information System and the National Registry of Health Establishments were used. The maternal and neonatal network of Brazilian Unified Health System consisted of 3,400 hospitals that performed at least 12 deliveries in 2009 or whose number of deliveries represented 10.0% or more of the total admissions in 2009. Relevance and reliability were defined as criteria for the selection of variables. Simple and composite indicators and the index of completeness were constructed and evaluated, and the distribution of maternal and neonatal hospital care was assessed in different regions of the country.RESULTS A total of 40 variables were selected, from which 27 single indicators, five composite indicators, and the index of completeness of care were built. Composite indicators were constructed by grouping simple indicators and included the following variables: hospital size, level of complexity, delivery care practice, recommended hospital practice, and epidemiological practice. The index of completeness of care grouped the five variables and classified them in ascending order, thereby yielding five levels of completeness of maternal and neonatal hospital care: very low, low, intermediate, high, and very high. The hospital network was predominantly of small size and low complexity, with inadequate child delivery care and poor development of recommended and epidemiological practices. The index showed that more than 80.0% hospitals had a low index of completeness of care and that most qualified heath care services were concentrated in the more developed regions of the country.CONCLUSIONS The index of completeness proved to be of great value for monitoring the maternal and neonatal hospital care of Brazilian Unified Health System and indicated that the quality of health care was unsatisfactory. However, its application does not replace specific evaluations.

Validity and reliability of a health care service evaluation instrument for tuberculosis

OBJECTIVE To evaluate the validity and reliability of an instrument that evaluates the structure of primary health care units for the treatment of tuberculosis.METHODS This cross-sectional study used simple random sampling and evaluated 1,037 health care professionals from five Brazilian municipalities (Natal, state of Rio Grande do Norte; Cabedelo, state of Paraíba; Foz do Iguaçu, state of Parana; Sao José do Rio Preto, state of Sao Paulo, and Uberaba, state of Minas Gerais) in 2011. Structural indicators were identified and validated, considering different methods of organization of the health care system in the municipalities of different population sizes. Each structure represented the organization of health care services and contained the resources available for the execution of health care services: physical resources (equipment, consumables, and facilities); human resources (number and qualification); and resources for maintenance of the existing infrastructure and technology (deemed as the organization of health care services). The statistical analyses used in the validation process included reliability analysis, exploratory factor analysis, and confirmatory factor analysis.RESULTS The validation process indicated the retention of five factors, with 85.9% of the total variance explained, internal consistency between 0.6460 and 0.7802, and quality of fit of the confirmatory factor analysis of 0.995 using the goodness-of-fit index. The retained factors comprised five structural indicators: professionals involved in the care of tuberculosis patients, training, access to recording instruments, availability of supplies, and coordination of health care services with other levels of care. Availability of supplies had the best performance and the lowest coefficient of variation among the services evaluated. The indicators of assessment of human resources and coordination with other levels of care had satisfactory performance, but the latter showed the highest coefficient of variation. The performance of the indicators “training” and “access to recording instruments” was inferior to that of other indicators.CONCLUSIONS The instrument showed feasibility of application and potential to assess the structure of primary health care units for the treatment of tuberculosis.

Reliability analysis of a timber truss system subjected to decay

Assessing the safety of existing timber structures is of paramount importance for taking reliable decisions on repair actions and their extent. The results obtained through semi-probabilistic methods are unrealistic, as the partial safety factors present in codes are calibrated considering the uncertainty present in new structures. In order to overcome these limitations, and also to include the effects of decay in the safety analysis, probabilistic methods, based on Monte-Carlo simulation are applied here to assess the safety of existing timber structures. In particular, the impact of decay on structural safety is analyzed and discussed, using a simple structural model, similar to that used for current semi-probabilistic analysis.

Reliability and convergent validity of the Childhood Anxiety Sensitivity Index in children and adolescents

OBJECTIVE: The purpose of this study was to examine the reliability and the convergent validity of the Children Anxiety Sensitivity Index (CASI) with DSM-IV anxiety disorder symptoms, by comparison with the Screen for Child Anxiety Related Emotional Disorders (SCARED), in a community sample of Brazilian children and adolescents. METHODS: Children and adolescents from five schools were selected from a larger study that aimed to assess different aspects of childhood anxiety disorders. All participants completed the CASI and the SCARED. RESULTS: This study supported the reliability of the CASI total score. Girls reported higher total anxiety sensitivity scores than boys and there were no differences in total anxiety sensitivity scores between children and adolescents. This study showed moderate to high correlations between the CASI scores with SCARED scores, all correlations coefficients being positive and significant. CONCLUSIONS: Our findings demonstrate an appropriate reliability and evidence of convergent validity in the CASI in a sample of Brazilian children and adolescents.

Validity, reliability and responsiveness of the French language translation of the Western Ontario Shoulder Instability Index (WOSI).

BACKGROUND: The WOSI (Western Ontario Shoulder Instability Index) is a self-administered quality of life questionnaire designed to be used as a primary outcome measure in clinical trials on shoulder instability, as well as to measure the effect of an intervention on any particular patient. It is validated and is reliable and sensitive. As it is designed to measure subjective outcome, it is important that translation should be methodologically rigorous, as it is subject to both linguistic and cultural interpretation. OBJECTIVE: To produce a French language version of the WOSI that is culturally adapted to both European and North American French-speaking populations. MATERIALS AND METHODS: A validated protocol was used to create a French language WOSI questionnaire (WOSI-Fr) that would be culturally acceptable for both European and North American French-speaking populations. Reliability and responsiveness analyses were carried out, and the WOSI-Fr was compared to the F-QuickDASH-D/S (Disability of the Arm, Shoulder and Hand-French translation), and Walch-Duplay scores. RESULTS: A French language version of the WOSI (WOSI-Fr) was accepted by a multinational committee. The WOSI-Fr was then validated using a total of 144 native French-speaking subjects from Canada and Switzerland. Comparison of results on two WOSI-Fr questionnaires completed at a mean interval of 16 days showed that the WOSI-Fr had strong reliability, with a Pearson and interclass correlation of r=0.85 (P=0.01) and ICC=0.84 [95% CI=0.78-0.88]. Responsiveness, at a mean 378.9 days after surgical intervention, showed strong correlation with that of the F-QuickDASH-D/S, with r=0.67 (P<0.01). Moreover, a standardized response means analysis to calculate effect size for both the WOSI-Fr and the F-QuickDASH-D/S showed that the WOSI-Fr had a significantly greater ability to detect change (SRM 1.55 versus 0.87 for the WOSI-Fr and F-QuickDASH-D/S respectively, P<0.01). The WOSI-Fr showed fair correlation with the Walch-Duplay. DISCUSSION: A French-language translation of the WOSI questionnaire was created and validated for use in both Canadian and Swiss French-speaking populations. This questionnaire will facilitate outcome assessment in French-speaking settings, collaboration in multinational studies and comparison between studies performed in different countries. TYPE OF STUDY: Multicenter cohort study. LEVEL OF EVIDENCE: II.

French version of the addiction severity index (5th Edition): validity and reliability among Swiss opiate-dependent patients. French validation of the Addiction Severity Index.

OBJECTIVE: The aim of the study was to validate a French adaptation of the 5th version of the Addiction Severity Index (ASI) instrument in a Swiss sample of illicit drug users. PARTICIPANTS AND SETTING: The participants in the study were 54 French-speaking dependent patients, most of them with opiates as the drug of first choice. Procedure: Analyses of internal consistency (convergent and discriminant validity) and reliability, including measures of test-retest and inter-observer correlations, were conducted. RESULTS: Besides good applicability of the test, the results on composite scores (CSs) indicate comparable results to those obtained in a sample of American opiate-dependent patients. Across the seven dimensions of the ASI, Cronbach's alpha ranged from 0.42 to 0.76, test-retest correlations coefficients ranged from 0.48 to 0.98, while for CSs, inter-observer correlations ranged from 0.76 to 0.99. CONCLUSIONS: Despite several limitations, the French version of the ASI presents acceptable criteria of applicability, validity and reliability in a sample of drug-dependent patients.

The Body Action and Posture Coding System (BAP): Development and Reliability

Several methods are available for coding body movement in nonverbal behavior research, but there is no consensus on a reliable coding system that can be used for the study of emotion expression. Adopting an integrative approach, we developed a new method, the Body Action and Posture (BAP) coding system, for the time-aligned micro description of body movement on an anatomical level (different articulations of body parts), a form level (direction and orientation of movement), and a functional level (communicative and self-regulatory functions). We applied the system to a new corpus of acted emotion portrayals, examined its comprehensiveness and demonstrated intercoder reliability at three levels: a) occurrence, b) temporal precision and c) segmentation. We discuss issues for further validation and propose some research applications.

The inter-rater reliability of the Pulmonary Embolism Severity Index.

The Pulmonary Embolism Severity Index (PESI) is a validated clinical prognostic model for patients with acute pulmonary embolism (PE). Our goal was to assess the PESI's inter-rater reliability in patients diagnosed with PE. We prospectively identified consecutive patients diagnosed with PE in the emergency department of a Swiss teaching hospital. For all patients, resident and attending physician raters independently collected the 11 PESI variables. The raters then calculated the PESI total point score and classified patients into one of five PESI risk classes (I-V) and as low (risk classes I/II) versus higher-risk (risk classes III-V). We examined the inter-rater reliability for each of the 11 PESI variables, the PESI total point score, assignment to each of the five PESI risk classes, and classification of patients as low versus higher-risk using kappa (κ) and intra-class correlation coefficients (ICC). Among 48 consecutive patients with an objective diagnosis of PE, reliability coefficients between resident and attending physician raters were > 0.60 for 10 of the 11 variables comprising the PESI. The inter-rater reliability for the PESI total point score (ICC: 0.89, 95% CI: 0.81-0.94), PESI risk class assignment (κ: 0.81, 95% CI: 0.66-0.94), and the classification of patients as low versus higher-risk (κ: 0.92, 95% CI: 0.72-0.98) was near perfect. Our results demonstrate the high reproducibility of the PESI, supporting the use of the PESI for risk stratification of patients with PE.

Validation of the instability shoulder index score in a multicenter reliability study in 114 consecutive cases.

BACKGROUND: Anterior shoulder stabilization surgery with the arthroscopic Bankart procedure can have a high recurrence rate in certain patients. Identifying these patients to modify outcomes has become a focal point of research. PURPOSE: The Instability Shoulder Index Score (ISIS) was developed to predict the success of arthroscopic Bankart repair. Scores range from 0 to 10, with higher scores predicting a higher risk of recurrence after stabilization. The interobserver reliability of the score is not known. STUDY DESIGN: Cohort study (diagnosis); Level of evidence, 2. METHODS: This is a prospective multicenter (North America and Europe) study of patients suffering from shoulder instability and waiting for stabilization surgery. Five pairs of independent evaluators were asked to score patient instability severity with the ISIS. Patients also completed functional scores (Western Ontario Shoulder Instability Index [WOSI], Disabilities of the Arm, Shoulder and Hand-short version [QuickDASH], and Walch-Duplay test). Data on age, sex, number of dislocations, and type of surgery were collected. The test-retest method and intraclass correlation coefficient (ICC: >0.75 = good, >0.85 = very good, and >0.9 = excellent) were used for analysis. RESULTS: A total of 114 patients with anterior shoulder instability were included, of whom 89 (78%) were men. The mean age was 28 years. The ISIS was very reliable, with an ICC of 0.933. The mean number of dislocations per patient was higher in patients who had an ISIS of ≥6 (25 vs 14; P = .05). Patients who underwent more complex arthroscopic procedures such as Hill-Sachs remplissage or open Latarjet had higher preoperative ISIS outcomes, with a mean score of 4.8 versus 3.4, respectively (P = .002). There was no correlation between the ISIS and the quality-of-life questionnaires, with Pearson correlations all >0.05 (WOSI = 0.39; QuickDASH = 0.97; Walch-Duplay = 0.08). CONCLUSION: Our results show that the ISIS is reliable when used in a multicenter study with anterior traumatic instability populations. There was no correlation between the ISIS and the quality-of-life questionnaires, but surgical decisions reflected its increased use.

Reliability and validity of the cross-culturally adapted French version of the Core Outcome Measures Index (COMI) in patients with low back pain.

PURPOSE: To conduct a cross-cultural adaptation of the Core Outcome Measures Index (COMI) into French according to established guidelines. METHODS: Seventy outpatients with chronic low back pain were recruited from six spine centres in Switzerland and France. They completed the newly translated COMI, and the Roland Morris disability (RMQ), Dallas Pain (DPQ), adjectival pain rating scale, WHO Quality of Life, and EuroQoL-5D questionnaires. After ~14 days RMQ and COMI were completed again to assess reproducibility; a transition question (7-point Likert scale; "very much worse" through "no change" to "very much better") indicated any change in status since the first questionnaire. RESULTS: COMI whole scores displayed no floor effects and just 1.5% ceiling effects. The scores for the individual COMI items correlated with their corresponding full-length reference questionnaire with varying strengths of correlation (0.33-0.84, P < 0.05). COMI whole scores showed a very good correlation with the "multidimensional" DPQ global score (Rho = 0.71). 55 patients (79%) returned a second questionnaire with no/minimal change in their back status. The reproducibility of individual COMI 5-point items was good, with test-retest differences within one grade ranging from 89% for 'social/work disability' to 98% for 'symptom-specific well-being'. The intraclass correlation coefficient for the COMI whole score was 0.85 (95% CI 0.76-0.91). CONCLUSIONS: In conclusion, the French version of this short, multidimensional questionnaire showed good psychometric properties, comparable to those reported for German and Spanish versions. The French COMI represents a valuable tool for future multicentre clinical studies and surgical registries (e.g. SSE Spine Tango) in French-speaking countries.

Scientific Literature on the Reliability and Validity of the Manchester Triage System (MTS) Protocol: A Integrative Literature Review

OBJECTIVE To analyze the scientific production about the validity and reliability of the Manchester Triage System (MTS) protocol. METHOD A descriptive study of an integrative literature review. Articles on the validity and reliability of the MTS developed with children and adults published between 1999 and 2013 were included. RESULTS 14 articles were selected from a total of 8438, nine of validity and five of reliability. The reliability of the MTS ranged from moderate to almost perfect, with higher intra-evaluation. Regarding validity, the results seem to point to equivalent and satisfactory sensibility and specificity levels of the MTS. The instrument proved to be a good predictor of the need for hospitalization and of hospital mortality. CONCLUSION The reliability and validity of the MTS obtained in the studies is varied. It is recommended that new studies indicate necessary modifications to the MTS so that it is more safely used by nurses.

Reliability of the image evaluation system PGMI

Purpose: Many countries used the PGMI (P=perfect, G=good, M=moderate, I=inadequate) classification system for assessing the quality of mammograms. Limits inherent to the subjectivity of this classification have been shown. Prior to introducing this system in Switzerland, we wanted to better understand the origin of this subjectivity in order to minimize it. Our study aimed at identifying the main determinants of the variability of the PGMI system and which criteria are the most subjected to subjectivity. Methods and Materials: A focus group composed of 2 experienced radiographers and 2 radiologists specified each PGMI criterion. Ten raters (6 radiographers and 4 radiologists) evaluated twice a panel of 40 randomly selected mammograms (20 analogic and 20 digital) according to these specified PGMI criteria. The PGMI classification was assessed and the intra- and inter-rater reliability was tested for each professional group (radiographer vs radiologist), image technology (analogic vs digital) and PGMI criterion. Results: Some 3,200 images were assessed. The intra-rater reliability appears to be weak, particularly in respect to inter-rater variability. Subjectivity appears to be largely independent of the professional group and image technology. Aspects of the PGMI classification criteria most subjected to variability were identified. Conclusion: Post-test discussions enabled to specify more precisely some criteria. This should reduce subjectivity when applying the PGMI classification system. A concomitant, important effort in training radiographers is also necessary.