Student's academic efficacy or Inefficacy? an example on how to evaluate the psychometric properties of a measuring instrument and evaluate the effects of item wording

This study evaluated the effect of item inversion on the construct validity and reliability of psychometric scales and proposed a theoretical framework for the evaluation of the psychometric properties of data gathered with psychometric instruments. To this propose, we used the Maslach Burnout Inventory, which is the most used psychometric inventory to measure burnout in different professional context (Students, Teachers, Police, Doctors, Nurses, etc…). The version of the MBI used was the MBI-Student Survey (MBI-SS). This inventory is composed of three key dimensions: Exhaustion, Cynicism and Professional Efficacy. The two first dimensions—which have positive formulated items—are moderate to strong positive correlated, and show moderate to strong negative correlations with the 3rd dimension—which has negative formulated items. We tested the hypothesis that, in college students, formulating the 3rd dimension of burnout as Inefficacy (reverting the negatively worded items in the Efficacy dimension) improves the correlation of the 3rd dimension with the other two dimensions, improves its internal consistency, and the overall MBI-SS’ construct validity and reliability. Confirmatory factor analysis results, estimated by Maximum Likelihood, revealed adequate factorial fit for both forms of the MBI-SS (with Efficacy) vs. the MBI-SSi (with Inefficacy). Also both forms showed adequate convergent and discriminant related validity. However, reliability and convergent validity were higher for the MBI-SSi. There were also stronger (positive) correlations between the 3 factors in MBI-SSi than the ones observed in MBI-SS. Results show that positively rewording of the 3rd dimension of the MBI-SS improves its validity and reliability. We therefore propose that the 3rd dimension of the MBI-SS should be named Professional Inefficacy and its items should be positively worded.

The hospital survey on patient safety culture in Portuguese hospitals: instrument validity and reliability

Purpose – Quantitative instruments to assess patient safety culture have been developed recently and a few review articles have been published. Measuring safety culture enables healthcare managers and staff to improve safety behaviours and outcomes for patients and staff. The study aims to determine the AHRQ Hospital Survey on Patient Safety Culture (HSPSC) Portuguese version's validity and reliability. Design/methodology/approach – A missing-value analysis and item analysis was performed to identify problematic items. Reliability analysis, inter-item correlations and inter-scale correlations were done to check internal consistency, composite scores. Inter-correlations were examined to assess construct validity. A confirmatory factor analysis was performed to investigate the observed data's fit to the dimensional structure proposed in the AHRQ HSPSC Portuguese version. To analyse differences between hospitals concerning composites scores, an ANOVA analysis and multiple comparisons were done. Findings – Eight of 12 dimensions had Cronbach's alphas higher than 0.7. The instrument as a whole achieved a high Cronbach's alpha (0.91). Inter-correlations showed that there is no dimension with redundant items, however dimension 10 increased its internal consistency when one item is removed. Originality/value – This study is the first to evaluate an American patient safety culture survey using Portuguese data. The survey has satisfactory reliability and construct validity.

Reliability and validity of the Alcohol Use Disorders Identification Test (AUDIT) imbedded within a general health risk screening questionnaire: results of a survey in 332 primary care patients.

BACKGROUND: Self-administered, general health risk screening questionnaires that are administered while patients wait in the doctor's office may be a reasonable and timesaving approach to address the requirements of preventive medicine in a typical 10-min medical visit. The psychometric characteristics of the Alcohol Use Disorders Identification Test (AUDIT) incorporated within a health questionnaire (H-AUDIT) have not been examined. METHODS: The reliability and validity of the self-administered AUDIT were compared between the H-AUDIT and the AUDIT used as a single scale (S-AUDIT) in 332 primary care patients. RESULTS: No major demographic or alcohol use characteristics were found between the 166 subjects who completed the H-AUDIT and the 166 individuals who completed the S-AUDIT. The test-retest reliability of the 166 subjects who completed the H-AUDIT [estimated by Spearman correlation coefficient at a 6-week interval (0.88), internal consistency (total correlation coefficients for all items ranged from 0.38 to 0.69; Cronbach alpha index 0.85), and the sensitivity and specificity of the H-AUDIT were used to identify at-risk drinkers' areas under receiver operating characteristic (0.77) and alcohol-dependent subjects' areas under receiver operating characteristic (0.89)] was similar to the same measurements obtained with the 166 individuals who completed the S-AUDIT. CONCLUSIONS: The AUDIT incorporated in a health risk screening questionnaire is a reliable and valid self-administered instrument to identify at-risk drinkers and alcohol-dependent individuals in primary care settings.

Predictive validity of the personal background and preparation survey and effectiveness of targeted interventions among health science community college students

Existing data, collected from 1st-year students enrolled in a major Health Science Community College in the south central United States, for Fall 2010, Spring 2011, Fall 2011 and Spring 2012 semesters as part of the "Online Navigational Assessment Vehicle, Intervention Guidance, and Targeting of Risks (NAVIGATOR) for Undergraduate Minority Student Success" with CPHS approval number HSC-GEN-07-0158, was used for this thesis. The Personal Background and Preparation Survey (PBPS) and a two-question risk self-assessment subscale were administered to students during their 1st-year orientation. The PBPS total risk score, risk self-assessment total and overall scores, and Under Representative Minority Student (URMS) status were recorded. The purpose of this study is to evaluate and report the predictive validity of the indicators identified above for Adverse Academic Status Events (AASE) and Nonadvancement Adverse Academic Status Events (NAASE) as well as the effectiveness of interventions targeted using the PBPS among a diverse population of health science community college students. The predictive validity of the PBPS for AASE has previously been demonstrated among health science professions and graduate students (Johnson, Johnson, Kim, & McKee, 2009a; Johnson, Johnson, McKee, & Kim, 2009b). Data will be analyzed using binary logistic regression and correlation using SPSS 19 statistical package. Independent variables will include baseline- versus intervention-year treatments, PBPS, risk self-assessment, and URMS status. The dependent variables will be binary AASE and NAASE status. ^ The PBPS was the first reliable diagnostic and prescriptive instrument to establish documented predictive validity for student Adverse Academic Status Events (AASE) among students attending health science professional schools. These results extend the documented validity for the PBPS in predicting AASE to a health science community college student population. Results further demonstrated that interventions introduced using the PBPS were followed by approximately one-third reduction in the odds of Nonadvancement Adverse Academic Status Events (NAASE), controlling for URMS status and risk self-assessment scores. These results indicate interventions introduced using the PBPS may have potential to reduce AASE or attrition among URMS and nonURMS attending health science community colleges on a broader scale; positively impacting costs, shortages, and diversity of health science professionals.^

Reliability and validity of the medical outcomes study social support survey (Chinese version)

No Abstract

Perceived need for mental health care, findings from the Australian National Survey of Mental Health and Well-being

Background. The Australian National Survey of Mental Health and Well-being was designed to detect and describe psychiatric morbidity, associated disability, service use and perceived need for care. The survey employed a single-phase interview methodology, delivering a field questionnaire to a clustered probability sample of 10641 Australians. Perceived need was sampled with an instrument designed for this survey, the Perceived Need for Care Questionnaire (PNCQ). This questionnaire gathers information about five categories of perceived need, assigning each to one of four levels of perceived need. Reliability and validity studies showed satisfactory performance of the instrument. Methods. Perceived need for mental health care in the Australian population has been analysed using PNCQ data, relating this to diagnostic and service utilization data from the above survey. Results. The survey findings indicate that an estimated 13.8 % of the Australian population have perceived need for mental health care. Those who met interview criteria for a psychiatric diagnosis and also expressed perceived need make up 9.9 % of the population. An estimated 11.0% of the population are cases of untreated prevalence, a minority (3.6% of the population) of whom expressed perceived need for mental health care. Among persons using services, those without a psychiatric diagnosis based on interview criteria (4.4% of the population), showed high levels of perceived met need. Conclusions. The overall rate of perceived need found by this methodology lies between those found in the USA and Canada. The findings suggest that service use in the absence of diagnosis elicited by survey questionnaires may often represent successful intervention. In the survey, untreated prevalence was commonly not accompanied by perceived need for mental health care.

Assessing perceived need for mental health care in a community survey: development of the Perceived Need for Care Questionnaire (PNCQ)

Background: The Perceived Need for Care Questionnaire (PNCQ) was designed for the Australian National Survey of Mental Health and Wellbeing. The PNCQ complemented collection of data on diagnosis and disability with the survey participants' perceptions of their needs for mental health care and the meeting of those needs. The four-stage design of the PNCQ mimics a conversational exploration of the topic of perceived needs. Five categories of perceived need are each assigned to one of four levels of perceived need (no need, unmet need, partially met need and met need). For unmet need and partially met need, information on barriers to care is collected, Methods: Inter-rater reliabilities of perceived needs assessed by the PNCQ were examined in a study of 145 anxiety clinic attenders. Construct validity of these items was tested, using a multi-trait multi-method approach and hypotheses regarding extreme groups, in a study with a sample of 51 general practice and community psychiatric service patients. Results: The instrument is brief to administer and has proved feasible for use in various settings. Inter-rater reliabilities for major categories, measured by the kappa statistic, exceeded 0.60 in most cases; for the summary category of all perceived needs, inter-rater reliability was 0.62. The multi-trait multi-method approach lent support to the construct validity of the instrument, as did findings in extreme groups. Conclusions: The PNCQ shows acceptable feasibility, reliability and validity, adding to the range of assessment tools available for epidemiological and health services research.

Characteristics of DSM-IV and ICD-10 cannabis dependence among Australian adults: results from the National Survey of Mental Health and Wellbeing

A representative sample (n = 10 641) of Australian adults completed a structured diagnostic interview assessing the prevalence of mental and substance use disorders in the last year. The prevalence of DSM-IV (1.5%) and ICD-10 (1.7%) cannabis dependence was similar. DSM-IV and ICD-10 dependence criteria comprised unidimensional syndromes. The most common symptoms among dependent and non-dependent users were difficulties with controlling use and withdrawal, although there were marked differences in symptom prevalence. Dependent users reported a median of four symptoms. There was good to excellent diagnostic concordance (kappas = 0.7-0.9) between systems for dependence but not for abuse/harmful use (Y = 0.4). These findings provide some support for the validity of cannabis dependence. (C) 2001 Elsevier Science Ireland Ltd. All rights reserved.

Cervical screening: population-based comparisons between self-reported survey and registry-derived Pap test rates

Objective: To compare self-reported cervical screening rates, as recorded in the 1998 NSW Health Survey, with registry-based cervical screening rates for NSW for the same period; and to examine factors associated with over-estimates of cervical screening rates by self-report. Methods: Self-reported cervical screening data was extracted from the 1998 NSW Health Survey, biennial screening rates estimated and compared with biennial cervical screening rates for 1997-98, as recorded on the NSW Pap Test Register (PTR). Rates and differences were related to socio-demographic characteristics of the 17 Area Health Services of NSW. Results: According to the 1998 NSW Health Survey, 74% of women reported having a Pap test during the previous two years. The equivalent rate recorded on the NSW PTR for 1997-98 was 62% (p

Non-linear pathways of corporate environmental management: a survey of ISO 14001-certified companies in Brazil

A study was conducted to verify whether the theory on the evolution of corporate environmental management (CEM) is applicable to organizations located in Brazil. Some of the most important proposals pertaining to the evolution of CEM were evaluated in a systematic fashion and integrated into a typical theoretical framework containing three evolutionary stages: reactive, preventive and proactive. The validity of this framework was tested by surveying 94 companies located in Brazil with ISO 14001 certification. Results indicated that the evolution of CEM tends to occur in a manner that is counter to what has generally been described in the literature. Two evolutionary stages were identified: 1) synergy for eco-efficiency and 2) environmental legislation view, which combine variables that were initially categorized into different theoretical CEM stages. These data, obtained from a direct study of Brazilian companies, suggest that the evolution of environmental management in organizations tends to occur in a non-linear fashion, requiring a re-analysis of traditional perceptions of CEM development, as suggested by Kolk and Mauser (2002). (C) 2010 Elsevier Ltd. All rights reserved.

Dietary trends: Estimates from food supply and survey data

Objective: To illustrate methodological issues involved in estimating dietary trends in populations using data obtained from various sources in Australia in the 1980s and 1990s. Methods: Estimates of absolute and relative change in consumption of selected food items were calculated using national data published annually on the national food supply for 1982-83 to 1992-93 and responses to food frequency questions in two population based risk factor surveys in 1983 and 1994 in the Hunter Region of New South Wales, Australia. The validity of estimated food quantities obtained from these inexpensive sources at the beginning of the period was assessed by comparison with data from a national dietary survey conducted in 1983 using 24 h recall. Results: Trend estimates from the food supply data and risk factor survey data were in good agreement for increases in consumption of fresh fruit, vegetables and breakfast food and decreases in butter, margarine, sugar and alcohol. Estimates for trends in milk, eggs and bread consumption, however, were inconsistent. Conclusions: Both data sources can be used for monitoring progress towards national nutrition goals based on selected food items provided that some limitations are recognized. While data collection methods should be consistent over time they also need to allow for changes in the food supply (for example the introduction of new varieties such as low-fat dairy products). From time to time the trends derived from these inexpensive data sources should be compared with data derived from more detailed and quantitative estimates of dietary intake.

Validity of the self reporting questionnaire-20 in epidemiological studies with older adults

This study examined the validity of the self-reporting questionnaire (SRQ-20) in a population-based survey with older adults in Sao Paulo, Brazil. The study sample consisted of 2,008 individuals aged 65 years or more who were living in an economically disadvantaged area of the city. The geriatric mental state (GMS) was used as the gold standard for the assessment of common mental disorders (CMD). The optimal SRQ-20 threshold for case definition was 4/5 (sensitivity = 76.1%, specificity = 74.6%, area under ROC curve = 0.82). Women, older participants and those with lower income were more likely to be misclassified by the questionnaire. The inclusion of older persons with dementia or psychosis did not change the ability of the SRQ-20 to identify cases of CMD. The SRQ-20 may be a useful instrument to be used in large epidemiological studies in resource-poor settings for the identification of cases of depression and anxiety in later life. The performance of the SRQ-20 among older adults was similar to that observed with the adult population in Brazil.