Student's academic efficacy or Inefficacy? an example on how to evaluate the psychometric properties of a measuring instrument and evaluate the effects of item wording

This study evaluated the effect of item inversion on the construct validity and reliability of psychometric scales and proposed a theoretical framework for the evaluation of the psychometric properties of data gathered with psychometric instruments. To this propose, we used the Maslach Burnout Inventory, which is the most used psychometric inventory to measure burnout in different professional context (Students, Teachers, Police, Doctors, Nurses, etc…). The version of the MBI used was the MBI-Student Survey (MBI-SS). This inventory is composed of three key dimensions: Exhaustion, Cynicism and Professional Efficacy. The two first dimensions—which have positive formulated items—are moderate to strong positive correlated, and show moderate to strong negative correlations with the 3rd dimension—which has negative formulated items. We tested the hypothesis that, in college students, formulating the 3rd dimension of burnout as Inefficacy (reverting the negatively worded items in the Efficacy dimension) improves the correlation of the 3rd dimension with the other two dimensions, improves its internal consistency, and the overall MBI-SS’ construct validity and reliability. Confirmatory factor analysis results, estimated by Maximum Likelihood, revealed adequate factorial fit for both forms of the MBI-SS (with Efficacy) vs. the MBI-SSi (with Inefficacy). Also both forms showed adequate convergent and discriminant related validity. However, reliability and convergent validity were higher for the MBI-SSi. There were also stronger (positive) correlations between the 3 factors in MBI-SSi than the ones observed in MBI-SS. Results show that positively rewording of the 3rd dimension of the MBI-SS improves its validity and reliability. We therefore propose that the 3rd dimension of the MBI-SS should be named Professional Inefficacy and its items should be positively worded.

Psychometric validation of the Portuguese version of the Neuropathic Pain Symptoms Inventory

Backgroud: It has been shown that different symptoms or symptom combinations of neuropathic pain (NeP) may correspond to different mechanistic backgrounds and respond differently to treatment. The Neuropathic Pain Symptom Inventory (NPSI) is able to detect distinct clusters of symptoms (i.e. dimensions) with a putative common mechanistic background. The present study described the psychometric validation of the Portuguese version (PV) of the NPSI. Methods: Patients were seen in two consecutive visits, three to four weeks apart. They were asked to: (i) rate their mean pain intensity in the last 24 hours on an 11-point (0-10) numerical scale; (ii) complete the PV-NPSI; (iii) provide the list of pain medications and doses currently in use. VAS and Global Impression of Change (GIC) were filled out in the second visit. Results: PV-NPSI underwent test-retest reliability, factor analysis, analysis of sensitivity to changes between both visits. The PV-NPSI was reliable in this setting, with a good intra-class correlation for all items. The factorial analysis showed that the PV-NPSI inventory assessed different components of neuropathic pain. Five different factors were found. The PV-NPSI was adequate to evaluate patients with neuropathic pain and to detect clusters of NeP symptoms. Conclusions: The psychometric properties of the PV-NPSI rendered it adequate to evaluate patients with both central and peripheral neuropathic pain syndromes and to detect clusters of NeP symptoms.

Psychometric properties of the Beck Depression Inventory-II: a comprehensive review

OBJECTIVE: To review the psychometric properties of the Beck Depression Inventory-II (BDI-II) as a self-report measure of depression in a variety of settings and populations. METHODS: Relevant studies of the BDI-II were retrieved through a search of electronic databases, a hand search, and contact with authors. Retained studies (k = 118) were allocated into three groups: non-clinical, psychiatric/institutionalized, and medical samples. RESULTS: The internal consistency was described as around 0.9 and the retest reliability ranged from 0.73 to 0.96. The correlation between BDI-II and the Beck Depression Inventory (BDI-I) was high and substantial overlap with measures of depression and anxiety was reported. The criterion-based validity showed good sensitivity and specificity for detecting depression in comparison to the adopted gold standard. However, the cutoff score to screen for depression varied according to the type of sample. Factor analysis showed a robust dimension of general depression composed by two constructs: cognitive-affective and somatic-vegetative. CONCLUSIONS: The BDI-II is a relevant psychometric instrument, showing high reliability, capacity to discriminate between depressed and non-depressed subjects, and improved concurrent, content, and structural validity. Based on available psychometric evidence, the BDI-II can be viewed as a cost-effective questionnaire for measuring the severity of depression, with broad applicability for research and clinical practice worldwide.

Working Alliance Inventory-Short Revised (WAI-SR): psychometric properties in outpatients and inpatients

The Working Alliance Inventory-Short Revised (WAI-SR) is a recently refined measure of the therapeutic alliance that assesses three key aspects of the therapeutic alliance: (a) agreement on the tasks of therapy, (b) agreement on the goals of therapy and (c) development of an affective bond. The WAI-SR demonstrated good psychometric properties in an initial validation in psychotherapy outpatients in the USA. The generalizability of these findings is limited because in some countries a substantial portion of individual psychotherapy is delivered in inpatient settings. This study investigated and compared the psychometric properties of the WAI-SR in German outpatients (N = 88) and inpatients (N = 243). In both samples reliability (alpha > 0.80) and convergent validity with the Helping Alliance Questionnaire were good (r > 0.64). Confirmatory factor analysis showed acceptable to good model fit for the proposed Bond-Task-Goal model in both samples. Multi-group analysis demonstrated that the same constructs were measured across settings. Alliance ratings of outpatients and inpatients differed regarding the overlap of alliance components and the magnitude of the alliance ratings: The differentiation of the alliance components was poorer in inpatients and they reported lower alliances. Unique aspects of the alliance in inpatient treatment are discussed and a need for further research on the alliance in inpatient settings is pointed out. Overall, the WAI-SR can be recommended for alliance assessment in both settings.

Investigating interocclusal perception in tactile teeth sensibility using symmetric and asymmetric analysis

The purpose of this clinical trial was to determine the active tactile sensibility of natural teeth and to obtain a statistical analysis method fitting a psychometric function through the observed data points. On 68 complete dentulous test persons (34 males, 34 females, mean age 45.9 ± 16.1 years), one pair of healthy natural teeth each was tested: n = 24 anterior teeth and n = 44 posterior teeth. The computer-assisted, randomized measurement was done by having the subjects bite on thin copper foils of different thickness (5-200 µm) inserted between the teeth. The threshold of active tactile sensibility was defined by the 50% value of correct answers. Additionally, the gradient of the sensibility curve and the support area (90-10% value) as a description of the shape of the sensibility curve were calculated. For modeling the sensibility curve, symmetric and asymmetric functions were used. The mean sensibility threshold was 14.2 ± 12.1 µm. The older the subject, the higher the tactile threshold (r = 0.42, p = 0.0006). The support area was 41.8 ± 43.3 µm. The higher the 50% threshold, the smaller the gradient of the curve and the larger the support area. The curves showing the active tactile sensibility of natural teeth demonstrate a tendency towards asymmetry, so that the active tactile sensibility of natural teeth can mathematically best be described by using the asymmetric Weibull function.

The self-administered 24-item geriatric pain measure (GPM-24-SA): psychometric properties in three European populations of community-dwelling older adults

OBJECTIVE: To explore the feasibility and psychometric properties of a self-administered version of the 24-item Geriatric Pain Measure (GPM-24-SA). DESIGN: Secondary analysis of baseline data from the Prevention in Older People-Assessment in Generalists' practices trial, an international multi-center study of a health-risk appraisal system. PARTICIPANTS: One thousand seventy-two community dwelling nondisabled older adults self-reporting pain from London, UK; Hamburg, Germany; and Solothurn, Switzerland. OUTCOME MEASURES: GPM-24-SA as part of a multidimensional Health Risk Appraisal Questionnaire including self-reported demographic and health-related information. RESULTS: Among the 1,072 subjects, 655 had complete GPM-24-SA data, 404 had 30% missing GPM-24-SA data. In psychometric analyses across the three European populations with complete GPM-24-SA data, the measure exhibited stable internal consistency, good convergent, divergent and discriminant validity, and produced stable pain measurements. However, factor analysis indicated differences in the GPM-24-SA across sites with discrepancies mainly related to items of a single subscale that failed to load appropriately. Analyses including imputation for subjects with psychometric properties comparable to complete data analyses suggesting that imputation in cases with psychometric evaluation of the GPM-24-SA is needed before it could be recommended for widespread use.

Psychometric properties of the Swiss version of the Nursing Home Survey on Patient Safety Culture

The Nursing Home Survey on Patient Safety Culture (NHSPSC) was specifically developed for nursing homes to assess a facility’s safety climate and it consists of 12 dimensions. After its pilot testing, however, no fur- ther psychometric analyses were performed on the instrument. For this study of safety climate in Swiss nursing home units, the NHSPSC was linguistically adapted to the Swiss context and to address the unit as well as facility level, with the aim of testing aspects of the validity and reliability of the Swiss version before its use in Swiss nursing home units. Psychometric analyses were performed on data from 367 nurs- ing personnel from nine nursing homes in the German-speaking part of Switzerland (response rate = 66%), and content validity (CVI) examined. The statistical influence of unit membership on respondents’ answers, and on their agreement concerning their units’ safety climate, was tested using intraclass corre- lation coefficients (ICCs) and the rWG(J) interrater agreement index. A multilevel exploratory factor analysis (MEFA) with oblimin rotation was applied to examine the questionnaire’s dimensionality. Cronbach’s alpha and Raykov’s rho were calculated to assess factor reliability. The relationship of safety climate dimensions with clinical outcomes was explored. Expert feedback confirmed the relevance of the instru- ment’s items (CVI = 0.93). Personnel showed strong agreement in their perceptions in three dimensions of the questionnaire. ICCs supported a multilevel analysis. MEFA produced nine factors at the within-level (in comparison to 12 in the original version) and two factors at the between-level with satisfactory fit statis- tics. Raykov’s Rho for the single level factors ranged between 0.67 and 0.86. Some safety climate dimen- sions show moderate, but non-significant correlations with the use of bedrails, physical restraint use, and fall-related injuries. The Swiss version of the NHSPSC needs further refinement and testing before its use can be recommended in Swiss nursing homes: its dimensionality needs further clarification, particularly to distinguish items addressing the unit-level safety climate from those at the facility level.

Moving beyond transition outcomes: meta-analysis of remission rates in individuals at high clinical risk for psychosis

Recent evidence suggests that transition risks from initial clinical high risk (CHR) status to psychosis are decreasing. The role played by remission in this context is mostly unknown. The present study addresses this issue by means of a meta-analysis including eight relevant studies published up to January 2012 that reported remission rates from an initial CHR status. The primary effect size measure was the longitudinal proportion of remissions compared to non-remission in subjects with a baseline CHR state. Random effect models were employed to address the high heterogeneity across studies included. To assess the robustness of the results, we performed sensitivity analyses by sequentially removing each study and rerunning the analysis. Of 773 subjects who met initial CHR criteria, 73% did not convert to psychosis along a 2-year follow. Of these, about 46% fully remitted from the baseline attenuated psychotic symptoms, as evaluated on the psychometric measures usually employed by prodromal services. The corresponding clinical remission was estimated as high as 35% of the baseline CHR sample. The CHR state is associated with a significant proportion of remitting subjects that can be accounted by the effective treatments received, a lead time bias, a dilution effect, a comorbid effect of other psychiatric diagnoses.

The dementia cognitive fluctuation scale, a new psychometric test for clinicians to identify cognitive fluctuations in people with dementia.

OBJECTIVES Cognitive fluctuation (CF) is a common feature of dementia and a core diagnostic symptom for dementia with Lewy bodies (DLB). CF remains difficult to accurately and reliably detect clinically. This study aimed to develop a psychometric test that could be used by clinicians to facilitate the identification of CF and improve the recognition and diagnosis of DLB and Parkinson disease, and to improve differential diagnosis of other dementias. METHODS We compiled a 17-item psychometric test for identifying CF and applied this measure in a cross-sectional design. Participants were recruited from the North East of England, and assessments were made in individuals' homes. We recruited people with four subtypes of dementia and a healthy comparison group, and all subjects were administered this pilot scale together with other standard ratings. The psychometric properties of the scale were examined with exploratory factor analysis. We also examined the ability of individual items to identify CF to discriminate between dementia subtypes. The sensitivity and specificity of discriminating items were explored along with validity and reliability analyses. RESULTS Participants comprised 32 comparison subjects, 30 people with Alzheimer disease, 30 with vascular dementia, 29 with DLB, and 32 with dementia associated with Parkinson disease. Four items significantly discriminated between dementia groups and showed good levels of sensitivity (range: 78.6%-80.3%) and specificity (range: 73.9%-79.3%). The scale had very good levels of test-retest (Cronbach's alpha: 0.82) and interrater (0.81) reliabilities. The four items loaded onto three different factors. These items were: 1) marked differences in functioning during the daytime; 2) daytime somnolence; 3) daytime drowsiness; and 4) altered levels of consciousness during the day. CONCLUSIONS We identified four items that provide valid, sensitive, and specific questions for reliably identifying CF and distinguishing the Lewy body dementias from other major causes of dementia (Alzheimer disease and vascular dementia).

Psychometric and biohormonal indices of dental anxiety in children. A prospective cohort study.

The stress of dental treatment often elicits negative emotions in children, expressed as dental fear or anxiety. Highly anxious children obstruct treatment and avoid therapy, further amplifying oral health problems. The aim of this study was to examine the neuroendocrine and autonomic nervous system responses to dental treatment and their possible interactions and associations with psychometric indices of anxiety, caries, previous dental experience, anesthesia, age and gender in school children. Upon informed consent, saliva was obtained from 97 children (59% males, mean age ±  SD: 89.73 ± 15 months) in the Clinic of pediatric dentistry before treatment, immediately post-treatment and at the recall visit to determine cortisol and salivary alpha-amylase (sAA) levels. Dental and general anxiety was assessed through specific questionnaires completed by the children. Compared to pre-treatment, cortisol levels were increased following treatment, while sAA levels were higher at the recall. Pre- and post-treatment cortisol and sAA responses were positively correlated. Dental and general anxiety questionnaire scores were also significantly correlated with each other. The integrated autonomic and neuroendocrine responses prior to treatment were correlated with state anxiety and those following treatment with dental anxiety. However, univariable and multivariable linear regression analysis associated post-treatment cortisol, but not sAA, levels with dental anxiety. No associations of cortisol or sAA responses with caries, age, gender, previous dental experience or anesthesia were detected. These data provide some evidence that both sAA and cortisol levels are altered in children in anticipation or during dental treatment, but only cortisol levels are associated to dental anxiety.

Elucidating the Functional Relationship Between Working Memory Capacity and Psychometric Intelligence: A Fixed-Links Modeling Approach for Experimental Repeated-Measures Designs

Numerous studies reported a strong link between working memory capacity (WMC) and fluid intelligence (Gf), although views differ in respect to how close these two constructs are related to each other. In the present study, we used a WMC task with five levels of task demands to assess the relationship between WMC and Gf by means of a new methodological approach referred to as fixed-links modeling. Fixed-links models belong to the family of confirmatory factor analysis (CFA) and are of particular interest for experimental, repeated-measures designs. With this technique, processes systematically varying across task conditions can be disentangled from processes unaffected by the experimental manipulation. Proceeding from the assumption that experimental manipulation in a WMC task leads to increasing demands on WMC, the processes systematically varying across task conditions can be assumed to be WMC-specific. Processes not varying across task conditions, on the other hand, are probably independent of WMC. Fixed-links models allow for representing these two kinds of processes by two independent latent variables. In contrast to traditional CFA where a common latent variable is derived from the different task conditions, fixed-links models facilitate a more precise or purified representation of the WMC-related processes of interest. By using fixed-links modeling to analyze data of 200 participants, we identified a non-experimental latent variable, representing processes that remained constant irrespective of the WMC task conditions, and an experimental latent variable which reflected processes that varied as a function of experimental manipulation. This latter variable represents the increasing demands on WMC and, hence, was considered a purified measure of WMC controlled for the constant processes. Fixed-links modeling showed that both the purified measure of WMC (β = .48) as well as the constant processes involved in the task (β = .45) were related to Gf. Taken together, these two latent variables explained the same portion of variance of Gf as a single latent variable obtained by traditional CFA (β = .65) indicating that traditional CFA causes an overestimation of the effective relationship between WMC and Gf. Thus, fixed-links modeling provides a feasible method for a more valid investigation of the functional relationship between specific constructs.

Elucidating underlying processes of psychometric intelligence: A comparison of behavioral response latency and P3 latency in a modified Hick reaction time paradigm

The mental speed approach explains individual differences in intelligence by faster information processing in individuals with higher compared to lower intelligence - especially in elementary cognitive tasks (ECTs). One of the most examined ECTs is the Hick paradigm. The present study aimed to contrast reaction time (RT) and P3 latency in a Hick task as predictors of intelligence. Although both, RT and P3 latency, are commonly used as indicators of mental speed, it is also known that they measure different aspects of information processing. Participants were 113 female students. RT and P3 latency were measured while participants completed the Hick task with four levels of complexity. Intelligence was assessed with Cattell's Culture Fair Test. A RT factor and a P3 factor were extracted by employing a PCA across complexity levels. There was no significant correlation between the factors. Commonality analysis was used to determine the proportions of unique and shared variance in intelligence explained by the RT and P3 latency factors. RT and P3 latency explained 5.5% and 5% of unique variance in intelligence. However, the two speed factors did not explain a significant portion of shared variance. This result suggests that RT and P3 latency in the Hick paradigm are measuring different aspects of information processing that explain different parts of variance in intelligence.

Psychometric behaviour of the strengths and difficulties questionnaire (SDQ) in the Spanish national health survey 2006

Background: The Strengths and Difficulties Questionnaire (SDQ) is a tool to measure the risk for mental disorders in children. The aim of this study is to describe the diagnostic efficiency and internal structure of the SDQ in the sample of children studied in the Spanish National Health Survey 2006. Methods: A representative sample of 6,773 children aged 4 to 15 years was studied. The data were obtained using the Minors Questionnaire in the Spanish National Health Survey 2006. The ROC curve was constructed and calculations made of the area under the curve, sensitivity, specificity and the Youden J indices. The factorial structure was studied using models of exploratory factorial analysis (EFA) and confirmatory factorial analysis (CFA). Results: The prevalence of behavioural disorders varied between 0.47% and 1.18% according to the requisites of the diagnostic definition. The area under the ROC curve varied from 0.84 to 0.91 according to the diagnosis. Factor models were cross-validated by means of two different random subsamples for EFA and CFA. An EFA suggested a three correlated factor model. CFA confirmed this model. A five-factor model according to EFA and the theoretical five-factor model described in the bibliography were also confirmed. The reliabilities of the factors of the different models were acceptable (>0.70, except for one factor with reliability 0.62). Conclusions: The diagnostic behaviour of the SDQ in the Spanish population is within the working limits described in other countries. According to the results obtained in this study, the diagnostic efficiency of the questionnaire is adequate to identify probable cases of psychiatric disorders in low prevalence populations. Regarding the factorial structure we found that both the five and the three factor models fit the data with acceptable goodness of fit indexes, the latter including an externalization and internalization dimension and perhaps a meaningful positive social dimension. Accordingly, we recommend studying whether these differences depend on sociocultural factors or are, in fact, due to methodological questions.

Psychometric properties of the Spanish adaptation of the School Attitude Assessment Survey-Revised

Background: The assessment of attitudes toward school with the objective of identifying adolescents who may be at risk of underachievement has become an important area of research in educational psychology, although few specific tools for their evaluation have been designed to date. One of the instruments available is the School Attitude Assessment Survey-Revised (SAAS-R). Method: The objective of the current research is to test the construct validity and to analyze the psychometric properties of the Spanish version of the SAAS-R. Data were collected from 1,398 students attending different high schools. Students completed the SAAS-R along with measures of the g factor, and academic achievement was obtained from school records. Results: Confirmatory factor analysis, multivariate analysis of variance and analysis of variance tests supported the validity evidence. Conclusions: The results indicate that the Spanish version of the SAAS-R is a useful measure that contributes to identification of underachieving students. Lastly, the results obtained and their implications for education are discussed.

Development and psychometric testing of the clinical learning environment, supervision and nurse teacher evaluation scale (CLES+T): The Spanish version

Background: The Clinical Learning Environment, Supervision and Nurse Teacher scale is a reliable and valid instrument to evaluate the quality of the clinical learning process in international nursing education contexts. Objectives: This paper reports the development and psychometric testing of the Spanish version of the Clinical Learning Environment, Supervision and Nurse Teacher scale. Design: Cross-sectional validation study of the scale. Setting: 10 public and private hospitals in the Alicante area, and the Faculty of Health Sciences (University of Alicante, Spain). Participants: 370 student nurses on clinical placement (January 2011–March 2012). Methods: The Clinical Learning Environment, Supervision and Nurse Teacher scale was translated using the modified direct translation method. Statistical analyses were performed using PASW Statistics 18 and AMOS 18.0.0 software. A multivariate analysis was conducted in order to assess construct validity. Cronbach’s alpha coefficient was used to evaluate instrument reliability. Results: An exploratory factorial analysis identified the five dimensions from the original version, and explained 66.4% of the variance. Confirmatory factor analysis supported the factor structure of the Spanish version of the instrument. Cronbach’s alpha coefficient for the scale was .95, ranging from .80 to .97 for the subscales. Conclusion: This version of the Clinical Learning Environment, Supervision and Nurse Teacher scale instrument showed acceptable psychometric properties for use as an assessment scale in Spanish-speaking countries.