Unresolved Problems with the "I", the "A" and the "T": Logical and Psychometric Critique of the Implicit Association Test (IAT)

The Implicit Association Test (IAT) had already gained the status of a prominent assessment procedure before its psychometric properties and underlying task structure were understood. The present critique addresses five major problems that arise when the IAT is used for diagnostic inferences: (1) the asymmetry of causal and diagnostic inferences; (2) the viability of the underlying association model; (3) the lack of a testable model underlying IAT-based inferences; (4) the difficulties of interpreting difference scores; and (5) the susceptibility of the IAT to deliberate faking and strategic processing. Based on a theoretical reflection of these issues, and a comprehensive survey of published IAT studies, it is concluded that a number of uncontrolled factors can produce (or reduce) significant IAT scores independently of the personality attribute that is supposed to be captured by the IAT procedure.

Individual differences in working memory capacity explain the relationship between general discrimination ability and psychometric intelligence

The close association between psychometric intelligence and general discrimination ability (GDA), conceptualized as latent variable derived from performance on different sensory discrimination tasks, is empirically well-established but theoretically widely unclear. The present study contrasted two alternative explanations for this association. The first explanation is based on what Spearman (1904) referred to as a central function underlying this relationship in the sense of the g factor of intelligence and becoming most evident in GDA. In this case, correlations between different aspects of cognitive abilities, such as working memory (WM) capacity, and psychometric intelligence should be mediated by GDA if their correlation is caused by g. Alternatively, the second explanation for the relationship between psychometric intelligence and GDA proceeds from fMRI studies which emphasize the role of WM functioning for sensory discrimination. Given the well-known relationship between WM and psychometric intelligence, the relationship between GDA and psychometric intelligence might be attributed to WM. The present study investigated these two alternative explanations at the level of latent variables. In 197 young adults, a model in which WM mediated the relationship between GDA and psychometric intelligence described the data better than a model in which GDA mediated the relationship between WM and psychometric intelligence. Moreover, GDA failed to explain portions of variance of psychometric intelligence above and beyond WM. These findings clearly support the view that the association between psychometric intelligence and GDA must be understood in terms of WM functioning.

Elucidating the Functional Relationship Between Working Memory Capacity and Psychometric Intelligence: A Fixed-Links Modeling Approach for Experimental Repeated-Measures Designs

Numerous studies reported a strong link between working memory capacity (WMC) and fluid intelligence (Gf), although views differ in respect to how close these two constructs are related to each other. In the present study, we used a WMC task with five levels of task demands to assess the relationship between WMC and Gf by means of a new methodological approach referred to as fixed-links modeling. Fixed-links models belong to the family of confirmatory factor analysis (CFA) and are of particular interest for experimental, repeated-measures designs. With this technique, processes systematically varying across task conditions can be disentangled from processes unaffected by the experimental manipulation. Proceeding from the assumption that experimental manipulation in a WMC task leads to increasing demands on WMC, the processes systematically varying across task conditions can be assumed to be WMC-specific. Processes not varying across task conditions, on the other hand, are probably independent of WMC. Fixed-links models allow for representing these two kinds of processes by two independent latent variables. In contrast to traditional CFA where a common latent variable is derived from the different task conditions, fixed-links models facilitate a more precise or purified representation of the WMC-related processes of interest. By using fixed-links modeling to analyze data of 200 participants, we identified a non-experimental latent variable, representing processes that remained constant irrespective of the WMC task conditions, and an experimental latent variable which reflected processes that varied as a function of experimental manipulation. This latter variable represents the increasing demands on WMC and, hence, was considered a purified measure of WMC controlled for the constant processes. Fixed-links modeling showed that both the purified measure of WMC (β = .48) as well as the constant processes involved in the task (β = .45) were related to Gf. Taken together, these two latent variables explained the same portion of variance of Gf as a single latent variable obtained by traditional CFA (β = .65) indicating that traditional CFA causes an overestimation of the effective relationship between WMC and Gf. Thus, fixed-links modeling provides a feasible method for a more valid investigation of the functional relationship between specific constructs.

At risk or not at risk? A meta-analysis of the prognostic accuracy of psychometric interviews for psychosis prediction

An accurate detection of individuals at clinical high risk (CHR) for psychosis is a prerequisite for effective preventive interventions. Several psychometric interviews are available, but their prognostic accuracy is unknown. We conducted a prognostic accuracy meta-analysis of psychometric interviews used to examine referrals to high risk services. The index test was an established CHR psychometric instrument used to identify subjects with and without CHR (CHR+ and CHR-). The reference index was psychosis onset over time in both CHR+ and CHR- subjects. Data were analyzed with MIDAS (STATA13). Area under the curve (AUC), summary receiver operating characteristic curves, quality assessment, likelihood ratios, Fagan's nomogram and probability modified plots were computed. Eleven independent studies were included, with a total of 2,519 help-seeking, predominately adult subjects (CHR+: N=1,359; CHR-: N=1,160) referred to high risk services. The mean follow-up duration was 38 months. The AUC was excellent (0.90; 95% CI: 0.87-0.93), and comparable to other tests in preventive medicine, suggesting clinical utility in subjects referred to high risk services. Meta-regression analyses revealed an effect for exposure to antipsychotics and no effects for type of instrument, age, gender, follow-up time, sample size, quality assessment, proportion of CHR+ subjects in the total sample. Fagan's nomogram indicated a low positive predictive value (5.74%) in the general non-help-seeking population. Albeit the clear need to further improve prediction of psychosis, these findings support the use of psychometric prognostic interviews for CHR as clinical tools for an indicated prevention in subjects seeking help at high risk services worldwide.

Adolescent tobacco use: Studies of prevalence, factorial validity, and factorial invariance between African Americans and Whites

In the last thirty years, increasing efforts have been made to reduce the prevalence of adolescent tobacco use in the United States. Although the prevalence has declined dramatically over the past decade, there are still sharp differences in adolescent smoking-initiation rates across racial/ethnic groups. Large-scale surveys frequently assess smoking-related attitudes, self-efficacy, and intentions to explain the differences in smoking rates between African Americans and Whites. However, there is little agreement about which constructs are significant. Moreover, the psychometric properties of smoking-related attitude, self-efficacy, and intention constructs have not been fully examined. More studies are needed to understand existing patterns of tobacco use and to validate and fully exploit the constructs' relationship to adolescent smoking initiation across racial/ethnic groups. ^ This dissertation reports on a secondary analysis of data from a large multi-ethnic convenience sample of sixth- through eighth-grade students in 22 schools in East Texas and the city of Houston. The specific aims of this dissertation were to (1) describe smoking and alternate tobacco product use rates by race/ethnicity, gender, age, and grade level (Article 1); (2) test the factorial validity of smoking-related attitudes, self-efficacy, and intentions using confirmatory factor analysis techniques (Article 2); and (3) test the factorial invariance of smoking-related attitudes, self-efficacy, and intentions between African Americans and Whites (Article 3). ^ The prevalence findings confirm the disparities in tobacco use among African American, Hispanic, and White adolescents that other surveys have reported (Article 1). This study also demonstrates the usefulness of examining use patterns of not only cigarettes but also alternative tobacco products in younger multiethnic populations, as well as of providing epidemiological data estimates about different phases of smoking. The confirmatory factor analysis provides evidence of construct validity of attitude, self-efficacy, and intention scales for the multiethnic sample (Article 2). Finally, the factorial invariance analyses indicates that some measures representing smoking-related attitudes, self-efficacy, and intentions may not be appropriate for use among both African Americans and Whites (Article 3). Additional research is needed to further our understanding of the patterns and predictors of youth tobacco use initiation. ^

Psychosocial measures reported by parents in studies of skin cancer prevention

Background. Because it is important to minimize children's sun exposure to reduce skin cancer risk, much of the extensive skin cancer prevention literature consists of studies of children's sun protection, sun avoidance and ultraviolet radiation (UVR) exposure. Little attention has been focused on the measurement of psychosocial constructs in these studies. Identification of the psychosocial correlates or determinants of children's skin cancer risk or risk-reduction behavior is critical to more fully understand and predict behavior. Furthermore, psychosocial variables may be influenced by interventions to reduce risk. Thus, it is important to examine the psychosocial measures used in studies of children's skin cancer prevention. Information on the validity and reliability of psychosocial measures may increase confidence in study findings based on these measures. In particular, self-efficacy and barriers are key constructs in several major theoretical frameworks and parental measures have been associated with children's sun protection. However, there is conceptual overlap of self-efficacy and barriers measures and little is known about the psychometric properties of these measures.^ Study Aims and Methods. The overall goal of this dissertation was to examine the measurement of psychosocial constructs relevant to children's skin cancer prevention. Because children depend primarily on their parents for skin cancer prevention, measures of parents' psychosocial constructs are the focus. Study 1 was a systematic review of parental psychosocial measures used in studies of children's sun protection, sun avoidance and UVR exposure. The specific aims of Study 1 were to (1) describe psychosocial measures reported by parents, including available information on the psychometric properties of these measures and their use in analyses and (2) provide recommendations for the development, refinement and standardized reporting of measures. ^ Study 2 examined the psychometric properties of measures of parental self-efficacy and barriers regarding children's sun protection. Melanoma patients (N=205) who were parents of children ≤ 12 years of age completed a telephone interview that included self-efficacy and barriers measures specific to sunscreen, clothing, shade and limiting time outdoors. The specific aims of Study 2 were to (1) use a confirmatory factor analytic approach to examine the factorial validity of parental self-efficacy and barriers measures, (2) examine the convergent and discriminant validity of behavior-specific measures of self-efficacy and barriers and (3) assess the reliability of item and scale measures.^ Results. In Study 1, a search of standard databases yielded 48 eligible studies. Most studies assessed only one or two psychosocial constructs. Knowledge was measured most frequently. There was little discussion of measure source, development, theoretical background or psychometric properties, besides internal consistency reliability. There was conceptual overlap of some measures. In Study 2, confirmatory factor analytic findings supported the factorial validity of the self-efficacy and barriers measures. When all eight self-efficacy and barriers measures were included in the same model, a modified eight-factor model adequately fit the data, providing preliminary evidence that the measures are distinct. Measure associations supported the convergent validity of all measures and the discriminant validity of most measures. The self-efficacy and barriers measures were reliable.^ Conclusions. Recommendations based on the literature review include developing and refining psychosocial measures based on theory. Describing a measure's theoretical basis and psychometric properties would facilitate critical evaluation. Standardized reporting of source, development, theory, construct, items and analytic role would facilitate comparison of findings, continual refinement and future applications of measures. In the validation study, self-efficacy and barriers measures were examined in a sample of parents with a personal history of melanoma. Findings suggested that these measures are valid and reliable for use in studies of children's sun protection. There was preliminary evidence that these measures are distinct but additional study is needed. ^

Assessment of problem solving in computing studies

The assessment of learning outcomes is a key concept in the European Credit Transfer and Accumulation System (ECTS) since credits are awarded when the assessment shows the competences which were aimed at have been developed at an appropriate level. This paper describes a study which was first part of the Bologna Experts Team-Spain project and then developed as an independent study. It was carried out with the overall goal to gain experience in the assessment of learning outcomes. More specifically it aimed at 1) designing procedures for the assessment of learning outcomes related to these compulsory generic competences; 2) testing some basic psychometric features that an assessment device with some consequences for the subjects being evaluated needs to prove; 3) testing different procedures of standard setting, and 4) using assessment results as orienting feedback to students and their tutors. The process of development of tests to carry out the assessment of learning outcomes is described as well as some basic features regarding their reliability and validity. First conclusions on the comparison of the results achieved at two academic levels are also presented.

Assessment of learning outcomes in computing studies

The assessment of learning outcomes is a key concept in the European Credit Transfer and Accumulation System (ECTS) since credits are awarded when the assessment shows the competences which were aimed at have been developed at an appropriate level. This paper describes a study which was first part of the project of the Bologna Experts Team-Spain and then developed as an independent study. It was carried out with the overall goal to gain experience in the assessment of learning outcomes. More specifically it aimed at 1) designing procedures for the assessment of learning outcomes related to these compulsory generic competences; 2) testing some basic psychometric features that an assessment device with some consequences for the subjects being evaluated needs to prove; 3) testing different procedures of standard setting, and 4) using assessment results as orienting feedback to students and their tutors. The process of development of tests to carry out the assessment of learning outcomes related to these competences, as well as some basic features regarding their reliability and validity is described and first results on the comparison of results achieved at two academic levels, will also be described at a later stage.

Development and psychometric testing of the clinical learning environment, supervision and nurse teacher evaluation scale (CLES+T): The Spanish version

Background: The Clinical Learning Environment, Supervision and Nurse Teacher scale is a reliable and valid instrument to evaluate the quality of the clinical learning process in international nursing education contexts. Objectives: This paper reports the development and psychometric testing of the Spanish version of the Clinical Learning Environment, Supervision and Nurse Teacher scale. Design: Cross-sectional validation study of the scale. Setting: 10 public and private hospitals in the Alicante area, and the Faculty of Health Sciences (University of Alicante, Spain). Participants: 370 student nurses on clinical placement (January 2011–March 2012). Methods: The Clinical Learning Environment, Supervision and Nurse Teacher scale was translated using the modified direct translation method. Statistical analyses were performed using PASW Statistics 18 and AMOS 18.0.0 software. A multivariate analysis was conducted in order to assess construct validity. Cronbach’s alpha coefficient was used to evaluate instrument reliability. Results: An exploratory factorial analysis identified the five dimensions from the original version, and explained 66.4% of the variance. Confirmatory factor analysis supported the factor structure of the Spanish version of the instrument. Cronbach’s alpha coefficient for the scale was .95, ranging from .80 to .97 for the subscales. Conclusion: This version of the Clinical Learning Environment, Supervision and Nurse Teacher scale instrument showed acceptable psychometric properties for use as an assessment scale in Spanish-speaking countries.

Psychometric properties of the drinking expectancy questionnaire: A review of the factor structure and a proposed new scoring method

Objective: Expectancies about the outcomes of alcohol consumption are widely accepted as important determinants of drinking. This construct is increasingly recognized as a significant element of psychological interventions for alcohol-related problems. Much effort has been invested in producing reliable and valid instruments to measure this construct for research and clinical purposes, but very few have had their factor structure subjected to adequate validation. Among them, the Drinking Expectancies Questionnaire (DEQ) was developed to address some theoretical and design issues with earlier expectancy scales. Exploratory factor analyses, in addition to validity and reliability analyses, were performed when the original questionnaire was developed. The object of this study was to undertake a confirmatory analysis of the factor structure of the DEQ. Method: Confirmatory factor analysis through LISREL 8 was performed using a randomly split sample of 679 drinkers. Results: Results suggested that a new 5-factor model, which differs slightly from the original 6-factor version, was a more robust measure of expectancies. A new method of scoring the DEQ consistent with this factor structure is presented. Conclusions: The present study shows more robust psychometric properties of the DEQ using the new factor structure.

Psychometric assessment of recovery from mild traumatic brain injury

The Rapid Screen of Concussion (RSC) is a brief psychometric test battery, designed to provide a functional criterion to aid clinical diagnosis of mild traumatic brain injury (mTBI). The present research aimed to examine the utility of this instrument for assessing recovery after mTBI. Three studies were conducted. In Study 1, Discriminant Function Analysis was performed to determine how well the RSC differentiated uninjured controls (N¼16), from mTBI patients (N¼22) and moderate to severe TBI patients (N¼14), several months post-injury. As predicted, moderate to severe TBI patients achieved lower scores than the mTBI and control groups. The RSC also successfully differentiated between each of the diagnostic groups, yielding an overall correct classification rate of 75%. Study 2 examined the predictive utility of the RSC in the mTBI sample (N¼22). Acute injury performance on the RSC was correlated with post-injury scores at an average of 5.5 months post-injury. Statistically significant partial correlation coefficients (r¼0.53r¼0.80) were found for each of the subtests, showing that low acute RSC scores were predictive of poor recovery scores on the RSC after mTBI. In the third study, Reliable Change Indices were calculated on the RSC subtests to examine individual patterns of recovery from mTBI. While 17 of the 23 participants made a significant improvement on their acute injury DSST scores (74%), only 13 of 25 made a significant improvement on the Rapid Sentence Judgement Test (52%), highlighting differential recovery of function, and challenging the notion of full recovery from mTBI within 3 months. These overall results offer support for the construct and predictive validity of the RSC and demonstrate that inexpensive tests of brain function may be useful for managing mTBI acutely for prognosis.

Sensory thresholds obtained from MEG data:cortical psychometric functions

Sensory sensitivity is typically measured using behavioural techniques (psychophysics), which rely on observers responding to very large numbers of stimulus presentations. Psychophysics can be problematic when working with special populations, such as children or clinical patients, because they may lack the compliance or cognitive skills to perform the behavioural tasks. We used an auditory gap-detection paradigm to develop an accurate measure of sensory threshold derived from passively-recorded MEG data. Auditory evoked responses were elicited by silent gaps of varying durations in an on-going noise stimulus. Source modelling was used to spatially filter the MEG data and sigmoidal ‘cortical psychometric functions’ relating response amplitude to gap duration were obtained for each individual participant. Fitting the functions with a curve and estimating the gap duration at which the evoked response exceeded one standard deviation of the prestimulus brain activity provided an excellent prediction of psychophysical threshold. Thus we have demonstrated that accurate sensory thresholds can be reliably extracted from MEG data recorded while participants listen passively to a stimulus. Because we required no behavioural task, the method is suitable for studies of populations where variations in cognitive skills or vigilance make traditional psychophysics unsuitable.

Shifts of the psychometric function: distinguishing bias from perceptual effects.

Morgan, Dillenburger, Raphael, and Solomon have shown that observers can use different response strategies when unsure of their answer, and, thus, they can voluntarily shift the location of the psychometric function estimated with the method of single stimuli (MSS; sometimes also referred to as the single-interval, two-alternative method). They wondered whether MSS could distinguish response bias from a true perceptual effect that would also shift the location of the psychometric function. We demonstrate theoretically that the inability to distinguish response bias from perceptual effects is an inherent shortcoming of MSS, although a three-response format including also an "undecided" response option may solve the problem under restrictive assumptions whose validity cannot be tested with MSS data. We also show that a proper two-alternative forced-choice (2AFC) task with the three-response format is free of all these problems so that bias and perceptual effects can easily be separated out. The use of a three-response 2AFC format is essential to eliminate a confound (response bias) in studies of perceptual effects and, hence, to eliminate a threat to the internal validity of research in this area.

Psychometric functions for detection and discrimination with and without flankers.

Recent studies have reported that flanking stimuli broaden the psychometric function and lower detection thresholds. In the present study, we measured psychometric functions for detection and discrimination with and without flankers to investigate whether these effects occur throughout the contrast continuum. Our results confirm that lower detection thresholds with flankers are accompanied by broader psychometric functions. Psychometric functions for discrimination reveal that discrimination thresholds with and without flankers are similar across standard levels, and that the broadening of psychometric functions with flankers disappears as standard contrast increases, to the point that psychometric functions at high standard levels are virtually identical with or without flankers. Threshold-versus-contrast (TvC) curves with flankers only differ from TvC curves without flankers in occasional shallower dippers and lower branches on the left of the dipper, but they run virtually superimposed at high standard levels. We discuss differences between our results and other results in the literature, and how they are likely attributed to the differential vulnerability of alternative psychophysical procedures to the effects of presentation order. We show that different models of flanker facilitation can fit the data equally well, which stresses that succeeding at fitting a model does not validate it in any sense.

Psychometric properties of the Drive for Muscularity Scale in Malay men

The Drive for Muscularity Scale (DMS) is a widely used measure in studies of men’s body image, but few studies have examined its psychometric properties outside English-speaking samples. Here, we assessed the factor structure of a Malay translation of the DMS. A community sample of 159 Malay men from Kuala Lumpur, Malaysia, completed the DMS, along with measures of self-esteem, body appreciation, and muscle discrepancy. Exploratory factor analysis led to the extraction of two factors, differentiating attitudes from behaviours, which mirrors the parent scale. Both factors also loaded on to a higher-order drive for muscularity factor. The subscales of the Malay DMS had adequate internal consistencies and good convergent validity, insofar as significant relationships were reported with self-esteem, body appreciation,muscle discrepancy, and body mass index. These results indicate that the Malay DMS has acceptable psychometric properties and can be used to assess body image concerns in Malay men.