The State of our Toolbox: A Meta-analysis of Reliability Measurement Precision

My study investigated internal consistency estimates of psychometric surveys as an operationalization of the state of measurement precision of constructs in industrial and organizational (I/O) psychology. Analyses were conducted of samples used in research articles published in the Journal of Applied Psychology between 1975 and 2010 in five year intervals (K = 934) from 480 articles yielding 1427 coefficients. Articles and their respective samples were coded for test-taker characteristics (e.g., age, gender, and ethnicity), research settings (e.g., lab and field studies), and actual tests (e.g., number of items and scale anchor points). A reliability and inter-item correlations depository was developed for I/O variables and construct groups. Personality measures had significantly lower inter-item correlations than other construct groups. Also, internal consistency estimates and reporting practices were evaluated over time, demonstrating an improvement in measurement precision and missing data.

Effect of performance feedback on perceived knowledge and likelihood to pursue continuing education

The purpose of this study was to assess the effect of performance feedback on Athletic Trainers’ (ATs) perceived knowledge (PK) and likelihood to pursue continuing education (CE). The investigation was grounded in the theories of “the definition of the situation” (Thomas & Thomas, 1928) and the “illusion of knowing,” (Glenberg, Wilkinson, & Epstein, 1982) suggesting that PK drives behavior. This investigation measured the degree to which knowledge gap predicted CE seeking behavior by providing performance feedback designed to change PK. A pre-test post-test control-group design was used to measure PK and likelihood to pursue CE before and after assessing actual knowledge. ATs (n=103) were randomly sampled and assigned to two groups, with and without performance feedback. Two independent samples t-tests were used to compare groups on the difference scores of the dependent variables. Likelihood to pursue CE was predicted by three variables using multiple linear regression: perceived knowledge, pre-test likelihood to pursue CE, and knowledge gap. There was a 68.4% significant difference (t101=2.72, p=0.01, ES=0.45) between groups in the change scores for likelihood to pursue CE because of the performance feedback (Experimental group=13.7% increase; Control group=4.3% increase). The strongest relationship among the dependent variables was between pre-test and post-test measures of likelihood to pursue CE (F2,102=56.80, p<0.01, r=0.73, R2=0.53). The pre- and post-test predictive relationship was enhanced when group was included in the model. In this model [YCEpost=0.76XCEpre-0.34 Xgroup+2.24+E], group accounted for a significant amount of unique variance in predicting CE while the pre-test likelihood to pursue CE variable was held constant (F3,102=40.28, p<0.01, r=0.74, R2=0.55). Pre-test knowledge gap, regardless of group allocation, was a linear predictor of the likelihood to pursue CE (F1,102=10.90, p=.01, r=.31, R2=.10). In this investigation, performance feedback significantly increased participants’ likelihood to pursue CE. Pre-test knowledge gap was a significant predictor of likelihood to pursue CE, regardless if performance feedback was provided. ATs may have self-assessed and engaged in internal feedback as a result of their test-taking experience. These findings indicate that feedback, both internal and external, may be necessary to trigger CE seeking behavior.

The therapeutic effects of Millon Clinical Multiaxial Inventory -III assessment feedback

A study was conducted to test the therapeutic effects of assessment feedback on rapport-building and self-enhancement variables (self-verification, self-discovery, self-esteem), as well as symptomatology. Assessment feedback was provided in the form of interpretive information based on the results of the Millon Clinical Multiaxial Inventory-III (MCMI-III). Participants (N = 89) were randomly assigned to three groups: a Feedback group, a Reflective-Counseling group, and a No-Feedback group. The Feedback group was provided with assessment feedback, the Reflective-Counseling group was asked to comment on the meaning of the taking the MCMI-III, the No-Feedback group received general information about the MCMI-III. Results revealed that assessment feedback, when provided in the form of interpretive interpretation positively affects rapport-building and self-enhancement variables (self-verification and self-discovery). No significant results were found in terms of self-esteem or symptom decrease as a function of feedback. However, a significant decrease in symptoms across groups was found. Results indicate that assessment feedback in the form of interpretive information can be used as a starting point in therapy. Implications of the findings are discussed with respect to theory and clinical practice. ^

The therapeutic effects of Millon Clinical Multiaxial Inventory-III assessment feedback

A study was conducted to test the therapeutic effects of assessment feedback on rapport-building and self-enhancement variables (self-verification, self-discovery, self-esteem), as well as symptomatology. Assessment feedback was provided in the form of interpretive information based on the results of the Millon Clinical Multiaxial Inventory- III (MCMI-III). Participants (N = 89) were randomly assigned to three groups: a Feedback group, a Reflective-Counseling group, and a No-Feedback group. The Feedback group was provided with assessment feedback, the Reflective-Counseling group was asked to comment on the meaning of the taking the MCMI-III, the No- Feedback group received general information about the MCMI-III. Results revealed that assessment feedback, when provided in the form of interpretive interpretation positively affects rapport-building and self-enhancement variables (self-verification and self-discovery). No significant results were found in terms of self-esteem or symptom decrease as a function of feedback. However, a significant decrease in symptoms across groups was found. Results indicate that assessment feedback in the form of interpretive information can be used as a starting point in therapy. Implications of the findings are discussed with respect to theory and clinical practice.

Effect of Performance Feedback on Perceived Knowledge and Likelihood to Pursue Continuing Education

The purpose of this study was to assess the effect of performance feedback on Athletic Trainers’ (ATs) perceived knowledge (PK) and likelihood to pursue continuing education (CE). The investigation was grounded in the theories of “the definition of the situation” (Thomas & Thomas, 1928) and the “illusion of knowing,” (Glenberg, Wilkinson, & Epstein, 1982) suggesting that PK drives behavior. This investigation measured the degree to which knowledge gap predicted CE seeking behavior by providing performance feedback designed to change PK. A pre-test post-test control-group design was used to measure PK and likelihood to pursue CE before and after assessing actual knowledge. ATs (n=103) were randomly sampled and assigned to two groups, with and without performance feedback. Two independent samples t-tests were used to compare groups on the difference scores of the dependent variables. Likelihood to pursue CE was predicted by three variables using multiple linear regression: perceived knowledge, pre-test likelihood to pursue CE, and knowledge gap. There was a 68.4% significant difference (t101= 2.72, p=0.01, ES=0.45) between groups in the change scores for likelihood to pursue CE because of the performance feedback (Experimental group=13.7% increase; Control group= 4.3% increase). The strongest relationship among the dependent variables was between pre-test and post-test measures of likelihood to pursue CE (F2,102=56.80, p<0.01, r=0.73, R2=0.53). The pre- and post-test predictive relationship was enhanced when group was included in the model. In this model [YCEpost=0.76XCEpre-0.34 Xgroup+2.24+E], group accounted for a significant amount of unique variance in predicting CE while the pre-test likelihood to pursue CE variable was held constant (F3,102=40.28, p<0.01,: r=0.74, R2=0.55). Pre-test knowledge gap, regardless of group allocation, was a linear predictor of the likelihood to pursue CE (F1,102=10.90, p=.01, r=.31, R2=.10). In this investigation, performance feedback significantly increased participants’ likelihood to pursue CE. Pre-test knowledge gap was a significant predictor of likelihood to pursue CE, regardless if performance feedback was provided. ATs may have self-assessed and engaged in internal feedback as a result of their test-taking experience. These findings indicate that feedback, both internal and external, may be necessary to trigger CE seeking behavior.