The refinement and validation of the critical decision making and problem solving scale moral dilema (CDP-MD)

This thesis extended previous research on critical decision making and problem solving by refining and validating a measure designed to assess the use of critical thinking and critical discussion in sociomoral dilemmas. The purpose of this thesis was twofold: 1) to refine the administration of the Critical Thinking Subscale of the CDP to elicit more adequate responses and for purposes of refining the coding and scoring procedures for the total measure, and 2) to collect preliminary data on the initial reliabilities of the measure. Subjects consisted of 40 undergraduate students at Florida International University. Results indicate that the use of longer probes on the Critical Thinking Subscale was more effective in eliciting adequate responses necessary for coding and evaluating the subjects performance. Analyses on the psychometric properties of the measure consisted of test-retest reliability and inter-rater reliability.

The development of rubrics to measure undergraduate students' global awareness and global perspective: A validity study

Higher education institutions across the United States have developed global learning initiatives to support student achievement of global awareness and global perspective, but assessment options for these outcomes are extremely limited. A review of research for a global learning initiative at a large, Hispanic-serving, urban, public, research university in South Florida found a lack of instruments designed to measure global awareness and global perspective in the context of an authentic performance assessment. This quasi-experimental study explored the development of two rubrics for the global learning initiative and the extent to which evidence supported the rubrics' validity and reliability. One holistic rubric was developed to measure students' global awareness and the second to measure their global perspective. The study utilized a pretest/posttest nonequivalent group design. Multiple linear regression was used to ascertain the rubrics' ability to discern and compare average learning gains of undergraduate students enrolled in two global learning courses and students enrolled in two non-global learning courses. Parallel pretest/posttest forms of the performance task required students to respond to two open-ended questions, aligned with the learning outcomes, concerning a complex case narrative. Trained faculty raters read responses and used the rubrics to measure students' global awareness and perspective. Reliability was tested by calculating the rates of agreement among raters. Evidence supported the finding that the global awareness and global perspective rubrics yielded scores that were highly reliable measures of students' development of these learning outcomes. Chi-square tests of frequency found significant rates of inter-rater agreement exceeding the study's .80 minimum requirement. Evidence also supported the finding that the rubrics yielded scores that were valid measures of students' global awareness and global perspective. Regression analyses found little evidence of main effects; however, post hoc analyses revealed a significant interaction between global awareness pretest scores and the treatment, the global learning course. Significant interaction was also found between global perspective pretest scores and the treatment. These crossover interactions supported the finding that the global awareness and global perspective rubrics could be used to detect learning differences between the treatment and control groups as well as differences within the treatment group.

Reliability of the Clinical Application of a Mechanical Inclinometer in Measuring Glenohumeral Motion

Objective: Establish intra- and inter-examiner reliability of glenohumeral range of motion (ROM) measures taken by a single-clinician using a mechanical inclinometer. Design: A single-session, repeated-measure, randomized, counterbalanced design. Setting: Athletic Training laboratory. Participants: Ten college-aged volunteers (9 right-hand dominant; 4 males, 6 females; age=23.2±2.4y, mass=73±16kg, height=170±8cm) without shoulder or neck injuries within one year. Interventions: Two Certified Athletic Trainers separately assessed passive glenohumeral (GH) internal (IR) and external (ER) rotation bilaterally. Each clinician secured the inclinometer to each subject’s distal forearm using elastic straps. Clinicians followed standard procedures for assessing ROM, with the participants supine on a standard treatment table with 90° of elbow flexion. A second investigator recorded the angle. Clinicians measured all shoulders once to assess inter-clinician reliability and eight shoulders twice to assess intra-clinician reliability. We used SPSS 14.0 (SPSS Inc., Chicago, IL) to calculate standard error of measure (SEM) and Intraclass Correlation Coefficients (ICC) to evaluate intra- and inter-clinician reliability. Main Outcome Measures: Dependent variables were degrees of IR, ER, glenohumeral internal rotation deficit (GIRD) and total arc of rotation. We calculated GIRD as the bilateral difference in IR (nondominant–dominant) and total arc for each shoulder (IR+ER). Results: Intra-clinician reliability for each examiner was excellent (ICC[1,1] range=0.90-0.96; SEM=2.2°-2.5°) for all measures. Examiners displayed excellent inter-clinician reliability (ICC[2,1] range=0.79-0.97; SEM=1.7°-3.0°) for all measures except nondominant IR which had good reliability(0.72). Conclusions: Results suggest that clinicians can achieve reliable measures of GH rotation and GIRD using a single-clinician technique and an inexpensive, readily available mechanical inclinometer.

Examination and Critique of Codebook for Textual Analysis

The researcher presents the details, findings, and critique of a pre-pilot study conducted on a codebook created for a textbook comparison. She used Cohen’s alpha and percent agreement to determine inter-rater reliabilities for coding categories. These values revealed changes needed in the coding scheme and in the coder training process for the future comparison study.

A model of school managerial control : the systematic analysis of managerial behaviors, processes, and indicators

The theoretical construct of control has been defined as necessary (Etzioni, 1965), ubiquitous (Vickers, 1967), and on-going (E. Langer, 1983). Empirical measures, however, have not adequately given meaning to this potent construct, especially within complex organizations such as schools. Four stages of theory-development and empirical testing of school building managerial control using principals and teachers working within the nation's fourth largest district are presented in this dissertation as follows: (1) a review and synthesis of social science theories of control across the literatures of organizational theory, political science, sociology, psychology, and philosophy; (2) a systematic analysis of school managerial activities performed at the building level within the context of curricular and instructional tasks; (3) the development of a survey questionnaire to measure school building managerial control; and (4) initial tests of construct validity including inter-item reliability statistics, principal components analyses, and multivariate tests of significance. The social science synthesis provided support of four managerial control processes: standards, information, assessment, and incentives. The systematic analysis of school managerial activities led to further categorization between structural frequency of behaviors and discretionary qualities of behaviors across each of the control processes and the curricular and instructional tasks. Teacher survey responses (N=486) reported a significant difference between these two dimensions of control, structural frequency and discretionary qualities, for standards, information, and assessments, but not for incentives. The descriptive model of school managerial control suggests that (1) teachers perceive structural and discretionary managerial behaviors under information and incentives more clearly than activities representing standards or assessments, (2) standards are primarily structural while assessments are primarily qualitative, (3) teacher satisfaction is most closely related to the equitable distribution of incentives, (4) each of the structural managerial behaviors has a qualitative effect on teachers, and that (5) certain qualities of managerial behaviors are perceived by teachers as distinctly discretionary, apart from school structure. The variables of teacher tenure and school effectiveness reported significant effects on school managerial control processes, while instructional levels (elementary, junior, and senior) and individual school differences were not found to be significant for the construct of school managerial control.

The State of our Toolbox: A Meta-analysis of Reliability Measurement Precision

My study investigated internal consistency estimates of psychometric surveys as an operationalization of the state of measurement precision of constructs in industrial and organizational (I/O) psychology. Analyses were conducted of samples used in research articles published in the Journal of Applied Psychology between 1975 and 2010 in five year intervals (K = 934) from 480 articles yielding 1427 coefficients. Articles and their respective samples were coded for test-taker characteristics (e.g., age, gender, and ethnicity), research settings (e.g., lab and field studies), and actual tests (e.g., number of items and scale anchor points). A reliability and inter-item correlations depository was developed for I/O variables and construct groups. Personality measures had significantly lower inter-item correlations than other construct groups. Also, internal consistency estimates and reporting practices were evaluated over time, demonstrating an improvement in measurement precision and missing data.