The development of rubrics to measure undergraduate students' global awareness and global perspective: A validity study

Higher education institutions across the United States have developed global learning initiatives to support student achievement of global awareness and global perspective, but assessment options for these outcomes are extremely limited. A review of research for a global learning initiative at a large, Hispanic-serving, urban, public, research university in South Florida found a lack of instruments designed to measure global awareness and global perspective in the context of an authentic performance assessment. This quasi-experimental study explored the development of two rubrics for the global learning initiative and the extent to which evidence supported the rubrics' validity and reliability. One holistic rubric was developed to measure students' global awareness and the second to measure their global perspective. The study utilized a pretest/posttest nonequivalent group design. Multiple linear regression was used to ascertain the rubrics' ability to discern and compare average learning gains of undergraduate students enrolled in two global learning courses and students enrolled in two non-global learning courses. Parallel pretest/posttest forms of the performance task required students to respond to two open-ended questions, aligned with the learning outcomes, concerning a complex case narrative. Trained faculty raters read responses and used the rubrics to measure students' global awareness and perspective. Reliability was tested by calculating the rates of agreement among raters. Evidence supported the finding that the global awareness and global perspective rubrics yielded scores that were highly reliable measures of students' development of these learning outcomes. Chi-square tests of frequency found significant rates of inter-rater agreement exceeding the study's .80 minimum requirement. Evidence also supported the finding that the rubrics yielded scores that were valid measures of students' global awareness and global perspective. Regression analyses found little evidence of main effects; however, post hoc analyses revealed a significant interaction between global awareness pretest scores and the treatment, the global learning course. Significant interaction was also found between global perspective pretest scores and the treatment. These crossover interactions supported the finding that the global awareness and global perspective rubrics could be used to detect learning differences between the treatment and control groups as well as differences within the treatment group.

Examining argumentative coherence in essays by undergraduate students of English as a foreign language in Mainland China and their English speaking peers in the United States

I conducted this study to provide insights toward deepening understanding of association between culture and writing by building, assessing, and refining a conceptual model of second language writing. To do this, I examined culture and coherence as well as the relationship between them through a mixed methods research design. Coherence has been an important and complex concept in ESL/EFL writing. I intended to study the concept of coherence in the research context of contrastive rhetoric, comparing the coherence quality in argumentative essays written by undergraduates in Mainland China and their U.S. peers. In order to analyze the complex concept of coherence, I synthesized five linguistic theories of coherence: Halliday and Hasan's cohesion theory, Carroll's theory of coherence, Enkvist's theory of coherence, Topical Structure Analysis, and Toulmin's Model. Based upon the synthesis, 16 variables were generated. Across these 16 variables, Hotelling t-test statistical analysis was conducted to predict differences in argumentative coherence between essays written by two groups of participants. In order to complement the statistical analysis, I conducted 30 interviews of the writers in the studies. Participants' responses were analyzed with open and axial coding. By analyzing the empirical data, I refined the conceptual model by adding more categories and establishing associations among them. The study found that U.S. students made use of more pronominal reference. Chinese students adopted more lexical devices of reiteration and extended paralleling progression. The interview data implied that the difference may be associated with the difference in linguistic features and rhetorical conventions in Chinese and English. As far as Toulmin's Model is concerned, Chinese students scored higher on data than their U.S. peers. According to the interview data, this may be due to the fact that Toulmin's Model, modified as three elements of arguments, have been widely and long taught in Chinese writing instruction while U.S. interview participants said that they were not taught to write essays according to Toulmin's Model. Implications were generated from the process of textual data analysis and the formulation of structural model defining coherence. These implications were aimed at informing writing instruction, assessment, peer-review, and self-revision.

Personal and institutional factors: Relationship to self-efficacy of persistence to the senior year in college among self-identified Black undergraduate students in a Hispanic serving institution

While undergraduate enrollment of all racial groups in United States higher education institutions has increased, 6-year graduation rates of Blacks (39%) remain low compared to other races; Asians (69%), Whites (62%), and Hispanics (50%; NCES, 2010). Women's graduation rate is higher than men's; 58% compared to men's at 53% in public institutions (IPEDS, 2011). Retention literature does not address the perceptions of Black ethnic groups' experiences in college, particularly in Hispanic serving institutions. Informed by Tinto's (1975, 1987, 1993) student academic and social integration model, Guiffrida's (2003, 2004, 2005, 2006) model of relationships while at college, and ex-post facto research design, the study investigated personal and institutional factors that relate to Black students' self-efficacy and persistence to the senior year in college. Data about Black ethnic undergraduate seniors' (N = 236) academic and social experiences in college were collected using the Student Institutional Integration Survey (SIIS), an online questionnaire. Descriptive statistics were used to collect background information about the sample, correlation was calculated to indicate the degree of relationship between the variables, and multiple linear regressions were used to identify variables that are predictors of self-efficacy of persistence. Independent samples t-test and analyses of variance were computed to determine whether differences in perceptions of personal and institutional factors that relate to self-efficacy of persistence to the senior year in college could be identified between gender and ethnicity. Frequency was summarized to identify themes of participants' primary motivation for finishing undergraduate degree programs. These themes were: (a) self-pride/personal goal, (b) professional aspiration/career (c) motivation to support family, (d) desire to have financial independence/better job, (e) to serve community, (f) opportunity to go to college, (g) being first-generation college student, and (h) prove to family the value of higher education. The research findings support the tenets of academic and social integration theories which suggest that students' interaction with peer and faculty, relationships with family and friends, and involvement in institutional activities and organizations influence their persistence in college. Implications based on the findings affect institutional policy, curriculum, and program improvements that relate to Black undergraduate students' academic and social support.

Personal and Institutional Factors: Relationship to Self-Efficacy of Persistence to the Senior Year in College among Self-Identified Black Undergraduate Students in a Hispanic Serving Institution

While undergraduate enrollment of all racial groups in United States higher education institutions has increased, 6-year graduation rates of Blacks (39%) remain low compared to other races; Asians (69%), Whites (62%), and Hispanics (50%; NCES, 2010). Women’s graduation rate is higher than men’s; 58% compared to men’s at 53% in public institutions (IPEDS, 2011). Retention literature does not address the perceptions of Black ethnic groups’ experiences in college, particularly in Hispanic serving institutions. Informed by Tinto’s (1975, 1987, 1993) student academic and social integration model, Guiffrida’s (2003, 2004, 2005, 2006) model of relationships while at college, and ex-post facto research design, the study investigated personal and institutional factors that relate to Black students’ self-efficacy and persistence to the senior year in college. Data about Black ethnic undergraduate seniors’ (N = 236) academic and social experiences in college were collected using the Student Institutional Integration Survey (SIIS), an online questionnaire. Descriptive statistics were used to collect background information about the sample, correlation was calculated to indicate the degree of relationship between the variables, and multiple linear regressions were used to identify variables that are predictors of self-efficacy of persistence. Independent samples t-test and analyses of variance were computed to determine whether differences in perceptions of personal and institutional factors that relate to self-efficacy of persistence to the senior year in college could be identified between gender and ethnicity. Frequency was summarized to identify themes of participants’ primary motivation for finishing undergraduate degree programs. These themes were: (a) self-pride/personal goal, (b) professional aspiration/career (c) motivation to support family, (d) desire to have financial independence/better job, (e) to serve community, (f) opportunity to go to college, (g) being first-generation college student, and (h) prove to family the value of higher education. The research findings support the tenets of academic and social integration theories which suggest that students’ interaction with peer and faculty, relationships with family and friends, and involvement in institutional activities and organizations influence their persistence in college. Implications based on the findings affect institutional policy, curriculum, and program improvements that relate to Black undergraduate students’ academic and social support.

A determination of the effects of a ropes course on the self -esteem of undergraduate and graduate students

The purpose of this study was to evaluate the impact of the ropes course on the self-esteem of undergraduate and graduate students. The ropes course provides challenging experiential activities that facilitate personal confidence and group teamwork. The study relates to adult education, experiential education, and program evaluation within the context of hospitality and tourism management education. Quantitative data were based on the assessment of self-esteem through the completion of the Coopersmith Self-Esteem Inventory (SEI-C) modified for this study. Qualitative data were based on assessment of participants' reflective papers stating their perceptions of the ropes course. ^ The study compared a treatment group (31 undergraduate and 25 graduate students) which participated in the ropes course, and a control group (31 undergraduate and 25 graduate students) which did not participate. Both groups completed the pre- and post-treatment SEI-C at the same time intervals. The quantitative data were analyzed using a two-way repeated-measures analysis of variance (ANOVA). The qualitative data, comprised of reflective papers voluntarily written by 44 (79%) of the treatment group, were coded using four major themes reflecting the students' perceptions about the ropes course. ^ Scores on the pretest and posttest of the SEI-C were not significantly different for the two groups. The qualitative data showed a favorable impact of the ropes course. The discrepancy in the outcomes based on these two measures suggests that the SEI-C self-report paper-and-pencil instrument may not be sufficiently refined to evaluate the complex issue of self-esteem. The SEI-C, if used, should be supplemented by other evaluation measures, since the two measures may be evaluating different components of self-esteem. They may be also differentially affected by bias in scoring, or by statistical characteristics of reliability and validity. ^

Examining the validity of the GRE General Test scores and undergraduate GPA for predicting success in graduate school at a large racially and ethnically diverse public university in Southeast Florida

The purpose of the study was to determine the degree of relationships among GRE scores, undergraduate GPA (UGPA), and success in graduate school, as measured by first year graduate GPA (FGPA), cumulative graduate GPA, and degree attainment status. A second aim of the study was to determine whether the relationships between the composite predictor (GRE scores and UGPA) and the three success measures differed by race/ethnicity and sex. A total of 7,367 graduate student records (masters, 5,990; doctoral: 1,377) from 2000 to 2010 were used to evaluate the relationships among GRE scores, UGPA and the three success measures. Pearson's correlation, multiple linear and logistic regression, and hierarchical multiple linear and logistic regression analyses were performed to answer the research questions. The results of the correlational analyses differed by degree level. For master's students, the ETS proposed prediction that GRE scores are valid predictors of first year graduate GPA was supported by the findings from the present study; however, for doctoral students, the proposed prediction was only partially supported. Regression and correlational analyses indicated that UGPA was the variable that consistently predicted all three success measures for both degree levels. The hierarchical multiple linear and logistic regression analyses indicated that at master's degree level, White students with higher GRE Quantitative Reasoning Test scores were more likely to attain a degree than Asian Americans, while International students with higher UGPA were more likely to attain a degree than White students. The relationships between the three predictors and the three success measures were not significantly different between men and women for either degree level. Findings have implications both for practice and research. They will provide graduate school administrators with institution-specific validity data for UGPA and the GRE scores, which can be referenced in making admission decisions, while they will provide empirical and professionally defensible evidence to support the current practice of using UGPA and GRE scores for admission considerations. In addition, new evidence relating to differential predictions will be useful as a resource reference for future GRE validation researchers.

Differences among undergraduate and graduate nursing students' cultural competency

This study examined differences in cultural competency levels between undergraduate and graduate nursing students (age, ethnicity, gender, language at home, education level, program standing, program track, diversity encounters, and previous diversity training). Participants were 83% women, aged 20 to 62; 50% Hispanic/Latino; with a Bachelor of Science in Nursing (n = 82) and a Master of Science in Nursing (n = 62). Degrees included high school diplomas, associate/diplomas, bachelors' degrees in or out of nursing, and medical doctorate degrees from outside the United States. Students spoke English (n = 82) or Spanish ( n = 54). The study used a cross-sectional design guided by the three-dimensional cultural competency model. The Cultural Competency Assessment (CCA) tool is composed of two subscales: Cultural Awareness and Sensitivity (CAS) and Culturally Competent Behaviors (CCB). Multiple regressions, Pearson's correlations, and ANOVAs determined relationships and differences among undergraduate and graduate students. Findings showed significant differences between undergraduate and graduate nursing students in CAS, p <.016. Students of Hispanic/White/European ethnicity scored higher on the CAS, while White/non-Hispanic students scored lower on the CAS, p < .05. One-way ANOVAs revealed cultural competency differences by program standing (grade-point averages), and by program tracks, between Master of Science in Nursing Advanced Registered Nurse Practitioners and both Traditional Bachelor of Science in Nursing and Registered Nurse-Bachelor of Science in Nursing. Univariate analysis revealed that higher cultural competency was associated with having previous diversity training and participation in diversity training as continuing education. After controlling for all predictors, multiple regression analysis found program level, program standing, and diversity training explained a significant amount of variance in overall cultural competency (p = .027; R2 = .18). Continuing education is crucial in achieving students' cultural competency. Previous diversity training, graduate education, and higher grade-point average were correlated with higher cultural competency levels. However, increased diversity encounters were not associated with higher cultural competency levels.^

It takes an institution's village to retain a student: A comprehensive look at two early warning system undergraduate retention programs and administrators' perceptions of students' experiences and the retention services they provide students in the early warning system retention programs

Institutions have implemented many campus interventions to address student persistence/retention, one of which is Early Warning Systems (EWS). However, few research studies show evidence of interventions that incorporate noncognitive factors/skills, and psychotherapy/psycho-educational processes in the EWS. A qualitative study (phenomenological interview and document analysis) of EWS at both a public and private 4-year Florida university was conducted to explore EWS through the eyes of the administrators of the ways administrators make sense of students' experiences and the services they provide and do not provide to assist students. Administrators' understanding of noncognitive factors and the executive skills subset and their contribution to retention and the executive skills development of at-risk students were also explored. Hossler and Bean's multiple retention lenses theory/paradigms and Perez's retention strategies were used to guide the study. Six administrators from each institution who oversee and/or assist with EWS for first time in college undergraduate students considered academically at-risk for attrition were interviewed. Among numerous findings, at Institution X: EWS was infrequently identified as a service, EWS training was not conducted, numerous cognitive and noncognitive issues/deficits were identified for students, and services/critical departments such as EWS did not work together to share students' information to benefit students. Assessment measures were used to identify students' issues/deficits; however, they were not used to assess, track, and monitor students' issues/deficits. Additionally, the institution's EWS did address students' executive skills function beyond time management and organizational skills, but did not address students' psychotherapy/psycho-educational processes. Among numerous findings, at Institution Y: EWS was frequently identified as a service, EWS training was not conducted, numerous cognitive and noncognitive issues/deficits were identified for students, and services/critical departments such as EWS worked together to share students' information to benefit students. Assessment measures were used to identify, track, and monitor students' issues/deficits; however, they were not used to assess students' issues/deficits. Additionally, the institution's EWS addressed students' executive skills function beyond time management and organizational skills, and psychotherapy/psycho-educational processes. Based on the findings, Perez's retention strategies were not utilized in EWS at Institution X, yet were collectively utilized in EWS at Institution Y, to achieve Hossler and Bean's retention paradigms. Future research could be designed to test the link between engaging in the specific promising activities identified in this research (one-to-one coaching, participation in student success workshops, academic contracts, and tutoring) and student success (e.g., higher GPA, retention). Further, because this research uncovered some concern with how to best handle students with physical and psychological disabilities, future research could link these same promising strategies for improving student performance for example among ADHD students or those with clinical depression.

Undergraduate Students’ Perceptions of Study Abroad and Their Level of Achievement of Global Learning Outcomes

This study expanded on current research on study abroad and global learning, using the Global Perspective Inventory (GPI), and conducted at Florida International University (FIU) in Miami, FL. The GPI assesses the holistic development of a global perspective in higher education within three domains and their respective FIU-determined equivalents: cognitive (global awareness), intrapersonal (global perspective), and interpersonal (global engagement). The main purpose of this study was to assess FIU’s undergraduate students’ perceptions of study abroad on their level of achievement of global awareness, global perspective, and global engagement. The secondary purpose was to determine how the students described their study abroad experience and achievement of global learning. The research design for this study consisted of parallel mixed methods. The quantitative component was an ex post facto with hypothesis design, using a pretest/posttest nonequivalent group methodology. FIU undergraduates (N=147) who studied abroad for one semester or more completed the GPI pre- and post-tests. Descriptive statistics and paired t-tests were conducted to compare the means. The interviews included 10 students, and were analyzed through Structural coding, Saldaña’s In Vivo coding, and Value coding. Quantitative analyses indicated positive changes in the students’ global awareness and global perspective. These analyses also showed that the FIU students achieved higher post-test means on all the domains of the GPI compared to other studies. Qualitative analyses showed that the students’ experiences incorporated all three global learning outcomes, most notably global awareness and perspective.

Online Education, Accrediting Standards, and Student Success: An Examination of the Relationship Between the Southern Association of Colleges and Schools Commission on Colleges Standards for Online Education and Student Success

Prior to 2000, there were less than 1.6 million students enrolled in at least one online course. By fall 2010, student enrollment in online distance education showed a phenomenal 283% increase to 6.1 million. Two years later, this number had grown to 7.1 million. In light of this significant growth and skepticism about quality, there have been calls for greater oversight of this format of educational delivery. Accrediting bodies tasked with this oversight have developed guidelines and standards for online education. There is a lack of empirical studies that examine the relationship between accrediting standards and student success. The purpose of this study was to examine the relationship between the presence of Southern Association of Colleges and Schools Commission on College (SACSCOC) standards for online education in online courses, (a) student support services and (b) curriculum and instruction, and student success. An original 24-item survey with an overall reliability coefficient of .94 was administered to students (N=464) at Florida International University, enrolled in 24 university-wide undergraduate online courses during fall 2014, who rated the presence of these standards in their online courses. The general linear model was utilized to analyze the data. The results of the study indicated that the two standards, student support services and curriculum and instruction were both significantly and positively correlated with student success but with small R2 and strengths of association less than .35 and .20 respectively. Mixed results were produced from Chi-square tests for differences in student success between higher and lower rated online courses when controlling for various covariates such as discipline, gender, race/ethnicity, GPA, age, and number of online courses previously taken. A multiple linear regression analysis revealed that the curriculum and instruction standard was the only variable that accounted for a significant amount of unique variance in student success. Another regression test revealed that no significant interaction effect exists between the two SACSCOC standards and GPA in predicting student success. The results of this study are useful for administrators, faculty, and researchers who are interested in accreditation standards for online education and how these standards relate to student success.

Nursing faculty intention to use service learning as pedagogy in higher education

The purpose of this study was to identify the factors that motivate nursing faculty to use service learning. The study was based on the theory of planned behavior (TPB), which implies that the target behavior of intention to use service learning in higher education is influenced by the predictor variables of behavior beliefs (attitude), normative beliefs (peer influence), and control beliefs (confidence and resources). External variables were also considered (years of teaching experience, tenure status, and the type of curriculum). Group interviews and a pilot test were conducted to create the instrument for the study, and Cronbach alpha were calculated for survey item reliability. The participants were full time undergraduate nursing faculty members (n=-160) in the Southeastern United States who taught in universities with accredited nurse education programs. Demographic data as well as scores on scaled survey responses were used to evaluate the intention of nursing faculty to use service learning in their classes. Pearson product moment correlation coefficient and path analysis were applied to the data. The correlation findings indicated that there were statistically significant relationships between behavior beliefs, normative beliefs, and control beliefs and nursing faculty intention to use service learning. The path analysis also indicated that behavior beliefs and normative beliefs were significant, while control beliefs were not a strong influence on intention to use service learning. Normative beliefs showed the strongest direct influence. The use of a community based curriculum also had a positive influence on intention, and faculty with tenure status were more likely to have positive behavior beliefs (attitude) towards service learning. Finally, as teaching experience increased, positive attitudes towards the intention to use service learning decreased. Seventy-nine percent of the variation in the intention to use service learning was explained by the theory of planned behavior, the type of curriculum, teaching experience, and tenure status. These results will assist nursing administration and faculty to design strategies to facilitate the implementation of service learning pedagogy, as well as a community based curriculum which will help meet the 21st century goals set forth from the American Association of Colleges of Nursing.

The relationship between higher education curricular variables and human flight performance in a preliminary flying training program

The ability of the United States Air Force (USAF) to sustain a high level of operational ability and readiness is dependent on the proficiency and expertise of its pilots. Recruitment, education, training, and retention of its pilot force are crucial factors in the USAF's attainment of its operational mission: defense of this nation and its allies. Failure of a student pilot during a training program does not only represent a loss of costly training expenditures to the American public, but often consists of loss of human life, aircraft, and property. This research focused on the Air Force Reserve Officer Training Corps' (AFROTC) selection method for student pilots for the light aircraft training (LATR) program. The LATR program is an intense 16 day flight training program that precedes the Air Force's undergraduate pilot training (UPT) program. The study subjects were 265 AFROTC cadets in the LATR program. A variety of independent variables from each subject's higher education curricular background as well as results of preselection tests, participation in varsity athletics, prior flying experience and gender were evaluated against subsequent performance in LATR. Performance was measured by a quantitative performance score developed by this researcher based on 28 graded training factors as well as overall pass or fail of the LATR program. Study results showed participation in university varsity athletics was very significantly and positively related to performance in the LATR program, followed by prior flying experience and to a very slight degree portions of the Air Force Officers Qualifying Test. Not significantly related to success in the LATR program were independent variables such as grade point average, scholastic aptitude test scores, academic major, gender and the AFROTC selection and ranking system.

Quality of student involvement and college contribution toward development of traditional and nontraditional undergraduate students

Nontraditional students differ from traditional students on characteristics such as age, employment status, marital status, and parental status. The quality of a student's experience is important as it relates to his or her transformation and is a reflection of the quality of the college. Using theory of involvement as a framework, the purpose of this study was to test if there were differences between traditional and nontraditional undergraduate students in their ratings of quality of college involvement (academic, co-curricular, student interactions, and faculty interactions) and perceptions of college contribution toward development (intellectual, personal, social, and career). A two part survey was distributed to a random cluster sample of sophomore and higher level undergraduate classes equaling 400 undergraduate students. Results of a 2 X 4 repeated measures ANOVA indicated that traditional students rated quality for co-curricular involvement and student involvement significantly higher than nontraditional students. Both traditional and nontraditional students had similar ratings of college contribution toward development. There were different patterns of correlations between involvement and development. Traditional students' ratings of academic and student involvement were more highly correlated with development than were the ratings of nontraditional students. However, nontraditional students' ratings of academic and faculty involvement were more highly correlated with development. When testing for differences in correlations between quality of involvement and college contribution toward development, the largest observed differences were quality of student involvement and college contribution toward personal and social development. Although not significantly different, traditional students had stronger correlations between those factors than did the nontraditional students. This research demonstrates the importance of using social role when defining student type. It contributes to involvement theory by explaining how traditional and nontraditional students differ in their ratings of quality of involvement. Further, it identifies different patterns of correlations between ratings of quality of involvement and college contribution toward development for the two types of students. While traditional students may need a more rounded college experience that includes more social and co-curricular experiences, nontraditional students use the classroom as their stage for learning.

Managing the multiplicities of graduate level education! Are Master’s students’ expectations matched by their experiences?

With rapid increases in student fees reflecting moves towards a QUASI Market model of Higher Education in the UK and across much of the Western World[1], many universities find themselves having to meet progressively higher levels of student expectations[2]. This is particularly the case at undergraduate level, where increases in fees over the past decade have far exceeded inflation. Yet with so much attention on ‘consumer savvy’ undergraduates, the question of whether Master’s level students’ expectations are matched by their experiences is one which remains largely unanswered. Grounded in an empirically grounded approach to learning and teaching developed by the paper authors[3], this paper sets out to being to answer this question. In doing so it makes a distinctive contribution to debates about graduate level engineering education and concludes with a number of recommendations. Discussion: The ‘MSc: Managing Expectations’ Project analyses the expectations and experiences of Graduate level Engineering Management Students over a two year period. Focusingon the ‘student experience’, three main concepts are identified as being particular relevant to enhancing learning [3]: Relationships: Variety: Synergy. Relationships: Based on empirical research, the significance of Relationships within the academic environment is discussed with particular attention being paid to the value of students’ social and academic support networks, including academic tutoring. Variety: Grounded in a statistical analysis of ‘engagement data’ together with survey and interview findings, the concept of variety critically examines students’ perspectives and experiencesof different approaches to learning and teaching. Synergy: Possibly the most important concept discussed within this paper, the need for constructively aligned curriculum is extended to reflect the students’ apriori knowledge and experienceas well as employer and societal demands and expectations. The conclusion brings the different concepts within the discussion together, providing a set of practical recommendations for colleagues working both at graduate and undergraduate level. References 1.Gibbs, P. (2001) "Higher education as a market: a problem or solution?." Studies in Higher Education 26. 1. pp. 85-94. 2.Tricker, T., (2005) Student Expectations-How do we measure up. University of Sheffield. Available from: http://www.persons.org.uk/tricker%20paper.pdf Accessed 9/10/14 3.Clark, R. & Andrews, J. (2014). Relationships, Variety & Synergy [RVS]: The Vital Ingredients for Scholarship in Engineering Education? A Case-Study. European Journal of Engineering Education. 39.6. pp. 585-600.

To what extent does the Health Professions Admission Test-Ireland predict performance in early undergraduate tests of communication and clinical skills? – An observational cohort study

Background: Internationally, tests of general mental ability are used in the selection of medical students. Examples include the Medical College Admission Test, Undergraduate Medicine and Health Sciences Admission Test and the UK Clinical Aptitude Test. The most widely used measure of their efficacy is predictive validity.A new tool, the Health Professions Admission Test- Ireland (HPAT-Ireland), was introduced in 2009. Traditionally, selection to Irish undergraduate medical schools relied on academic achievement. Since 2009, Irish and EU applicants are selected on a combination of their secondary school academic record (measured predominately by the Leaving Certificate Examination) and HPAT-Ireland score. This is the first study to report on the predictive validity of the HPAT-Ireland for early undergraduate assessments of communication and clinical skills. Method. Students enrolled at two Irish medical schools in 2009 were followed up for two years. Data collected were gender, HPAT-Ireland total and subsection scores; Leaving Certificate Examination plus HPAT-Ireland combined score, Year 1 Objective Structured Clinical Examination (OSCE) scores (Total score, communication and clinical subtest scores), Year 1 Multiple Choice Questions and Year 2 OSCE and subset scores. We report descriptive statistics, Pearson correlation coefficients and Multiple linear regression models. Results: Data were available for 312 students. In Year 1 none of the selection criteria were significantly related to student OSCE performance. The Leaving Certificate Examination and Leaving Certificate plus HPAT-Ireland combined scores correlated with MCQ marks.In Year 2 a series of significant correlations emerged between the HPAT-Ireland and subsections thereof with OSCE Communication Z-scores; OSCE Clinical Z-scores; and Total OSCE Z-scores. However on multiple regression only the relationship between Total OSCE Score and the Total HPAT-Ireland score remained significant; albeit the predictive power was modest. Conclusion: We found that none of our selection criteria strongly predict clinical and communication skills. The HPAT- Ireland appears to measures ability in domains different to those assessed by the Leaving Certificate Examination. While some significant associations did emerge in Year 2 between HPAT Ireland and total OSCE scores further evaluation is required to establish if this pattern continues during the senior years of the medical course.