Fourth generation evaluation revisited: Testing a revised pause model

This research provides data which investigates the feasibility of using fourth generation evaluation during the process of instruction. A semester length course entitled "Multicultural Communications", (PUR 5406/4934) was designed and used in this study, in response to the need for the communications profession to produce well-trained culturally sensitive practitioners for the work force and the market place. A revised pause model consisting of three one-on-one indepth interviews conducted outside of the class, three reflections periods during the class and a self-reflective essay prepared one week before the end of the course was analyzed. Narrative and graphic summaries of participant responses produced significant results. The revised pause model was found to be an effective evaluation method for use in multicultural education under certain conditions as perceived by the participants in the study. participant self-perceived behavior change and knowledge acquisition was identified through use of the revised pause model. Study results suggest that by using the revised pause model of evaluation, instructors teaching multicultural education in schools of journalism and mass communication is yet another way of enhancing their ability to become both the researcher and the research subject. In addition, the introduction of a qualitative model has been found to be a more useful way of generating participant involvement and introspection. Finally, the instructional design of the course used in the study provides communication educators with a practical way of preparing their students be effective communicators in a multicultural world.

An empirical study of Kirkpatrick's evaluation model in the hospitality industry

This study examined Kirkpatrick’s training evaluation model (Kirkpatrick & Kirkpatrick, 2006) by assessing a sales training program conducted at an organization in the hospitality industry. The study assessed the employees’ training outcomes of knowledge and skills, job performance, and the impact of the training upon the organization. By assessing these training outcomes and their relationships, the study demonstrated whether Kirkpatrick’s theories are supported and the lower evaluation levels can be used to predict organizational impact. The population for this study was a group of reservations sales agents from a leading luxury hotel chain’s reservations center. During the study period from January 2005 to May 2007, there were 335 reservations sales agents employed in this Global Reservations Center (GRC). The number of reservations sales agents who had completed a sales training program/intervention during this period and had data available for at least two months pre and post training composed the sample for this study. The number of agents was 69 ( N = 69). Four hypotheses were tested through paired-samples t tests, correlation, and hierarchical regression analytic procedures. Results from the analyses supported the hypotheses in this study. The significant improvement in the call score supported hypothesis one that the reservations sales agents who completed the training improved their knowledge of content and required skills in handling calls (Level 2). Hypothesis two was accepted in part as there was significant improvement in call conversion, but there was no significant improvement of time usage. The significant improvement in the sales per call supported hypothesis three that the reservations agents who completed the training contributed to increased organizational impact (Level 4), i.e., made significantly more sales. Last, findings supported hypothesis four that Level 2 and Level 3 variables can be used for predicting Level 4 organizational impact. The findings supported the theory of Kirkpatrick’s evaluation model that in order to expect organizational results, a positive change in behavior (job performance) and learning must occur. The examinations of Levels 2 and 3 helped to partially explain and predict Level 4 results.

A study of the implementation of junior college *evaluation system in Taiwan

Since the establishment of the evaluation system in 1975, the junior colleges in the Republic of China (Taiwan), have gone through six formal evaluations. We know that evaluation in schooling, like quality control in businesses, should be a systematic, formal, and a continual process. It can doubtless serve as a strategy to refine the quality of education. The purpose of this research is to explore the current practice of junior college evaluation in Taiwan. This provides insight into the development of and quality of the current evaluation system. Moreover, this study also identified the source of problems with the current evaluation system and provided suggestion for improvements.^ In order to attain the above purposes, this research was undertaken in both theoretical and practical ways. First, theoretically, on the basis of a literature review, the theories of educational evaluation and, according to the course and principles of development, a view of the current practice in Taiwan. Secondly, in practice, by means of questionnaires, an analysis of the views of evaluation committeemen, junior college presidents, and administrators were obtained on evaluation models, methods, contents, organization, functions, criteria, grades reports, and others with suggestions for improvement. The summary of findings concludes that most evaluators and evaluatees think the purpose of evaluation can help the colleges explore their difficulties and problems. In addition, it was found that there is significant difference between the two groups regarding the evaluation methods, contents, organization, functions, criteria, grades reports and others, while analyzing these objective data forms the basis for an improved method of evaluation for Junior Colleges in Taiwan. ^

GRE as a predictor of graduate student success at a Hispanic serving institution of higher education

Accurately predicting the success of graduate students is an important aspect of determining which students should be admitted into graduate programs. The GRE is a pivotal factor to examine since it is one of the most widely used criteria for graduate school admission. Even though the GRE is advertised as an accurate tool for predicting first year graduate GPA, there is a lack of research on long term success factors such as time to degree and graduate rate (Luthy, 1996; Powers, 2004). Furthermore, since most studies have low minority sample sizes, the validity of the GRE may not be the same across all groups (ETS, 2008b; Kuncel, Hezlett, & Ones, 2001). Another gap in GRE studies is that few researchers analyze student characteristics, which may alter or moderate the prediction validity of the GRE. Thus, student characteristics such as degree of academic involvement, mentorship interactions, and other academic and social experiences have not been widely examined in this context. These gaps in the analysis of GRE validity are especially relevant given the high attrition rates within of some graduate programs (e.g., an estimated 68% of doctoral student never complete their programs in urban universities; Lovitts, 2001). A sequential mixed methods design was used to answer the research questions in two phases. The quantitative phase used student data files to analyze the relationship of two success variables (graduation rate and graduate GPA) to the GRE scores as well as other academic and demographic graduate student characteristics. The qualitative phase served to complement the first phase by describing a wider range of characteristics from the 11 graduate students who were interviewed. Both proximal and distal moderators influence student behaviors and success in graduate school. In the first phase of the study, the GRE was the distal facilitator under analysis. Findings suggested that both the GRE Quantitative and the GRE Verbal were predictors of success for master’s students, but the GRE Quantitative was not predictive of success for doctoral students. Other student characteristics such as demographic variables and disciplinary area were also predictors of success for the population of students studied. In the second phase of the study, it was inconclusive whether the GRE was predictive of graduate student success; though it did influence access to graduate programs. Furthermore, proximal moderators such as student involvement, faculty/peer interactions, motivational factors, and program structure were perceived to be facilitators and/or detractors for success.

It takes an institution's village to retain a student: A comprehensive look at two early warning system undergraduate retention programs and administrators' perceptions of students' experiences and the retention services they provide students in the early warning system retention programs

Institutions have implemented many campus interventions to address student persistence/retention, one of which is Early Warning Systems (EWS). However, few research studies show evidence of interventions that incorporate noncognitive factors/skills, and psychotherapy/psycho-educational processes in the EWS. A qualitative study (phenomenological interview and document analysis) of EWS at both a public and private 4-year Florida university was conducted to explore EWS through the eyes of the administrators of the ways administrators make sense of students' experiences and the services they provide and do not provide to assist students. Administrators' understanding of noncognitive factors and the executive skills subset and their contribution to retention and the executive skills development of at-risk students were also explored. Hossler and Bean's multiple retention lenses theory/paradigms and Perez's retention strategies were used to guide the study. Six administrators from each institution who oversee and/or assist with EWS for first time in college undergraduate students considered academically at-risk for attrition were interviewed. Among numerous findings, at Institution X: EWS was infrequently identified as a service, EWS training was not conducted, numerous cognitive and noncognitive issues/deficits were identified for students, and services/critical departments such as EWS did not work together to share students' information to benefit students. Assessment measures were used to identify students' issues/deficits; however, they were not used to assess, track, and monitor students' issues/deficits. Additionally, the institution's EWS did address students' executive skills function beyond time management and organizational skills, but did not address students' psychotherapy/psycho-educational processes. Among numerous findings, at Institution Y: EWS was frequently identified as a service, EWS training was not conducted, numerous cognitive and noncognitive issues/deficits were identified for students, and services/critical departments such as EWS worked together to share students' information to benefit students. Assessment measures were used to identify, track, and monitor students' issues/deficits; however, they were not used to assess students' issues/deficits. Additionally, the institution's EWS addressed students' executive skills function beyond time management and organizational skills, and psychotherapy/psycho-educational processes. Based on the findings, Perez's retention strategies were not utilized in EWS at Institution X, yet were collectively utilized in EWS at Institution Y, to achieve Hossler and Bean's retention paradigms. Future research could be designed to test the link between engaging in the specific promising activities identified in this research (one-to-one coaching, participation in student success workshops, academic contracts, and tutoring) and student success (e.g., higher GPA, retention). Further, because this research uncovered some concern with how to best handle students with physical and psychological disabilities, future research could link these same promising strategies for improving student performance for example among ADHD students or those with clinical depression.

The relationship between selected standardized test scores and performance in advanced placement math and science exams: Analyzing the differential effectiveness of scores for course identification and placement

There is a national need to increase the STEM-related workforce. Among factors leading towards STEM careers include the number of advanced high school mathematics and science courses students complete. Florida's enrollment patterns in STEM-related Advanced Placement (AP) courses, however, reveal that only a small percentage of students enroll into these classes. Therefore, screening tools are needed to find more students for these courses, who are academically ready, yet have not been identified. The purpose of this study was to investigate the extent to which scores from a national standardized test, Preliminary Scholastic Assessment Test/ National Merit Qualifying Test (PSAT/NMSQT), in conjunction with and compared to a state-mandated standardized test, Florida Comprehensive Assessment Test (FCAT), are related to selected AP exam performance in Seminole County Public Schools. An ex post facto correlational study was conducted using 6,189 student records from the 2010 - 2012 academic years. Multiple regression analyses using simultaneous Full Model testing showed differential moderate to strong relationships between scores in eight of the nine AP courses (i.e., Biology, Environmental Science, Chemistry, Physics B, Physics C Electrical, Physics C Mechanical, Statistics, Calculus AB and BC) examined. For example, the significant unique contribution to overall variance in AP scores was a linear combination of PSAT Math (M), Critical Reading (CR) and FCAT Reading (R) for Biology and Environmental Science. Moderate relationships for Chemistry included a linear combination of PSAT M, W (Writing) and FCAT M; a combination of FCAT M and PSAT M was most significantly associated with Calculus AB performance. These findings have implications for both research and practice. FCAT scores, in conjunction with PSAT scores, can potentially be used for specific STEM-related AP courses, as part of a systematic approach towards AP course identification and placement. For courses with moderate to strong relationships, validation studies and development of expectancy tables, which estimate the probability of successful performance on these AP exams, are recommended. Also, findings established a need to examine other related research issues including, but not limited to, extensive longitudinal studies and analyses of other available or prospective standardized test scores.

Online education, accrediting standards, and student success: An examination of the relationship between the southern association of colleges and schools commission on colleges standards for online education and student success

Prior to 2000, there were less than 1.6 million students enrolled in at least one online course. By fall 2010, student enrollment in online distance education showed a phenomenal 283% increase to 6.1 million. Two years later, this number had grown to 7.1 million. In light of this significant growth and skepticism about quality, there have been calls for greater oversight of this format of educational delivery. Accrediting bodies tasked with this oversight have developed guidelines and standards for online education. ^ There is a lack of empirical studies that examine the relationship between accrediting standards and student success. The purpose of this study was to examine the relationship between the presence of Southern Association of Colleges and Schools Commission on College (SACSCOC) standards for online education in online courses, (a) student support services and (b) curriculum and instruction, and student success. An original 24-item survey with an overall reliability coefficient of .94 was administered to students (N=464) at Florida International University, enrolled in 24 university-wide undergraduate online courses during fall 2014, who rated the presence of these standards in their online courses. The general linear model was utilized to analyze the data. The results of the study indicated that the two standards, student support services and curriculum and instruction were both significantly and positively correlated with student success but with small R2 and strengths of association less than .35 and .20 respectively. Mixed results were produced from Chi-square tests for differences in student success between higher and lower rated online courses when controlling for various covariates such as discipline, gender, race/ethnicity, GPA, age, and number of online courses previously taken. A multiple linear regression analysis revealed that the curriculum and instruction standard was the only variable that accounted for a significant amount of unique variance in student success. Another regression test revealed that no significant interaction effect exists between the two SACSCOC standards and GPA in predicting student success. ^ The results of this study are useful for administrators, faculty, and researchers who are interested in accreditation standards for online education and how these standards relate to student success.^

Evaluation of a six-week professional education curriculum designed to retain registered psychiatric nurses

During the past few years there has been a drastic shortage of registered nurses in the field. The shortage appears to have affected the field of psychiatric mental health nursing most intensely. The psychiatric nursing shortage is a multifaceted problem grounded in decreasing federal funds for advanced clinical training, inadequate undergraduate psychiatric experiences, lack of a well prepared articulate role model, the integrated curriculum and the confusion and blurring associated with the roles and functions of the psychiatric mental health nurse.^ This dissertation will describe the current nursing shortage; the decline in enrollment to nursing programs; the history of psychiatric nursing as a discipline; the shortage of psychiatric mental health nurses; factors contributing to the psychiatric nursing shortage and a plan for a solution to the nursing shortage in psychiatry.^ The paper focuses on an evaluation conducted on an internship curriculum designed to facilitate effective nursing care in the treatment of clients who exhibit emotional problems. The purpose of this study was to attract and retain nurses to employment opportunities in four Hospital Corporation of America (HCA) facilities, using a six week internship program.^ The study will yield an analysis of the effect of combining psychodynamic principles and knowledge with skills in the clinical area. The demands of educational practice have been merged with the discipline of psychiatric nursing in the development of this curriculum. ^

The effect of mentoring electronic mail on student achievement and attitudes in a graduate course in educational research

This exploratory study of a classroom with mentoring and neutral e-mail was conducted in a public commuter state university in South Florida between January 1996 and April 1996. Sixteen males and 83 females from four graduate level educational research classes participated in the study.^ Two main hypotheses were tested. Hypothesis One was that those students receiving mentoring e-mail messages would score significantly higher on an instrument measuring attitude toward educational research (ATERS) than those not receiving mentoring e-mail messages. Hypothesis Two was that those students receiving mentoring e-mail would score significantly higher on objective exams covering the educational research material than those not receiving mentoring e-mail.^ Results of factorial analyses of variance showed no significant differences between the treatment groups in achievement or in attitudes toward educational research. Introverts had lower attitudes and lower final exam grades in both groups, although introverts in the mentored group scored higher than those introverts in the neutral group.^ A t test of the means of total response to e-mail from the researcher showed a significant difference between the mentored and neutral e-mail groups. Introverts responded more often than extraverts in both groups.^ Teacher effect was significant in determining class response to e-mail messages. Responses were most frequent in the researcher's classes.^ Qualitative analyses of the e-mail and course evaluation survey and of the content of e-mail messages received by the researcher were then grouped into basic themes and discussed.^ A qualitative analysis of an e-mail and course evaluation survey revealed that students from both the neutral and mentoring e-mail groups appreciated teacher feedback. A qualitative analysis of the mentoring and neutral e-mail replies divided the responses into those pertaining to the class, such as test and research paper questions, and more personal items, such as problems in the class and personal happenings.^ At this point in time, e-mail is not a standard way of communicating in classes in the college of education at this university. As this technology tool of communication becomes more popular, it is anticipated that replications of this study will be warranted. ^

An evaluation of English language instruction in universities/colleges in Taiwan

The purpose of this study is about to survey the effectiveness of English teaching in universities/colleges in Taiwan and provide suggestions to the educational authorities for upgrading or improving the English language teaching techniques or methods currently in use. On the one hand, the suggestions of English teachers in universities/colleges are the main concerns in this study. On the other hand, equal attention is given to the opinions and learning attitudes of the students, including the interaction between the teachers and the students.^ A social investigation methodology, consisting of questionnaire and data analysis, is devoted to this evaluation and analysis of teaching effectiveness. Among the significant findings reported in this study is the fact that both students and teachers consider evaluation of teaching effectiveness as a very important aspect of English Language programs at the university/college level. Another significant finding is that students are less concerned about the grades earned or awarded in English Language courses and more concerned about how much they learned in these classes.^ The analysis and suggestions based on the results for English teaching evaluation of universities/colleges in this dissertation can help teachers modify their teaching techniques and methods, the students to adjust their learning attitudes, and the authorities to schematize the English teaching plan that can facilitate the teaching goals to be realized. ^

Parent involvement on school advisory councils: A process evaluation at the middle school level

Increasing parental involvement was made an important goal for all Florida schools in educational reform legislation in the 1990's. A forum for this input was established and became known as the School Advisory Council (SAC). To demonstrate the importance of process and inclusion, a south Florida school district and its local teacher's union agreed on the following five goals for SACs: (a) to foster an environment of professional collaboration among all stakeholders, (b) to assist in the preparation and evaluation of the school improvement plan, (c) to address all state and district goals, (d) to serve as the avenue for authentic and representative input from all stakeholders, and (e) to ensure the continued existence of the consensus-building process on all issues related to the school's instructional program. ^ The purpose of this study was to determine to what extent and in what ways the parent members of one south Florida middle school's SAC achieved the five district goals during its first three years of implementation. The primary participants were 16 parents who served as members of the SAC, while 16 non-parent members provided perspective on parent involvement as “outside sources.” Being qualitative by design, factors such as school climate, leadership styles, and the quality of parental input were described from data collected from four sources: parent interviews, a questionnaire of non-parents, researcher observations, and relevant documents. A cross-case analysis of all data informed a process evaluation that described the similarities and differences of intended and observed outcomes of parent involvement from each source using Stake's descriptive matrix model. A formative evaluation of the process compared the observed outcomes with standards set for successful SACs, such as the district's five goals. ^ The findings indicated that parents elected to the SACs did not meet the intended goals set by the state and district. The school leadership did not foster an environment of professional collaboration and authentic decision-making for parents and other stakeholders. The overall process did not include consensus-building, and there was little if any input by parents on school improvement and other important issues relating to the instructional program. Only two parents gave the SAC a successful rating for involving parents in the decision-making process. Although compliance was met in many of the procedural transactions of the SAC, the reactions of parents to their perceived role and influence often reflected feelings of powerlessness and frustration with a process that many thought lacked meaningfulness and productivity. Two conclusions made from this study are as follows: (a) that the role of the principal in the collaborative process is pivotal, and (b) that the normative-re-educative approach to change would be most appropriate for SACs. ^

A follow-up outcome evaluation of independent living services for foster care youth

This study aimed to evaluate the effectiveness of the Independent Living (IL) program targeting foster youths to prepare them for effective adulthood. The study employed a pre-post, two-group comparative research design. The IL group was composed of 49 young adults who participated in the IL or SIL (subsidized IL) program prior to their emancipation from foster care (mean age, 20.6 years). The comparison was made up of 18 young adults who experienced foster care but never participated in the IL or SIL programs (mean age, 20.2 years). Data were collected via a mailed survey that included the Daniel Memorial Independent Living Assessment (DMILA) and an additional questionnaire developed by the researcher. The study also examined: (1) why youth in foster care do not participate in IL programs, (2) how participating youth evaluate IL services and what recommendations they make to improve services, and (3) the internal consistency of the DMILA. Results suggest that the DMILA assessment has mediocre reliability. IL program participation is associated with better educational, employment, income, housing, early parenting-prevention, transportation, anger control, criminal-prevention, and self-evaluation outcomes. However, IL participation is not associated with better social support, perceived parenting competence, substance abuse-prevention, sexual risk-prevention outcomes, increased knowledge in money management skills, job seeking and job maintenance skills, interpersonal skills, or lower depression. Results also suggest that the outreach activities of the IL program may be flawed. IL participants reported the IL program was doing best in educational preparation, criminal involvement prevention, and money management preparation and least well in parenting preparation, housing preparation, employment preparation, and substance abuse prevention. To improve services, youths recommended primarily that IL counselors develop closer relationships with youths, that IL training better address organizational skills, and that monthly subsidy be raised and SIL eligibility requirements softened. The study's political context and limitations are also discussed and implications are derived regarding prevention, intervention, outreach, mentorship, empowerment, cross-systems collaboration, and future research. ^

A study of the relationship between educational placement and the achievement of urban low socioeconomic Hispanic middle school students with and without specific learning disabilities

Public schools traditionally have been held accountable for educating the majority of the nation’s school children, and through the years, these schools have been evaluated in a variety of ways. Currently, evaluation measures for accountability purposes consist solely of standardized test scores. In the past, only test scores of general education students were analyzed. Laws governing the education of students with disabilities, however, have extended accountability measures not only to include those students, but to report their scores in a disaggregated form (No Child Left Behind Act, 2001). The recent emphasis on accountability and compliance has resulted in the need for schools to carefully examine how programs, services, and policies impact student achievement (Bowers & Figgers, 2003). ^ Standard-based school reform and accountability systems have raised expectations about student learning outcomes for all students, including those with disabilities and minority students. Yet, overall, racial/ethnic minority students are performing well below their White non-Hispanic peers in most academic areas. Additionally, with respect to special education, there exists an enduring problem of disproportionate representation of racial/ethnic minority students (National Research Council, 2000). ^ This study examined classroom placement (inclusive versus non-inclusive) relative to academic performance of urban, low socioeconomic Hispanic students with and without disabilities in secondary content area classrooms. A mixed method research design was used to investigate this important issue using data from a local school district and results from field observations. The study compared performance levels of four middle school Hispanic student subgroups (students with disabilities in inclusive settings, students without disabilities in inclusive settings, students with disabilities in resource settings, and student without disabilities in general education settings) each in their respective placements for two consecutive years, exploring existing practices within authentic settings. ^ Significant differences were found in the relationship of educational placement and achievement between grade level and disability in the areas of math and reading. Additionally, clear and important differences were observed in student-teacher interactions. Recommendations for further researchers and stakeholders include soliciting responses from teams at the schools composed of general education and special education teachers, administrative personnel, and students as well as broadening the study across grade levels and exceptionalities. ^

The early development and pilot -testing of a case management intervention with victims of stalking: An evaluation study

This study evaluated the early development and pilot-testing of Project IMPACT, a case management intervention for victims of stalking. The Design and Development framework (Rothman & Thomas, 1994) was used as a guide for program development and evaluation. Nine research questions examined the processes and outcomes associated with program implementation. ^ The sample included all 36 clients who participated in Project IMPACT between February of 2000 and June of 2001, as well as the victim advocates who provided them with services. Quantitative and qualitative data were drawn from client case files, participant observation field notes and interview transcriptions. Quantitative data were entered into three databases where: (1) clients were the units of analysis (n = 36), (2) services were the units of analysis (n = 1146), and (3) goals were the units of analysis (n = 149). These data were analyzed using descriptive statistics, Pearson's Chi-square, Spearman's Rho, Phi, Cramer's V, Wilcoxon's Matched Pairs Signed-Ranked Test and McNemar's Test Statistic. Qualitative data were reduced via open, axial and selective coding methods. Grounded theory and case study frameworks were utilized to analyze these data. ^ Results showed that most clients noted an improved sense of well-being and safety, although residual symptoms of trauma remained for numerous individuals. Stalkers appeared to respond to criminal and civil justice-based interventions by reducing violent and threatening behaviors; however, covert behaviors continued. The study produced findings that provided preliminary support for the use of several intervention components including support services, psycho-education, safety planning, and boundary spanning. The psycho-education and safety planning in particular seemed to help clients cognitively reframe their perceptions of the stalking experience and gain a sense of increased safety and well-being. A 65% level of satisfactory goal achievement was observed overall, although goals involving justice-based organizations were associated with lower achievement. High service usage was related to low-income clients and those lacking in social support. Numerous inconsistencies in program implementation were found to be associated with the skills and experiences of victim advocates. Thus, recommendations were made to further refine, develop and evaluate the intervention. ^

TEACHNJ: An Evaluation of Two Years Implementation

The Teacher Effectiveness and Accountability for the Children of New Jersey (TEACHNJ) Act was adopted by the New Jersey legislature in August 2012 with the intent to raise student achievement by improving the overall quality of instruction. As a result of this act, new teacher evaluation systems are being introduced in school districts across the state in an effort to more accurately assess teacher performance. The new teacher evaluations will be based on multiple classroom observations as well as the academic achievement of their students as measured on standardized tests. In addition, professional development opportunities are likely to change under this legislation, with schools customizing professional development programs to more effectively meet the needs of their teachers. The overarching question that informs our research is what impact will TEACH NJ have on the overall value of teacher evaluations and the quality of professional development opportunities offered to teachers. Data collected through survey research presents the pre-implementation practices (2011-2012 school year) as well as one year post-implementation practices (2013-2014) taking place in school districts throughout New Jersey. The findings reflect teachers’ perceptions of the value of the current teacher evaluation practices, the quality of the current professional development opportunities and the value the school administration places on teacher evaluations.