Assessing the effectiveness of five-year mechanical engineering technology programs of junior colleges in Taiwan, R.O.C.: An application of the CIPP evaluation model

The purpose of this study was to evaluate the mechanical engineering technology curriculum effectiveness at the junior college in Taiwan by using the CIPP evaluation model. The study concerned the areas of the curriculum, curriculum materials, individualized instruction, support services, teaching effectiveness, student achievement, and job performance. A descriptive survey method was used with questionnaires for data collection from faculty, students, graduates, and employers.^ All categories of respondents tended to agree that the curriculum provides appropriate occupational knowledge and skills. Students, graduates, and faculty tended to be satisfied with the curriculum; faculty tended to be satisfied with student achievement; graduates tended to be satisfied with their job preparation; and employers were most satisfied with graduates' job performance.^ Conclusions were drawn in the context, input, process, and product of the CIPP model. In Context area: Students were dissatisfied with curriculum flexibility in students characteristics. Graduates were dissatisfied with curriculum design for student's adaptability in new economic and industrial conditions; practicum flexibility in student characteristics; and course overlap. Both students and graduates were dissatisfied with practicum credit hours. Both faculty and students were dissatisfied with the number of required courses.^ In Input area: Students, faculty, and graduates perceived audiovisuals and manipulative aids positively. Faculty and students perceive CAI implementation positively. Students perceived textbooks negatively.^ In Process area: Faculty, students, and graduates perceived all support service negatively. Faculty tended to perceive the ratios of graduates who enter advanced study and related occupation, and who passed the professional skills certification, negatively. Students tended to perceive teaching effectiveness in terms of instructional strategies, the quality of instruction, overall suitability, and receivable, negatively. Graduates also tended to identify the instructional strategies as a negative perception. Faculty and students perceived curriculum objectives and practicum negatively. Both faculty and students felt that instructors should be more interested in making the courses a useful learning experience.^ In Product area: Employers were satisfied with graduates' academic preparation and job performance, adaptability, punctuality, and their ability to communicate, cooperate, and meet organization needs. Graduates were weak in terms of equipment familiarity and supervisory ability.^ In sum, the curriculum of the five-year mechanical engineering technology programs of junior college in Taiwan has served adequately up to this time in preparing a work force to enter industry. It is now time to look toward the future and adapt the curriculum and instruction for the future needs of this high-tech society. ^

Assessing the effectiveness of two-year banking and insurance technology programs of junior colleges in Taiwan, R.O.C.: An application of the Context Input Process Product (CIPP) evaluation model

The purpose of this descriptive study was to evaluate the banking and insurance technology curriculum at ten junior colleges in Taiwan. The study focused on curriculum, curriculum materials, instruction, support services, student achievement and job performance. Data was collected from a diverse sample of faculty, students, alumni, and employers. ^ Questionnaires on the evaluation of curriculum at technical junior colleges were developed for use in this specific case. Data were collected from the sample described above and analyzed utilizing ANOVA, T-Tests and crosstabulations. Findings are presented which indicate that there is room for improvement in terms of meeting individual students' needs. ^ Using Stufflebeam's CIPP model for curriculum evaluation it was determined that the curriculum was adequate in terms of the knowledge and skills imparted to students. However, students were dissatisfied with the rigidity of the curriculum and the lack of opportunity to satisfy the individual needs of students. Employers were satisfied with both the academic preparation of students and their on the job performance. ^ In sum, the curriculum of the two-year banking and insurance technology programs of junior college in Taiwan was shown to have served adequately preparing a work force to enter businesses. It is now time to look toward the future and adapt the curriculum and instruction for the future needs of the ever evolving high-tech society. ^

An empirical study of Kirkpatrick's evaluation model in the hospitality industry

This study examined Kirkpatrick’s training evaluation model (Kirkpatrick & Kirkpatrick, 2006) by assessing a sales training program conducted at an organization in the hospitality industry. The study assessed the employees’ training outcomes of knowledge and skills, job performance, and the impact of the training upon the organization. By assessing these training outcomes and their relationships, the study demonstrated whether Kirkpatrick’s theories are supported and the lower evaluation levels can be used to predict organizational impact. The population for this study was a group of reservations sales agents from a leading luxury hotel chain’s reservations center. During the study period from January 2005 to May 2007, there were 335 reservations sales agents employed in this Global Reservations Center (GRC). The number of reservations sales agents who had completed a sales training program/intervention during this period and had data available for at least two months pre and post training composed the sample for this study. The number of agents was 69 ( N = 69). Four hypotheses were tested through paired-samples t tests, correlation, and hierarchical regression analytic procedures. Results from the analyses supported the hypotheses in this study. The significant improvement in the call score supported hypothesis one that the reservations sales agents who completed the training improved their knowledge of content and required skills in handling calls (Level 2). Hypothesis two was accepted in part as there was significant improvement in call conversion, but there was no significant improvement of time usage. The significant improvement in the sales per call supported hypothesis three that the reservations agents who completed the training contributed to increased organizational impact (Level 4), i.e., made significantly more sales. Last, findings supported hypothesis four that Level 2 and Level 3 variables can be used for predicting Level 4 organizational impact. The findings supported the theory of Kirkpatrick’s evaluation model that in order to expect organizational results, a positive change in behavior (job performance) and learning must occur. The examinations of Levels 2 and 3 helped to partially explain and predict Level 4 results.

Instructional change through collaboration and reflection

The purpose of this study was to investigate the effectiveness of the Facilitator-Collaborative-Reflective Model, a strategic plan for changing teachers' practices and beliefs, on a selected group of middle school teachers. This model of staff development training was based on Corey's (1953) Cooperative-Action-Research Model and Anders and Richardson (1994) Collaborative-Reflective Model. It supports the notion that earning a teachers' commitment to change by focusing on collaboration, reflection and the normative-re-educative process aids in altering teachers' beliefs and practices especially crucial to the change process. The year-long training provided for reflection, inquiry, and learning that was useful to teachers as they pursued their goals with their students. The lead teacher, as a change agent and transformational leader, assisted in this commitment to change by improving the teachers' self-concepts as they slowly changed. The collaborative and receptive environment of the staff development fostered acceptance and stimulation of ideas.^ Given the collaborative nature of the change process, qualitative research methods were used in the investigation. The research process was based on Stufflebeam's Context, Input, Process and Product (CIPP) Evaluation Model (Madaus, Scriven & Stufflebeam, 1983). This allowed for all three factors of the staff development model to be evaluated. The case studies and focus group were effective in discerning any actual change in practices or beliefs.^ The findings of the qualitative evaluation, consisting of a baseline survey, case studies, questionnaire and modified focus group interviews, concluded that all of the teachers were strongly influenced by the intervention model which was the subject of this study. From this evaluation, three distinct indicators were looked at to determine if any change in the teachers' practices and beliefs emerged: (1) change in practice and belief, (2) reflective feedback and (3) collaborative reflection. These indicators were common throughout the teacher responses thus substantiating the infusion of the Facilitator-Collaborative-Reflective Model at the school level for effective staff development. ^

A curricular analysis of the servicing of students who are hearing impaired in Dade County, Florida

Deafness is an invisible handicap which makes education difficult due to its effect on language acquisition and development, speech acquisition, and ability to communicate with others. Teachers of students who are hearing impaired have been at odds for more than a century as to the best method of communication to use in teaching students who are hearing impaired, and in providing these students with communication systems that will enable them to be effective communicators. This dissertation utilized qualitative research methods to analyze whether the Dade County Public Schools Procedures for Providing Special Education for Exceptional Students-Hearing Impaired (DCPS), (1976-77; 1992-93) show evidence of an appropriate curriculum and instructional program that is responsive to the conditions facing exceptional education students identified as hearing impaired. Results indicate that many of the curriculum and instructional program requirements are not identified nor described when analyzed by Tyler's "Rationale" for Curriculum and Stufflebeam's Improvement-Oriented Evaluation Model, better known as Stufflebeam's CIPP Model. Recommendations to improve and enhance programs for students who are hearing impaired are offered based on this analysis. ^

An investigation of multiple pathways of developmental intervention change

Convergence among treatment, prevention, and developmental intervention approaches has led to the recognition of the need for evaluation models and research designs that employ a full range of evaluation information to provide an empirical basis for enhancing the efficiency, efficacy, and effectiveness of prevention and positive development interventions. This study reports an investigation of a positive youth development program using an Outcome Mediation Cascade (OMC) evaluation model, an integrated model for evaluating the empirical intersection between intervention and developmental processes. The Changing Lives Program (CLP) is a community supported positive youth development intervention implemented in a practice setting as a selective/indicated program for multi-ethnic, multi-problem at risk youth in urban alternative high schools. This study used a Relational Data Analysis integration of quantitative and qualitative data analysis strategies, including the use of both fixed and free response measures and a structural equation modeling approach, to construct and evaluate the hypothesized OMC model. Findings indicated that the hypothesized model fit the data (χ2 (7) = 6.991, p = .43; RMSEA = .00; CFI = 1.00; WRMR = .459). Findings also provided preliminary evidence consistent with the hypothesis that in addition to having effects on targeted positive outcomes, PYD interventions are likely to have progressive cascading effects on untargeted problem outcomes that operate through effects on positive outcomes. Furthermore, the general pattern of findings suggested the need to use methods capable of capturing both quantitative and qualitative change in order to increase the likelihood of identifying more complete theory informed empirically supported models of developmental intervention change processes.

A developmental intervention science outreach research approach to promoting positive youth development

Recent intervention efforts in promoting positive identity in troubled adolescents have begun to draw on the potential for an integration of the self-construction and self-discovery perspectives in conceptualizing identity processes, as well as the integration of quantitative and qualitative data analytic strategies. This study reports an investigation of the Changing Lives Program (CLP), using an Outcome Mediation (OM) evaluation model, an integrated model for evaluating targets of intervention, while theoretically including a Self-Transformative Model of Identity Development (STM), a proposed integration of self-discovery and self-construction identity processes. This study also used a Relational Data Analysis (RDA) integration of quantitative and qualitative analysis strategies and a structural equation modeling approach (SEM), to construct and evaluate the hypothesized OM/STM model. The CLP is a community supported positive youth development intervention, targeting multi-problem youth in alternative high schools in the Miami Dade County Public Schools (M-DCPS). The 259 participants for this study were drawn from the CLP’s archival data file. The model evaluated in this study utilized three indices of core identity processes (1) personal expressiveness, (2) identity conflict resolution, and (3) informational identity style that were conceptualized as mediators of the effects of participation in the CLP on change in two qualitative outcome indices of participants’ sense of self and identity. Findings indicated the model fit the data (χ2 (10) = 3.638, p = .96; RMSEA = .00; CFI = 1.00; WRMR = .299). The pattern of findings supported the utilization of the STM in conceptualizing identity processes and provided support for the OM design. The findings also suggested the need for methods capable of detecting and rendering unique sample specific free response data to increase the likelihood of identifying emergent core developmental research concepts and constructs in studies of intervention/developmental change over time in ways not possible using fixed response methods alone.

Cognitive competence and life course change in multi-problem adolescents

The dissertation reports on two studies. The purpose of Study I was to develop and evaluate a measure of cognitive competence (the Critical Problem Solving Skills Scale – Qualitative Extension) using Relational Data Analysis (RDA) with a multi-ethnic, adolescent sample. My study builds on previous work that has been conducted to provide evidence for the reliability and validity of the RDA framework in evaluating youth development programs (Kurtines et al., 2008). Inter-coder percent agreement among the TOC and TCC coders for each of the category levels was moderate to high, with a range of .76 to .94. The Fleiss' kappa across all category levels was from substantial agreement to almost perfect agreement, with a range of .72 to .91. The correlation between the TOC and the TCC demonstrated medium to high correlation, with a range of r(40)=.68, p<.001 to r(40)=.79, p<.001. Study II reports an investigation of a positive youth development program using an Outcome Mediation Cascade (OMC) evaluation model, an integrated model for evaluating the empirical intersection between intervention and developmental processes. The Changing Lives Program (CLP) is a community supported positive youth development intervention implemented in a practice setting as a selective/indicated program for multi-ethnic, multi-problem at risk youth in urban alternative high schools in the Miami Dade County Public Schools (M-DCPS). The 259 participants for this study were drawn from the CLP's archival data file. The study used a structural equation modeling approach to construct and evaluate the hypothesized model. Findings indicated that the hypothesized model fit the data (χ2 (7) = 5.651, p = .83; RMSEA = .00; CFI = 1.00; WRMR = .319). My study built on previous research using the OMC evaluation model (Eichas, 2010), and the findings are consistent with the hypothesis that in addition to having effects on targeted positive outcomes, PYD interventions are likely to have progressive cascading effects on untargeted problem outcomes that operate through effects on positive outcomes.

Cognitive Competence and Life Course Change in Multi-Problem Adolescents

The dissertation reports on two studies. The purpose of Study I was to develop and evaluate a measure of cognitive competence (the Critical Problem Solving Skills Scale – Qualitative Extension) using Relational Data Analysis (RDA) with a multi-ethnic, adolescent sample. My study builds on previous work that has been conducted to provide evidence for the reliability and validity of the RDA framework in evaluating youth development programs (Kurtines et al., 2008). Inter-coder percent agreement among the TOC and TCC coders for each of the category levels was moderate to high, with a range of .76 to .94. The Fleiss’ kappa across all category levels was from substantial agreement to almost perfect agreement, with a range of .72 to .91. The correlation between the TOC and the TCC demonstrated medium to high correlation, with a range of r(40)=.68, p Study II reports an investigation of a positive youth development program using an Outcome Mediation Cascade (OMC) evaluation model, an integrated model for evaluating the empirical intersection between intervention and developmental processes. The Changing Lives Program (CLP) is a community supported positive youth development intervention implemented in a practice setting as a selective/indicated program for multi-ethnic, multi-problem at risk youth in urban alternative high schools in the Miami Dade County Public Schools (M-DCPS). The 259 participants for this study were drawn from the CLP’s archival data file. The study used a structural equation modeling approach to construct and evaluate the hypothesized model. Findings indicated that the hypothesized model fit the data (χ2 (7) = 5.651, p = .83; RMSEA = .00; CFI = 1.00; WRMR = .319). My study built on previous research using the OMC evaluation model (Eichas, 2010), and the findings are consistent with the hypothesis that in addition to having effects on targeted positive outcomes, PYD interventions are likely to have progressive cascading effects on untargeted problem outcomes that operate through effects on positive outcomes.

Fourth generation evaluation revisited: Testing a revised pause model

This research provides data which investigates the feasibility of using fourth generation evaluation during the process of instruction. A semester length course entitled "Multicultural Communications", (PUR 5406/4934) was designed and used in this study, in response to the need for the communications profession to produce well-trained culturally sensitive practitioners for the work force and the market place. A revised pause model consisting of three one-on-one indepth interviews conducted outside of the class, three reflections periods during the class and a self-reflective essay prepared one week before the end of the course was analyzed. Narrative and graphic summaries of participant responses produced significant results. The revised pause model was found to be an effective evaluation method for use in multicultural education under certain conditions as perceived by the participants in the study. participant self-perceived behavior change and knowledge acquisition was identified through use of the revised pause model. Study results suggest that by using the revised pause model of evaluation, instructors teaching multicultural education in schools of journalism and mass communication is yet another way of enhancing their ability to become both the researcher and the research subject. In addition, the introduction of a qualitative model has been found to be a more useful way of generating participant involvement and introspection. Finally, the instructional design of the course used in the study provides communication educators with a practical way of preparing their students be effective communicators in a multicultural world.