Fourth generation evaluation revisited: Testing a revised pause model

This research provides data which investigates the feasibility of using fourth generation evaluation during the process of instruction. A semester length course entitled "Multicultural Communications", (PUR 5406/4934) was designed and used in this study, in response to the need for the communications profession to produce well-trained culturally sensitive practitioners for the work force and the market place. A revised pause model consisting of three one-on-one indepth interviews conducted outside of the class, three reflections periods during the class and a self-reflective essay prepared one week before the end of the course was analyzed. Narrative and graphic summaries of participant responses produced significant results. The revised pause model was found to be an effective evaluation method for use in multicultural education under certain conditions as perceived by the participants in the study. participant self-perceived behavior change and knowledge acquisition was identified through use of the revised pause model. Study results suggest that by using the revised pause model of evaluation, instructors teaching multicultural education in schools of journalism and mass communication is yet another way of enhancing their ability to become both the researcher and the research subject. In addition, the introduction of a qualitative model has been found to be a more useful way of generating participant involvement and introspection. Finally, the instructional design of the course used in the study provides communication educators with a practical way of preparing their students be effective communicators in a multicultural world.

Assessing the effectiveness of five-year mechanical engineering technology programs of junior colleges in Taiwan, R.O.C.: An application of the CIPP evaluation model

The purpose of this study was to evaluate the mechanical engineering technology curriculum effectiveness at the junior college in Taiwan by using the CIPP evaluation model. The study concerned the areas of the curriculum, curriculum materials, individualized instruction, support services, teaching effectiveness, student achievement, and job performance. A descriptive survey method was used with questionnaires for data collection from faculty, students, graduates, and employers.^ All categories of respondents tended to agree that the curriculum provides appropriate occupational knowledge and skills. Students, graduates, and faculty tended to be satisfied with the curriculum; faculty tended to be satisfied with student achievement; graduates tended to be satisfied with their job preparation; and employers were most satisfied with graduates' job performance.^ Conclusions were drawn in the context, input, process, and product of the CIPP model. In Context area: Students were dissatisfied with curriculum flexibility in students characteristics. Graduates were dissatisfied with curriculum design for student's adaptability in new economic and industrial conditions; practicum flexibility in student characteristics; and course overlap. Both students and graduates were dissatisfied with practicum credit hours. Both faculty and students were dissatisfied with the number of required courses.^ In Input area: Students, faculty, and graduates perceived audiovisuals and manipulative aids positively. Faculty and students perceive CAI implementation positively. Students perceived textbooks negatively.^ In Process area: Faculty, students, and graduates perceived all support service negatively. Faculty tended to perceive the ratios of graduates who enter advanced study and related occupation, and who passed the professional skills certification, negatively. Students tended to perceive teaching effectiveness in terms of instructional strategies, the quality of instruction, overall suitability, and receivable, negatively. Graduates also tended to identify the instructional strategies as a negative perception. Faculty and students perceived curriculum objectives and practicum negatively. Both faculty and students felt that instructors should be more interested in making the courses a useful learning experience.^ In Product area: Employers were satisfied with graduates' academic preparation and job performance, adaptability, punctuality, and their ability to communicate, cooperate, and meet organization needs. Graduates were weak in terms of equipment familiarity and supervisory ability.^ In sum, the curriculum of the five-year mechanical engineering technology programs of junior college in Taiwan has served adequately up to this time in preparing a work force to enter industry. It is now time to look toward the future and adapt the curriculum and instruction for the future needs of this high-tech society. ^

Assessing the effectiveness of two-year banking and insurance technology programs of junior colleges in Taiwan, R.O.C.: An application of the Context Input Process Product (CIPP) evaluation model

The purpose of this descriptive study was to evaluate the banking and insurance technology curriculum at ten junior colleges in Taiwan. The study focused on curriculum, curriculum materials, instruction, support services, student achievement and job performance. Data was collected from a diverse sample of faculty, students, alumni, and employers. ^ Questionnaires on the evaluation of curriculum at technical junior colleges were developed for use in this specific case. Data were collected from the sample described above and analyzed utilizing ANOVA, T-Tests and crosstabulations. Findings are presented which indicate that there is room for improvement in terms of meeting individual students' needs. ^ Using Stufflebeam's CIPP model for curriculum evaluation it was determined that the curriculum was adequate in terms of the knowledge and skills imparted to students. However, students were dissatisfied with the rigidity of the curriculum and the lack of opportunity to satisfy the individual needs of students. Employers were satisfied with both the academic preparation of students and their on the job performance. ^ In sum, the curriculum of the two-year banking and insurance technology programs of junior college in Taiwan was shown to have served adequately preparing a work force to enter businesses. It is now time to look toward the future and adapt the curriculum and instruction for the future needs of the ever evolving high-tech society. ^

An empirical study of Kirkpatrick's evaluation model in the hospitality industry

This study examined Kirkpatrick’s training evaluation model (Kirkpatrick & Kirkpatrick, 2006) by assessing a sales training program conducted at an organization in the hospitality industry. The study assessed the employees’ training outcomes of knowledge and skills, job performance, and the impact of the training upon the organization. By assessing these training outcomes and their relationships, the study demonstrated whether Kirkpatrick’s theories are supported and the lower evaluation levels can be used to predict organizational impact. The population for this study was a group of reservations sales agents from a leading luxury hotel chain’s reservations center. During the study period from January 2005 to May 2007, there were 335 reservations sales agents employed in this Global Reservations Center (GRC). The number of reservations sales agents who had completed a sales training program/intervention during this period and had data available for at least two months pre and post training composed the sample for this study. The number of agents was 69 ( N = 69). Four hypotheses were tested through paired-samples t tests, correlation, and hierarchical regression analytic procedures. Results from the analyses supported the hypotheses in this study. The significant improvement in the call score supported hypothesis one that the reservations sales agents who completed the training improved their knowledge of content and required skills in handling calls (Level 2). Hypothesis two was accepted in part as there was significant improvement in call conversion, but there was no significant improvement of time usage. The significant improvement in the sales per call supported hypothesis three that the reservations agents who completed the training contributed to increased organizational impact (Level 4), i.e., made significantly more sales. Last, findings supported hypothesis four that Level 2 and Level 3 variables can be used for predicting Level 4 organizational impact. The findings supported the theory of Kirkpatrick’s evaluation model that in order to expect organizational results, a positive change in behavior (job performance) and learning must occur. The examinations of Levels 2 and 3 helped to partially explain and predict Level 4 results.