Problems in assessing the impact of Education Division programs on girls and women for the National Advisory Council on Women's Educational Programs /

Issued October 1977.

Evaluation forum /

"A review journal for employment and training professionals focusing on policy analysis and evaluation research in education, occupational training, work and welfare, and economic development."

Federal aid for education. Hearings before the Committee on education and labor, U. S. Senate, 79th Congress, 1st session, on S. 181, a bill to authorize the appropriation of funds to assist the states and territories in more adequately financing their systems of public education during emergency, and in reducing the inequalities of educational opportunities through public elementary and secondary schools...

James E. Murray, chairman.

Factors affecting student attitudes toward flexible online learning in management education

In response to recent technological advances and the trend toward flexible learning in education, the authors examined the factors affecting student satisfaction with flexible online learning. The authors identified 2 key student attributes of student satisfaction: (a) positive perceptions of technology in terms of ease of access and use of online flexible learning material and (b) autonomous and innovative learning styles. The authors derived measures of perceptions of technology from research on the Technology Acceptance Model and used locus of control and innovative attitude as indicators of an autonomous and innovative learning mode. First-year students undertaking an introductory management course completed surveys at the beginning (n = 248) and at the end (n = 256) of course work. The authors analyzed the data by using structural equation modeling. Results suggest that student satisfaction is influenced by positive perceptions toward technology and an autonomous learning mode.

Measuring performance of virtual learning environment system in higher education

Purpose – The purpose of this paper is to measure the performance of commercial virtual learning environment (VLE) systems, which helps the decision makers to select the appropriate system for their institutions. Design/methodology/approach – This paper develops an integrated multiple criteria decision making approach, which combines the analytic hierarchy process (AHP) and quality function deployment (QFD), to evaluate and select the best system. The evaluating criteria are derived from the requirements of those who use the system. A case study is provided to demonstrate how the integrated approach works. Findings – The major advantage of the integrated approach is that the evaluating criteria are of interest to the stakeholders. This ensures that the selected system will achieve the requirements and satisfy the stakeholders most. Another advantage is that the approach can guarantee the benchmarking to be consistent and reliable. From the case study, it is proved that the performance of a VLE system being used at the university is the best. Therefore, the university should continue to run the system in order to support and facilitate both teaching and learning. Originality/value – It is believed that there is no study that measures the performance of VLE systems, and thus decision makers may have difficulties in system evaluation and selection for their institutions.

The development and evaluation of distance learning materials for administrative personnel in a multi-national company

There has been substantial research into the role of distance learning in education. Despite the rise in the popularity and practice of this form of learning in business, there has not been a parallel increase in the amount of research carried out in this field. An extensive investigation was conducted into the entire distance learning system of a multi-national company with particular emphasis on the design, implementation and evaluation of the materials. In addition, the performance and attitudes of trainees were examined. The results of a comparative study indicated that trainees using distance learning had significantly higher test scores than trainees using conventional face-to-face training. The influence of the previous distance learning experience, educational background and selected study environment of trainees was investigated. Trainees with previous experience of distance learning were more likely to complete the course and with significantly higher test scores than trainees with no previous experience. The more advanced the educational background of trainees, the greater the likelihood of their completing the course, although there was no significant difference in the test scores achieved. Trainees preferred to use the materials at home and those opting to study in this environment scored significantly higher than those studying in the office, the study room at work or in a combination of environments. The influence of learning styles (Kolb, 1976) was tested. The results indicated that the convergers had the greatest completion rates and scored significantly higher than trainees with the assimilator, accommodator and diverger learning styles. The attitudes of the trainees, supervisors and trainers were examined using questionnaire, interview and discussion techniques. The findings highlighted the potential problems of lack of awareness and low motivation which could prove to be major obstacles to the success of distance learning in business.

A comparison of a distance education and locally based course in an urban university setting

The purpose of this research was to compare the academic performance and attitudes of students at the instructor-based site of a televised course and the distant site. An earlier pilot program indicated the need for certain technical and structural interventions at the distant site such as multiple "press-to-touch" microphones, a site-administrator and participative seating arrangements. At the beginning of the class, demographic data were collected from the students at both sites through a questionnaire and supplemented with information from students' records. Factors such as age, gender, ethnicity, marital status, number of children, current class status, major, work status and CLAST scores (achievement tests) were examined. There were no significant differences between the students at the sites except ethnicity and reading CLAST scores. The instructor-based site had a higher percentage of Hispanic students and the distant site had a larger percentage of Caucasian and Black Americans. The distant site scored significantly better on the reading section of the CLAST achievement test. An evaluation instrument was distributed to both sites, at the midpoint of the semester, measuring their attitude toward the organizational, technical, and pedagogical factors of the course. A second evaluation instrument, measuring similar factors, but more in-depth, was distributed to both sites near the end of the term. Nine students at the distant site were interviewed along with the site administrator to collect additional information.^ Course completion rates, dropout rates, pass rates and final grades of students at both sites were compared. There were no significant differences in academic performance between the students at both sites, however, there were significant differences in their attitudes. Those at the instructor-based site gave better ratings to most of items in the evaluation instruments. Problems at the distant site included audio and visual clarity, lack of available assistance, too much nonrelated talking, not enough opportunities to ask questions or to interact with the instructor during class. ^

Adult education directors' perceptions of performance indicators for adult education programs in the state of Florida

Current demands for accountability in education emphasize outcome-based program evaluation and tie program funding to individual student performance. As has been the case for elementary and secondary programs, demands for accountability have increased pressure on adult educators to show evidence of the benefits of their programs in order to justify their financial support. In Florida, recent legislation fundamentally changes the delivery of adult education in the state by establishing a performance-based funding system that is based on outcomes related to the retention, completion, and employment of program participants.^ A performance-based funding system requires an evaluation process that stresses outcome indicators over indicators that focus on program context or process. Although the state has adopted indicators of program quality to evaluate its adult education programs, these indicators focus mostly on program processes rather than student outcomes. In addition, the indicators are not specifically tied to workforce development outcomes, a priority to federal and local funding agents.^ Improving the accountability of adult education programs and defining the role of these programs in Florida's Workforce Development System has become a priority to policy makers across the state. Another priority has been to involve adult education practitioners in every step of this process.^ This study was conducted in order to determine what performance indicators, as judged by the directors and supervisors of adult education programs in the state of Florida, are important and feasible in measuring the quality and effectiveness of these programs. The results of the study indicated that, both statewide and by region, the respondents consistently gave the highest ratings on both importance and feasibility to the indicators of Program Context, which reflect the needs, composition, and structure of the programs, and to the indicators of Educational Gain, which reflect learner progress in the attainment of basic skills and competencies. In turn, the respondents gave the lowest ratings on both importance and feasibility to the indicators in the areas of Return on State's Investment, Efficiency, Retention, and Workforce Training. In general, the indicators that received high ratings for importance also received high ratings for feasibility. ^

An empirical study of Kirkpatrick's evaluation model in the hospitality industry

This study examined Kirkpatrick’s training evaluation model (Kirkpatrick & Kirkpatrick, 2006) by assessing a sales training program conducted at an organization in the hospitality industry. The study assessed the employees’ training outcomes of knowledge and skills, job performance, and the impact of the training upon the organization. By assessing these training outcomes and their relationships, the study demonstrated whether Kirkpatrick’s theories are supported and the lower evaluation levels can be used to predict organizational impact. The population for this study was a group of reservations sales agents from a leading luxury hotel chain’s reservations center. During the study period from January 2005 to May 2007, there were 335 reservations sales agents employed in this Global Reservations Center (GRC). The number of reservations sales agents who had completed a sales training program/intervention during this period and had data available for at least two months pre and post training composed the sample for this study. The number of agents was 69 ( N = 69). Four hypotheses were tested through paired-samples t tests, correlation, and hierarchical regression analytic procedures. Results from the analyses supported the hypotheses in this study. The significant improvement in the call score supported hypothesis one that the reservations sales agents who completed the training improved their knowledge of content and required skills in handling calls (Level 2). Hypothesis two was accepted in part as there was significant improvement in call conversion, but there was no significant improvement of time usage. The significant improvement in the sales per call supported hypothesis three that the reservations agents who completed the training contributed to increased organizational impact (Level 4), i.e., made significantly more sales. Last, findings supported hypothesis four that Level 2 and Level 3 variables can be used for predicting Level 4 organizational impact. The findings supported the theory of Kirkpatrick’s evaluation model that in order to expect organizational results, a positive change in behavior (job performance) and learning must occur. The examinations of Levels 2 and 3 helped to partially explain and predict Level 4 results.

Development of a new student evaluation instrument of instructor effectiveness in online courses

The purpose of this study was to (a) develop an evaluation instrument capable of rating students' perceptions of the instructional quality of an online course and the instructor’s performance, and (b) validate the proposed instrument with a study conducted at a major public university. The instrument was based upon the Seven Principles of Good Practice for Undergraduate Education (Chickering & Gamson, 1987). The study examined four specific questions. 1. Is the underlying factor structure of the new instrument consistent with Chickering and Gamson's Seven Principles? 2. Is the factor structure of the new instrument invariant for male and female students? 3. Are the scores on the new instrument related students’ expected grades? 4. Are the scores on the new instrument related to the students' perceived course workload? ^ The instrument was designed to measure students’ levels of satisfaction with their instruction, and also gathered information concerning the students’ sex, the expected grade in the course, and the students’ perceptions of the amount of work required by the course. A cluster sample consisting of an array of online courses across the disciplines yielded a total 297 students who responded to the online survey. The students for each course selected were asked to rate their instructors with the newly developed instrument. ^ Question 1 was answered using exploratory factor analysis, and yielded a factor structure similar to the Seven Principles.^ Question 2 was answered by separately factor-analyzing the responses of male and female students and comparing the factor structures. The resulting factor structures for men and women were different. However, 14 items could be realigned under five factors that paralleled some of the Seven Principles. When the scores of only those 14 items were entered in two principal components factor analyses using only men and only women, respectively and restricting the factor structure to five factors, the factor structures were the same for men and women.^ A weak positive relationship between students’ expected grades and their scores on the instrument was found (Question 3). There was no relationship between students’ perceived workloads for the course and their scores on the instrument (Question 4).^

Cooperative or collaborative learning: Is there a difference in university students’ perceptions?

The hypothesis that the same educational objective, raised as cooperative or collaborative learning in university teaching does not affect students’ perceptions of the learning model, leads this study. It analyses the reflections of two students groups of engineering that shared the same educational goals implemented through two different methodological active learning strategies: Simulation as cooperative learning strategy and Problem-based Learning as a collaborative one. The different number of participants per group (eighty-five and sixty-five, respectively) as well as the use of two active learning strategies, either collaborative or cooperative, did not show differences in the results from a qualitative perspective.

Evaluation of online higher education: learning, interaction and technology

The work presents a theoretical framework for the evaluation of e-Teaching that aims at positioning the online activities designed and developed by the teacher as to the Learning, Interaction and Technology Dimensions. The theoretical research that underlies the study was developed reflecting current thinking on the promotion of quality of teaching and of the integration of information and communication tools into the curriculum in Higher Education (HE), i.e., bearing in mind some European guidelines and policies on this subject. This way, an answer was sought to be given to one of the aims put forward in this study, namely to contribute towards the development of a conceptual framework to support research on evaluation of e-teaching in the context of HE. Based on the theoretical research carried out, an evaluation tool (SCAI) was designed, which integrates the two questionnaires developed to collect the teachers' and the students' perceptions regarding the development of e-activities. Consequently, an empirical study was structured and carried out, allowing SCAI tool to be tested and validated in real cases. From the comparison of the theoretical framework established and the analysis of the data obtained, we found that the differences in teaching should be valued and seen as assets by HE institutions rather than annihilated in a globalizing perspective.

Multimediale, interaktive und patientennahe Lehrszenarien in der medizinischen Ausbildung

MeduMobile ist ein im Rahmen der Ausschreibung „Notebook-University“ vom bmb+f gefördertes Projekt. Ziel des Projektes ist es, die Ausbildung am Krankenbett im Medizinstudium zu verbessern, indem bestimmte Lehrveranstaltungen mit Hilfe von WLAN und Notebook ubiquitär auf dem Campus verfügbar gemacht werden. Hierzu werden neue, so genannte OnCall-Lehrszenarien entwickelt und erprobt, bei denen die auf Abruf bereit stehenden Medizinstudierenden alarmiert und zur Teilnahme gebeten werden, wenn akute und/oder seltene Fälle in die Klinik eingeliefert werden. Die Studierenden nehmen aktiv an den vielfach interdisziplinären Lehrveranstaltungen teil. Der Unterricht findet vor, während und nach der Live-Session statt. Der Hochschullehrer und die Studierenden können den Fall gemeinsam besprechen und die Diagnose bzw. Therapie u.a. an Hand bildgebender Verfahren (CT, Mikroskop, Röntgen, Ultraschall, ...) erarbeiten. Parallel dazu können die Studierenden Lehrmaterialen aus multimedialen Datenbanken, Medline und Internet nutzen sowie eigene Videokonferenzen für die Gruppenarbeit einsetzen. Eine solche Lehrveranstaltung kann somit mehrere didaktische Elemente beinhalten: instruktives, konstruktives, kognitives und kooperatives Lernen. An der Erprobung nehmen etwa 80 Studierende an Veranstaltungen aus 8 Fachgebieten teil. Es werden Studien zur Evaluation des didaktischen Mehrwerts sowie technischer und organisatorischer Qualität durchgeführt. Endgültige Ergebnisse werden Ende 2003 vorliegen.(DIPF/Orig.)

Assessing the Role of General Chemistry Learning in Higher Education

The inclusion of General Chemistry (GC) in the curricula of higher education courses in science and technology aims, on the one hand, to develop students' skills necessary for further studies and, on the other hand, to respond to the need of endowing future professionals of knowledge to analyze and solve multidisciplinary problems in a sustainable way. The participation of students in the evaluation of the role played by the GC in their training is crucial, and the analysis of the results can be an essential tool to increase success in the education of students and improving practices in various professions. Undeniably, this work will be focused on the development of an intelligent system to assess the role of GC. The computational framework is built on top of a Logic Programming approach to Knowledge Representation and Reasoning, complemented with a problem solving methodology moored on Artificial Neural Networks. The results so far obtained show that the proposed model stands for a good start, being its overall accuracy higher than 95%.