A longitudinal evaluation of a project-based learning initiative in an engineering undergraduate programme

Project-based learning (PBL) is a well-known student-centred methodology for engineering design education. The methodology claims to offer a number of educational benefits. This paper evaluates the student perceptions of the initial and second offering of a first-year design unit at Griffith University in Australia. It builds on an earlier evaluation conducted after the initial offering of the unit. It considers the implementation of the recommended changes. Evaluations of the two offerings reveal that students (in both the initial and second offering) generally enjoyed the experience, but that the second offering was found to be a significantly more enjoyable learning experience. Students in the second offering also reported a significantly better understanding of what they needed to do for the design projects and where to find the requisite information. The oral presentation aspect of the initial and second offerings received the lowest satisfaction rating. The inclusion (and delivery) of the computer-aided drawing component of the unit is seen as a positive aspect by some students, but many others comment on it negatively. The best aspects of the PBL unit and those aspects needing further improvement were similar to the findings of other investigations documented in the literature.

Program evaluation for a summer science internship: Short and long term benefits as reported by interns

Undergraduate research programs have been used as a tool to attract and retain student interest in science careers. This study evaluates the short and long-term benefits of a Summer Science Internship (SSI) at the University of Texas Health Science Center at Houston– School of Public Health – in Brownsville, Texas, by analyzing survey data from alumni. Questions assessing short-term program impact were aimed at three main topics, student: satisfaction with program, self-efficacy for science after completing the program, and perceived benefits. Long-term program impact was assessed by looking at student school attendance and college majors along with perceived links between SSI and future college plans. Students reported high program satisfaction, a significant increase in science self-efficacy and high perceived benefits. At the time data were collected for the study, one-hundred percent of alumni were enrolled in school (high school or college). The majority of students indicated they were interested in completing a science major/career, heavily influenced by their participation in the program.^

Evaluation of the Alcohol Use and Traffic Safety Curriculum (K-12). Section II - volume 7a. Final report.

National Highway Traffic Safety Administration, Office of Driver and Pedestrian Programs, Washington, D.C.

Evaluation of special consumer-homemaking programs in Illinois high schools /

"Prepared for and funded by the Board of Vocational Education and Rehabilitation, Division of Vocational and Technical Education, Special Programs Unit, Springfield, Illinois."

An empirical study of Kirkpatrick's evaluation model in the hospitality industry

This study examined Kirkpatrick’s training evaluation model (Kirkpatrick & Kirkpatrick, 2006) by assessing a sales training program conducted at an organization in the hospitality industry. The study assessed the employees’ training outcomes of knowledge and skills, job performance, and the impact of the training upon the organization. By assessing these training outcomes and their relationships, the study demonstrated whether Kirkpatrick’s theories are supported and the lower evaluation levels can be used to predict organizational impact. The population for this study was a group of reservations sales agents from a leading luxury hotel chain’s reservations center. During the study period from January 2005 to May 2007, there were 335 reservations sales agents employed in this Global Reservations Center (GRC). The number of reservations sales agents who had completed a sales training program/intervention during this period and had data available for at least two months pre and post training composed the sample for this study. The number of agents was 69 ( N = 69). Four hypotheses were tested through paired-samples t tests, correlation, and hierarchical regression analytic procedures. Results from the analyses supported the hypotheses in this study. The significant improvement in the call score supported hypothesis one that the reservations sales agents who completed the training improved their knowledge of content and required skills in handling calls (Level 2). Hypothesis two was accepted in part as there was significant improvement in call conversion, but there was no significant improvement of time usage. The significant improvement in the sales per call supported hypothesis three that the reservations agents who completed the training contributed to increased organizational impact (Level 4), i.e., made significantly more sales. Last, findings supported hypothesis four that Level 2 and Level 3 variables can be used for predicting Level 4 organizational impact. The findings supported the theory of Kirkpatrick’s evaluation model that in order to expect organizational results, a positive change in behavior (job performance) and learning must occur. The examinations of Levels 2 and 3 helped to partially explain and predict Level 4 results.

Development of a new student evaluation instrument of instructor effectiveness in online courses

The purpose of this study was to (a) develop an evaluation instrument capable of rating students' perceptions of the instructional quality of an online course and the instructor’s performance, and (b) validate the proposed instrument with a study conducted at a major public university. The instrument was based upon the Seven Principles of Good Practice for Undergraduate Education (Chickering & Gamson, 1987). The study examined four specific questions. 1. Is the underlying factor structure of the new instrument consistent with Chickering and Gamson's Seven Principles? 2. Is the factor structure of the new instrument invariant for male and female students? 3. Are the scores on the new instrument related students’ expected grades? 4. Are the scores on the new instrument related to the students' perceived course workload? ^ The instrument was designed to measure students’ levels of satisfaction with their instruction, and also gathered information concerning the students’ sex, the expected grade in the course, and the students’ perceptions of the amount of work required by the course. A cluster sample consisting of an array of online courses across the disciplines yielded a total 297 students who responded to the online survey. The students for each course selected were asked to rate their instructors with the newly developed instrument. ^ Question 1 was answered using exploratory factor analysis, and yielded a factor structure similar to the Seven Principles.^ Question 2 was answered by separately factor-analyzing the responses of male and female students and comparing the factor structures. The resulting factor structures for men and women were different. However, 14 items could be realigned under five factors that paralleled some of the Seven Principles. When the scores of only those 14 items were entered in two principal components factor analyses using only men and only women, respectively and restricting the factor structure to five factors, the factor structures were the same for men and women.^ A weak positive relationship between students’ expected grades and their scores on the instrument was found (Question 3). There was no relationship between students’ perceived workloads for the course and their scores on the instrument (Question 4).^

GRE as a predictor of graduate student success at a Hispanic serving institution of higher education

Accurately predicting the success of graduate students is an important aspect of determining which students should be admitted into graduate programs. The GRE is a pivotal factor to examine since it is one of the most widely used criteria for graduate school admission. Even though the GRE is advertised as an accurate tool for predicting first year graduate GPA, there is a lack of research on long term success factors such as time to degree and graduate rate (Luthy, 1996; Powers, 2004). Furthermore, since most studies have low minority sample sizes, the validity of the GRE may not be the same across all groups (ETS, 2008b; Kuncel, Hezlett, & Ones, 2001). Another gap in GRE studies is that few researchers analyze student characteristics, which may alter or moderate the prediction validity of the GRE. Thus, student characteristics such as degree of academic involvement, mentorship interactions, and other academic and social experiences have not been widely examined in this context. These gaps in the analysis of GRE validity are especially relevant given the high attrition rates within of some graduate programs (e.g., an estimated 68% of doctoral student never complete their programs in urban universities; Lovitts, 2001). A sequential mixed methods design was used to answer the research questions in two phases. The quantitative phase used student data files to analyze the relationship of two success variables (graduation rate and graduate GPA) to the GRE scores as well as other academic and demographic graduate student characteristics. The qualitative phase served to complement the first phase by describing a wider range of characteristics from the 11 graduate students who were interviewed. Both proximal and distal moderators influence student behaviors and success in graduate school. In the first phase of the study, the GRE was the distal facilitator under analysis. Findings suggested that both the GRE Quantitative and the GRE Verbal were predictors of success for master’s students, but the GRE Quantitative was not predictive of success for doctoral students. Other student characteristics such as demographic variables and disciplinary area were also predictors of success for the population of students studied. In the second phase of the study, it was inconclusive whether the GRE was predictive of graduate student success; though it did influence access to graduate programs. Furthermore, proximal moderators such as student involvement, faculty/peer interactions, motivational factors, and program structure were perceived to be facilitators and/or detractors for success.

Respectful relationships education in schools: the beginnings of change

Children and young people have become one of the most important populations for the prevention of gender-based violence. The unacceptably high rates of gender-based violence, in particular, violence against women and their children, are well established and there is a clear opportunity to change the story for future generations via the education system. There is no single cause of gender-based violence, however, the latest international evidence shows that it is primarily driven by a range of social norms, institutional structures, and organizational or community practices relating to gender inequality. Gender inequality manifests in every aspect of life, from our relationships through to our institutions, including schools. Shifting the pattern of violence will require cultural transformation- and new research shows that this change is possible. Governments have seized this opportunity, and through the Australian Curriculum, schools have been directed to consider their role in the prevention of gender-based violence through the incorporation of Respectful Relationships Education. Schools play a central role in the intellectual, social and emotional development of children and young people. The education system sets the foundation for creating future generations of successful learners, confident and creative individuals, and active and informed citizens. Recent international evidence shows that the impact of school based Respectful Relationships Education – if implemented according to good practice standards – can be profound. Schools are ‘mini communities’ where respect and equality can be modelled to help shape positive attitudes and behaviours at an early stage of life. As workplaces and community hubs, schools have spheres of influence which extend to a workforce of over 40,000 teaching and non-teaching staff in Victorian schools, and into every Victorian community. As such, their reach and potential to drive cultural change to prevent gender-based violenceis significant. This report presents the findings of the Respectful Relationships Education in Schools (RREiS) pilot as one of the first evaluations in Australia to examine the impact of Respectful Relationships Education across the whole school – from the classroom, through to the staff room and broader school culture and ethos.

Prisons, dietetic services, research and training : win win

Learning Outcome: Gain knowledge in the area of dietetic training in Australia and the benefits of collaborative partnerships between government and universities to achieve improvements in dietetic service delivery, evidenced based practice, and student placements. Prisoners have high rates of chronic disease, however dietetic services and research in this sector is limited. Securing high quality professional practice placements for dietetic training in Australia is competitive, and prisons provide exciting opportunities. Queensland University of Technology (QUT) has a unique twenty year partnership with Queensland Corrective Services (QCS) with a service learning model placing final year dietetic students within prisons. Building on this partnership, in 2007 a new joint position was funded to establish dietetic services to over 5500 prisoners and support viable best practice dietetic education. Evaluation of the past three years of this partnership has shown an expansion of QUT student placements in Queensland prisons, with a third of final year students each undertaking 120 hours of foodservice management practicum. Student evaluations of placement over this period are much higher than the University average. Through the joint position student projects have been targeted on strategic areas to support nutrition and dietetic policy and practice. Projects have been broadened from menu reviews to more comprehensive quality improvement and dietetic research activities, with all student learning activities transferrable to other foodservice settings. Student practice in the prisons has been extended beyond foodservice management to include group education and dietetic counseling. For QCS, student placements have equated to close to a full-time dietitian position, with nutrition policy now being implemented as an outcome of this support. This innovative partnership has achieved a sustainable student placement model, supported research, whilst delivering dietetic services to a difficult to access group. Funding Disclosure: None

Apprendre à bien lire : une étude coût efficacité des Actions Lecture

How to Improve Pupils' Literacy? A Cost-Effectiveness Analysis of the Action Lecture This article presents a cost-effectiveness analysis of an innovative teaching method run in some nursery and primary schools in Paris. This project, named Action Lecture, is designed to improve pupils' abilities and taste on literacy. We report the results of an evaluation of this program. We describe the experimental protocol that was built to estimate the program's impact on several types of indicators (academic abilities, attitude toward reading, school life) by comparing the evolution of treatment schools and control schools. Data were processed following a Differences-in-Differences (DID) method. Then we use the estimation of the impact on academic achievement to conduct a cost-effectiveness analysis and take a reduction of the class size program as a benchmark. The results are positive for the Action Lecture program both in terms of impact's evaluation and in terms of cost-effectiveness ratio.

Kaleidoscope : shaping a better future through a stakeholder approach to widespread organisational change

The main focus of ‘Kaleidoscope: Reframing evaluation through a stakeholder approach to sustainable, cultural change in Higher Education’ is to develop a set of principles to guide user-led engagement in widespread organisational change and maximise its impact. The word kaleidoscope represents the unique lens through which each institution will need to view their cultural specificity and local context through an extensive process of collaboration and engagement, followed by communication and dissemination. Kaleidoscope has particular relevance when new approaches to learning and teaching evaluation are introduced by tertiary institutions. Building on the Reframe Project, which involved three years of user-led consultation and was designed to meet stakeholders’ needs, QUT successfully introduced a new evaluation framework in 2013 across the university. Reframe was evidence based, involved scholarly reflection and was founded on a strong theoretical framework. The evolution of the evaluation framework included analysis of scholarly literature and environmental scans across the higher education sector (Alderman, et al., 2012), researched development of conceptual theory (Alderman, et al., in press 2013), incorporated the stakeholder voice and framed within project management principles (Alderman & Melanie, 2012). Kaleidoscope’s objectives are for QUT to develop its research-based stakeholder approach to distil the successful experience exhibited in the Reframe Project into a transferable set of guidelines for use by other tertiary institutions across the sectors. These guidelines will assist others to design, develop, and deploy, their own culturally specific widespread organisational change informed by stakeholder engagement and organisational buy-in. It is intended that these guidelines will promote, support and enable other tertiary institutions to embark on their own projects and maximise the impact. In correlation with a our conference paper, this round table presents the Draft Guidelines and Framework ready for external peer review by evaluation practitioners, as part of Kaleidoscope’s dissemination (Hinton & Gannaway, 2011) applying illuminative evaluation theory (Parlett & Hamilton, 1976), through conference workshops and linked round table discussions (Shapiro, et al., 1983; Jacobs, 2000).

How to meet stakeholders' expectations, not manage them?

Evaluation practices in the higher education sector have been criticised for having unclear purpose and principles; ignoring the complexity and changing nature of learning and teaching and the environments in which they occur; relying almost exclusively on student ratings of teachers working in classroom settings; lacking reliability and validity; using data for inappropriate purposes; and focusing on accountability and marketing rather than the improvement of learning and teaching. In response to similar criticism from stakeholders, in 2011 Queensland University of Technology began a project, entitled REFRAME, to review its approach to evaluation, particularly the student survey system it had been using for the past five years. This presentation will outline the scholarly, evidence based methodology used to undertake institution-wide change, meet the needs of stakeholders suitable to the cultural needs of the institution. It is believed that this approach is broadly applicable to other institutions contemplating change with regard to evaluation of learning and teaching.

Framing student evaluations of university learning and teaching : discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice, that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual, gestural) that inform and are produced by, for and through evaluative practices are important as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.

Evaluative decision-making for high-quality professional development: Cultivating an evaluative stance

Unprecedented policy attention to early childhood education internationally has highlighted the crucial need for a skilled early years workforce. Consequently, professional development of early years educators has become a global policy imperative. At the same time, many maintain that professional development research has reached an impasse. In this paper, we offer a new approach to addressing this impasse. In contrast to calls for a redesign of comparative studies of professional development programs, or for the refinement of researcher-constructed professional development evaluation frameworks, we argue the need to cultivate what we refer to as an ‘evaluative stance’ amongst all involved in making decisions about professional development in the early years – from senior bureaucrats with responsibilities for funding professional development programs to individual educators with choices about which professional development opportunities to take up. Drawing on three bodies of literature -- evaluation capacity building, personal epistemology, and co-production -- that, for the most part, have been overlooked with respect to early years professional learning this paper proposes a conceptual framework to explain why cultivating an evaluative stance in professional development decision-making has rich possibilities for systemic, sustainable, and transformative change in early years education.