Medical Student Evaluation - Table of Contents

Documents pertaining to the organization of the College of Medicine, Medical Education, the Office of Student Affairs, requirements for acceptance into the College, and other documents related to the College of Medicine.

Development of a new student evaluation instrument of instructor effectiveness in online courses

The purpose of this study was to (a) develop an evaluation instrument capable of rating students' perceptions of the instructional quality of an online course and the instructor’s performance, and (b) validate the proposed instrument with a study conducted at a major public university. The instrument was based upon the Seven Principles of Good Practice for Undergraduate Education (Chickering & Gamson, 1987). The study examined four specific questions. 1. Is the underlying factor structure of the new instrument consistent with Chickering and Gamson's Seven Principles? 2. Is the factor structure of the new instrument invariant for male and female students? 3. Are the scores on the new instrument related students’ expected grades? 4. Are the scores on the new instrument related to the students' perceived course workload? ^ The instrument was designed to measure students’ levels of satisfaction with their instruction, and also gathered information concerning the students’ sex, the expected grade in the course, and the students’ perceptions of the amount of work required by the course. A cluster sample consisting of an array of online courses across the disciplines yielded a total 297 students who responded to the online survey. The students for each course selected were asked to rate their instructors with the newly developed instrument. ^ Question 1 was answered using exploratory factor analysis, and yielded a factor structure similar to the Seven Principles.^ Question 2 was answered by separately factor-analyzing the responses of male and female students and comparing the factor structures. The resulting factor structures for men and women were different. However, 14 items could be realigned under five factors that paralleled some of the Seven Principles. When the scores of only those 14 items were entered in two principal components factor analyses using only men and only women, respectively and restricting the factor structure to five factors, the factor structures were the same for men and women.^ A weak positive relationship between students’ expected grades and their scores on the instrument was found (Question 3). There was no relationship between students’ perceived workloads for the course and their scores on the instrument (Question 4).^

Assessing the effectiveness of five-year mechanical engineering technology programs of junior colleges in Taiwan, R.O.C.: An application of the CIPP evaluation model

The purpose of this study was to evaluate the mechanical engineering technology curriculum effectiveness at the junior college in Taiwan by using the CIPP evaluation model. The study concerned the areas of the curriculum, curriculum materials, individualized instruction, support services, teaching effectiveness, student achievement, and job performance. A descriptive survey method was used with questionnaires for data collection from faculty, students, graduates, and employers.^ All categories of respondents tended to agree that the curriculum provides appropriate occupational knowledge and skills. Students, graduates, and faculty tended to be satisfied with the curriculum; faculty tended to be satisfied with student achievement; graduates tended to be satisfied with their job preparation; and employers were most satisfied with graduates' job performance.^ Conclusions were drawn in the context, input, process, and product of the CIPP model. In Context area: Students were dissatisfied with curriculum flexibility in students characteristics. Graduates were dissatisfied with curriculum design for student's adaptability in new economic and industrial conditions; practicum flexibility in student characteristics; and course overlap. Both students and graduates were dissatisfied with practicum credit hours. Both faculty and students were dissatisfied with the number of required courses.^ In Input area: Students, faculty, and graduates perceived audiovisuals and manipulative aids positively. Faculty and students perceive CAI implementation positively. Students perceived textbooks negatively.^ In Process area: Faculty, students, and graduates perceived all support service negatively. Faculty tended to perceive the ratios of graduates who enter advanced study and related occupation, and who passed the professional skills certification, negatively. Students tended to perceive teaching effectiveness in terms of instructional strategies, the quality of instruction, overall suitability, and receivable, negatively. Graduates also tended to identify the instructional strategies as a negative perception. Faculty and students perceived curriculum objectives and practicum negatively. Both faculty and students felt that instructors should be more interested in making the courses a useful learning experience.^ In Product area: Employers were satisfied with graduates' academic preparation and job performance, adaptability, punctuality, and their ability to communicate, cooperate, and meet organization needs. Graduates were weak in terms of equipment familiarity and supervisory ability.^ In sum, the curriculum of the five-year mechanical engineering technology programs of junior college in Taiwan has served adequately up to this time in preparing a work force to enter industry. It is now time to look toward the future and adapt the curriculum and instruction for the future needs of this high-tech society. ^

The effect of mentoring electronic mail on student achievement and attitudes in a graduate course in educational research

This exploratory study of a classroom with mentoring and neutral e-mail was conducted in a public commuter state university in South Florida between January 1996 and April 1996. Sixteen males and 83 females from four graduate level educational research classes participated in the study.^ Two main hypotheses were tested. Hypothesis One was that those students receiving mentoring e-mail messages would score significantly higher on an instrument measuring attitude toward educational research (ATERS) than those not receiving mentoring e-mail messages. Hypothesis Two was that those students receiving mentoring e-mail would score significantly higher on objective exams covering the educational research material than those not receiving mentoring e-mail.^ Results of factorial analyses of variance showed no significant differences between the treatment groups in achievement or in attitudes toward educational research. Introverts had lower attitudes and lower final exam grades in both groups, although introverts in the mentored group scored higher than those introverts in the neutral group.^ A t test of the means of total response to e-mail from the researcher showed a significant difference between the mentored and neutral e-mail groups. Introverts responded more often than extraverts in both groups.^ Teacher effect was significant in determining class response to e-mail messages. Responses were most frequent in the researcher's classes.^ Qualitative analyses of the e-mail and course evaluation survey and of the content of e-mail messages received by the researcher were then grouped into basic themes and discussed.^ A qualitative analysis of an e-mail and course evaluation survey revealed that students from both the neutral and mentoring e-mail groups appreciated teacher feedback. A qualitative analysis of the mentoring and neutral e-mail replies divided the responses into those pertaining to the class, such as test and research paper questions, and more personal items, such as problems in the class and personal happenings.^ At this point in time, e-mail is not a standard way of communicating in classes in the college of education at this university. As this technology tool of communication becomes more popular, it is anticipated that replications of this study will be warranted. ^

Design and evaluation of a consulting system for database design

Database design is a difficult problem for non-expert designers. It is desirable to assist such designers during the problem solving process by means of a knowledge based (KB) system. A number of prototype KB systems have been proposed, however there are many shortcomings. Few have incorporated sufficient expertise in modeling relationships, particularly higher order relationships. There has been no empirical study that experimentally tested the effectiveness of any of these KB tools. Problem solving behavior of non-experts, whom the systems were intended to assist, has not been one of the bases for system design. In this project a consulting system for conceptual database design that addresses the above short comings was developed and empirically validated.^ The system incorporates (a) findings on why non-experts commit errors and (b) heuristics for modeling relationships. Two approaches to knowledge base implementation--system restrictiveness and decisional guidance--were used and compared in this project. The Restrictive approach is proscriptive and limits the designer's choices at various design phases by forcing him/her to follow a specific design path. The Guidance system approach which is less restrictive, provides context specific, informative and suggestive guidance throughout the design process. The main objectives of the study are to evaluate (1) whether the knowledge-based system is more effective than a system without the knowledge-base and (2) which knowledge implementation--restrictive or guidance--strategy is more effective. To evaluate the effectiveness of the knowledge base itself, the two systems were compared with a system that does not incorporate the expertise (Control).^ The experimental procedure involved the student subjects solving a task without using the system (pre-treatment task) and another task using one of the three systems (experimental task). The experimental task scores of those subjects who performed satisfactorily in the pre-treatment task were analyzed. Results are (1) The knowledge based approach to database design support lead to more accurate solutions than the control system; (2) No significant difference between the two KB approaches; (3) Guidance approach led to best performance; and (4) The subjects perceived the Restrictive system easier to use than the Guidance system. ^

A study of the effects of a service-learning experience on student success at an urban community college

The purpose of this study was to explore the effects of a service-learning experience on student success as measured by class attendance, course completion, final course grades, and end-of-term evaluation data.^ Though many outcomes of service-learning experiences have been studied, including ethical values, self-esteem, student personal development, and career preparation, relatively few studies have been conducted on the effects of such experiences on academic achievement, and the studies that have been done have primarily studied students at traditional, four-year, residential universities.^ The study consisted of 286 students enrolled in six paired courses taught by five instructors at a community college in the Fall term 1996. One section of each pair (the control group) was taught using traditional subject matter and course materials and the other section of each pair (the treatment group) participated in a 20-hour required service-learning activity in addition to the regular course curriculum. The courses in the study included American History, Sociology, College Preparatory English, and Introduction to English Composition.^ The results of this study indicate that, overall, students who participated in a class in which service-learning was a requirement, achieved higher final course grades and reported greater satisfaction with the course, the instructor, the reading assignments, and the grading system, and the treatment section of one course pair had fewer absences. In addition, the faculty members reported that, in the treatment sections, class discussions were more stimulating, the sections seemed more vital in terms of student involvement, the students seemed more challenged academically, more motivated to learn, and seemed to exert more effort in the course. ^

Planning and evaluation of Business English course in compliance with internationalization in Taiwan: A case study of Tainan Women's College of Arts & Technology

In compliance with the economic internationalization movement and the development of Asia-Pacific Regional Operation Center (APROC) in Taiwan, international business has become more and more important. To sustain favorable trade balances every year and the promotion of APROC in Taiwan, more and more talent with knowledge and skills of Business English are needed. As a consequence, it is necessary to make Business English curriculum appropriate to meet the emerging needs.^ Two groups, experimental and control, received the revised or traditional Business English course to answer the question, "Does the Business English curriculum at Tainan Woman's College of Arts & Technology (TWCAT) meet the needs of students?" Ninety-five subjects were randomly selected from the commercial departments at TWCAT and then randomly assigned to the two groups. In addition, the Business English scores of the subjects' previous semester were collected and analyzed to justify the random selection and assignment. The finding was that their initial equivalence was proved.^ A questionnaire for students and another one for the business community were administered to facilitate data collection and analysis. The results of the questionnaires were used to modify the curriculum content of Business English.^ A final-term examination was given to the subjects at the end of the pilot study of Business English in early May of 1998. The resulting scores of the examination were used to determine if there was a significant difference in learning achievement between the students of the two groups.^ Using Independent Samples Test, significant results indicated that the experimental group had higher level of learning Business English than the control group. The finding supports the hypothesis of this study.^ Recommendations based on these results are that the revised curriculum be adapted and used by TWCAT because it better meets student needs. ^

Assessing the effectiveness of two-year banking and insurance technology programs of junior colleges in Taiwan, R.O.C.: An application of the Context Input Process Product (CIPP) evaluation model

The purpose of this descriptive study was to evaluate the banking and insurance technology curriculum at ten junior colleges in Taiwan. The study focused on curriculum, curriculum materials, instruction, support services, student achievement and job performance. Data was collected from a diverse sample of faculty, students, alumni, and employers. ^ Questionnaires on the evaluation of curriculum at technical junior colleges were developed for use in this specific case. Data were collected from the sample described above and analyzed utilizing ANOVA, T-Tests and crosstabulations. Findings are presented which indicate that there is room for improvement in terms of meeting individual students' needs. ^ Using Stufflebeam's CIPP model for curriculum evaluation it was determined that the curriculum was adequate in terms of the knowledge and skills imparted to students. However, students were dissatisfied with the rigidity of the curriculum and the lack of opportunity to satisfy the individual needs of students. Employers were satisfied with both the academic preparation of students and their on the job performance. ^ In sum, the curriculum of the two-year banking and insurance technology programs of junior college in Taiwan was shown to have served adequately preparing a work force to enter businesses. It is now time to look toward the future and adapt the curriculum and instruction for the future needs of the ever evolving high-tech society. ^

An analysis of student and instructor perceptions of students' writing skills at the university level

This study examined university students' writing skills as perceived by university students and their English instructors. The goal of the study was to provide English instructors with objective, quantified information about writing perceptions from both the students' and instructors' viewpoints. ^ A survey instrument was developed based on a survey instrument created by Newkirk, Cameron, and Selfe (1977) to identify instructors' perceived knowledge of student writing skills. The present study used a descriptive statistical design. It examined five writing skill areas: attitude, content, grammar and mechanics, literary considerations, and the writing process through a questionnaire completed by a convenience sample of summer and fall admitted freshmen who were enrolled in Essay Writing and Freshman Composition courses and English Department instructors at a large South Florida public university. ^ The study consisted of five phases. The first phase was modifying of the Newkirk, Cameron, and Selfe (1977) questionnaire. Two versions of the revised survey were developed - one for instructors and one for students. The second phase was pilot testing the questionnaire for evaluation of administration and scoring. The third phase was administering the questionnaire to 1,280 students and 48 instructors. The fourth phase was analyzing the data. The study found a significant difference in the perceptions of students and instructors in all areas of writing skills examined by the survey. Responses to 29 of 30 questions showed that students felt they had better attitudes toward writing and better writing skills than instructors thought. ^ The final phase was developing recommendations for practice. Based on findings and theory and empirical evidence drawn from the fields of adult education and composition research, learner-centered, self-directed curriculum guidelines are offered. ^ By objectively quantifying student and instructor perceptions of students' writing skills, this study contributes to a growing body of literature that: (a) encourages instructors to acknowledge the perception disparities between instructors and students; (b) gives instructors a better understanding of how to communicate with students; and (c) recommends the development of new curriculum, placement tests, and courses that meet the needs of students and enables English instructors to provide meaningful instruction. ^

GRE as a predictor of graduate student success at a Hispanic serving institution of higher education

Accurately predicting the success of graduate students is an important aspect of determining which students should be admitted into graduate programs. The GRE is a pivotal factor to examine since it is one of the most widely used criteria for graduate school admission. Even though the GRE is advertised as an accurate tool for predicting first year graduate GPA, there is a lack of research on long term success factors such as time to degree and graduate rate (Luthy, 1996; Powers, 2004). Furthermore, since most studies have low minority sample sizes, the validity of the GRE may not be the same across all groups (ETS, 2008b; Kuncel, Hezlett, & Ones, 2001). Another gap in GRE studies is that few researchers analyze student characteristics, which may alter or moderate the prediction validity of the GRE. Thus, student characteristics such as degree of academic involvement, mentorship interactions, and other academic and social experiences have not been widely examined in this context. These gaps in the analysis of GRE validity are especially relevant given the high attrition rates within of some graduate programs (e.g., an estimated 68% of doctoral student never complete their programs in urban universities; Lovitts, 2001). A sequential mixed methods design was used to answer the research questions in two phases. The quantitative phase used student data files to analyze the relationship of two success variables (graduation rate and graduate GPA) to the GRE scores as well as other academic and demographic graduate student characteristics. The qualitative phase served to complement the first phase by describing a wider range of characteristics from the 11 graduate students who were interviewed. Both proximal and distal moderators influence student behaviors and success in graduate school. In the first phase of the study, the GRE was the distal facilitator under analysis. Findings suggested that both the GRE Quantitative and the GRE Verbal were predictors of success for master’s students, but the GRE Quantitative was not predictive of success for doctoral students. Other student characteristics such as demographic variables and disciplinary area were also predictors of success for the population of students studied. In the second phase of the study, it was inconclusive whether the GRE was predictive of graduate student success; though it did influence access to graduate programs. Furthermore, proximal moderators such as student involvement, faculty/peer interactions, motivational factors, and program structure were perceived to be facilitators and/or detractors for success.

Developing a process to create and validate an instrument assessing student attainment of competencies at an intercultural university in Mexico

This study took place at one of the intercultural universities (IUs) of Mexico that serve primarily indigenous students. The IUs are pioneers in higher education despite their numerous challenges (Bertely, 1998; Dietz, 2008; Pineda & Landorf, 2010; Schmelkes, 2009). To overcome educational inequalities among their students (Ahuja, Berumen, Casillas, Crispín, Delgado et al., 2004; Schmelkes, 2009), the IUs have embraced performance-based assessment (PBA; Casillas & Santini, 2006). PBA allows a shared model of power and control related to learning and evaluation (Anderson, 1998). While conducting a review on PBA strategies of the IUs, the researcher did not find a PBA instrument with valid and reliable estimates. The purpose of this study was to develop a process to create a PBA instrument, an analytic general rubric, with acceptable validity and reliability estimates to assess students' attainment of competencies in one of the IU's majors, Intercultural Development Management. The Human Capabilities Approach (HCA) was the theoretical framework and a sequential mixed method (Creswell, 2003; Teddlie & Tashakkori, 2009) was the research design. IU participants created a rubric during two focus groups, and seven Spanish-speaking professors in Mexico and the US piloted using students' research projects. The evidence that demonstrates the attainment of competencies at the IU is a complex set of actual, potential and/or desired performances or achievements, also conceptualized as "functional capabilities" (FCs; Walker, 2008), that can be used to develop a rubric. Results indicate that the rubric's validity and reliability estimates reached acceptable estimates of 80% agreement, surpassing minimum requirements (Newman, Newman, & Newman, 2011). Implications for practice involve the use of PBA within a formative assessment framework, and dynamic inclusion of constituencies. Recommendations for further research include introducing this study's instrument-development process to other IUs, conducting parallel mixed design studies exploring the intersection between HCA and assessment, and conducting a case study exploring assessment in intercultural settings. Education articulated through the HCA empowers students (Unterhalter & Brighouse, 2007; Walker, 2008). This study aimed to contribute to the quality of student learning assessment at the IUs by providing a participatory process to develop a PBA instrument.

TEACHNJ: An Evaluation of Two Years Implementation

The Teacher Effectiveness and Accountability for the Children of New Jersey (TEACHNJ) Act was adopted by the New Jersey legislature in August 2012 with the intent to raise student achievement by improving the overall quality of instruction. As a result of this act, new teacher evaluation systems are being introduced in school districts across the state in an effort to more accurately assess teacher performance. The new teacher evaluations will be based on multiple classroom observations as well as the academic achievement of their students as measured on standardized tests. In addition, professional development opportunities are likely to change under this legislation, with schools customizing professional development programs to more effectively meet the needs of their teachers. The overarching question that informs our research is what impact will TEACH NJ have on the overall value of teacher evaluations and the quality of professional development opportunities offered to teachers. Data collected through survey research presents the pre-implementation practices (2011-2012 school year) as well as one year post-implementation practices (2013-2014) taking place in school districts throughout New Jersey. The findings reflect teachers’ perceptions of the value of the current teacher evaluation practices, the quality of the current professional development opportunities and the value the school administration places on teacher evaluations.

A study of the effects of a service-learning experience on student success at an urban community college

The purpose of this study was to explore the effects of a service-learning experience on student success as measured by class attendance, course completion, final course grades, and end-of-term evaluation data. Though many outcomes of service-learning experiences have been studied, including ethical values, self-esteem, student personal development, and career preparation, relatively few studies have been conducted on the effects of such experiences on academic achievement, and the studies that have been done have primarily studied students at traditional, four-year, residential universities. The study consisted of 286 students enrolled in six paired courses taught by five instructors at a community college in the Fall term 1996. One section of each pair (the control group) was taught using traditional subject matter and course materials and the other section of each pair (the treatment group) participated in a 20-hour required service- learning activity in addition to the regular course curriculum. The courses in the study included American History, Sociology, College Preparatory English, and Introduction to English Composition. The results of this study indicate that, overall, students who participated in a class in which service-learning was a requirement, achieved higher final course grades and reported greater satisfaction with the course, the instructor, the reading assignments, and the grading system, and the treatment section of one course pair had fewer absences. In addition, the faculty members reported that, in the treatment sections, class discussions were more stimulating, the sections seemed more vital in terms of student involvement, the students seemed more challenged academically, more motivated to learn, and seemed to exert more effort in the course.

Online Education, Accrediting Standards, and Student Success: An Examination of the Relationship Between the Southern Association of Colleges and Schools Commission on Colleges Standards for Online Education and Student Success

Prior to 2000, there were less than 1.6 million students enrolled in at least one online course. By fall 2010, student enrollment in online distance education showed a phenomenal 283% increase to 6.1 million. Two years later, this number had grown to 7.1 million. In light of this significant growth and skepticism about quality, there have been calls for greater oversight of this format of educational delivery. Accrediting bodies tasked with this oversight have developed guidelines and standards for online education. There is a lack of empirical studies that examine the relationship between accrediting standards and student success. The purpose of this study was to examine the relationship between the presence of Southern Association of Colleges and Schools Commission on College (SACSCOC) standards for online education in online courses, (a) student support services and (b) curriculum and instruction, and student success. An original 24-item survey with an overall reliability coefficient of .94 was administered to students (N=464) at Florida International University, enrolled in 24 university-wide undergraduate online courses during fall 2014, who rated the presence of these standards in their online courses. The general linear model was utilized to analyze the data. The results of the study indicated that the two standards, student support services and curriculum and instruction were both significantly and positively correlated with student success but with small R2 and strengths of association less than .35 and .20 respectively. Mixed results were produced from Chi-square tests for differences in student success between higher and lower rated online courses when controlling for various covariates such as discipline, gender, race/ethnicity, GPA, age, and number of online courses previously taken. A multiple linear regression analysis revealed that the curriculum and instruction standard was the only variable that accounted for a significant amount of unique variance in student success. Another regression test revealed that no significant interaction effect exists between the two SACSCOC standards and GPA in predicting student success. The results of this study are useful for administrators, faculty, and researchers who are interested in accreditation standards for online education and how these standards relate to student success.