Factors associated with prompt difficulty in automated essay scoring

This study explores factors related to the prompt difficulty in Automated Essay Scoring. The sample was composed of 6,924 students. For each student, there were 1-4 essays, across 20 different writing prompts, for a total of 20,243 essays. E-rater® v.2 essay scoring engine developed by the Educational Testing Service was used to score the essays. The scoring engine employs a statistical model that incorporates 10 predictors associated with writing characteristics of which 8 were used. The Rasch partial credit analysis was applied to the scores to determine the difficulty levels of prompts. In addition, the scores were used as outcomes in the series of hierarchical linear models (HLM) in which students and prompts constituted the cross-classification levels. This methodology was used to explore the partitioning of the essay score variance.^ The results indicated significant differences in prompt difficulty levels due to genre. Descriptive prompts, as a group, were found to be more difficult than the persuasive prompts. In addition, the essay score variance was partitioned between students and prompts. The amount of the essay score variance that lies between prompts was found to be relatively small (4 to 7 percent). When the essay-level, student-level-and prompt-level predictors were included in the model, it was able to explain almost all variance that lies between prompts. Since in most high-stakes writing assessments only 1-2 prompts per students are used, the essay score variance that lies between prompts represents an undesirable or "noise" variation. Identifying factors associated with this "noise" variance may prove to be important for prompt writing and for constructing Automated Essay Scoring mechanisms for weighting prompt difficulty when assigning essay score.^

Explaining the effects of class size on faculty and students

This study explored the effects of class size on faculty and students. Specifically, it examined the relationship of class size and students' participation in class, faculty interactive styles, and academic environment and how these behaviors affected student achievement (percentage of students passing). The sample was composed of 629 students in 30 sections of Algebra I at a large, urban community college. A survey was administered to the students to solicit their perceptions on their participation in class, their faculty interaction style, and the academic environment in their classes. Selected classes were observed to triangulate the findings. The relationship of class size to student participation, faculty interactive styles, and academic environment was determined by using hierarchical linear modeling (HLM). A significant difference was found on the participation of students related to class size. Students in smaller classes participated more and were more engaged than students in larger classes. Regression analysis using the same variables in small and large classes showed that faculty interactive styles significantly predicted student achievement. Stepwise regression analyses of student and faculty background variables showed that (a) students' estimate of GPA was significantly related to their achievement (r = .63); (b) older students reported more participation than did younger ones, (c) students in classes taught by female, Hispanic faculty earned higher passing grades, and (d) students' participation was greater with adjunct professors. Class observations corroborated these findings. The analysis and observational data provided sufficient evidence to warrant the conclusion that small classes were not always most effective in promoting achievement. It was found that small classes may be an artifact of ineffectual teaching, actual or by reputation. While students in small classes participate and are more engaged than students in larger classes, the class-size effect is essentially due to what happens in instruction to promote learning. The interaction of the faculty with students significantly predicted students' achievement regardless of class size. Since college students select their own classes, students do not register for classes taught by faculty with poor teaching reputation, thereby leading to small classes. Further studies are suggested to determine reasons why classes differ in size.