UNDERSTANDING ESSAY RATING AS A SOCIALLY MEDIATED ACTIVITY: THE CASE OF A HIGH-STAKES ENGLISH TEST IN CHINA

Most essay rating research in language assessment has examined human raters’ essay rating as a cognitive process, thus overlooking or oversimplifying the interaction between raters and sociocultural contexts. Given that raters are social beings, their practices have social meanings and consequences. Hence it is important to situate essay rating within its sociocultural context for a more meaningful understanding. Drawing on Engeström’s (1987, 2001) cultural-historical activity theory (CHAT) framework with a sociocultural perspective, this study reconceptualized essay rating as a socially mediated activity with both cognitive (individual raters’ goal-directed decision-making actions) and social layers (raters’ collective object-oriented essay rating activity at related settings). In particular, this study explored raters’ essay rating at one provincial rating centre in China within the context of a high-stakes university entrance examination, the National Matriculation English Test (NMET). This study adopted a multiple-method multiple-perspective qualitative case study design. Think-aloud protocols, stimulated recalls, interviews, and documents served as the data sources. This investigation involved 25 participants at two settings (rating centre and high schools), including rating centre directors, team leaders, NMET essay raters who were high school teachers, and school principals and teaching colleagues of these essay raters. Data were analyzed using Strauss and Corbin’s (1990) open and axial coding techniques, and CHAT for data integration. The findings revealed the interaction between raters and the NMET sociocultural context. Such interaction can be understood through a surface structure (cognitive layer) and a deep structure (social layer) concerning how raters assessed NMET essays, where the surface structure reflected the “what” and the deep structure explained the “how” and “why” in raters’ decision-making. This study highlighted the roles of goals and rules in rater decision-making, rating tensions and raters’ solutions, and the relationship between essay rating and teaching. This study highlights the value of a sociocultural view to essay rating research, demonstrates CHAT as a sociocultural approach to investigate essay rating, and proposes a direction for future washback research on the effect of essay rating. This study also provides support for NMET rating practices that can potentially bring positive washback to English teaching in Chinese high schools.

LibQUAL+ Survey 2004 @ Queen's University Library : analysis of results

Queen's University Library was one of 202 libraries, including 57 members of the Association of Research Libraries (ARL), to survey its users in spring 2004 using the LibQUAL+ survey instrument. LibQUAL+ was designed by ARL to assist libraries in assessing the quality of their services and identifying areas for improvement. # Overall: Queen's scored higher than the average for all ARL participants and 1st among the 2004 Canadian participants. This relatively high rating is due to very high scores in the dimensions of Library as Place and Affect of Service. However, there is considerable need for improvement in the area of Information Control where Queen's rated well below the ARL average. # Affect of Service: Queen's strong overall ratings are supported by the many respondent comments praising customer service throughout the system. The ratings and survey comments indicate greatest appreciation by faculty and more experienced students (e.g. graduate students) for the instruction and on-site services provided by the libraries. The ratings also indicate that undergraduates, growing up with the web, want and expected to be able to access library resources independently and do not value these services as highly. The comments also indicated some specific areas for improvement throughout the library system. # Library as Place : All Queen's libraries except for Law ranked well above the ARL and Canadian averages. Overall, Library as Place ranked lowest in importance among the service dimensions for all ARL participants including Queen's. Comparative analysis of LibQUAL results since the survey began shows a decline in “desired” ratings for Library as Place. However, undergraduates continue to give strong "desired" ratings to certain aspects of Library as Place and a relatively high rating for "minimum expected" service. The comments from Queen's survey respondents and ARL's analyses of focus groups indicate that undergraduates value the library much more as a place to study and work with peers rather than for its on-site resources and services. # Information Control: This is the area in greatest need of attention. While it ranked highest in importance for all user groups by a wide margin, Queen's performed poorly in this category. Overall, Queen's ranked far below both the ARL average and the top three Canadian scores. However, the major dissatisfaction was concentrated in the humanities/social sciences (Stauffer primary users) and the health sciences (Bracken primary users) where the overall rating of perceived service quality ranked below the minimum expected service rating. Primary users of the Education, Engineering/Science and Law libraries rated this service dimension higher than the ARL average. The great success of the Canadian National Site License Program (CNSLP) is reflected in the high overall rating generated by Engineering/Science Library users. The low ratings from the humanities and social sciences are supported by respondents' comments and are generally consistent with other ARL participants.