Global and local challenges for best practices in assessment. The 9th Conference on the International Test Commission. Book of abstracts

The 9th International Test Commission Conference (ITC) took place at the Miramar Palace in San Sebastian, Spain, between the 2nd and 5th of July, 2014. The Conference was titled, “Global and Local Challenges for Best Practices in Assessment.” The International Test Commission, ITC (www.intestcom.org), is an association of national psychological associations, test commissions, publishers, and other organizations, as well as individuals who are committed to the promotion of effective testing and assessment policies and to the proper development, evaluation, and uses of educational and psychological instruments. The ITC facilitates the exchange of information among members and stimulates their cooperation on problems related to the construction, distribution, and uses of psychological and educational tests and other psychodiagnostic tools. This volume contains the abstracts of the contributions presented at the 9th International Test Commission Conference. The four themes of the Conference were closely linked to the goals of the ITC: - Challenges and Opportunities in International Assessment. - Application of New Technoloogies and New Psychometric Models in Testing. - Standards and Guidelines for Best Testing Practices. - Testing in Multilingual and Multicultural Contexts.

Team-teaching for inclusive learning: Purposes, practices and perceptions of a team-teaching initiative in Irish post-primary schools

Background. Schools unequivocally privilege solo-teaching. This research seeks to enhance our understanding of team-teaching by examining how two teachers, working in the same classroom at the same time, might or might not contribute to the promotion of inclusive learning. There are well-established policy statements that encourage change and moves towards the use of team-teaching to promote greater inclusion of students with special educational needs in mainstream schools and mainstream classrooms. What is not so well established is the practice of team-teaching in post-primary settings, with little research conducted to date on how it can be initiated and sustained, and a dearth of knowledge on how it impacts upon the students and teachers involved. Research questions and aims. In light of the paucity and inconclusive nature of the research on team-teaching to date (Hattie, 2009), the orientating question in this study asks ‘To what extent, can the introduction of a formal team-teaching initiative enhance the quality of inclusive student learning and teachers’ learning at post-primary level?’ The framing of this question emerges from ongoing political, legal and educational efforts to promote inclusive education. The study has three main aims. The first aim of this study is to gather and represent the voices and experiences of those most closely involved in the introduction of team-teaching; students, teachers, principals and administrators. The second aim is to generate a theory-informed understanding of such collaborative practices and how they may best be implemented in the future. The third aim is to advance our understandings regarding the day-to-day, and moment-to-moment interactions, between teachers and students which enable or inhibit inclusive learning. Sample. In total, 20 team-teaching dyads were formed across seven project schools. The study participants were from two of the seven project schools, Ash and Oak. It involved eight teachers and 53 students, whose age ranged from 12-16 years old, with 4 teachers forming two dyads per school. In Oak there was a class of first years (n=11) with one dyad and a class of transition year students (n=24) with the other dyad. In Ash one class group (n=18) had two dyads. The subjects in which the dyads engaged were English and Mathematics. Method. This research adopted an interpretive paradigm. The duration of the fieldwork was from April 2007 to June 2008. Research methodologies included semi-structured interviews (n=44), classroom observation (n=20), attendance at monthly teacher meetings (n=6), questionnaires and other data gathering practices which included school documentation, assessment findings and joint examination of student work samples (n=4). Results. Team-teaching involves changing normative practices, and involves placing both demands and opportunities before those who occupy classrooms (teachers and students) and before those who determine who should occupy these classrooms (principals and district administrators). This research shows how team-teaching has the potential to promote inclusive learning, and when implemented appropriately, can impact positively upon the learning experiences of both teachers and students. The results are outlined in two chapters. In chapter four, Social Capital Theory is used in framing the data, the change process of bonding, bridging and linking, and in capturing what the collaborative action of team-teaching means, asks and offers teachers; within classes, between classes, between schools and within the wider educational community. In chapter five, Positioning Theory deductively assists in revealing the moment-to-moment, dynamic and inclusive learning opportunities, that are made available to students through team-teaching. In this chapter a number of vignettes are chosen to illustrate such learning opportunities. These two theories help to reveal the counter-narrative that team-teaching offers, regarding how both teachers and students teach and learn. This counter-narrative can extend beyond the field of special education and include alternatives to the manner in which professional development is understood, implemented, and sustained in schools and classrooms. Team-teaching repositions teachers and students to engage with one another in an atmosphere that capitalises upon and builds relational trust and shared cognition. However, as this research study has found, it is wise that the purposes, processes and perceptions of team-teaching are clear to all so that team-teaching can be undertaken by those who are increasingly consciously competent and not merely accidentally adequate. Conclusions. The findings are discussed in the context of the promotion of effective inclusive practices in mainstream settings. I believe that such promotion requires more nuanced understandings of what is being asked of, and offered to, teachers and students. Team-teaching has, and I argue will increasingly have, its place in the repertoire of responses that support effective inclusive learning. To capture and extend such practice requires theoretical frameworks that facilitate iterative journeys between research, policy and practice. Research to date on team-teaching has been too focused on outcomes over short timeframes and not focused enough on the process that is team-teaching. As a consequence team-teaching has been under-used, under-valued, under-theorised and generally not very well understood. Moving from classroom to staff room and district board room, theoretical frameworks used in this research help to travel with, and understand, the initiation, engagement and early consequences of team-teaching within and across the educational landscape. Therefore, conclusions from this study have implications for the triad of research, practice and policy development where efforts to change normative practices can be matched by understandings associated with what it means to try something new/anew, and what it means to say it made a positive difference.

An investigation of assessment policy and practices in terms of broad social and educational purposes

Based upon relevant literature, this study investigated the assessment policy and practices for the BSc (Hons) Computing Science programme at the University of Greenwich (UOG), contextualising these in terms of broad social and educational purposes. It discusses Assessment, and then proceeds to give a critical evaluation of the assessment policy and practices at the UOG. Although this is one case study, because any of the features of the programme are generic to other programmes and institutions, it is of wider value and has further implications. The study was concluded in the summer of 2002. It concludes that overall, the programme's assessment policy and practices are well considered in terms of broad social and educational purposes, although it identifies and outlines several possible improvements, as well as raising some major issues still to be addressed which go beyond assessment practices.

An assessment of the business value of traceability practices in the italian fishery processing industry

Traceability is often perceived by food industry executives as an additional cost of doing business, one to be avoided if possible. However, a traceability system can in fact comply the regulatory requirements, increase food safety and recall performance, improving marketing performances and, as well as, improving supply chain management. Thus, traceability affects business performances of firms in terms of costs and benefits determined by traceability practices. Costs and benefits affect factors such as, firms’ characteristics, level of traceability and ,lastly, costs and benefits perceived prior to traceability implementation. This thesis was undertaken to understand how these factors are linked to affect the outcome of costs and benefits. Analysis of the results of a plant level survey of the Italian ichthyic processing industry revealed that processors generally adopt various level of traceability while government support appears to increase the level of traceability and the expectations and actual costs and benefits. None of the firms’ characteristics, with the exception of government support, influences costs and level of traceability. Only size of firms and level of QMS certifications are linked with benefits while precision of traceability increases benefits without affecting costs. Finally, traceability practices appear due to the request from “external“ stakeholders such as government, authority and customers rather than “internal” factors (e.g. improving the firm management) while the traceability system does not provide any added value from the market in terms of price premium or market share increase.

Assessment of soil erosion and soil conservation practices in Angereb watershed, Ethiopia: Technological and land user context

Soil conservation technologies that fit well to local scale and are acceptable to land users are increasingly needed. To achieve this at small-holder farm level, there is a need for an understanding of specific erosion processes and indicators, the land users’ knowledge and their willingness, ability and possibilities to respond to the respective problems to decide on control options. This study was carried out to assess local erosion and performance of earlier introduced conservation terraces from both technological and land users’ points of view. The study was conducted during July to August 2008 at Angereb watershed on 58 farm plots from three selected case-study catchments. Participatory erosion assessment and evaluation were implemented along with direct field measurement procedures. Our focus was to involve the land users in the action research to explore with them the effectiveness of existing conservation measures against the erosion hazard. Terrace characteristics measured and evaluated against the terrace implementation guideline of Hurni (1986). The long-term consequences of seasonal erosion indicators had often not been known and noticed by farmers. The cause and effect relationships of the erosion indicators and conservation measures have shown the limitations and gaps to be addressed towards sustainable erosion control strategies. Less effective erosion control has been observed and participants have believed the gaps are to be the result of lack of landusers’ genuine participation. The results of both local erosion observation and assessment of conservation efficacy using different aspects show the need to promote approaches for erosion evaluation and planning of interventions by the farmers themselves. This paper describes the importance of human factor involving in the empirical erosion assessment methods towards sustainable soil conservation.

'Worldmarks': Web guidelines for socially and culturally responsive assessment in university classrooms

In the early 21st century, we need to prepare university students to navigate local and global cultures effectively and sensitively. These future professionals must develop comprehensive intercultural communication skills and understanding. Yet university assessment in Australia is often based on a western template of knowledge, which automatically places International, Indigenous, as well as certain groups of local students at a study disadvantage. It also ensures that Australian students from dominant groups are not given the opportunity to develop these vital intercultural skills. This paper explores the issues embedded in themes 1 and 4 of this conference and provides details of an innovative website developed at Queensland University of Technology in Brisbane, Australia, which encourages academic staff to investigate the hidden assumptions that can underpin their assessment practices. The website also suggests strategies academics can use to ensure that their assessment becomes more socially and culturally responsive.

Viewbrics: Formative Assessment of Complex Skills with Video-Enhanced Rubrics (VER) in Dutch Secondary Education

To learn complex skills, like collaboration, learners need to acquire a concrete and consistent mental model of what it means to master this skill. If learners know their current mastery level and know their targeted mastery level, they can better determine their subsequent learning activities. Rubrics support learners in judging their skill performance as they provide textual descriptions of skills’ mastery levels with performance indicators for all constituent subskills. However, text-based rubrics have a limited capacity to support the formation of mental models with contextualized, time-related and observable behavioral aspects of a complex skill. This paper outlines the design of a study that intends to investigate the effect of rubrics with video modelling examples compared to text-based rubrics on skills acquisition and feedback provisioning. The hypothesis is that video-enhanced rubrics, compared to text based rubrics, will improve mental model formation of a complex skill and improve the feedback quality a learner receives (from e.g. teachers, peers) while practicing a skill, hence positively effecting final mastery of a skill.

ESL teacher profiles of ICT integration in their classroom practices and assessment activities : a portrait viewed through the lens of some Quebec teachers’ social representations

Abstract : Information and communication technologies (ICTs, henceforth) have become ubiquitous in our society. The plethora of devices competing with the computer, from iPads to the Interactive whiteboard, just to name a few, has provided teachers and students alike with the ability to communicate and access information with unprecedented accessibility and speed. It is only logical that schools reflect these changes given that their purpose is to prepare students for the future. Surprisingly enough, research indicates that ICT integration into teaching activities is still marginal. Many elementary and secondary schoolteachers are not making effective use of ICTs in their teaching activities as well as in their assessment practices. The purpose of the current study is a) to describe Quebec ESL teachers’ profiles of using ICTs in their daily teaching activities; b) to describe teachers’ ICT integration and assessment practices; and c) to describe teachers’ social representations regarding the utility and relevance of ICT use in their daily teaching activities and assessment practices. In order to attain our objectives, we based our theoretical framework, principally, on the social representations (SR, henceforth) theory and we defined most related constructs which were deemed fundamental to the current thesis. We also collected data from 28 ESL elementary and secondary school teachers working in public and private sectors. The interview guide used to that end included a range of items to elicit teachers’ SR in terms of ICT daily use in teaching activities as well as in assessment practices. In addition, we carried out our data analyses from a textual statistics perspective, a particular mode of content analysis, in order to extract the indicators underlying teachers’ representations of the teachers. The findings suggest that although almost all participants use a wide range of ICT tools in their practices, ICT implementation is seemingly not exploited to its fullest potential and, correspondingly, is likely to produce limited effects on students’ learning. Moreover, none of the interviewees claim that they use ICTs in their assessment practices and they still hold to the traditional paper-based assessment (PBA, henceforth) approach of assessing students’ learning. Teachers’ common discourse reveals a gap between the positive standpoint with regards to ICT integration, on the one hand, and the actual uses of instructional technology, on the other. These results are useful for better understanding the way ESL teachers in Quebec currently view their use of ICTs, particularly for evaluation purposes. In fact, they provide a starting place for reconsidering the implementation of ICTs in elementary and secondary schools. They may also be useful to open up avenues for the development of a future research program in this regard.

Respecting local, cultural contexts for assessment practice in an era of globalization

Dissatisfaction, internationally, with existing educational practices and outcomes since the early 1990s has led to increased educational reform. At the same time, there has also been a worldwide shift in control of education away from teachers toward the state for the purposes of restructuring economies. More bureaucratic forms of curriculum and assessment have resulted, with a return to the use of more techno-rational discourse in assessment and evaluation for purposes of efficiency, accountability, impact, and performance management. There has also been an increase in the use of economic and productivity models to study educational outcomes. These models fail to account for the range of outcomes achieved and fail to identify the factors responsible for such diversity in performance.

Assessment and curriculum reform in China : the college English test and tertiary English as a foreign language education.

Educational assessment was a worldwide commonplace practice in the last century. With the theoretical underpinnings of education shifting from behaviourism and social efficiency to constructivism and cognitive theories in the past two decades, the assessment theories and practices show a widespread changing movement. The emergent assessment paradigm, with a futurist perspective, indicates a deviation away from the prevailing large scale high-stakes standardised testing and an inclination towards classroom-based formative assessment. Innovations and reforms initiated in attempts to achieve better education outcomes for a sustainable future via more developed learning and assessment theories have included the 2007 College English Reform Program (CERP) in Chinese higher education context. This paper focuses on the College English Test (CET) - the national English as a Foreign Language (EFL) testing system for non-English majors at tertiary level in China. It seeks to explore the roles that the CET played in the past two College English curriculum reforms, and the new role that testing and assessment assumed in the newly launched reform. The paper holds that the CET was operationalised to uplift the standards. However, the extended use of this standardised testing system brings constraints as well as negative washback effects on the tertiary EFL education. Therefore in the newly launched reform -CERP, a new assessment model which combines summative and formative assessment approaches is proposed. The testing and assessment, assumed a new role - to engender desirable education outcomes. The question asked is: will the mixed approach to formative and summative assessment provide the intended cure to the agony that tertiary EFL education in China has long been suffering - spending much time, yet achieving little effects? The paper reports the progresses and challenges as informed by the available research literature, yet asserts a lot needs to be explored on the potential of the assessment mix in this examination tradition deep-rooted and examination-obsessed society.

Supervision practices in allied mental health : relationships of supervision characteristics to perceived impact and job satisfaction

This study examined whether supervision characteristics impacted on mental health practice and morale, and developed a new Supervision Attitude Scale (SAS). Telephone surveys were conducted with a representative sample of 272 staff from public mental health services across Queensland. Although supervision was widely received and positively rated, it had low average intensity, and assessment and training of skills was rarely incorporated. Perceived impact on practice was associated with acquisition of skills and positive attitudes to supervisors, but extent of supervision was related to impact only if it was from within the profession. Intention to resign was unrelated to extent of supervision, but was associated with positive attitudes to supervisors, accessibility, high impact, and empathy or praise in supervision sessions. The SAS had high internal consistency, and its intercorrelations were consistent with it being a measure of relationship positivity. The study supported the role of supervision in retention and in improving practice. It also highlighted supervision characteristics that might be targeted in training, and provided preliminary data on a new measure.

Governmentality and the reflection of legal educators : assessment practices as a case study

This paper presents a conceptual framework, informed by Foucault’s work on governmentality, which allows for new kinds of reflection on the practice of legal education. Put simply, this framework suggests that legal education can be understood as a form of government that relies on a specific rationalisation and programming of the activities of legal educators, students, and administrators, and is implemented by harnessing specific techniques and bodies of ‘know-how’. Applying this framework to assessment at three Australian law schools, this paper highlights how assessment practices are rationalised, programmed, and implemented, and points out how this government shapes students’ legal personae. In particular, this analysis focuses on the governmental effects of pedagogical discourses that are dominant within the design and scholarship of legal education. It demonstrates that the development of pedagogically-sound regimes of assessment has contributed to a reformulation of the terrain of government, by providing the conditions under which forms of legal personae may be more effectively shaped, and extending the power relations that achieve this. This analysis provides legal educators with an original way of reflecting on the power effects of teaching the law, and new opportunities for thinking about what is possible in legal education.