How to write criterion-referenced tests for Spanish-English bilingual programs /

"The project reported herein was performed pursuant to a grant from the U.S. Office of Education, Department of Health, Education and Welfare."

Engaging Students in the Implementation of Criterion Referenced Assessment in First Year Law

The Queensland University of Technology (QUT) University Academic Board approved a new QUT Assessment Policy in September 2003, which requires a criterion-referenced approach as opposed to a norm-referenced approach to assessment across the university(QUT,MOPP,2003). In 2004, the QUT Law School embarked upon a process of awareness raising about criterion-referenced assessment amongst staff and from 2004 – 2005 staggered the implementation of criterion-referenced assessment in all first year core undergraduate law units. This paper will briefly discuss the benefits and potential pitfalls of criterion referenced assessment and the context for implementing it in the first year law program, report on student’s feedback on the introduction of criterion referenced assessment and the strategies adopted in 2005 to engage students more fully in criterion referenced assessment processes to enhance their learning outcomes.

Is implementing criterion-referenced assessment worth the effort with gen Y?

Well implemented criterion-referenced assessment (CRA) requires dedicated time and effort, especially in describing realistic expectations of evidence of achievement to students in the form of criteria sheets (or grading rubrics). It is also takes time out of delivering content to teach students how to judge their own work using criteria sheets. In 2007, to engage third year Microbiology students in using criteria sheets for the first time in their degree, we devised an innovative assessment tutorial supported by online resources. We were sceptical of much of the literature that reported ‘agreed’ characteristics of our predominantly gen Y cohort, because of the older ages of the majority of authors. These authors claim gen Y has a propensity for digital media, overconfidence in their own abilities and a collaborative orientation. We rejected this stereotype when developing the tutorial. Evaluations by students were positive and there was no dramatic change to grades for the unit. These results are similar to those in the literature for non gen Y cohorts. This lends support to our claim that giving students control over their own learning, irrespective of their generational label, is worth the time and effort.

Assessing reflection skills in law using criterion-referenced assessment

This paper focuses on the assessment of reflective practice, an issue that has not been fully explored within legal education literature. While the issue of how reflective practice should be taught is one that requires careful consideration, it is beyond the scope of this paper to consider both the teaching and the assessment of reflective practice. Part II of this paper conceptualises reflective practice, and Part III explores the benefits of reflective practice in legal education and the use of reflective writing to assess experiential learning in a legal context. Part IV considers the diverse issues that arise in assessing reflective practice and whether there is an objective method for assessing reflection. Part V of the paper examines the assessment of reflective practice in the context of an exemplar undergraduate law subject that uses a reflective report to assess students’ experiential learning during a court visit.14 Finally, Part VI offers a rubric to facilitate criterion-referenced assessment of reflective practice and thereby provides a framework for assessing reflection skills. It is suggested that the rubric is transferable not only to other law subjects but also to subjects in other disciplines.

Does the summative assessment of real world learning using criterion-referenced assessment need to be discipline specific?

This paper synthesises the existing literature on the contemporary conception of ‘real world’ and compares it with similar notions such as ‘authentic’ and ‘work integrated learning’. While the term ‘real world’ may be partly dependent on the discipline, it does not necessarily follow that the criterion-referenced assessment of ‘real world’ assessment must involve criteria and performance descriptors that are discipline specific. Two examples of summative assessment (court report and trial process exercise) from a final year core subject at the Queensland University of Technology, LWB432 Evidence, emphasise real world learning, are authentic, innovative and better prepare students for the transition into the workplace than more generic forms of assessment such as tutorial participation or oral presentations. The court report requires students to attend a criminal trial in a Queensland Court and complete a two page report on what they saw in practice compared with what they learned in the classroom. The trial process exercise is a 50 minute written closed book activity conducted in tutorials, where students plan questions that they would ask their witness in examination-in-chief, plan questions that they would ask their opponent’s witness in cross-examination, plan questions that they would ask in reexamination given what their opponent asked in cross-examination, and prepare written objections to their opponent’s questions. The trial process exercise simulates the real world, whereas the court report involves observing the real world, and both assessment items are important to the role of counsel. The design of the criterion-referenced assessment rubrics for the court report and trial process exercise is justified by the literature. Notably, the criteria and performance descriptors are not necessarily law specific and this paper highlights the parts that may be easily transferred to other disciplines.

Towards a classification of criterion referenced assessment models in mathematics courses : student and academic perspectives

We present the findings of a study into the implementation of explicitly criterion- referenced assessment in undergraduate courses in mathematics. We discuss students' concepts of criterion referencing and also the various interpretations that this concept has among mathematics educators. Our primary goal was to move towards a classification of criterion referencing models in quantitative courses. A secondary goal was to investigate whether explicitly presenting assessment criteria to students was useful to them and guided them in responding to assessment tasks. The data and feedback from students indicates that while students found the criteria easy to understand and useful in informing them as to how they would be graded, it did not alter the way the actually approached the assessment activity.

Enhancing fuzzy inference system based criterion-referenced assessment with an application

An important and difficult issue in designing a Fuzzy Inference System (FIS) is the specification of fuzzy sets, and fuzzy rules. The aim of this paper is to demonstrate how an additional qualitative information, i.e., monotonicity property, can be exploited and extended to be part of an FIS designing procedure (i.e., fuzzy sets and fuzzy rules design). In this paper, the FIS is employed as an alternative to the use of addition in aggregating the scores from test items/tasks in a Criterion-Referenced Assessment (CRA) model. In order to preserve the monotonicity property, the sufficient conditions of the FIS is proposed. Our proposed FIS based CRA procedure can be viewed as an enhancement for the FIS based CRA procedure, where monotonicity property is preserved. We demonstrate the applicability of the proposed approach with a case study related to a laboratory project assessment task at a university, and the results indicate the usefulness of the proposed approach in the CRA domain.

A construção e validação de um teste com referência a critério para medir as competências básicas de leitura e escrita para o Programa de Alfabetização Funcional do MOBRAL

A necessidade de se desenvolverem instrumentos adequados a avaliação de programas educacionais é premente no momento atual. Segundo os estudos mais recentes no campo da medida educacional, os testes com referência a critério são os que atendem a esta necessidade. Este estudo teve por objetivo a construção de dois testes deste tipo para a avaliação do Programa de Alfabetização Funcional do MOBRAL nos dominios da aquisição das competências basicas em leitura e escrita. Inicialmente foram selecionadas e descritas quatro competências para serem medidas por cada teste. As competências foram selecionadas tendo como base os objetivos do programa apresentados no Roteiro de Orientação ao Alfabetizador (1977). A segunda etapa do estudo consistiu na criação das especificações para cada competência selecionadas nos dois dominios. A partir destas especificações foram construidos 32 itens para o teste de leitura e 40 para o de escrita. Estes itens foram, distribuidos randomicamente em 5 formas experimentais para cada teste. O teste de leitura é composto de itens de múltipla escolha e o teste de escrita de itens de resposta construida. Os criterios de correção e os padrões minimos de desempenho foram estabelecidos para cada teste. Para a analise destes padrões minimos foram calculados as medias e os desvios-padrão para cada forma aplicada. Estes resultados permitem a manutenção dos padrões minimos usados em estudos posteriores. A análise dos itens foi feita em duas etapas: na primeira foram calculados os indices de discriminância pelo processo tradicional e na segunda os indices de sensitivida de pelo processo desenvolvido por Gronlund (1976). Os resultados obtidos permitiram o aproveitamento de todos os itens. Os itens foram divididos em duas formas equivalentes para cada teste, cada um pode ser subdividido em três subtestes para a medida isolada das competências descritas. Para o exame de validade foram estabelecidos os indices de validade descritiva e de seleção de dominio. A fidedignidade dos testes ficou assegurada com a construção de mais de 5 itens para competência Gronlund (1976).

Improving math and science assessment : report on the Secretary's third conference on mathematics and science education /

"PIP 94-1201"--P. [4] of cover.

The assessment of GCSE art: Criterion-referencing and cognitive abilities

In this paper Cognitive Abilities Test scores are compared directly with moderated GCSE scores awarded to the same group of pupils. For ease of interpretation the comparisons are presented in a graphical form. Whilst some provisional and tentative conclusions are drawn about the reliability of GCSE art, questions are raised about the general validity of criterion-referenced assessment in this area.

The effects of school mobility in pre -assessment grade levels on student performance and school accountability in urban Title I schools

This study examined the effects of student mobility and educational enrollment experiences on academic achievement. The educational progress, school enrollments and transfers of inner-city elementary students were tracked over a four-year period. Student achievement was measured by criterion-referenced reading tests administered in the second semester of the third grade. It further analyzed the degree to which the switch to different basal reading textbooks interrupted the continuity of education thereby contributing to the detrimental effects of intra-district mobility. ^ Mobility histories of 2,913 third grade students were collected to evaluate the number of times each student entered or withdrew from a Miami-Dade County Public School beginning in August 2000 through March 2004, and distinguished between transfers that occurred during the academic school year and those that occurred during summer months. Data were analyzed using Pearson correlations and multiple regressions to determine if school mobility contributed to performance on the Florida Comprehensive Assessment Third Grade Reading Test (FCAT). Transferring from one school to another was found to have a significant negative impact on student test scores. Transfers within the academic school year were more detrimental than transfers that occurred during the summer months. Third grade students who transferred into schools that used the same reading textbook series were found to have significantly higher FCAT reading scores than third graders who transferred into schools that used different reading textbooks. ^ The effects of mobility rates on overall school performance were also examined. Data was collected on 124 Title I elementary schools to determine the degree to which mobility affected school accountability scores. Title I schools with high student mobility rates had significantly lower accountability scores than schools with lower student mobility rates. ^ The results of this study highlight the impact of education and housing policy and imply a need for programs and practices that promote stability in the early elementary years. ^

Item and task difficulty in a B2 reading examination: perceptions of test-takers and CEFR alignment experts compared with psychometric measurements

The article presents a study of a CEFR B2-level reading subtest that is part of the Slovenian national secondary school leaving examination in English as a foreign language, and compares the test-taker actual performance (objective difficulty) with the test-taker and expert perceptions of item difficulty (subjective difficulty). The study also analyses the test-takers’ comments on item difficulty obtained from a while-reading questionnaire. The results are discussed in the framework of the existing research in the fields of (the assessment of) reading comprehension, and are addressed with regard to their implications for item-writing, FL teaching and curriculum development.