Sustainable assessment for large science classes : non-multiple choice, randomised assignments through a Learning Management System

This paper reports on the development of a tool that generates randomised, non-multiple choice assessment within the BlackBoard Learning Management System interface. An accepted weakness of multiple-choice assessment is that it cannot elicit learning outcomes from upper levels of Biggs’ SOLO taxonomy. However, written assessment items require extensive resources for marking, and are susceptible to copying as well as marking inconsistencies for large classes. This project developed an assessment tool which is valid, reliable and sustainable and that addresses the issues identified above. The tool provides each student with an assignment assessing the same learning outcomes, but containing different questions, with responses in the form of words or numbers. Practice questions are available, enabling students to obtain feedback on their approach before submitting their assignment. Thus, the tool incorporates automatic marking (essential for large classes), randomised tasks to each student (reducing copying), the capacity to give credit for working (feedback on the application of theory), and the capacity to target higher order learning outcomes by requiring students to derive their answers rather than choosing them. Results and feedback from students are presented, along with technical implementation details.

Multiple-choice versus short-response items: Differences in omit behavior

The overall rate of omission of items for 28,331 17 year old Australian students on a high stakes test of achievement in the common elements or cognitive skills of the senior school curriculum is reported for a subtest in multiple choice format and a subtest in short response format. For the former, the omit rates were minuscule and there was no significant difference by gender or by type of school attended. For the latter, where an item can be 'worth' up to five times that of a single multiple choice item, the omit rates were between 10 and 20 times that for multiple choice and the difference between male and female omit rate was significant as was the difference between students from government and non-government schools. For both formats, females from single sex schools omitted significantly fewer items than did females from co-educational schools. Some possible explanations of omit behaviour are alluded to.

Optimal Correction for Guessing in Multiple-Choice Tests

Building on Item Response Theory we introduce students’ optimal behavior in multiple-choice tests. Our simulations indicate that the optimal penalty is relatively high, because although correction for guessing discriminates against risk-averse subjects, this effect is small compared with the measurement error that the penalty prevents. This result obtains when knowledge is binary or partial, under different normalizations of the score, when risk aversion is related to knowledge and when there is a pass-fail break point. We also find that the mean degree of difficulty should be close to the mean level of knowledge and that the variance of difficulty should be high.

On the strategic equivalence of multiple-choice test scoring rules

A disadvantage of multiple-choice tests is that students have incentives to guess. To discourage guessing, it is common to use scoring rules that either penalize wrong answers or reward omissions. These scoring rules are considered equivalent in psychometrics, although experimental evidence has not always been consistent with this claim. We model students' decisions and show, first, that equivalence holds only under risk neutrality and, second, that the two rules can be modified so that they become equivalent even under risk aversion. This paper presents the results of a field experiment in which we analyze the decisions of subjects taking multiple-choice exams. The evidence suggests that differences between scoring rules are due to risk aversion as theory predicts. We also find that the number of omitted items depends on the scoring rule, knowledge, gender and other covariates.

Type Systems for a Network Specification Language With Multiple-Choice Let

When analysing the behavior of complex networked systems, it is often the case that some components within that network are only known to the extent that they belong to one of a set of possible "implementations" – e.g., versions of a specific protocol, class of schedulers, etc. In this report we augment the specification language considered in BUCSTR-2004-021, BUCS-TR-2005-014, BUCS-TR-2005-015, and BUCS-TR-2005-033, to include a non-deterministic multiple-choice let-binding, which allows us to consider compositions of networking subsystems that allow for looser component specifications.

Investigating Multiple-Choice Assessment: Secondary School Teachers' Perceptions and Practices

Multiple-choice assessment is used within nearly all levels of education and is often heavily relied upon within both secondary and postsecondary institutions in determining a student’s present and future success. Understanding why it is effective or ineffective, how it is developed, and when it is or is not used by teachers can further inform teachers’ assessment practices, and subsequently, improve opportunities for student success. Twenty-eight teachers from 3 secondary schools in southern Ontario were interviewed about their perceptions and use of multiple-choice assessment and participated in a single-session introductory workshop on this topic. Perceptions and practices were revealed, discussed, and challenged through the use of a qualitative research method and examined alongside existing multiple-choice research. Discussion centered upon participants’ perspectives prior to and following their participation in the workshop. Implications related to future assessment practices and research in this field of assessment were presented. Findings indicated that many teachers utilized the multiple-choice form of assessment having had very little teacher education coursework or inservice professional development in the use of this format. The findings also revealed that teachers were receptive to training in this area but simply had not been exposed to or been given the opportunity to further develop their understanding. Participants generally agreed on its strengths (e.g., objectivity) and weaknesses (e.g., development difficulty). Participants were particularly interested in the potential for this assessment format to assess different levels of cognitive difficulty (i.e., levels beyond remembering of Bloom’s revised taxonomy), in addition to its potential to perhaps provide equitable means for assessing students of varying cultures, disabilities, and academic streams.

Evaluation of five guidelines for option development in multiple-choice item-writing. 'Evaluaci??n de cinco directrices para la construcci??n de opciones en ??tems de elecci??n m??ltiple'

Resumen tomado de la publicaci??n

Are multiple-choice tests a suitable method to evaluate students' knowledge? : against the extensive use of multiple-choice evaluation test. '¿Son los test de opción múltiple un método adecuado para evaluar el conocimiento de los estudiantes? : en contra del uso extensivo del test de evaluación de opción múltiple'.

Resumen basado en el de la publicación

Verbal reasoning : multiple choice : 11 + practice papers.

Son pruebas de inteligencia objetivas para evaluar las habilidades verbales de los alumnos en un sentido más amplio que el que proporciona el contenido específico de un programa de estudios. Su objetivo es tratar de que identifiquen modelos, similitudes y diferencias entre palabras, y demostrar, además, su comprensión de las reglas y del significado específico del lenguaje en diferentes contextos.

Non-verbal reasoning : multiple choice : 11 + practice papers.

Son pruebas de inteligencia cuyas preguntas no tienen una solución que puede ser aprendida de antemano.Son utilizadas, entre otras finalidades, para conocer de los escolares de ocho a catorce años, sus capacidades para comprender y asimilar información novedosa, independientemente de sus habilidades lingüísticas.

English : multiple choice : 11 + practice papers.

Estos tests evalúan el nivel de inglés alcanzado por los niños durante la educación primaria, y les proporciona, además, la práctica para las pruebas de selección de la escuela secundaria.

Practice multiple-choice papers suitable for : key skills level 1 communication, level 1 adult literacy (basic skills), and ESOL skills for life level 1 reading.

Este recurso puede ser utilizado tanto por los tutores como por los estudiantes que siguen las habilidades básicas, alumnos adultos de alfabetización en la evaluación final en el nivel 1 y los estudiantes de ESOL (English for speakers of other languages) en lectura. Contiene 12 cuestionarios, cada uno con 40 preguntas de opción múltiple para evaluar la capacidad del candidato para identificar los puntos e ideas principales de los escritos, entender el significado de los documentos y su estilo de escritura, interpretar la información de tablas, gráficos y mapas, y reconocer la ortografía, la gramática y la puntuación. También, incluye las respuestas y las hojas de evaluación para fotocopiar.

Practice multiple-choice papers suitable for : key skills level 2 communication, level 2 adult literacy (basic skills), and ESOL skills for life level 2 reading.

Este recurso puede ser utilizado tanto por los tutores como por los estudiantes que siguen las habilidades básicas, alumnos adultos de alfabetización en la evaluación final en el nivel 2 y los estudiantes de ESOL (English for speakers of other languages) en lectura. Contiene 12 cuestionarios, cada uno con 40 preguntas de opción múltiple para evaluar la capacidad del candidato para identificar los puntos e ideas principales de los escritos, entender el significado de los documentos y su estilo de escritura, interpretar la información de tablas, gráficos y mapas, y reconocer la ortografía, la gramática y la puntuación. También, incluye las respuestas y las hojas de evaluación para fotocopiar.

Practice multiple-choice papers suitable for : key skills level 2 application of number and level 2 adult numeracy.

Este recurso puede ser utilizado por los tutores que imparten las habilidades básicas y la aritmética elemental para adultos como por los estudiantes en la evaluación final del nivel 2. Contiene 12 cuestionarios, cada uno de ellos con 40 preguntas de opción múltiple y cuya finalidad es evaluar la capacidad de los candidatos para interpretar la información de tablas, diagramas, cuadros y gráficos y utilizar los cálculos que implican, entre otras, las siguientes habilidades: números, fracciones, decimales y porcentajes; números enteros, positivos y negativos; decimales y porcentajes; peso y capacidad; área, perímetro y volumen; fórmulas. También, incluye las respuestas y las hojas de evaluación para fotocopiar.