Item and task difficulty in a B2 reading examination: perceptions of test-takers and CEFR alignment experts compared with psychometric measurements
| Data(s) |
26/09/2016
|
|---|---|
| Resumo |
The article presents a study of a CEFR B2-level reading subtest that is part of the Slovenian national secondary school leaving examination in English as a foreign language, and compares the test-taker actual performance (objective difficulty) with the test-taker and expert perceptions of item difficulty (subjective difficulty). The study also analyses the test-takers’ comments on item difficulty obtained from a while-reading questionnaire. The results are discussed in the framework of the existing research in the fields of (the assessment of) reading comprehension, and are addressed with regard to their implications for item-writing, FL teaching and curriculum development. |
| Formato |
application/pdf |
| Identificador |
http://revistas.ucm.es/index.php/CLAC/article/view/53487 10.5209/CLAC.53487 |
| Publicador |
Ediciones Complutense |
| Relação |
http://revistas.ucm.es/index.php/CLAC/article/view/53487/49044 /*ref*/Alderson, J. C., and Lukmani, Y. (1989) Cognition and Reading: Cognitive Levels as Embodied in Test Questions. Reading in a Foreign Language 5 (2), 253-270. /*ref*/Alderson, J. C. (2000) Assessing Reading. Cambridge: Cambridge University Press. /*ref*/Alderson, J. C., Figueras, N., Kuijper, H., Nold, G., Takala, S., and Tardieu, C. (2004) The development of specifications for item development and classification within The Common European Framework of Reference for Languages: Learning, Teaching, Assessment: Reading and Listening: Final report of The Dutch CEF Construct Project.Working paper. Retrieved February 26, 2014, from Lancaster University Publications & Outputs: http://eprints.lancs.ac.uk/44/1/final_report.pdf /*ref*/Apostolou, E. (2010) Comparing Perceived and Actual Task and Text Difficulty in the Assessment of Listening Comprehension. Papers from the LAEL PG 2010 (pp. 26-47). Lancaster: Department of Linguistics and English Language (LAEL), Lancaster University. Retrieved April 7, 2014, from http://www.ling.lancs.ac.uk/pgconference/v05/Apostolou.pdf /*ref*/Bachman, L. F. (1990) Fundamental Considerations in Language Testing. Oxford: Oxford University Press. /*ref*/Bachman, L. F., and Palmer, A. S. (1996) Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press. /*ref*/Baily, K. M. (1998) Learning about language assessment: Dilemmas, decisions, and directions. Boston: Heinle Cengage Learning. /*ref*/Bitenc Peharc, S., and Tratnik, A. (2014) Umestitev nacionalnih izpitov iz angleščine v skupni evropski okvir. Zaključno poročilo o izvedbi projekta. Ljubljana: Državni izpitni center. /*ref*/Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956) Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York: David McKay Company. /*ref*/Brunfaut, T. and McCray, G. (2015) Looking into test-takers’ cognitive processes whilst completing reading tasks: A mixed-method eye-tracking and stimulated recall study. (ARAGs Research Reports – Online. Vol. 1, No. 1). London: British Council. /*ref*/Cizek, G. J. (Ed.). (2001) Setting Perfomance Standards, Concepts, Methods, and Perspectives. Mahwah, New Jersey, London: Lawrence Erlbaum Associates, Publishers. /*ref*/Council of Europe. (2001) Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press. /*ref*/Council of Europe. (2009) Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR). A Manual. Strasbourg: Language Policy Division. /*ref*/Efklides, A. (2002) Feelings and judgments as subjective evaluations of cognitive processing: how reliable are they? Psychology: The Journal of the Hellenic Psychological Society 9, 163e184. /*ref*/Figueras, N. (2012) The impact of the CEFR. ELT Journal 66 (4), 477-485. DOI: 10.1093/elt/ccs037 /*ref*/Fortus, R., Coriat, R. and Fund, S. (1998) Prediction of item difficulty in the English subtest of Israel’s inter-university psychometric entrance test. In A. Kunnan (Ed.), Validation in Language Assessment: Selected Papers from the 17th Language Research Colloquium (pp. 61-87). Long Beach, Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc. /*ref*/Freedle, R. O., and Kostin, I. W. (1993) Item Difficulty for Expository Prose Passages for Three Item Types: Main Idea, Inference, and Supporting Idea Items. Princeton, New Jersey: Educational Testing Service. /*ref*/Freedle, R., and Kostin, I. (1999) Does the text matter in a multiple-choice test of comprehension? the case for the construct validity of TOEFL's minitalks. Language Testing 16 (2), 2-32. Retrieved April 17, 2014, from http://ltj.sagepub.com/content/16/1/2. DOI: 10.1177/026553229901600102 /*ref*/Fulcher, G. (2004) Deluded by Artifices? The Common European Framework and Harmonization. Language Assessment Quarterly 1(4), 253-266. DOI: 10.1207/s15434311laq0104_4 /*ref*/Fulmer, S. M., and Tulis, M. (2013) Changes in interest and affect during a difficult reading task: Relationships with perceived difficulty and reading fluency. Learning and Instruction 27, 11-20. DOI:10.1016/j.learninstruc.2013.02.001 /*ref*/Grabe, W. (2009) Reading in a Second Language. Moving from Theory to Practice. Cambridge: Cambridge University Press. /*ref*/Grabe, W. (1991) Current developments in second-language reading research. TESOL Quarterly 25 (3), 375–406. /*ref*/Green, A. B. (2014) Exploring Language Assessment and Testing: Language in Action. London and New York: Routledge. /*ref*/Hambleton, R., and Jirka, S. (2006) Anchor-based methods for judgmentally estimating item statistics. In S. Downing and T. Haladyna (Eds.), Handbook of test development (pp. 399-420). Mahwah, Nj: Erlbaum. /*ref*/Harding, L., Alderson, C. and Brunfaut, T. (2015) Diagnostic assessment of reading and listening in a second or foreign language: elaborating on diagnostic principles. Language Testing. Prepublished January 27, 2015. DOI:10.1177/0265532214564505. /*ref*/Hinkel, E. (2010) Integrating the four skills: Current and historical perspectives. In R. B. Kaplan (Ed.), Oxford Handbook in Applied Linguistics, Second Edition (pp. 110-126). New York: Oxford University Press. /*ref*/Ilc, G. and Stopar, A. (2014) Validating the Slovenian national alignment to CEFR: The case of the B2 reading comprehension examination in English. Language Testing. Prepublished December 23, 2014. DOI:10.1177/0265532214562098. /*ref*/Ilc, G., Rot Gabrovec, V., and Stopar, A. (2014) Relating the Slovenian secondary-school English language national examinations to the CEFR: Findings and implications. Linguistica 54 (1), 293-308. DOI:10.4312/linguistica.54.1.293-308. /*ref*/Kaftandijeva, F. (2010) Methods for Setting Cut Scores in Criterion-referenced Achievement Tests. A comparative analysis of six recent methods with an application to tests of reading in EFL. Cito, Arnhem: EALTA. Retrieved February 5, 2014, from http://www.ealta.eu.org/documents/resources/FK_second_doctorate.pdf /*ref*/Khalifa, H., and Weir, C. J. (2009) Examining Reading: Research and Practice in Assessing Second Language Reading. Cambridge: Cambridge University Press. /*ref*/Koda, K. (2005) Insights into Second Laguage Reading. A Cross-Linguistic Approach. Cambridge: Cambridge University Press. /*ref*/Little, D. (2007) The Common European Framework of Reference for Languages: Perspectives on the Making of Supranational Language Education Policy. The Modern Language Journal 91 (4), 645–655. DOI:10.1111/j.1540-4781.2007.00627_2.x /*ref*/Lund, R. J. (1991) A Comparison of Second Language Listening and Reading Comprehension. The Modern Language Journal, 75(2), 196-204. Retrieved April 28, 2014, from http://www.jstor.org/stable/328827. DOI:10.1111/j.1540-4781.1991.tb05350.x /*ref*/Martyniuk, W. (Ed.). (2010) Aligning Tests with the CEFR: Reflections on Using the Council of Europe's Draft Manual (Studies in Language Testing). Cambridge: Cambridge University Press. /*ref*/Miles, M. B. and Huberman, A. M. (1994) An Expanded Sourcebook: Qualitative Data Analysis. Second Edition. London: London Sage Publications. /*ref*/Munby, J. (1978) Communicative Syllabus Design. Cambridge: Cambridge University Press. /*ref*/Robinson, P. (2001) Task Complexity, Task Difficulty, and Task Production: Exploring Interactions in a Componential Framework. Applied Linguistics 22 (1), 27-57. DOI:10.1093/applin/22.1.27 /*ref*/Spaan, M. (2007) Evolution of a test item. Language Assessment Quarterly 4, 279-293. DOI: 10.1080/15434300701462937 /*ref*/Sydorenko, T. (2011) Item Writer Judgments of Item Difficulty Versus Actual Item Difficulty: A Case Study. Language Assessment Quarterly 8, 34-52. DOI: 10.1080/15434303.2010.536924 /*ref*/Šifrar Kalan, M., and Trenc, A. (2014) Relating reading comprehension in the Spanish as a foreign language national exam to the CEFR: some aspects of evaluation. Linguistica 54 (1), 309-323. DOI:10.4312/linguistica.54.1.309-323. /*ref*/Urquhart, A. H., and Weir, C. (1998) Reading in a Second Language: Process, Product and Practice. London, New York: Longman. /*ref*/Wu, R. Y. (2011) Establishing the Validity of the General English Proficiency Test Reading Component through a Critical Evaluation on Alignment with the Common European Framework of Reference (Unpublished doctoral dissertation). University of Bedfordshire, Bedford, Luton Bedfordshire, and Milton Keynes, Buckinghamshire, UK. Retrieved June 12, 2014, from http://uobrep.openrepository.com/uobrep/bitstream/10547/223000/1/wu%20ESTABLISHING%20THE%20VALIDITY%20OF%20THE%20GENERAL.pdf |
| Direitos |
LICENCE OF USE: The full text articles included on the Scientific Journals of the Complutense website are open access and the property of their authors and/or publishers. Therefore, any reproduction, distribution, public communication and/or total or partial transformation requires their express and written consent. Links to the full text of the articles on the Scientific Journals of the Complutense website should be to the official URL of the Complutense University of Madrid. LICENCIA DE USO: Los artículos a texto completo incluidos en el Portal de Revistas Científicas Complutenses son de acceso libre y propiedad de sus autores y/o editores. Por tanto, cualquier acto de reproducción, distribución, comunicación pública y/o transformación total o parcial requiere el consentimiento expreso y escrito de aquéllos. Cualquier enlace al texto completo de los artículos del Portal de Revistas Científicas Complutenses debe efectuarse a la URL oficial de la Universidad Complutense de Madrid |
| Fonte |
Círculo de Lingüística Aplicada a la Comunicación; Vol 67 (2016); 318-342 |
| Palavras-Chave | #reading comprehension; difficulty perception; CEFR alignment judges; psychometric measurement |
| Tipo |
info:eu-repo/semantics/article Artículo revisado por pares info:eu-repo/semantics/publishedVersion |
