Quality Assurance in the Assessment of Receptive Language Skills:
Test Construction and Item Writing
Monique Reichert & Philipp Sonnleitner, University of Luxembourg, Luxembourg
This workshop is designed to familiarize participants with the basic notions of item writing and test construction in the context of reading and listening skills assessments. Workshop participants will learn how to deal with and consider different item response formats, and reflect on the construct that can be tackled by a variety of items. The workshop will also offer scope to reflect on those aspects that are important in the construction of a test by imparting basic notions about test quality properties and criteria (e.g., test reliability), and the analysis of test results. After the workshop, each participant should be able to develop, deliver, and analyse assessments of receptive language skills according to up-to-date quality standards. Crucially, the workshop places special emphasis on learning transfer to the participants’ own field of activity.
The workshop is designed for a maximum number of 30 participants. Participants do not need to have any previous experience in designing items or constructing a test; however, they are encouraged to bring their ideas or even their own test items or draft test material. Although the workshop will be conducted in English, participants can also bring their material in French or German, since the workshop leaders are familiar with French, German and English. Active participation is expected and highly welcomed.
Quite a few challenges are related to writing qualitatively good items and composing tests of receptive language skills. The first stage within the workshop will consist in dealing with and specifying the construct to be measured, which will be followed by the development of concrete items in small groups. This second, practical stage is embedded in a number of theoretical considerations that aim at making aware of the most crucial aspects that need to be reflected on in the item writing process. This step involves the discussion of different response formats, their relation and interaction with the tested construct, the cognitive basis of certain errors and mistakes in the test-taking process, but also the balancing of item difficulty within the test compilation. By exchanging developed items among the groups, this third stage will highlight the particular value of establishing peer review practices, and aims at consolidating previous discussed topics.
The final phase of the workshop will tackle strategies for time-efficient coding and analysis of test takers’ responses. In the closing session, illustrative items will be presented to the participants together with authentic test taker responses. This will not only stress the central importance of analysing item responses for future test development(s), but also show how they can be used to further learn about the tested construct itself.
The workshop consists of alternating theoretical and practical parts. By explicitly drawing on prior experiences and knowledge from the participants, the transfer to everyday testing will be assured. After each theoretical block, structured and guided discussions among the participants will allow to share and develop the participants’ understanding and reflection on item and test quality. Participants will also be asked to develop their own items, and the workshop organizers will provide a number of illustrative item examples, too. This will allow illustrating different ways of assessing receptive language skills in an optimal way. To further base the workshop’s content on participants’ prior knowledge and expectations, participants will be asked prior to the workshop to answer a small questionnaire provided by the workshop leaders.
Dr MONIQUE REICHERT is a psychologist specialized in language assessment, cognitive science issues and empirical methods. From 2002 to 2004 she worked as a research assistant at LIFE Research & Consult, where she was involved in research in the domain of second and foreign language learning. In 2004, she joined the EMACS Research Unit of the University of Luxembourg (today: Luxembourg Centre for Educational Testing – LUCET), where her main areas of work lie in the development and appraisal of language assessment methods. Since 2006, she is responsible for the development of language tests – in particular French and German reading comprehension and C-tests – used in the context of large scale monitoring projects aiming at evaluating specific aspects of the Luxembourg educational system. She took part in different European projects relating to language evaluation, such as in the EBAFLS, and the CEF-ESTIM project (http://cefestim.ecml.at), and is regularly training teachers in the context of the development of language tests. Together with Philipp, she is managing and coordinating the ‘Test Development Lab’, which aims at implementing quality assurance procedures regarding item and test development. She received her PhD for her work on the validity of C-tests.
Dr PHILIPP SONNLEITNER specializes in psychological assessment and was involved in developing large-scale assessment instruments in various domains (e.g. mathematics, reading comprehension, listening comprehension, collaborative problem-solving) for the Austrian School Monitoring program, PISA, and PIAAC. Since 2012, he is working at the Luxembourg Centre for Educational Testing (LUCET) and applying his expertise to the Luxembourg School Monitoring program. Together with Monique, he manages and coordinates the ‘Test Development Lab’, which aims at implementing quality assurance procedures regarding item and test development. Throughout his career, he has taught numerous workshops and courses on fair assessment and test development at universities, teacher training colleges, and schools. By his research and teaching activities, he aims at making the world (of assessment) a better place. Philipp received his Masters degree in Psychology from the University of Vienna in 2007 and is a licensed work- and occupational psychologist. His master thesis received the German Society of Psychology’s excellence award. In 2015, he received his PhD from the Free University of Berlin for his work on the computer-based assessment of complex problem-solving.