Speaking Assessment:
Oral Examiner Training

Cathy Taylor & Anthea Wilson, Trinity College London, UK

At the end of the two-day workshop the participants will have explored the key issues involved in recruiting, training and standardising oral examiners for face to face, direct tests. For those not directly involved in testing, for example teachers, the workshop will also address the application of testing principles to their contexts and how to incorporate these into their teaching.

Participants will have the opportunity to prepare, discuss and present a programme of oral examiner recruitment, training and standardisation appropriate for their tests and/or a programme to incorporate systematised oral assessment into a language course or test preparation course. The intention is for participants to be able to use the plans they develop in the workshop to review and develop current practices in their testing organisations and teaching contexts.

Participants will be required to work in oral testing or as teachers who prepare learners to take a specific test or who use tests in their classes or have experience of recruiting, training and standardising examiners or teachers. They will be expected to have an overview of the key challenges faced by their testing organisation, testing within a language course or in test preparation. As the participants will leave with an outline for examiner training or a plan to include testing as part of a language course, it is advisable that their role in their testing organisation or school/college will allow them to put this into practice. Please note this workshop will focus on f2f direct testing rather than online, however, there will be coverage of remote online training, standardisation and interviewing.

Tues 22 May:  2 – 5pm

Introduction to and discussion of the issues involved in recruiting, training and standardising oral examiners.

  • Overview and learning aims of the workshop
  • What kind of oral tests do the participants deliver?
  • What kind of tests do they currently use or would like to use?
  • Do they prepare learners for a specific test?
  • What are the challenges they face?
  • How have they dealt with the challenges so far?
  • Overview of the Trinity College London approach to oral examiner training

Wed 23 May: All day

Stage 1: Refer back to Trinity model from previous day, analyse the processes involved. At the end of each section participants discuss applicability to their context.

1. Recruiting examiners and those who use tests in a learning context – some approaches

  • Application form
  • Skype, 1-1, f2f interviews
  • Training on delivery and assessment – f2f and remote
  • 1-1 interviews
  • Online standardisation
  • References/police check/conflict of interest
  • Final selection process

2. Training oral examiners and teachers

  • Method of delivery f2f training, remote
  • Regularity of training – to fit with frequency of exam sessions
  • Training materials – use of live materials or not, ensuring confidentiality
  • Training content overview:
    1. Test purpose, test takers, test users, philosophy of the testing organisation, washback of the test on teaching, test impact
    2. Test format and procedures, examiner role and responsibilities
      • Assessment training – training on rating scales, video/audio benchmarked performances
  • Training documentation e.g. handbooks, support materials
  • Equipment and training
  • Follow up support for examiners/teachers

3. Selecting trainers & monitors (see below under QA for monitors)

  • Familiarity and association with test and organisation
  • Examining experience
  • Experience preparing learners to take a test
  • Experience using tests in a learning context
  • Training credentials: f2f, online
  • Conflict of interest – do the trainers also prepare candidates for the test?

4. Standardisation

  • Distinction between training and standardisation
  • Methods for standardisation: f2f, online
  • Number and level of assessment samples for analysis
  • Feedback to examiners and others who deliver the test
  • Support/further training for substandard assessments

5. Quality assurance for examiners and those involved in test preparation

  • Monitoring: f2f/remote
  • Security of live test items
  • Examiner management: contractual obligation, sanctions
  • Examiner performance review/ professional development
  • Support and development for substandard examining and assessment
  • Research and data collection and analysis

Stage 2: Participants review the stages above and begin preparation on a programme suitable to their contexts. 

Thurs 24 May: half a day
Participants present their outline programmes to the group/s. Presentations are followed by a Q &A session to give helpful suggestions to presenters.
Round up and summary
A closing session will address any outstanding queries.

The training will be a combination of whole group, small group and paired collaboration and discussion.  Some parts will be trainer-led but these will lead into group work. Video and audio performances of exams will be used for illustrative purposes. There will be handouts with key information and processes.

Participants need to review the training processes and issues they face in their organisations and be prepared to discuss in the workshop.  There is no specific reading list although the following will give context:

  • EALTA guidelines for Good Practice
  • Alderson, J. C., Clapham, C., Wall, D. 1995. Language Test Construction and Evaluation. Chapter 5. Cambridge, Cambridge University Press.

CATHY TAYLOR worked in teaching, teacher training and academic management in SE Asia, Australia, Europe and the UK for 20+ years. She has also worked as an examiner for International testing organisations. Since 2005 Cathy has worked at Trinity College London in Test Production and examiner training. She has an MA in Language Testing from Lancaster University and has presented and delivered workshops at international conferences including EALTA and ALTE. Cathy is currently head of Examiner development and standardisation and is responsible for the recruitment, training and standardisation of all Trinity examiners.

ANTHEA WILSON has taught ESOL to children and adults for over twenty years in South Africa, Portugal and the UK in a range of settings. As a teacher and manager she developed and managed a range of internal assessments and tests, and has been an examiner and marker for Cambridge ESOL and for Trinity College London. She has an MA in Language Testing from Lancaster University, and has presented on assessment issues at national and international conferences. Anthea has a research interest in the assessment of speaking, the roles of interlocutor and rater, and the interaction of the component parts in a speaking and listening exam. Currently Anthea is Head of test production for English language tests at Trinity College London.