Darren Perrett is Senior Assessment Services, Recognition and Pedagogy Manager. He heads up a team of experts at Cambridge responsible for assessment, pedagogy, and recognition in the region and has been working in language assessment for over 10 years. He holds a master’s degree in Language Testing from Lancaster University and is currently in his final year of his PhD focused on AI language testing validation techniques.
Is performance in exams affected by candidates’ cognitive development or by other factors?
This is an important consideration, especially in a country like Greece, where children tend to start learning English and taking exams early. This is why we take great care to develop age-appropriate tests which provide progression for learners as their level of English and their cognitive maturity develop. Our exams for young learners - Pre-A1 Starters, A1 Movers and A2 Flyers - are a fantastic example of this approach. These exams are designed to not put too much pressure on children and to make learning English enjoyable and fun.
Text by: Anastasia Spyropoulou
(For more information on the cognitive development of children and assessment of different skills you can read SILT 47 Examining Young Learners.)
Moving up the levels we then have our ‘for schools’ series of tests, starting at A2 Key for Schools and finishing in B2 First for Schools. In these tests, the content is written to suit the development level of school-aged learners. In C1 Advanced and C2 Proficiency, the scope widens and is more appropriate for teenagers or adults preparing for university entrance. These are really in-depth qualifications that show you have the language skills that employers and universities are looking for.
Do ELT exams assess cognitive skills such as critical thinking, problem-solving, and creativity?
Our exams focus on real life English language skills, and do not assess life competencies, such as critical-thinking and problem solving. However, it is important to note that preparing for our exams helps to develop a wide range of learning and thinking skills, as well as the teamwork involved in a paired speaking test, for example. We have developed the Cambridge Life Competencies Framework which is used as a basis of our curriculum design of our learning materials, which is also used to support teachers in this area. It’s quite a hot topic at the moment and the question of how to assess life competencies was raised at the Association of Language Testers in Europe (ALTE) conference in Madrid, as it is becoming clear that stakeholders across Europe and beyond see this as the next step in reporting candidates’ performance in language learning.
In which B2 Level skill(s) do Greek candidates perform better? In which skill(s) are they weak?
Our most recent figures show more than 90% of Greek candidates were successful in their B2 First for Schools exam, which is really impressive. There are small differences in scores across the papers, with slightly better average scores in Speaking and slightly lower in Writing, but overall we can see that learners and their teachers are successfully developing all four skills.
At level B2 and above, in writing we have noticed that the “language” and “organisation” sub-skills need some attention. Students answer the task really well in “content” and have good “communicative achievement” scores. Therefore, if teachers target the other sub-skills, student scores will get better. In reading, it is more generalised, and we believe this is due to processing speeds of very complex texts. By reading more complex texts with advanced lexis and grammar, not only will students be able to get higher scores in reading, but the more that less frequent language is encountered and processed, the more likely students will also be able to use it in writing and speaking.
Do you see significant differences in pass rates across different countries or regions, and what factors do you think contribute to these variations?
It’s very difficult to compare performance across countries due to varying sample sizes, different ages, and so on. If we look at the example of B2 First for schools, Greece has a mean score which is the same as Sweden. However, it is difficult to compare the two. For example, in Greece there are thousands of candidates, but in other countries there might only be a small number.
- *How do you ensure that pass rates are consistent and fair across different testing locations and populations?
When a new test is introduced, and periodically on an ongoing basis, we carry out standard setting procedures to align test scores to the CEFR. This is a rigorous process undertaken by our world leading assessment experts.
Their task, for example, is to identify what constitutes a B2 at grade A, B and C in a multiple-matching task type. This is conducted for all task types in every test from Cambridge and it helps us to ensure that the mapped CEFR score is as valid and reliable as possible, helping us to provide consistency on a global scale. For example, we can be confident that a candidate in Greece with a B2 First grade A is the same as a candidate from Mexico of the same level, and also that they would receive the same score if they sat another B2 test.
We also take a rigorous approach to marking. Most items are marked by computer, and only the short-answer and writing and speaking responses are marked by humans. Short-answer responses have a fixed number of possible answers which are listed for examiners to mark against ensuring 100% reliability. For the open-ended writing responses in the writing part of the test, examiners are trained and standardised for each of the CEFR levels to maximise reliability scores. The examiners mark against a standardised mark scheme and are monitored for their performance on a regular basis. Furthermore, an examiner’s leniency or stringency is adjusted post-test to ensure more reliable marking.
How do you work with language educators to help students better prepare for language exams, and what strategies do you recommend to improve pass rates?
We carry out a programme of research on how people learn and use English so that our unique approach to learning and assessment is effective, efficient, and enjoyable. Working with teachers and partners, we ensure we're learning from your classrooms so we can give you the best quality help and support available. This also creates insights that we use to make sure that our exams are fair, accurate and meaningful.
We offer specific direct support to educators in the region through our series of professional learning and development webinars and face-to-face events, customized to your requirements in order to help your English language world grow, and to help students achieve their peak performance on the exam day.
The question of how to increase pass rates is an interesting one. Within our tests we have specific tasks which are intended to test a specific part of the construct of a given proficiency level. For example, in the multiple matching task type (B2 First Part 7) it is intended to test expeditious reading rather than careful local reading. It is a reading task testing for specific information, detail, opinion, and attitude. If a student approaches this task type reading each word, clause, and sentence carefully, they might find they run out of time and be forced to guess answers.
Therefore, my main advice for teachers and students is to fully understand what Cambridge are testing for. Look at each of the task types, understand the correct approach for each and prepare in a timed manner, under exam conditions. It is also important that the student understands at the item level what is being tested. For example, if a question starts with “what is the author’s opinion…” then they should know that this is a careful global reading; they have to bring together meaning from across all paragraphs, not only answer based on text in the last paragraph. Unfortunately, it is not possible for me to list out all of my recommendations here in this interview, but look out for events and webinars we’re holding across our regions for further updates!
( https://www.cambridgeenglish.org/gr/events/ )
How do you ensure that exam questions are culturally appropriate and do not disadvantage test takers from specific cultural backgrounds?
We have detailed and rigorous test specifications, which are used by teams of experts with many years’ experience from across the world. These experts are responsible for ensuring that test content is not culturally offensive and does not bias a particular test taker cohort. We spend many hours researching and debating this topic and do our very best to ensure that what is in the final test, is of the highest possible quality. Further to these human checks, as part of the pretesting stage of item production, we receive statistical information from across multiple demographics. This allows us to analyse the variation between the language families and spot if the item is biasing, negatively or positively, a particular sub-set. When the item is live in a test, we also analyse live data to spot any discrepancies and make validation and marking adjustments if necessary, or in extreme cases, remove the item from any future tests.
What points should teachers who read your interview take away?
It is our job at Cambridge to provide you with the most valid and reliable test of English. As part of Cambridge University, we help to deliver its mission which is to contribute to society through the pursuit of education, learning and research at the highest international levels of excellence. As such we are a not-for-profit organisation and are one of only two English language exam boards in the world to have the ALTE Q-Mark, which is a sign of quality in language assessment. It is due to this quality that we have global worldwide recognition. This means that students can use their certificate and test score throughout the world, to gain entrance to university, and to prove their skills to future employers.
Greek students are some of the most committed learners of English in the world! In fact, you have the highest percentage takers of C2 Proficiency in the world. And where would students be without amazing teachers? So well done Greek students and teachers!
I’d like to end by saying best of luck to all students currently preparing for Cambridge English tests this year! Personally, I am very much looking forward to having further discussions with Greek teachers again soon to show them that Cambridge is where your world grows!
