One of the claims of language tests administered in Greece is that topical knowledge is not tested so that all candidates may stand the same chance of performing well. However, topical knowledge, especially in the form of intercultural knowledge, is one of the constructs that should be tested according to the CEFR by which all language tests are supposed to abide. In addition, speaking topics come from a variety of areas and interests, which, by nature, would mean that familiarity with a particular topic could constitute an advantage for some candidates in relation to others who lack it. Within this framework, this article will present a longitudinal study conducted among Greek students who were preparing for the University of Michigan ECCE and ECPE in a language school. During this study, the students were given a range of topics from a variety of areas. The results which will be presented were based both on the students’ performance according to the criteria set for the specific speaking exam, as well as on interviews conducted with the students following each test on a particular topic, in the hope of shedding light on whether topical knowledge can actually have an effect on candidate performance in the speaking module of a language test.
By Panagiota Frytzala, Lecturer BA in TESOL, New York College, Athens, Greece
The study
The aim of the study was to investigate the effect of topical knowledge on test-takers’ performance in a high-stakes speaking test. Within this framework, the main research questions focused on examining whether poor performances can be attributed to the lack of topical knowledge and what would then be the implications for test development as well as for teaching.
The reason why the ECCE and ECPE exams were chosen is that they are high-stakes exams during which the candidates don’t have a choice of topic as in other exams, but instead the topic is administered in a mostly random way. In addition, the topics are not differentiated by age or nationality or any other criterion, which made these exams more appropriate for this particular study.
As for the methodology, the sample was four students in an ECCE class and four students in an ECPE class in the same language school. They were all teenagers except for one student who was an adult preparing for the ECPE, and there were two boys and six girls. The study lasted six months, namely from November to April, and 10 ECCE prompts and 7 ECPE ones were administered. The main criterion in the selection of these prompts was the different kind of topical knowledge needed in relation to the students’ interests and real world knowledge taking into account mainly their age. The procedure involved first of all the familiarization of the students with the format of the test and the scoring criteria, and then the administration of the test followed by interviews with students after every test on which parts of the prompt presented difficulties for them. Finally, students’ scores were compared for each prompt taking into account the relevant comments made by them during the interviews.
Results of the study
What became obvious through both the scores of students and their comments during the follow-up interviews is that familiarity with the topic did boost their performance. This was linked to two factors, namely how close to their interests the topic was and how much intercultural knowledge was needed to tackle the task. In some cases, when topical knowledge was not completely lacking in students, they were still able to get a passing grade, but there were cases when lack of topical knowledge would actually result in a failing grade. However, this was more obvious for the ECPE students rather than for the ECCE ones, mainly because the prompts for the latter are by definition less demanding than for the former.
To illustrate, starting with ECCE, the topic which was clearly quite difficult for all of them was the one asking them to choose a location for a training conference for managers of local banks. This was due to the fact that the idea of a training conference is not very familiar to teenagers, so some of them even misunderstood the topic and believed that the task involved choosing a holiday destination. However, even in this case, students were able to get through the task successfully though their grade was lower.
In the case of ECPE, topical knowledge played a more decisive role in the performance of the students who faced quite a few problems with the vocabulary in prompts. More specifically, vocabulary pertaining to university studies such as “majored in” or “PhD” rendered the task more challenging, as was the case with work-related vocabulary or points. For example, one of the points presented as a disadvantage was the fact that a person changed jobs quite often, but teenage students failed to see why this would be a negative point for a person applying for a job. However, they were still able to get a lower passing grade, which was not the case with the topic on choosing a location for a business conference as they had difficulty understanding what this involved. For example, points pertaining to the electronic equipment available in different locations was seen as a means to entertain the participants and not as an essential part of the conference.
Conclusion
As shown by this study, topical knowledge does have an effect on candidates’ performance, especially when it requires some more specific intercultural elements. This has implications for both test developers and teachers. In the case of the former, it means that topical knowledge, especially in relation to the L2 culture, should be tested but kept to a minimum, and that topics should be tailored to the average age and interests of test takers so as to ensure the validity, authenticity, interactiveness and impact of the test. In addition, since topical content is linked to the ability to interpret discourse, which in turn is linked to language ability, such knowledge should not be excluded from tests, which is something that should also concern teachers who prepare students for exams. More specifically, it would be a good idea for teachers to cover as many topics as possible and focus especially on topics which students are not very familiar with due to their young age or even their different culture so as for students to be better prepared to deal with more challenging topics in the speaking test.
