The topics of the panel discussions
• Many exam bodies have introduced computer-based versions of their tests which are administered in a secure environment. Will computer-based tests become more popular than paper-based ones in the near future?
All panellists agreed that the future of the exams is digital. The vast majority of the people who voted on our online survey believes the same.
• Computer-based tests are more cost effective. Do you consider lowering the fees?
All panellists agreed that the development of computerized tests has the same challenges as the development of paper and pencil tests.
Issues relating to security, psychometric editing, reliability etc are common in both forms of testing. In addition new issues arise with computer-based testing (CBT) such as an increased risk of candidate cheating or item overexposure.
To make CBT as reliable as their paper and pencil tests testing organizations invest huge amounts of money in computer software and infrastructure.
So lowering the fees is not always possible. Prof. Bessie Dendrinos said that the fees for the KPG exams are very low and affordable even in harsh times such as the ones we are experiencing.
George Vassilakis said that the City & Guilds exams have reduced their fees by 30% and continue to offer a free retaking of one component of the exam for the immediate next session.
The above diagram however illustrates that people believe that computer-based testing is more cost effective probably due to the lack of administration costs.
• Does the cut score fluctuate from one session of the exam to the other? Is it due to the fact that items are analysed and their performance is measured after the exam administration?
All panellists said that the cut score of their exams does not fluctuate from one session of the exam to the other. However the people who took part in our online survey have a different opinion.
They think that in some exam sessions the tests are either easier or more difficult; in other words cut scores fluctuate.
• How do you ensure that your exams are valid and reliable? What processes are in place to ensure validity and reliability?
Reliability refers to the extent to which assessments are consistent. All panellists said that they strive to have reliable, consistent instruments to measure student achievement.
The values for reliability coefficients range from 0 to 1.0. A coefficient of 0 means no reliability and 1.0 means perfect reliability. Since all tests have some error, reliability coefficients never reach 1.0. Generally, if the reliability of a standardized test is above .80, it is said to have very good reliability; if it is below .50, it would not be considered a very reliable test.
Validity refers to the accuracy of an assessment; whether or not it measures what it is supposed to measure. Even if a test is reliable, it may not provide a valid measure. Since teachers, parents, and schools make decisions about students based on assessments, the validity inferred from the assessments is essential - even more crucial than the reliability. Also, if a test is valid, it is almost always reliable.
There are three ways in which validity can be measured. In order to have confidence that a test is valid (and therefore the inferences we make based on the test scores are valid), all three kinds of validity evidence should be considered.
| Type of Validity | Definition |
| Content | The extent to which the content of the test matches the instructional objectives. |
| Criterion | The extent to which scores on the test are in agreement with (concurrent validity) or predict (predictive validity) an external criterion. |
| Construct | The extent to which an assessment corresponds to other variables, as predicted by some rationale or theory. |
Testing organizations provide information about the validity and reliability of their exams on their websites. The procedures they follow are publicised and accessible to stakeholders.
The majority of our voters appear to have a sound knowledge of test validity and reliability and that procedures followed by examination bodies are transparent to test takers as the diagram below illustrates.
However there are many teachers who think differently. This should alarm examination bodies.
• Are your exams pretested? How important do you think it is?
All panellists agreed that pretesting is a vital element in the examination development process. Trialling exam materials with students before they are used in exams, helps them ensure that:
1. their exams are accurate and fair, with each individual item pitched at the right level
2. test content is appropriate and that people from all backgrounds can do equally well.
After the pretesting session, extensive analysis of the level and quality of test tasks and items is carried out and further editing of tasks takes place. The best quality tasks are then selected for the actual examination.
About 55% of the voters know that examination bodies pretest questions and content but there is a considerable number of teachers who probably do not know that such a procedure exists.
• Are you aware of the uses to which your exam results are put? Do you think the uses made of your tests are fair?
All panellists admitted that language exams results are very important for test takers.
In many cases they determine whether a person would be allowed to enter a country for working or study purposes.
In the case of Greece, C2 Level exams results determine whether a test taker would get a teaching license and make a living in the private sector.
Again about 55% of the voters think the results are used fairly but there are those who have either no opinion or are rather sceptical about the use and misuse of language tests.
• What would you advise Foreign Language School Owners to do with borderline candidates, which seem to constitute the vast majority of test-takers in Greece taking into account that students’ test scores shape the reputation of schools and the communities they serve?
The above diagram speaks for itself. Foreign Language Schools place great importance on exams results for their survival as businesses.
There is no doubt that a lot of classroom time is spent on exam preparation. A huge exam machine is set in operation every day in every corner of the country.
Students and teachers strive to achieve the best possible results. All panellists focused on the achievements of the Greek candidates and praised their teachers for their commitment and effort.
