You are here: Home / The IPN / Archive / Knowledge or Strategy?

Knowledge or Strategy?

How strategies can help improve test results

Gun-Brit Thoma and Olaf Köller

Knowledge, learning outcomes and competences are often measured by tests. Multiple choice questions are an important test format. There is much to be said for their use, because tests using multiple-choice format questions are easy to use, economical and objective in their evaluation. Multiple-choice tests have established themselves in many areas, such as the theoretical driving test, medical examinations or large-scale comparative education studies such as PISA. Multiple-choice tests are also used at school to check learning outcomes. However, multiple-choice tests also pose problems, for example candidates can use certain strategies to answer questions successfully, even if they lack the topic knowledge. We took a closer look at this problem in our study.

Multiple-choice tests are designed to reflect the level of knowledge people have on a particular subject or to show what skills and competences they possess. It is assumed that the more questions a person answers correctly, the higher the level of knowledge or competence. Test-wiseness has also been reported in the literature since the middle of the last century in connection with multiple-choice tasks. Test-wiseness refers to a person's ability to achieve a high score in a multiple-choice test by identifying the correct answer using specific clues and characteristics of the question. These pointers can occur accidentally during test construction.

In the Anglo-American region, the degree of test-wiseness is also partly assessed in examinations. Although the significance of test-wiseness is well known, the topic seems to play a subordinate role in German-speaking countries. This is shown, among other things, by the fact that there is no adequate translation of the term into German. To date, for example, there has also been no German-language test to capture test-wiseness.

>> A high score in a multiple-choice test can be achieved by answering the question correctly using specific pointers and characteristics of the questions. <<

What is meant by test-wiseness?

As mentioned above, test-wiseness refers to the ability of a person to use certain strategies to correctly answer as many questions as possible in a test, independently of the topic knowledge, by means of unintentional hints from the authors of the questions. As early as the 1960s, two categories of test-wiseness were described in the Anglo-American region. The first includes strategies independent of test purpose and test construction, the second includes strategies dependent on test purpose and test development.

Time use and error avoidance strategies as well as strategies which affect the intention help the examinee to not achieve a bad test result for reasons other than a lack of topic knowledge. Guessing, inference and strategies using hints are used to correctly answer questions in addition to those that can be answered based on topic knowledge. Several of these strategies can be applied during a test.

For example, a person can first answer all the questions they are convinced they know the right answer to. Then, for those questions that have not yet been answered, they can search for clues in the answers to answer them correctly. Test-wiseness thus has an effect on the test results, since the strategies mentioned can lead to the right solution, although the correct answer is not known. It is uncertain how strong this influence is. In addition, the question arises as to whether test-wiseness and the topic knowledge of a person are really independent of each other. It seems more likely that test-wiseness is at least to some extent related to topic knowledge.

Is test-wiseness measurable?

As early as the 1960s, a test was developed in the Anglo-American region with which test-wiseness can be measured. It is a multiple-choice test that can only be answered correctly when applying test-wiseness. The test contains questions on historical facts and interpretations that cannot be solved with general knowledge. In this test the questions have different "hints" as they can unintentionally appear in knowledge tests:

-       Similarities between question text and possible answers are found in the right answers rather than in the wrong ones.

-       Answer options unsuitable for the question or completely absurd indicate wrong answers.

-       Specific words such as "any", "never" or "completely" indicate incorrect answers.

-       The correct answer is usually more specific and more precise than false answers.

-       The longest answer is often the right answer.

-       Grammatical mistakes are more likely to be found in incorrect answers than in correct ones.

-       The correct answer to one question can be inferred from another question in the test.

Test-wiseness in the German-speaking area

As mentioned above, thus far test-wiseness has hardly been the focus of attention in Germany. We have therefore developed a German test with which test-wiseness can be measured in this country. The test proved useful but compared to results from international tests, test persons didn't know how to handle some of the questions as well. 

Possibly the formulations and/or reference words in the answer possibilities weren't as clear in German as in the English original test, so that the wrong answers seemed more attractive than the right answer. Our statistical analyses show, however, that a test has nonetheless been created that corresponds to the quality of existing international tests. The execution and evaluation of the test requires little effort.

Topical knowledge or test-wiseness?

We also explored whether subject knowledge has an influence on the results of the test used to measure test-wiseness. Our study involved third-year history students. It can be assumed that this group has subject knowledge, as the questions described above deal with difficult historical facts and interpretations.

Our results show that the historians in the test-wiseness test performed better than the other participants in the test, who did not have a pronounced knowledge of history. This result suggests that test-wiseness is not independent of a person's knowledge of the subject matter, and that the questions are solved both on the basis of the subject knowledge as well as with test-wiseness.