Writing Multiple-Choice Questions That Demand Critical Thinking
Writing Multiple-Choice Questions That Demand Critical Thinking
Writing Multiple-Choice Questions That Demand Critical Thinking
Critical Thinking
Adapted from: http://darkwing.uoregon.edu/~tep/assessment/mc4critthink.html
Important Considerations What role should testing play in the learning process? How
can tests create a real dialogue between ourselves and our students about what students
do and do not understand? How can we avoid using tests to simply punish or reward
cramming?
Some suggestions:
Use frequent, small quizes and tests rather than monolithic once-or-twice per-term
exams.
Give students instant feedback on their performance (for example, putting the correct
answers up on an overhead after all the tests are turned in.)
Consider allowing students to take quizzes first as individuals and then the same quiz
again in groups.
Multiple-choice questions are easiest to write when there is a definitively right or wrong
answer. Multiple-choice testing of more interpretive material should always include an
appeal mechanism in which students can and must make a written, evidence-supported
case for their answer.
Learning objectives at this level: know common terms, know specific facts, know
methods and procedures, know basic concepts, know principles.
Question verbs: Define, list, state, identify, label, name, who? when? where? what?
2. Comprehension
The ability to grasp the meaning of material. Translating material from one form to
1
another (words to numbers), interpreting material (explaining or summarizing),
estimating future trends (predicting consequences or effects). Goes one step beyond the
simple remembering of material, and represent the lowest level of understanding.
Learning objectives at this level: understand facts and principles, interpret verbal
material, interpret charts and graphs, translate verbal material to mathematical formulae,
estimate the future consequences implied in data, justify methods and procedures.
Question verbs: Explain, predict, interpret, infer, summarize, convert, translate, give
example, account for, paraphrase x?
3. Application
The ability to use learned material in new and concrete situations. Applying rules,
methods, concepts, principles, laws, and theories.
Learning outcomes in this area require a higher level of understanding than those under
comprehension.
Learning objectives at this level: apply concepts and principles to new situations, apply
laws and theories to practical situations, solve mathematical problems, construct graphs
and charts, demonstrate the correct usage of a method or procedure.
Question verbs: How could x be used to y? How would you show, make use of, modify,
demonstrate, solve, or apply x to conditions y?
4. Analysis
The ability to break down material into its component parts. Identifying parts, analysis of
relationships between parts, recognition of the organizational principles involved.
Learning outcomes here represent a higher intellectual level than comprehension and
application because they require an understanding of both the content and the structural
form of the material.
5. Synthesis
(by definition, synthesis cannot be assessed with multiple-choice questions.
It appears here to complete Bloom's taxonomy.) The ability to put parts together to form
a new whole.
This may involve the production of a unique communication (theme or speech), a plan
2
of operations (research proposal), or a set of abstract relations (scheme for classifying
information).
Learning outcomes in this area stress creative behaviors, with major emphasis on the
formulation of new patterns or structure.
Learning objectives at this level: write a well organized paper, give a well organized
speech, write a creative short story (or poem or music), propose a plan for an experiment,
integrate learning from different areas into a plan for solving a problem, formulate a new
scheme for classifying objects (or events, or ideas).
Question verbs: Design, construct, develop, formulate, imagine, create, change, write a
short story and label the following elements:
6. Evaluation
The ability to judge the value of material (statement, novel, poem, research report) for
a given purpose. The judgments are to be based on definite criteria, which may be
internal (organization) or external (relevance to the purpose). The student may determine
the criteria or be given them. Learning outcomes in this area are highest in the cognitive
hierarchy because they contain elements of all the other categories, plus conscious value
judgments based on clearly defined criteria.
Learning objectives at this level: judge the logical consistency of written material,
judge the adequacy with which conclusions are supported by data, judge the value of a
work (art, music, writing) by the use of internal criteria, judge the value of a work (art,
music, writing) by use of external standards of excellence.
Question verbs: Justify, appraise, evaluate, judge x according to given criteria. Which
option would be better/preferable to party y?
General Suggestions
1) Do not write the test in one day. Spread the work out over time. Questions demanding
high-level thinking take longer to craft-professional item writers often write only 3 or 4
per day. Write one or two questions after each class, so it becomes a simple matter of
assembling them into an exam. Some teachers keep a rubber-banded stack of note cards
in their desk for this purpose.
2) If students are to hand-write the letters of their chosen answers, ask them to use
CAPITAL LETTERS. The handwritten, lower-case letters "a" and "d" and "c" and "e"
can be difficult to distinguish when scoring.
3
Writing the Stem
3) Include any language in the stem that you would have to repeat in each answer
option. For example, a stem such as "Biology is defined as the scientific study of:" keeps
you from having to repeat "is the scientific study of" at the beginning of each option.
Answer Options
1) Avoid lifting phrases directly from text or lecture. This becomes a simple recall
activity for the student. Use new language as frequently as possible.
2) Most literature recommends writing the correct answer before writing the
distracters. This makes sure you pay enough attention to formulating the one clearly
correct answer.
3) Answer options should be about the same length and parallel in grammatical
structure. Too much detail or different grammatical structure can give the answer away.
For example, the specificity and grammatical structure of the first option here are dead
give-aways:
4) Limit the number of answer options. Research shows that three-choice items are
about as effective as four-choice items. Four choice items are the most popular, and never
give more than five alternatives.
4
5) Distracters must be incorrect, but plausible. If you can, include among the
distracters options that contain common errors. Students will then be motivated to listen
to your explanations of why those options are incorrect.
6) To make distracters more plausible, use words that should be familiar to students.
7) If a recognizable key word appears in the correct answer, it should appear in some or
all of the distracters as well. Don't let a verbal clue decrease the accuracy of your exam.
For example, someone with no biology background would not have to think very hard to
make a correct guess on this question:
Every organism is made of cells and every cell comes from another cell. This is the:
a) Relativity Theory
b) Evolution Theory
c) Heat Theory
d) Cell Theory
For example: "Which of the following is NOT an explicit norm?" Likewise, when you
ask a similarly-worded question about two different things, always highlight the
difference between the questions.
If it is too easy to eliminate one or two options, then the question loses much of its
measurement value. If energy or time is limited and you must come up with one more
distracter, consider either offering a true statement that does not answer the question
and/or a jargon-ridden option that is meaningless to someone who understands the
concept.
Extreme words like "all," "always" and "never" (generally a wrong answer). Vague words
or phrases like "usually," "typically" and "may be" (generally a correct answer).
"All of the above" - eliminating one distracter immediately eliminates this, too.
"None of the above" - use only when the correct answer can be absolutely correct, such
as in math, grammar, historical dates, geography, etc.. Do not use with negatively-stated
stems, as the resulting double-negative is confusing. Studies do show that using "None of
the above" does make a question more difficult, and is a better choice when the
alternative is a weak distracter.
5
IV. Some Techniques for Writing Multiple-Choice Items that Demand
Critical Thinking
1) Premise - Consequence
Students must identify the correct outcome of a given circumstance.
Example: If nominal gross national product (GNP) increases at a rate of 10% per year
and the GNP deflator increases at 8% per year, then real GNP:
a) Remains constant.
b) Rises by 10%.
c) Falls by 8%.
d) Rises by 2%.
2) Analogy
Students must map the relationship between two items into a different context:
3) Case study A single, well-written paragraph can provide material for several follow-
up questions.
Example:
2) Alice, Barbara, and Charles own a small business: the Chock-Full-o-Goodness Cookie
Company. Because Charles has many outside commitments and Barbara has a few, Alice
tends to be most in touch with the daily operations of Chock-Full-o-Goodness. As a
result, when financial decisions come down to a vote at their monthly meeting, they have
decided that Alice gets 8 votes, Barbara gets 7, and Charles gets 2-with 9 being required
to make the decision.
a) Alice
b) Barbara
c) Charles
d) No trend toward any specific person.
6
3) In the scenario in question 2, according to minimum-power coalition theory, who is
most likely to be courted for their vote?
a) Alice
b) Barbara
c) Charles
d) No trend toward any specific person.
4) Incomplete Scenario
Students must respond to what is missing or needs to be changed within a provided
scenario.
Note: when using a graph or image, try to lay it out differently than how the students
have seen it. This is equivalent to using new language to present a familiar concept and
prevents students from using rote memorization to answer the question.
For example, the diagram below may originally have been split left to right instead of top
to bottom, and this diagram may not be as detailed as the diagram they saw in the book.)
2) If the Applications resided below the heavy black line, they would:
a) be open to hackers on the network.
b) compete with the OS for memory.
c) be preemptively multi-tasked.
d) launch in individual NTVDMs.
Example: A student was asked the following question: "Briefly list and explain the
various stages of the creative process."
As an answer, this student wrote the following:
"The creative process is believed to take place in five stages, in the following order:
orientation, when the problem must be identified and defined, preparation, when all the
possible information about the problem is collected, incubation, when no solution seems
in sight and the person is often busy with other tasks, illumination, when the person
experiences a general idea of how to arrive at a solution to the problem, and finally
verification, when the person determines whether the solution is the right one for the
problem."
7
How would you judge this student' s answer?
a) EXCELLENT (all stages correct in the right order with clear and correct explanations)
b) GOOD (all stages correct in the right order, but the explanations are not as clear as
they should be)
c) MEDIOCRE (one or two stages are missing OR the stages are in the wrong order, OR
the explanations are not clear OR the explanations are irrelevant)
d) UNACCEPTABLE (more than two stages are missing AND the order is incorrect
AND the explanations are not clear AND/OR they are irrelevant)
Books:
Articles:
Web-sites:
University of Capetown's Guide to Designing and Managing Multiple Choice Questions
An excellent site from which much of our workshop material was adapted.