Questionnaire

Design
Unit 3
Metric Vs Non metric Data
Metric Non Metric
Metric data is what most people Nonmetric data includes
mean when they talk about information that is ranked (which is
‘numbers’, the sorts of numbers called ordinal) and information that
we collect when we measure has no linear pattern to it (which is
something called nominal or categorical).

How old is somebody? Which region do you live in? Asia,

Australasia, Europe, North America,
South America, Other

Two key ones are ratio scales Can use non-parametric statistics,
and interval scales. such as counts, percentages, chi-
square, mode and median

Many of the statistical With non-metric data we can’t use

techniques we use in market tools like mean, standard
research, including the mean deviation, z tests, or factor analysis
and standard deviation only
2
‘work’ with metric data.
Tools for measurement
 Qualitative methods include case studies,
interviews , questionnaires and focus
groups.
 Quantitative methods are experimentation,

observation, surveys etc.

3
Questionnaires in Social
Research
 Much of the data in social science
research is gathered using questionnaires
or interviews.
 The validity of the results depends on the
quality of these instruments.
◦Good questionnaires are difficult to
construct;
◦Bad questionnaires are difficult to analyze.

4
Questionnaires in Social
Research
 Difficult to design for several reasons:
◦ Each question must provide a valid and reliable
measure.
◦ The questions must clearly communicate the
research intention to the survey respondent.
◦ The questions must be assembled into a logical,
clear instrument that flows naturally and will keep
the respondent sufficiently interested to continue to
cooperate.

5
Reliability and
Validity
Reliability and validity are concepts used to
evaluate the quality of research.

 They indicate how well a method, technique

or test measures something.

6
Quality aims in survey
research
Goal is to collect information that is:
 Valid: Measures the quantity or concept
that is supposed to be measured. Validity is
about the accuracy of a measure.

 Reliable: Measures the quantity or

concept in a consistent or reproducible
manner. Reliability is about
the consistency of a measure

7
Reliability Vs. Validity
Reliability Validity
What does it tell you? The extent to which The extent to which
the results can be the results really
reproduced when the measure what they
research is repeated are supposed to
under the same measure.
conditions.
How is it assessed? By checking the By checking how well
consistency of results the results correspond
across time, across to established
different observers, theories and other
and across parts of measures of the same
the test itself. concept.
How do they relate? A reliable A valid measurement
measurement is not is generally reliable: if
always valid: the a test produces
results might be accurate results, they
reproducible, but should be
they’re not reproducible.
necessarily correct.
8
Steps to design a
questionnaire:
1. Write out the primary and secondary aims
of your study.
2. Write out concepts/information to be
collected that relates to these aims.
3. Review the current literature to identify
already validated questionnaires that
measure your specific area of interest.
4. Compose a draft of your questionnaire.
5. Revise the draft.
6. Assemble the final questionnaire.

9
Step 1: Define the aims of the
study
 Write out the problem and primary and
secondary aims using one sentence per
aim. Formulate a plan for the statistical
analysis of each aim.
 Make sure to define the target population in

your aim(s).

10
Step 2: Define the variables to
be collected
 Write a detailed list of the information to be collected and
the concepts to be measured in the study. Are you trying
to identify:
◦ Attitudes
◦ Needs
◦ Behavior
◦ Demographics
◦ Some combination of these concepts
 Translate these concepts into variables that can be
measured.
 Define the role of each variable in the statistical analysis:
◦ Predictor
◦ Outcome

11
Step 3: Review the
literature
 Review current literature to identify related surveys
and data collection instruments that have measured
concepts similar to those related to your study’s aims.
 Saves development time and allows for comparison

with other studies if used appropriately.

 Proceed with caution if using only a subset of an

existing questionnaire as this may change the meaning

of the scores. Contact the authors of the questionnaire
to determine if a smaller version of the instrument
exists that has also been validated.

12
Step 4: Compose a draft
 Determine the mode of survey administration: face-to-
face interviews, telephone interviews, self-completed
questionnaires, computer-assisted approaches.
 Write more questions than will be included in the final
draft.
 Format the draft as if it were the final version with
appropriate white space to get an accurate estimate as to
its length – longer questionnaires reduce the response rate.
 Place the most important items in the first half of the
questionnaire to increase response on the important
measures even in partially completed surveys.
 Make sure questions flow naturally from one to another.

13
Compose a draft
 Question: How many cups of coffee or tea
do you drink in a day?
 Principle: Ask for an answer in only one

dimension.
 Solution: Separate the question into two –

◦ (1) How many cups of coffee do you drink during

a typical day?
◦ (2) How many cups of tea do you drink during a
typical day?

14
Compose a draft
 Question: What brand of computer do you own?
◦ (A) IBM PC
◦ (B) Apple
 Principle: Avoid hidden assumptions. Make sure to
accommodate all possible answers.

 Solution:
◦ (1) Make each response a separate dichotomous item
 Do you own an IBM PC? (Circle: Yes or No)
 Do you own an Apple computer? (Circle: Yes or No)
◦ (2) Add necessary response categories and allow for
multiple responses.

 What brand of computer do you own? (Circle all that

apply)
 Do not own computer
 IBM PC
 Apple
 Other 15
Compose a draft
 Question: Have you had pain in the last
week?
[ ] Never [ ] Seldom [ ] Often [ ] Very
often
 Principle: Make sure question and answer
options match.
 Solution: Reword either question or answer

to match.
◦ How often have you had pain in the last week?
[ ] Never [ ] Seldom [ ] Often [ ] Very
Often

16
Compose a draft
 Question: Where did you grow up?
◦ Village
◦ Farm
◦ City
 Principle: Avoid questions having non-
mutually exclusive answers.
 Solution: Design the question with mutually
exclusive options.
◦ Where did you grow up?
 House in the village
 Farm in the village
 City

17
Compose a draft
 Question: Which one of the following do you think
increases a person’s chance of having a heart attack
the most? (Check one.)
[ ] Smoking [ ] Being overweight [ ] Stress
 Principle: Encourage the respondent to consider each
possible response to avoid the uncertainty of whether a
missing item may represent either an answer that does
not apply or an overlooked item.
 Solution: Which of the following increases the chance
of having a heart attack?
◦ Smoking: [ ] Yes [ ] No [ ] Don’t know
◦ Being overweight: [ ] Yes [ ] No [ ] Don’t know
◦ Stress: [ ] Yes [ ] No [ ] Don’t know

18
Compose a draft
 Question:
◦ (1) Do you currently have a life insurance policy?
(Circle: Yes or No)
◦ If no, go to question 3.
◦ (2) How much is your annual life insurance
premium?
 Principle: Avoid branching as much as
possible to avoid confusing respondents.
 Solution: If possible, write as one question.

◦ How much did you spend last year for life

insurance? (Write 0 if none).

19
Answer key
 Question: Are you against drug abuse?
(Circle: Yes or No)
 Principle: Write questions that will produce

variability in the responses.

 Solution: Eliminate the question.

20
Step 5: Revise
 Shorten the set of questions for the study. If
a question does not address one of your
aims, discard it.
 Refine the questions included and their

wording by testing them with a variety of

respondents.
◦ Ensure the flow is natural.
◦ Verify that terms and concepts are familiar and
easy to understand for your target audience.
◦ Keep recall to a minimum and focus on the recent
past.

21
Step 6: Assemble the final
questionnaire
 Decide whether you will format the questionnaire
yourself or use computer-based programs for assistance:
◦ SurveyMonkey.com
◦ Adobe Live Cycle Designer 7.0
◦ Google forms
 At the top, clearly state:
◦ The purpose of the study
◦ How the data will be used
◦ Instructions on how to fill out the questionnaire
◦ Your policy on confidentiality
 Include identifying data on each page of a multi-page,
paper-based questionnaire such as a respondent ID
number in case the pages separate.

22
Assemble the final
questionnaire
 Group questions concerning major subject
areas together and introduce them by
heading or short descriptive statements.

 Order questions in order to stimulate recall.

 Order and format questions to ensure

unbiased and balanced results.

23
Assemble the final
questionnaire
 Include white space to make answers clear
and to help increase response rate.
 Space response scales widely enough so
that it is easy to circle or check the correct
answer without the mark accidentally
including the answer above or below.
◦ Open-ended questions: the space for the
response should be big enough to allow
respondents with large handwriting to write
comfortably in the space.
◦ Closed-ended questions: line up answers
vertically and precede them with boxes or
brackets to check, or by numbers to circle, rather
than open blanks.
 Use larger font size (e.g., 14) and high
contrast (black on white). 24
Enhance response rate
 When writing questions and assembling the
final questionnaire, edit with a view towards
saliency: apparent relevance, importance,
and interest of the survey to the respondent
 Consider either pre-notifying those in your
sample or sending reminders to those who
received the survey (if self-administered).
Studies have shown that making contact with
the sampled individuals increases the
response rate.
 If possible, offer an incentive.

25
Non-responders
 Understanding the characteristics of those
who did not respond to the survey is
important to quantify what, if any, bias
exists in the results.
 To quantify the characteristics of the non-

responders to postal surveys, Moser and

Kalton suggest tracking the length of time it
takes for surveys to be returned. Those
who take the longest to return the survey
are most like the non-responders. This
result may be situation-dependent.
26
Conclusions
 You need plenty of time!
◦ Design your questionnaire from research
hypotheses that have been carefully studied and
thought out.
◦ Discuss the research problem with colleagues and
subject matter experts is critical to developing
good questions.
◦ Review, revise and test the questions on an
iterative basis.
◦ Examine the questionnaire as a whole for flow
and presentation.

27
EFFECTIVE RESEARCH INTERVIEWS
 Interviews are particularly useful for getting
the story behind a participant's
experiences.
 The interviewer can pursue in-depth

information around a topic.

 Interviews may be useful as follow-up to

certain respondents to questionnaires, e.g.,

to further investigate their responses.
 Usually open-ended questions are asked

during interviews.

28
Preparation for Interview

 Choose a setting with little distraction.

 Explain the purpose of the interview.
 Address terms of confidentiality.
 Explain the format of the interview.
 Indicate how long the interview usually takes.
 Tell them how to get in touch with you later if
they want to.
 Ask them if they have any questions
 Don't count on your memory to recall their
answers.

29
Types of Interviews

 Informal, conversational interview

 General interview guide approach
 Standardized, open-ended interview
 Closed, fixed-response interview

30
Types of Topics in Questions

Behaviors - about what a person has done or is

doing
 Opinions/values - about what a person thinks
about a topic
 Feelings - note that respondents sometimes
respond with "I think ..." so be careful to note that
you're looking for feelings
 Knowledge - to get facts about a topic
 Sensory - about what people have seen,
touched, heard, tasted or smelled
 Background/demographics - standard
background questions, such as age, education,
etc.
31
Sequence of Questions

 Get the respondents involved in the interview as soon

as possible.
 Before asking about controversial matters (such as

feelings and conclusions), first ask about some

facts. With this approach, respondents can more easily
engage in the interview before warming up to more
personal matters.
 Intersperse fact-based questions throughout the
interview to avoid long lists of fact-based questions,
which tends to leave respondents disengaged.

32
Sequence of Questions

 Ask questions about the present before questions

about the past or future. It's usually easier for them to
talk about the present and then work into the past or
future.
 The last questions might be to allow respondents to

provide any other information they prefer to add and

their impressions of the interview.

33
Wording of Question

 Wording should be open-ended. Respondents should be

able to choose their own terms when answering
questions.

 Questions should be as neutral as possible. Avoid

wording that might influence answers, e.g., evocative,
judgmental wording.

 Questions should be asked one at a time.

34
Wording of Question

 Questions should be worded clearly. This includes

knowing any terms particular to the program or the
respondents' culture.
 Be careful asking "why" questions. This type of

question infers a cause-effect relationship that may not

truly exist. These questions may also cause respondents
to feel defensive, e.g., that they have to justify their
response, which may inhibit their responses to this and
future questions.

35
Conducting Interview

 Occasionally verify the tape recorder (if used) is

working.
 Ask one question at a time.
 Attempt to remain as neutral as possible. That is, don't

show strong emotional reactions to their responses.

 Encourage responses with occasional nods of the head,

"uh huh"s, etc.

36
Conducting Interview
 Be careful about the appearance when note taking. That
is, if you jump to take a note, it may appear as if you're
surprised or very pleased about an answer, which may
influence answers to future questions.

 Provide transition between major topics, e.g., "we've

been talking about (some topic) and now I'd like to
move on to (another topic).“

 Don't lose control of the interview. This can occur

when respondents stray to another topic, take so long to
answer a question that times begins to run out, or even
begin asking questions to the interviewer
37
Immediately After Interview

 Verify if the tape recorder, if used, worked

throughout the interview.
 Make any notes on your written notes, e.g., to clarify

any scratchings, ensure pages are numbered, fill out

any notes that don't make senses, etc.
 Write down any observations made during the

interview. For example, where did the interview occur

and when, was the respondent particularly nervous at
any time? Were there any surprises during the
interview? Did the tape recorder break?

38