Data Collection Techniques

Week
Introduction

 Data Collection Strategies

 Characteristics of Good Measures
 Quantitative and Qualitative Data
 Tools for Collecting Data

2 IPDET © 2009
Data Collection Strategies
 No one best way: decision depends on:
 What you need to know: numbers or stories
 Where the data reside: environment, files, people
 Resources and time available
 Complexity of the data to be collected
 Frequency of data collection
 Intended forms of data analysis

3 IPDET © 2009
Rules for Collecting Data

 Use multiple data collection methods

 Use available data, but need to know
 how the measures were defined
 how the data were collected and cleaned
 the extent of missing data
 how accuracy of the data was ensured

4 IPDET © 2009
Rules for Collecting Data
 If must collect original data:
 be sensitive to burden on others
 pre-test, pre-test, pre-test
 establish procedures and follow them (protocol)
 maintain accurate records of definitions and coding
 verify accuracy of coding, data input

5 IPDET © 2009
Structured Approach
 All data collected in the same way
 Especially important for multi-site and cluster evaluations
so you can compare
 Important when you need to make comparisons with
alternate interventions

6 IPDET © 2009
Use Structured Approach When:
 need to address extent questions
 have a large sample or population
 know what needs to be measured
 need to show results numerically
 need to make comparisons across different sites or
interventions

7 IPDET © 2009
Semi-structured Approach
 Systematic and follow general procedures but data are not
collected in exactly the same way every time
 More open and fluid
 Does not follow a rigid script
 may ask for more detail
 people can tell what they want in their own way

8 IPDET © 2009
Use Semi-structured Approach when:
 conducting exploratory work
 seeking understanding, themes, and/or issues
 need narratives or stories
 want in-depth, rich, “backstage” information
 seek to understand results of data that are unexpected

9 IPDET © 2009
Characteristics of Good Measures
 Is the measure relevant?
 Is the measure credible?
 Is the measure valid?
 Is the measure reliable?

10 IPDET © 2009
Relevance
Does the measure Do not measure what
capture what matters? is easy instead of
what is needed

11 IPDET © 2009
Credibility
Is the measure believable? Will it be viewed
as a reasonable and appropriate way to
capture the information sought?

12 IPDET © 2009
Internal Validity
How well does the Are waiting lists a
measure capture what valid measure of
it is supposed to? demand?

13 IPDET © 2009
Reliability
A measure’s How reliable are:
precision and  birth weights of
stability- extent to newborn infants?
which the same result  speeds measured by a

would be obtained stopwatch?

with repeated trials

14 IPDET © 2009
Quantitative Approach
 Data in numerical form
 Data that can be precisely measured
 age, cost, length, height, area, volume, weight, speed, time, and
temperature
 Harder to develop
 Easier to analyze

15 IPDET © 2009
Qualitative Approach
 Data that deal with description
 Data that can be observed or self-reported, but not
always precisely measured
 Less structured, easier to develop
 Can provide “rich data” — detailed and widely
applicable
 Is challenging to analyze
 Is labor intensive to collect
 Usually generates longer reports

16 IPDET © 2009
Which Data?

If you: Then Use:

- want to conduct statistical analysis
- want to be precise Quantitative
- know what you want to measure
- want to cover a large group
- want narrative or in-depth information
- are not sure what you are able to measure Qualitative
- do not need to quantify the results

17 IPDET © 2009
Obtrusive vs. Unobtrusive Methods

Obtrusive Unobtrusive
data collection methods data collection methods
that directly obtain that do not collect
information from those information directly
being evaluated from evaluees
e.g. interviews, surveys, focus e.g., document analysis,
groups GoogleEarth, observation
at a distance, trash of the
stars

18 IPDET © 2009
How to Decide on Data Collection Approach
 Choice depends on the situation
 Each technique is more appropriate in some situations
than others
 Caution: All techniques are subject to bias

19 IPDET © 2009
Triangulation to Increase Accuracy of Data
 Triangulation of methods
 collection of same information using different methods
 Triangulation of sources
 collection of same information from a variety of sources
 Triangulation of evaluators
 collection of same information from more than one evaluator

20 IPDET © 2009
Data Collection Tools
 Participatory Methods
 Records and Secondary Data
 Observation
 Surveys and Interviews
 Focus Groups
 Diaries, Journals, Self-reported Checklists
 Expert Judgment
 Delphi Technique
 Other Tools

21 IPDET © 2009
Tool 1: Participatory Methods
 Involve groups or communities heavily in data collection
 Examples:
 community meetings
 mapping
 transect walks

22 IPDET © 2009
Community Meetings
 One of the most common participatory methods
 Must be well organized
 agree on purpose
 establish ground rules
 who will speak
 time allotted for speakers
 format for questions and answers

23 IPDET © 2009
Mapping
 Drawing or using existing maps
 Useful tool to involve stakeholders
 increases understanding of the community
 generates discussions, verifies secondary sources of
information, perceived changes
 Types of mapping:
 natural resources, social, health, individual or civic assets,
wealth, land use, demographics

24 IPDET © 2009
Transect Walks
 Evaluator walks around community observing people,
surroundings, and resources
 Need good observation skills
 Walk a transect line through a map of a community —
line should go through all zones of the community

25 IPDET © 2009
Tool 2: Records and Secondary Data
 Examples of sources:
 files/records
 computer data bases
 industry or government reports
 other reports or prior evaluations
 census data and household survey data
 electronic mailing lists and discussion groups
 documents (budgets, organizational charts, policies and
procedures, maps, monitoring reports)
 newspapers and television reports

26 IPDET © 2009
Using Existing Data Sets

Key issues: validity, reliability, accuracy, response

rates, data dictionaries, and missing data rates

27 IPDET © 2009
Advantage/Challenge: Available Data

Advantages Often less expensive and faster

than collecting the original data
again

Challenges There may be coding errors or

other problems. Data may not be
exactly what is needed. You may
have difficulty getting access. You
have to verify validity and
reliability of data
28 IPDET © 2009
Tool 3: Observation
 See what is happening
 traffic patterns
 land use patterns
 layout of city and rural areas
 quality of housing
 condition of roads
 conditions of buildings
 who goes to a health clinic

29 IPDET © 2009
Observation is Helpful when:
 need direct information
 trying to understand ongoing behavior
 there is physical evidence, products, or outputs than can
be observed
 need to provide alternative when other data collection is
infeasible or inappropriate

30 IPDET © 2009
Degree of Structure of Observations
 Structured: determine, before the observation,
precisely what will be observed before the observation
 Unstructured: select the method depending upon the
situation with no pre-conceived ideas or a plan on
what to observe
 Semi-structured: a general idea of what to observe but
no specific plan

31 IPDET © 2009
Google Earth
 Maps and satellite images for complex or pinpointed
regional searches
 Has an Advanced version and an Earth Outreach version
 Web site for Google Earth
 [Link]

32 IPDET © 2009
Ways to Record Information from Observations
 Observation guide
 printed form with space to record
 Recording sheet or checklist
 Yes/no options; tallies, rating scales
 Field notes
 least structured, recorded in narrative, descriptive style

33 IPDET © 2009
Guidelines for Planning Observations
 Have more than one observer, if feasible
 Train observers so they observe the same things
 Pilot test the observation data collection
instrument
 For less structured approach, have a few key
questions in mind

34 IPDET © 2009
Advantages and Challenges: Observation

Advantages Collects data on actual vs. self-

reported behavior or perceptions. It is
real-time vs. retrospective

Challenges Observer bias, potentially unreliable;

interpretation and coding challenges;
sampling can be a problem; can be
labor intensive; low response rates

35 IPDET © 2009
Tool 4: Surveys and Interviews
 Excellent for asking people about:
 perceptions, opinions, ideas
 Less accurate for measuring behavior
 Sample should be representative of the whole
 Big problem with response rates

36 IPDET © 2009
Structures for Surveys
 Structured:
 Precisely worded with a range of pre-determined
responses that the respondent can select
 Everyone asked exactly the same questions in exactly the
same way, given exactly the same choices
 Semi-structured
 Asks same general set of questions but answers to the
questions are predominantly open-ended

37 IPDET © 2009
Structured vs. Semi-structured Surveys

Structured harder to develop

easier to complete
easier to analyze
more efficient when working with large numbers
Semi- easier to develop: open ended questions
structured more difficult to complete: burdensome for
people to complete as a self-administrated
questionnaire
harder to analyze but provide a richer source of
data, interpretation of open-ended responses
subject to bias
38 IPDET © 2009
Modes of Survey Administration
 Telephone surveys
 Self-administered questionnaires distributed by mail, e-
mail, or websites
 Administered questionnaires, common in the development
context
 In development context, often issues of language and
translation

39 IPDET © 2009
Mail / Phone / Internet Surveys
 Literacy issues
 Consider accessibility
 reliability of postal service
 turn-around time
 Consider bias
 What population segment has telephone access? Internet
access?

40 IPDET © 2009
Advantages and Challenges of Surveys

Advantages Best when you want to know what

people think, believe, or perceive,
only they can tell you that
Challenges People may not accurately recall their
behavior or may be reluctant to reveal
their behavior if it is illegal or
stigmatized. What people think they
do or say they do is not always the
same as what they actually do.
41 IPDET © 2009
Interviews
 Often semi-structured
 Used to explore complex issues in depth
 Forgiving of mistakes: unclear questions can be
clarified during the interview and changed for
subsequent interviews
 Can provide evaluators with an intuitive sense of the
situation

42 IPDET © 2009
Challenges of Interviews
 Can be expensive, labor intensive, and time consuming
 Selective hearing on the part of the interviewer may miss
information that does not conform to pre-existing beliefs
 Cultural sensitivity: e.g., gender issues

43 IPDET © 2009
Tool 5: Focus Groups
 Type of qualitative research where small homogenous
groups of people are brought together to informally
discuss specific topics under the guidance of a moderator
 Purpose: to identify issues and themes, not just interesting
information, and not “counts”

44 IPDET © 2009
Focus Groups Are Inappropriate when:
 language barriers are insurmountable
 evaluator has little control over the situation
 trust cannot be established
 free expression cannot be ensured
 confidentiality cannot be assured

45 IPDET © 2009
Focus Group Process
Phase Action

1 Opening Ice-breaker; explain purpose; ground rules;

introductions
2 Warm- Relate experience; stimulate group interaction;
up start with least threatening and simplest questions
3 Main Move to more threatening or sensitive and
body complex questions; elicit deep responses; connect
emergent data to complex, broad participation
4 Closure End with closure-type questions; summarize and
refine; present theories, etc; invite final comments
or insights; thank participants

46 IPDET © 2009
Advantages and Challenges of Focus Groups

Advantages Can be conducted relatively quickly and

easily; may take less staff time than in-depth,
in-person interviews; allow flexibility to make
changes in process and questions; can
explore different perspectives; can be fun

Challenges Analysis is time consuming; participants not

be representative of population, possibly
biasing the data; group may be influenced by
moderator or dominant group members

47 IPDET © 2009
Tool 6: Diaries and Self-Reported Checklists
 Use when you want to capture information about events in
people’s daily lives
 Participants capture experiences in real-time not later in a
questionnaire
 Used to supplement other data collection

48 IPDET © 2009
Guidelines for Diaries or Journals
Step Process

1 Recruit people face-to-face

• encourage participation, appeal to altruism, assure
confidentiality, provide incentive
2 Provide a booklet to each participant
• cover page with clear instructions, definitions, example
• short memory-joggers, explain terms, comments on last
page , calendar
3 Consider the time-period for collecting data
• if too long, may become burdensome or tedious
• if too short may miss the behavior or event
49 IPDET © 2009
Self-reported Checklists
 Cross between a questionnaire and a diary
 The evaluator specifies a list of behaviors or events and
asks the respondents to complete the checklist
 Done over a period of time to capture the event or
behavior
 More quantitative approach than diary

50 IPDET © 2009
Advantages and Challenges of Diaries and
Self-reported Checklists

Advantages Can capture in-depth, detailed data that might be

otherwise forgotten
Can collect data on how people use their time
Can collect sensitive information
Supplements interviews provide richer data

Challenges Requires some literacy

May change behavior
Require commitment and self-discipline
Data may be incomplete or inaccurate
Poor handwriting, difficult to understand phrases

51 IPDET © 2009
Tool 7: Expert Judgment
Use of experts, one-on-one Can be structured or
or as a panel unstructured
E.g., Government task Issues in selecting experts
forces, Advisory Groups

52 IPDET © 2009
Selecting Experts
 Establish criteria for selecting experts not only on
recognition as expert but also based on:
 areas of expertise
 diverse perspectives
 diverse political views
 diverse technical expertise

53 IPDET © 2009
Advantages and Challenges of Expert
Judgment

Advantages Fast, relatively inexpensive

Challenges Weak for impact evaluation

May be based mostly on perceptions
Value of data depends on how
credible the experts are perceived to
be

54 IPDET © 2009
Tool 8: Delphi Technique
 Enables experts to engage remotely in a dialogue
and reach consensus, often about priorities
 Experts asked specific questions; often rank
choices
 Responses go to a central source, are summarized
and fed back to the experts without attribution
 Experts can agree or argue with others’ comments
 Process may be iterative

55 IPDET © 2009
Advantages and Challenges of Delphi
Technique

Advantages Allows participants to remain anonymous

Is inexpensive
Is free of social pressure, personality influence,
and individual dominance
Is conducive to independent thinking
Allows sharing of information
Challenges May not be representative
Has tendency to eliminate extreme positions
Requires skill in written communication
Requires time and participant commitment

56 IPDET © 2009
Other Measurement Tools
- scales (weight) - health testing tools:
- tape measure i.e. blood pressure
- stop watches - aptitude and
- chemical tests : achievement tests
i.e. quality of water -citizen report cards

57 IPDET © 2009
Data Collection Summary
Choose more than one data collection technique
No “best” tool
Do not let the tool drive your work but rather choose the
right tool to address the evaluation question

58 IPDET © 2009
.
.

Questions?

59 IPDET © 2009