Systematic Review

Reference
Participant files of CSRTP program of JBI
Introduction
Systematic Review

Systematic reviews aim to provide a comprehensive,

unbiased synthesis of many relevant studies in a single
document using rigorous and transparent methods
(Aromataris & Pearson, 2014)
Why do a Systematic Review?
The aims of a systematic review may be to:
1. uncover the international evidence
2. confirm current practice/address any variation
3. identify areas for future research
4. investigate conflicting results
5. produce statements to guide decision making
Systematic Review Standards
Quality depends on the extent to which methods are followed
to minimize risk of error and bias
SRs require explicit and exhaustive reporting of methods
Reporting standards exist to guide review reports
⁃ PRISMA 2020
⁃ ENTREQ
⁃ JBI Manual for Evidence Synthesis and JBI
Evidence Synthesis
⁃ Cochrane Handbook and MECIR
(Methodological Expectations of Cochrane Intervention
Reviews)
Steps
Formulate Question
Define Inclusion & Exclusion Criteria
Locate Studies (Searching)
Select Studies
Critical Appraisal
Data Extraction
Data Synthesis
Present Results
Interpret / Establish Confidence in Results
Systematic Review Logistics
Consider human and technical resources
A JBI review requires at least 2 reviewers
Consider expertise of topic and expertise of review
process
Library support (database access, searching expertise)
Statistician support
Methodologist support
Software access
Choosing a Systematic Review Topic

Preliminary investigation of literature required to

determine if papers are available on topic of interest
Have there been any reviews already conducted on your
topic of interest? If so how are they different?
Will depend on your topic but consider JBI Evidence
Synthesis, Cochrane database, PubMed, PROSPERO
Duplicate reviews need to be justified
Formulate Question
Types of Systematic Reviews
1. Effectiveness Reviews
2. Qualitative Reviews
3. Costs/Economics Reviews
4. Prevalence or Incidence Reviews
5. Diagnostic Test Accuracy Reviews
6. Etiology and Risk Reviews
7. Textual Synthesis Reviews
8. Mixed Methods Reviews
9. Umbrella Reviews
10. Scoping Reviews
Frameworks exist to help reviewers structure the
question:
⁃ PICO most common for effectiveness reviews
⁃ PICo for qualitative reviews
⁃ CoCoPop for prevalence and incidence reviews
⁃ PIRD for diagnostic test accuracy reviews
⁃ PEO for etiology and risk reviews
⁃ PCC for scoping reviews
Effectiveness Reviews
Effectiveness can be defined as “the extent to which an
intervention, when used appropriately, achieves the intended
effect”

Question Development:

PICO
 Population

 Intervention

 Comparator/control

 Outcome
Example:
Are non-pharmacological interventions compared
with control interventions (e.g. usual care) effective for
treating depressive symptoms of older adults with
depressive disorders?
Qualitative Reviews
Aim - To investigate the experience or meaningfulness of
a particular phenomenon. Focusing on analyzing human
experiences and cultural and social phenomena

Example: PICo
What is the experience of recovering from burns injuries in
adult (over 18 years of age) patients during their stay in a
hospital burns care unit?
Prevalence or Incidence Reviews
Aim - To determine the prevalence and/or incidence of a
certain condition (measure of disease burden). Enable
governments, policy makers, health professionals and the
general population to inform the development and
delivery of health services.
Example: CoCOPop
What is the prevalence and incidence of peri-natal
depression among women in Australia?
Diagnostic Test Accuracy Reviews
- To determine how well a diagnostic test works in terms
of its sensitivity and specificity for a particular diagnosis.
Important for clinicians to determine the accuracy of the
diagnostic tests they use
Example: PIRD
What is the diagnostic accuracy of currently available
laboratory tests for swine flu (H1N1) compared to viral
culture as a reference test amongst people presenting
with suspected flu?
Inclusion Criteria
 Participants/population characteristics
 Intervention, interest, exposure or phenomenon under investigation
 Comparators
 Outcomes
 Context
 Condition
 Types of studies to be included

The aspects that will not be covered in the review form your exclusion criteria
Searching for Studies
Searching for Studies
A key characteristic of systematic reviews is a
comprehensive search
The attempt to locate ALL published and unpublished
evidence relevant to a review question
Keyword/free-text word
Index terms/subject headings/controlled vocabularies
A comprehensive search strategy should consist of BOTH
keywords/free-text words AND index terms

Truncation and Wildcard Symbols

Boolean Operators (AND, OR, NOT)
Limits
• Date
• Other
⁃ Study design
⁃ Population
Logic Grid
Example Final Search Query - PubMed
(Dementia[tw] OR Alzheimer*[tw] OR Huntington*[tw] OR Kluver[tw] OR
Lewy[tw] OR Dementia[mh] OR Alzheimer disease[mh]) AND ((Animal
assisted therap*[tw] OR Animal therap*[tw] OR Pet therap*[tw] OR Animal
assisted activit*[tiab] OR Animal assisted intervention*[tiab] OR Dog
therap*[tw] OR Dogassisted therap*[tw] OR Canine-assisted therap*[tw] OR
Aquarium[tw] OR Animal-Assisted Therapy[mh] OR Pets[mh] OR Dogs[mh]
OR Cats[mh] OR Birds[mh] OR Bonding, Human-Pet[mh] OR Animals,
Domestic[mh]) OR (Music*[tw] OR Singing[tw] OR Sing[tw] OR Auditory
stimulation[tw] OR Music[mh] OR Music Therapy[mh] OR Acoustic
Stimulation[mh] OR Singing[mh])) AND (Aggression[tw] OR Behavior*[tiab]
OR Behaviour*[tiab] OR Neuropsychiatric[tiab] OR Apathy inventory[tiab] OR
Cornell scale[tiab] OR Cohen Mansfield[tiab] OR BEHAVE-AD[tiab] OR CERAD-
BRSD[tiab] OR Aggression[mh] OR Personality inventory[mh] OR
Psychomotor agitation[mh])
Study Selection
Study selection is an initial assessment that occurs
following the review search
It addresses the question of whether the full text of the
paper should be retrieved?
It is essential to use two assessors in the selection
process to limit the risk of error and bias. This should be
done independently
Stages of Study Selection
1. Collate all records
2. Remove duplicate records that refer to the same report
3. Screen records for potential retrieval of reports
(e.g. Title and Abstract Screening)

4. Screen reports for inclusion/ exclusion in the review

(e.g. Full Text Screening)

5. Repeat steps 1-4 for records identified from websites,

organisations and citation searching
6. If doing an update of a review, identify the number of
studies and reports included from the previous version of
the review
PRISMA 2020 Searching Process
Conducting SR of Quantitative Evidence
(SR of Effectiveness )
Quantitative Research
Quantitative research has been defined as “a formal,
objective, systematic process in which numerical data are
used to obtain information about the world” (Grove et al
2015 p. 20)
This type of research is used:
⁃ to describe variables
⁃ to examine relationships among variables
⁃ to determine cause-and-effect relationships
between variables
Critical Appraisal
Also referred to as:
• Assessment of risk of bias
• Assessment of methodological quality
Effectiveness research in healthcare

questions of effectiveness
Evidence about the effects of interventions may come
from three main categories of studies:
⁃ experimental studies
⁃ quasi-experimental studies
⁃ observational studies
Experimental Studies
In experimental studies causality between the
intervention and outcome variables is examined under
highly controlled conditions
The defining features of “true” experimental studies
include:
Researcher manipulation of at least one treatment variable
Control group(s)
Random assignment
Randomization
Stratified - Randomization

Group 1 Results
Sample Randomizatio
n

Group 2 Results
RCT variants
- Cross-over trials – subjects are allocated to study arms
where each arm consists of a sequence of two or more
treatments given consecutively
- Cluster RCTs – groups of individuals rather than individuals
are randomized to different interventions
- Stepped wedge trials – an intervention is rolled-out
sequentially to participants over a number of periods. Data
is collected at each point where a new group receives the
intervention

Pseudo-RCTs
Quasi-experimental Studies
Quasi-experimental studies can be similar to
experimental studies, however, the study participants are
not randomly allocated to an intervention or control
group
Allocation in quasi-experimental studies is by the means
of self-selection, or by means of researchers selection
Observational Studies
Types of studies:
⁃ Cohort
⁃ Case control
⁃ Cross sectional
⁃ Case series/case report
Levels of Evidence
• Level 1.a – Systematic review of Randomized Controlled Trials (RCTs)
• Level 1.b – Systematic review of RCTs and other study designs
Experiment • Level 1.c – RCT
al Studies • Level 1.d – Pseudo-RCTs

• Level 2.a – Systematic review of quasi-experimental studies

Quasi- • Level 2.b – Systematic review of quasi-experimental and other lower
Experimental study designs
• Level 2.c – Quasi-experimental prospectively controlled study
Studies • Level 2.d – Pre-test - post-test or historic/ retrospective control group
study
• Level 3.a – Systematic review of comparable cohort studies
Observation • Level 3.b – Systematic review of comparable cohort and other
lower study designs
al Analytical • Level 3.c – Cohort study with control group
Studies • Level 3.d – Case controlled study
• Level 3.e – Observational study without a control group
• Level 4.a – Systematic review of descriptive studies
Observational • Level 4.b – Cross sectional study
• Level 4.c – Case series
Descriptive • Level 4.d – Case report
Studies
• Level 5.a – Systematic review of expert opinion
Expert • Level 5.b – Expert consensus
• Level 5.c – Bench research/ single expert
Opinion opinion
Why is it important?
Combining results of poor quality research may lead to
biased or misleading results
 Aim is to establish validity
Ensures that the review is credible and useful for
informing healthcare policy, clinical practice and future
research
Facilitates deeper knowledge of the included studies
Validity
Two main types of validity:
Internal and External
Establishing Internal Validity
Major sources of bias which may affect internal validity:
⁃ Selection bias
⁃ Performance bias
⁃ Detection bias
⁃ Attrition bias
How to Critically Appraise?
Many validated standardized tools / instruments /
checklists available
JBI checklists are based on study design
Criteria scored as being met, not met, unclear or not
applicable
Conducted by at least two members of your review team
independently
JBI RCT Appraisal Tool
 1. Was true randomization used for assignment of participants to treatment
groups?
 2. Was allocation to treatment groups concealed?
 3. Were treatment groups similar at the baseline?
 4. Were participants blind to treatment assignment?
 5. Were those delivering treatment blind to treatment assignment?
 6. Were outcomes assessors blind to treatment assignment?
 7. Were treatment groups treated identically other than the intervention of
interest?
 8. Was follow up complete and if not, were differences between groups in terms of
their follow up adequately described and analyzed?
 9. Were participants analyzed in the groups to which they were randomized?
 10.Were outcomes measured in the same way for treatment groups?
 11.Were outcomes measured in a reliable way?
 12.Was appropriate statistical analysis used?
 13.Was the trial design appropriate, and any deviations from the standard RCT
design (individual randomization, parallel groups) accounted for in the conduct and
analysis of the trial?
Was appropriate statistical analysis used?
Statistical Conclusion Validity ( s/a power, type I & II
error)
JBI Cohort Study Critical Appraisal Tool
1. Were the two groups similar and recruited from the same population?
2. Were the exposures measured similarly to assign people to both
exposed and unexposed groups?
3. Was the exposure measured in a valid and reliable way?
4. Were confounding factors identified?
5. Were strategies to deal with confounding factors stated?
6. Were the groups/participants free of the outcome at the start of the
study (or at the moment of exposure)?
7. Were the outcomes measured in a valid and reliable way?
8. Was the follow up time reported and sufficient to be long enough for
outcomes to occur?
9. Was follow up complete, and if not, were the reasons to loss to follow up
described and explored?
10.Were strategies to address incomplete follow up utilized?
11.Was appropriate statistical analysis used?
Data Extraction & Meta-analysis
Data Extraction
Data extraction is the process of sourcing and recording
relevant results and details from the primary studies
included in the systematic review
Standardized data extraction tools facilitate the
extraction of the same types of data across all of the
included studies and are required for JBI systematic
reviews
Data extraction occurs in two phases
Phase 1
Extraction of study characteristics
⁃ Citation
⁃ Study design and method
⁃ Setting/Context
⁃ Population characteristics
⁃ Intervention / Comparator / Condition / Exposure / Tests
⁃ Outcomes
Make sure that you are including enough information so
that readers of your review do not have to refer back to
the primary research!
Phase 2
Extraction of results data
What is a measure of effect?
Important measures of effect include:
⁃ Relative Risk (RR) Risk Difference (RD) Odds Ratio (OR)
⁃ Weighted Mean Difference (WMD)
⁃ Standardized Mean Difference (SMD)
Can be either dichotomous or continuous in nature
Expressed in either relative or absolute terms
Interpreting the measures of effect
P values
⁃ Probability that the observed results of the study
are due to random chance, rather than the actual effect

Confidence Intervals
⁃ Provide us with a range within which we can
expect the final effect to lie.
Heterogeneity
Interpreting I2

if the value is zero there are no worries

if the value is 25% there is only little concern
if the value is 50% there is concern
if the value is 75% we should be very concerned
if the value is 100% it indicates substantial heterogeneity
and the appropriateness of undertaking meta-analysis
should be questioned
(Murad et al, 2015)
Meta-analysis approaches
The model for meta-analysis is commonly one of two
approaches:
⁃ Fixed effect model
⁃ Random effects model

Fixed effect models only assume within-study variability

Random effects model assume within and between study
variability
Forest Plot example: Dichotamus Data
Forest Plot example: Continuous Data
Reporting
Reporting Methods of Data Synthesis

 Software used
 Approaches for dichotomous and
continuous data
 Meta-analysis model utilized
 Test for statistical heterogeneity
 Point were statistical heterogeneity
was considered significant
 Specific methods of meta-analysis
used for different types of data

From Davies et al 2015

Assessment of Methodological Quality
Conducting SR of Qualitative Evidence
Overview

JBI approach to the pooling/synthesis of qualitative

studies is meta-aggregation
Requires reviewers to identify and extract the findings
from studies included in the review; to categorize these
study findings; and to aggregate these categories to
develop synthesized findings
Mirrors the processes of a quantitative review
Qualitative Methodologies
Interpretive
⁃ generating knowledge through eliciting
understandings from people; focuses on the meaning of the
phenomena to those involved
Critical
⁃ focuses on moving beyond understanding people
in their cultural contexts and seeks to enable human
agency/emancipation
Interpretive Methodologies
Phenomenology
Ethnography
Grounded theory
Descriptive approaches
Critical Methodologies
Action research
Feminist research
Discourse analysis
Congruity between Paradigm, Methodology and Methods

Paradigm Methodology Methods

Positivist Clinical Trials Measurement of outcomes

Cohort Studies
Other methodologies using
numerical data

Interpretive Phenomenology Interviews

Ethnography Observation
Grounded Theory Field Work
Descriptive approaches Case study

Critical Action Research Group Processes

Reflective Journaling
Assessing qualitative research
The traditional terms used to measure quality or rigor in
experimental research are:
⁃ Reliability
⁃ Internal validity (risk of bias)
⁃ External validity (generalizability)
qualitative evidence:
⁃ Dependability
⁃ Credibility
⁃ Transferability
Critical Appraisal
Critical appraisal therefore focuses on:
⁃ congruity between philosophical position adopted
in the study, study methodology, study methods,
representation of the data and the interpretation of the
results
⁃ the degree to which the biases of the researcher
are made explicit ⁃ the relationship between what the
participants are reported to have said and the conclusions
drawn in analysis
The JBI Approach
JBI SUMARI incorporates a checklist to appraise rigor
generically across methodologies
Establishing credibility in qualitative research requires:
A standardized approach that is clearly defined,
articulated, and applied, and takes into consideration the
relationship between:
▪ Methodology and methods
▪ Methods and data
▪ Data and conclusions
JBI Appraisal Tool for Qualitative Research
 1. Is there congruity between the stated philosophical perspective and the research
methodology?
 2. Is there congruity between the research methodology and the research question
or objectives?
 3. Is there congruity between the research methodology and the methods used to
collect data?
 4. Is there congruity between the research methodology and the representation
and analysis of data?
 5. Is there congruity between the research methodology and the interpretation of
results?
 6. Is there a statement locating the researcher culturally or theoretically?
 7. Is the influence of the researcher on the research, and vice-versa, addressed?
 8. Are participants, and their voices, adequately represented?
 9. Is the research ethical according to current criteria or, for recent studies, and is
there evidence of ethical approval by an appropriate body?
 10. Do the conclusions drawn in the research report flow from the analysis, or
interpretation, of the data?
The Critical Appraisal Process
All studies can be included, regardless of critical appraisal
results
Alternatively, the decision to include/exclude can be
based on:
⁃ Meeting a predetermined proportion of all criteria
⁃ Meeting certain criteria
⁃ Weighing criteria differently
Whatever the approach it needs to be transparent and
documented in the protocol
Data Extraction
Two phases:
• Study characteristics
• Findings, illustrations and levels of credibility
Example 1 Finding/Illustration (Theme Level)
A forever healing process
Venous ulcer healing extends over lengthy periods. In
the larger sample of 54 patients, for the 38 ulcers that healed,
the mean healing time was 31.37 weeks, and more than half
of the patients were experiencing their second, third, or fourth
ulcer. When asked about their first leg ulcer experience, many
patients could not even remember their first ulcer experience.
"It's been so long." The constant prevalence of the ulcer was
referred to as an important aspect of living with the condition.
"It's like having something that you cannot get rid of. It seems
like it to me. It's like...In fact, I wonder if it will ever heal.“
Another man living with the condition for years said, "It's like a
forever healing process, not getting better, not getting worse.“
From: Chase et al 1997
Approaches to Qualitative Synthesis

Thematic synthesis
Narrative synthesis
Realist synthesis
Content analysis
Meta-ethnography
Meta-aggregation (JBI approach)
Meta-aggregation
Meta-aggregation focuses on study findings not study
data
This means differing methodologies (e.g.
phenomenology, ethnography or grounded theory), using
different methods, can be mixed in a single synthesis of
qualitative studies as long as they focus on the same
phenomena of interest
Meta-aggregative Flowchart
Questions and Discussion
THANK YOU