Indriasari2020 Article GamificationOfStudentPeerRevie

Education and Information Technologies (2020) 25:5205–5234

https://doi.org/10.1007/s10639-020-10228-x

Gamification of student peer review in education:

A systematic literature review
Theresia Devi Indriasari 1,2 & Andrew Luxton-Reilly 1 & Paul Denny 1
Received: 19 September 2019 / Accepted: 17 May 2020 / Published online: 22 May 2020
# Springer Science+Business Media, LLC, part of Springer Nature 2020
Abstract
We present the first systematic review of the use of gamification in educational peer
review activities. The goal of this work is to understand how gamification has been
used to engage students in peer review activities and to summarize the empirical
evidence for its effectiveness. Our main contribution is the presentation of a general
model of the peer review process that captures the students’ activities and an exami-
nation of the specific actions within this model that have been gamified in the current
literature. We also summarize the commonly used game mechanics and the context and
year level of courses in which prior research has been conducted, along with the
reported effects on student behavior. We find that artifact assessment and artifact
creation are the two most commonly gamified actions with respect to our peer review
model and that the quantity and quality of both the artifacts and the generated feedback
are the most popular reward criteria. In addition, Science, Technology, Engineering and
Mathematics (STEM) are the discipline areas in which gamified peer review activities
are most often reported. In general, while the existing peer review literature reports
mostly positive effects of gamification on student engagement, the range of student
actions which have been incentivized remains narrow. Key activities, such as student
reflection on the feedback received, have been largely unexplored with respect to
gamification and thus present useful avenues for future work.
Keywords Gamification . Education . Peer review . Peer assessment . Systematic literature

review . Systematic review
* Theresia Devi Indriasari

[email protected]; [email protected]
1
School of Computer Science, Faculty of Science, The University of Auckland, 38 Princes St,
Auckland 1010, New Zealand
2
Department of Informatics, Universitas Atma Jaya Yogyakarta, Jalan Babarsari 43,
Yogyakarta 55281, Indonesia
5206 Education and Information Technologies (2020) 25:5205–5234
1 Introduction
Within education, peer review is a collaborative activity in which students consider,

assess, and generate feedback on one another’s work, as well as receive reviews from
peers on their own work. Peer review in education has been an active area of research
for at least 20 years (Dochy and McDowell 1997; Falchikov 1995; Pond et al. 1995). In
an early review of the literature on the use of peer review in college and university
settings, Topping (1998) identifies many benefits, including the development of cog-
nitive, social, affective, and transferable skills as students learn to deliver and accept
feedback, justify arguments and refuse suggestions. Subsequent research has provided
further evidence for these benefits (see, for example, Anvari et al. 2019; Dickson et al.
2019; Hamnett and McKie 2019; Nicol et al. 2014; Seifert and Feliks 2019). In a
review of student formative peer assessment, Søndergaard and Mulder (2012) highlight
skills that can be developed through peer assessment activities and that are greatly
appreciated in the workplace. These include analysis and communication skills.
Despite its benefits, peer review presents several challenges that must be carefully
considered by teachers. In larger classes, administrative tasks such as assignment
distribution and review collection could potentially be a problem, although this has
largely been addressed by the now-widespread use of online tools for learning man-
agement (Fu et al. 2019; Luxton-Reilly 2009). One of the most significant unresolved
challenges posed by the use of peer review in the classroom is how to effectively
motivate students (Hamer et al. 2005; Pond et al. 1995; Sitthiworachart and Joy 2004).
The benefits that arise from receiving feedback depend on the quality of that feedback,
and the quality of peer reviews rely on the students’ engagement in the review process
(Moccozet et al. 2013). Ensuring that students are engaged in the process of peer
review is therefore of utmost importance. Turner et al. (2011) highlighted the need for
external motivation in the review process so that students manage to complete the
reviews on time. Thus, further research is warranted to identify effective ways to
motivate students in peer review activities.
One technique that has been successfully used to promote motivation and increase
participant engagement is gamification (Heylighen et al. 2013). Deterding et al. (2011)
define gamification as the use of game design elements in a non-game environment.
The main reasons for implementing gamification are because of its potential to motivate
and improve positive behavior (Hamari et al. 2014). Gamification has been adopted for
motivating particular behavior changes in health and well-being (Johnson et al. 2016),
marketing (Conaway and Garay 2014), tourism (Xu et al. 2017), crowdsourcing
(Morschheuser et al. 2016), business (Kumar 2013), commerce (Adaji and Vassileva
2017), and education (Dicheva et al. 2015).
Gamification has been of considerable interest to research communities and has led
to several in-depth reviews of the literature. Hamari et al. (2014) reviewed 24 empirical
studies on gamification to evaluate its effectiveness. The results indicate that
gamification often has a positive impact on engagement and enjoyment, but the
effects depend on the context of implementation, and the participants. Hamari et al.
(2014) found that most of the studies that implement gamification in learning contexts
resulted in positive changes in engagement. Dicheva et al. (2015) conducted a mapping
study of gamification in education that investigated current empirical research on its
implementation. They found that reviewed papers reported promising results, but, most
Education and Information Technologies (2020) 25:5205–5234 5207
of the empirical studies did not provide a rigorous evaluation, making it difficult to
understand the reasons behind the positive or negative results. Therefore, Dicheva et al.
(2015) suggest more empirical studies are needed to investigate the motivating effects
of using single game elements in different educational contexts, and for specific types
of learners. In a review of gamification in software engineering education, Alhammad
and Moreno (2018) found that research on gamification in software engineering
contexts is still in its infancy, and the most common reasons for implementing
gamification are to increase student engagement and improve student knowledge or
performance. Subhash and Cudney (2018) conducted a systematic review of game-
based learning and the implementation of gamification in higher education. They
analyzed 41 papers and reported that points, badges, and leaderboards were the most
popular game elements used, and suggested that creative teaching and learning methods
using gamification in higher education should be explored to increase engagement,
motivation, and student performance. However, as they only reviewed papers that
included the phrase “Higher Education” in the title or abstract, this limited the selection
of primary studies and thus the generalizability of the findings. In a much more
extensive systematic review, Koivisto and Hamari (2019) examined 514 empirical
and non-empirical research papers on gamification. They found that most of the
research was conducted in education contexts, and the most common methods of
implementing gamification take the form of points, badges, and leaderboards. Studies
commonly found positive student perceptions of gamified systems, and desirable
changes in student behavior.
Researchers have adopted gamification in the classroom because of its potential to
improve behavior and performance (Denny et al. 2018; Landers et al. 2019; Ortiz-Rojas
et al. 2019). Gamification could, therefore, be an effective strategy for tackling the
difficult problem of low motivation in peer review activities. Gamification is growing
and maturing (Nacke and Deterding 2017), and for many kinds of tasks, there are
sufficient studies to warrant a systematic review of the application of gamification to
those tasks. In this paper, we conduct a systematic review of the literature on the use of
gamification within peer review tasks. We use the term peer review to define activities
where students are making a judgement or considering another students’ work for either
summative or formative purposes. We are interested in understanding how gamification
can be effectively used to engage students in peer review activities, potentially leading
to improved learning outcomes.
We conduct a large-scale systematic literature review, which to the best of our
knowledge, is the first systematic review of gamification applied to peer review
activities. The main contribution of this review is the identification of primary studies
that present empirical evidence relating to the use of gamification within peer review
tasks. We describe a model for classifying the interactions that occur within a typical
peer review process, and we use this model to analyze the literature. We also examine
the context of the gamified peer review studies and the evidence for the impact of
gamification.
The remainder of this paper is organized as follows. Section 2 outlines our research
questions and explains our research methodology for performing the systematic liter-
ature review, including the processes for selection of primary studies, data extraction,
and analysis of results. Section 3 explains the peer review model that is used in the
analysis process, and we present our results in Section 4. Section 5 discusses our key
findings, and we outline threats to validity in Section 6. Section 7 concludes with the
trends and gaps identified from the study, which may form the basis of future work.
2 Research methodology
In conducting this systematic literature review, we follow the guidelines proposed by

Kitchenham (2004). This section describes each step of the study, including the
selection of research questions, the search process, the selection process, and the data
extraction and analysis processes.
2.1 Research questions
The purpose of this study is to explore how gamification has been adapted for use in
student peer review activities and to understand its effectiveness as reported by the
research literature. A goal of this work is to identify potential gaps and opportunities for
future study. Table 1 summarizes the research questions used in this systematic
literature review.
The goal of the first research question (RQ1) is to understand the various ways in
which gamification has been used in peer review contexts. We split this broad question
into three distinct aspects of the implementation: activities in peer review that have
been gamified; methods to gamify peer review activities; game elements that have been
used. The second research question (RQ2) focuses on identifying the context of the
peer review, including the types of subjects/courses and the level of education of the
students. The last research question (RQ3) examines the reported evidence for the
impact of gamification. Together, this information will be valuable for understanding
how the gamification of peer review activities can affect the learning process.
2.2 Search process
The search process used in this study is described in Fig. 1. The first step of the search
process was defining candidate main search terms. The search string was composed of
three main terms: “gamification”, “peer review”, and “education”.
In the second step, each main term was expanded into multiple synonymous terms.
Synonyms for “gamification” were “gamified”, “gamify”, “gamifies”, “gamifying”,
Table 1 Research questions
ID Research Question
RQ1 How have peer review activities been gamified?

RQ1.1 What actions have been gamified?
RQ1.2 What methods have been used to gamify the actions?
RQ1.3 What game elements have been used?
RQ2 In which contexts (course/subject) and levels (year of study) have peer review been gamified?
RQ3 What is the evidence and impact of the gamified peer review studies?
Fig. 1 The seven-step search process
“gameification”, “gameified”, “gameify”, “gameifies”, and “gameifying”. Synonyms

selected for the term “peer review” were “peer assessment”, “peer learning”, “peer
feedback”, “peer grading”, “peer mentoring”, “peer help”, “code review”, and
“crowdsource”. To identify peer review studies in the educational context, we used
supplementary terms that included: “learning”, “student”, “course”, “software”, and
“task”. These search terms were then linked in the third step of the process using the
logical operators “AND” and “OR”, with logical “AND” used for the three primary
parts of the search string and logical “OR” for the similar phrases and keywords.
In the fourth step, seven databases were chosen as primary sources: ACM Digital
Library, IEEE Xplore, ScienceDirect, Springer Link, Scopus, Web of Science Core
Collection, and ERIC. Each of the databases has different characteristics that require
minor adjustments to the search. The search process in this study was intended to
capture as many relevant studies as possible, so the following conditions were applied
to the respective databases:
1. ACM Digital Library: search in the full text

2. IEEE Xplore: search in the metadata and full text
3. Science Direct: search in all parts of the document
4. SpringerLink: search in the title, the abstract and the full text (including authors,
affiliations, and references
5. Scopus: search in the title, abstract, and keyword
6. Web of Science Core Collection: search in the topic and title
7. ERIC: search anywhere
Before the search strings were applied, there was a step to finalize the search terms.
This fifth step was to identify which of the synonymous terms contributed to the
effectiveness of the search. The first experiment added each of the terms synonymous
with gamification to the search string and applied the search on each database to
identify if any additional relevant papers were identified using that term. The search
strings: (gamification OR X) AND (“peer review” OR “peer assessment” OR “peer
learning” OR “peer feedback” OR “peer grading” OR “peer mentoring” OR “peer
help” OR “code review” OR crowdsource) AND (education OR learning OR student

OR course OR software OR task) were applied to each database, with X replaced with
the synonymous term of gamification. The result of the experiment showed that none of
the proposed gamification synonyms were effective in identifying any additional
relevant papers. The second and third experiments followed a similar process to
eliminate ineffective terms synonymous with peer review and education, respectively.
In the sixth step, the search string was finalized as follows: (gamification) AND (“peer
review” OR “peer assessment” OR “peer learning” OR “peer feedback” OR “peer
grading” OR “peer mentoring” OR “peer help” OR “code review” OR crowdsource)
AND (education OR learning OR student OR software OR task).
The last step of the process involved searching all databases using the search string.
The search was applied to the ACM Digital Library, IEEE Xplore, ScienceDirect,
Springer Link, and Scopus on 30 July 2018 and also in the Web of Science Core
Collection and ERIC on 10 August 2018. A total of 1281 publications were returned as
a result of this search, with publication years ranging from 2011 to 2018.
2.3 Selection process
The process of selecting primary studies from the full list of candidate publica-
tions consisted of four screening stages. In the first stage, several criteria were
applied initially to exclude studies. Articles of less than three pages (e.g. extended
abstracts, posters or short papers), articles that are not written in English, full-text
articles that are not accessible, duplicate titles, and articles that are not peer
reviewed were excluded.
The second stage of the screening process involved reading the title, keywords, and
abstract to identify studies that are relevant to gamification of peer review in an
education context. Furthermore, articles that reported serious games, online games, or
game-based learning studies were excluded. In addition, articles that discussed
gamification and peer review in a non-educational context, such as health, commerce,
or tourism were excluded. As mentioned earlier in the introduction section, this paper
used the term peer review to define activities where students are making a judgement or
considering another students’ work for either summative or formative purposes. There-
fore, studies that included peer review activities in their learning process (such as
posting and replying in a forum discussion) were included.
The third stage involved reading the full text in detail. In this stage, in-depth criteria
were needed to capture primary studies that are relevant to the research questions of this
study. During the third stage, the criteria required articles to have reported empirical
studies, where participants were students enrolled in an institution or organization, and
that the article provided sufficient details of the gamified peer review activity (such as
explicitly describing the peer review activity as well as the reward(s) given to students
for doing the activity). Table 2 lists the inclusion and exclusion criteria that have been
applied across these four screening stages.
The final stage involved a snowballing process. Snowballing refers to the identifi-
cation of additional papers using the reference list of an article or the citations to the
article (Wohlin 2014). This review used the reference list for backward snowballing to
capture relevant studies that were cited by the primary studies. For each entry in the
reference list of each primary study, the four-stage process described for selection of
Table 2 Inclusion and exclusion criteria
Inclusion Criteria Exclusion Criteria
An article that has at least three pages Poster or short paper that has less than three pages
Written in English Not written in English
Accessible Not Accessible
Peer reviewed articles Non-peer reviewed articles
Only one copy of each title was included Duplicated titles
Studies that describe gamification and peer Studies that describe serious games and/or do not explicitly
review in education contexts use gamification and/or there is no peer review activity
and/or in non-education contexts
Empirical studies Non-empirical studies
Participants of the experiment were enrolled as Participants of the experiment were not enrolled students of
students in an institution or organization an educational institution
Studies that provide detailed information Studies that provide insufficient detail of the gamified peer
regarding the gamified peer review activity review activity
primary studies was applied, until there were no further papers added through
snowballing.
Figure 2 shows the selection process and the number of papers identified in each
stage. In summary, using the selection process described above, 1248 articles were
excluded, leaving 33 studies. Six additional studies were identified from the backward
snowballing process, bringing the total number of primary studies to 39.
2.4 Data extraction and analysis process
The data analysis was informed by several existing frameworks. To answer RQ1.1 and
RQ1.2, a gamified peer review model (described further in section 3) was adopted from
Hamer et al. (2007), while a framework used by Zichermann and Cunningham (2011),
and Bunchball (2010), was used to categorize game mechanics to answer RQ1.3.
Finally, to categorize higher education contexts for RQ2, the National Survey of
Fig. 2 Selection process

Student Engagement (NSSE) 2018 Codebook (National Survey of Student

Engagement 2018) was used.
The data extraction process was divided into two parts: the design of the data
extraction form and the data extraction itself. An electronic data extraction form was
created to record the data, and included the following key fields:
1. Gamified actions and methods used to gamify. This field records the activities
in the peer review process that were gamified, as well as the methods used
to gamify each action. The peer review model presented in Fig. 3 was used
to categorize the activities that occur during a peer review task. Each arc in
the model represents a different peer review action that could potentially be
gamified.
a. Arc A (Author develops, creates, or improves artifact)

b. Arc B (Reviewer views, or considers artifact from the author)
c. Arc C (Reviewer judges, reviews, or assesses artifact from the author)
d. Arc D (Author reads or reflects on feedback from the reviewer)
e. Arc E (Rater reads or considers feedback from the reviewer)
f. Arc F (Rater judges, reviews, or assesses feedback from the reviewer)
g. Arc G (Reviewer reads or reflects on feedback from the rater)
2. Game elements. This field records the gamification mechanics used.

3. Artifacts. This field records the kind of artifacts developed by authors as part of the
peer review task. Artifacts are categorized as one of text, multimedia, source code,
or mixed (combination).
4. Context and level. This field records the subject of the course or discipline, and also
the level of education where gamified peer review was applied.
5. Sample size. This field records the number of participants in the empirical study.
6. Research measure. This field records the metric used by primary studies to
evaluate the impact of gamification.
7. Impact. This field gathers information regarding the impact of gamified peer
review activities.
3 Peer review
A peer review process involves several activities or actions, such as creating an

artifact, reading an artifact, or delivering feedback to peers. A peer assessment
model from Hamer et al. (2007) has been adapted to analyze the gamified peer
review activities. The original model describes different activities that may form
part of a peer review process and is intended to capture a wide range of
activities completed by both students and instructors. The current study focuses
on gamifying students’ interactions in peer review rather than instructors’
activities. Consequently, three arcs and two nodes (representing dispute resolu-
tion and grading) have been removed from the peer assessment model described
by Hamer et al. (2007), as they related to instructor activities rather than
student activities that could be gamified.
Fig. 3 Gamified Peer Review Model
3.1 Gamified peer review model
This section explains the “Gamified Peer Review Model” (Fig. 3) that we use in our
analysis. It captures the activities that may occur during a student peer review process.
The circular nodes represent the different roles that students adopt. The rectangular
nodes symbolize objects. The arcs represent actions taken by students, mediated
through the peer review system.
An author is a person who creates or improves an artifact. A reviewer is a peer that
reviews the artifact and delivers feedback to the author. A rater is a person who reviews
the feedback from reviewers. In this model, each role could potentially be filled by one
or more students (i.e., authors could also be reviewers and raters).
An artifact could be in any form such as questions (problems), answers (solutions),
files (such as audio, video, or image), essays, bookmarks, wiki pages, web pages, or
computer source code. Feedback could be in the form of Boolean (True or False, such
as: correct or wrong), Fixed Group (selection criteria values, such as: correct, partially
correct, or wrong), Interval (numbers in a range with a minimum and maximum value,
where any number in the range is allowed), or Text (free criteria, such as: any text in a
text box). Within this peer review model, the reviewer could give feedback to the
author in any form such as a grade (interval), a comment (text), a vote (boolean), or a
rating (fixed group).
The seven arcs in this model represent actions in the peer review process and are
identified using letters from A to G. The letter does not represent the order of the action
but is arbitrarily allocated to distinguish between different activities. Arc A represents
the development of artifacts where authors create new products or make improvements
to existing products. Arcs B and E represent similar actions in which a reviewer views
an artifact produced by an author, or a rater views the feedback produced by a reviewer.
Arcs C and F involve judgment, either by a reviewer (on an artifact) or by a rater (on a
review), in which content is reviewed and feedback is generated. Arcs D and G
represent reflection, either by an author (on the reviewer’s feedback on their artifact)
or by a reviewer (on the rater’s feedback on their review) in which the recipients reflect
on the feedback they have received which may then be used to inform future work.
To illustrate the application of this model, we describe two common scenarios that
involve peer review processes.
3.2 Example scenarios
The first scenario involves arcs A, B, C, and D. The first scenario is initiated when an
author creates an artifact (arc A) and submits it to the system. The system then delivers
the artifact to peers of the author, who are expected to review the artifact. A reviewer
will first view the artifact (arc B) and will then evaluate the artifact using some criteria
(arc C). Finally, the author can read the feedback and use that feedback to reflect on the
reviewed artifact (arc D).
The second scenario includes arcs A, B, C, E, F, and G. Like the first
scenario; it begins when an author creates an artifact (arc A). Then, reviewers
will view the artifact (arc B), evaluate the artifact, and deliver feedback to the
author (arc C). In this scenario, authors (who are playing the role of a rater),
subsequently view the feedback (arc E), then evaluate the quality of the
feedback (arc F) that they have received. Finally, the reviewers read and reflect
on the feedback from the raters (arc G).
Theoretically, all actions in Fig. 3 could be gamified.
4 Results
4.1 General results
This section presents the general descriptive data gathered from the study. Figure 4
depicts the distribution of primary studies by publication years from 2013 to mid-2018.
The first three studies that adopted gamified peer review activities in education were
dated back to 2013. Over the next several years, the number of published studies
suggests a growing interest in the topic, bearing in mind that the search process was run
in the middle of 2018. Journals and conference proceedings were the dominant venues,
each accounting for 46% of the studies. Only 3 of the 39 articles were published as
book chapters.
4.2 RQ 1 – The implementation of gamified peer review activities
RQ 1 addresses the implementation of gamification in peer review activities. As

mentioned previously, in this study, we consider the following three aspects: activities
or actions that have been gamified, mechanisms for gamifying those activities, specific
game elements that have been used.
4.2.1 RQ1.1 – Gamified peer review activities
RQ1.1 aims to identify the different activities or actions in peer review tasks that have
been gamified. We present these results using the peer review model introduced in
Fig. 4 Distribution of primary studies by year
Section 3.1. Fig. 5 illustrates the arcs in the model that have been targeted by the
primary studies, with line thickness denoting frequency.
Theoretically, all actions in the peer review model could be gamified to motivate
specific behaviors. Our findings, presented in Fig. 5, show that the majority of studies
focused on two actions. Arc C, the action in which a reviewer judges or assesses an
artifact, was targeted by nearly 90% of the primary studies, while arc A, the artifact
creation action, was targeted by 82% of the studies. Very few studies gamified arcs B or
F (only 3 of 39 involved each arc), and no primary studies attempted to gamify arcs D,
E, or G.
Fig. 5 Gamified peer review actions

4.2.2 RQ1.2 – Method or technique to gamify an activity
RQ 1.2 aims to identify the methods or techniques used to gamify peer review
activities. These methods can be classified as follows:
1. Quality: methods that measure the grade or excellence of artifacts produced by

authors or feedback delivered by reviewers or raters.
2. Quantity: methods that count the number of artifacts produced by authors or
feedback delivered by reviewers or raters.
3. Time: methods that measure duration or rank the order of submitted artifacts or
feedback.
Table 3 presents the methods that were used to gamify an action. As Table 3 shows,
most primary studies (24 out of 39) used gamification to motivate students to create
high-quality artifacts rather than to produce a certain quantity of artifacts or to deliver
artifacts on time. In contrast, most primary studies (26 out of 39) rewarded reviewers
based on the quantity of the feedback they produced, rather than on the quality or
timeliness of their feedback. Interestingly, very few studies rewarded students for
reading their peers’ artifacts (arc B; 3 out of 39) or for reviewing feedback produced
by their peers about their own work (arc F; 3 out of 39).
Table 3 provides more detailed information regarding the methods used by the
primary studies to reward actions performed by authors, reviewers, or raters. In
addition, this table presents the kind of game elements received. During analysis, we
identified points, leaderboards, badges, progress bars, levels, virtual gifts, and prizes as
elements that have been used to gamify specific actions in peer review. Authors,
reviewers, or raters received one or more of them as consequences of their actions
based on qualitative, quantitative, or time-based metrics. Overall, badges, points, and
leaderboards were the most popular rewards, followed by virtual gifts, progress bars,
levels, and prizes. This result is consistent with findings from Hamari et al. (2014), who
explored gamification in a broader range of contexts. We found that several studies
only used one game element. However, more than half of the studies combined several
game elements as rewards to authors, reviewers or raters, such as points-leaderboards
(Otake et al. 2015; Zimmerling et al. 2018), badges-points-progress bars-leaderboards
(Ding et al. 2017, 2018), badges-leaderboards (Huang and Hew 2018; Morales and
Amado-salvatierra 2016; Rizzardini et al. 2016), points-leaderboards-virtual gifts (De-
Marcos et al. 2016b, 2017; Ding et al. 2017), and points-leaderboards-badges (Li et al.
2013; Usami et al. 2015).
The creation (or modification) of artifacts (arc A) by authors was gamified in several
ways. First, based on qualitative metrics, authors received rewards such as points or
badges when they revised their assignments based on peer feedback (Wang et al. 2016),
or they submitted deliverables that received positive reviews based on specific criteria
(Borras-Gene et al. 2016; Boticki et al. 2015; De-Marcos et al. 2016b; Denny 2013;
Palomo-Duarte et al. 2014; Sunikova et al. 2017). Other examples include posts made
to forums which earned positive votes or reactions from peers (De-Marcos et al. 2016a;
Ding et al. 2018; Morales and Amado-salvatierra 2016; Rizzardini et al. 2014;
Ruipérez-Valiente et al. 2016; Staubitz et al. 2017). Second, based on quantitative
metrics, authors earned rewards such as points or badges when they performed at least
Table 3 Method or technique to gamify an action and game element received
Arc Method N Game Elements
Points Leaderboards Badges Progress Level V i r t u a l Prize

Bar Gift
A Quality 24 ID3, ID4, ID5, ID4, ID6, ID5, ID6, ID9, ID5, ID6 ID6 ID16, ID15
ID6, ID8, ID7, ID9, ID10, ID14, ID17
ID11, ID16, ID10, ID16, ID24,
ID17, ID19, ID16, ID34, ID35,
ID20, ID21, ID17 ID37, ID38
ID22, ID27,
ID32
Quantity 16 ID1, ID2, ID5, ID7, ID11, ID2, ID5, ID11, ID5, ID7 ID39 ID16,
ID11, ID12, ID12, ID12, ID13, ID17,
ID15, ID16, ID15, ID26, ID35, ID18
ID17, ID18, ID16, ID39
ID19, ID26, ID17,
ID28, ID39 ID18,
ID26,
ID39
Time 5 ID36 ID10, ID36 ID10, ID14,
ID34, ID35,
ID36
B Quantity 3 ID1, ID31 ID31 ID35
C Quality 19 ID5, ID11, ID19, ID5, ID9, ID5, ID8, ID9, ID5, ID30 ID6
ID22, ID23, ID10, ID10, ID11, ID30
ID25, ID26, ID11, ID14, ID19,
ID28, ID30, ID23, ID26, ID30,
ID32, ID36 ID26, ID34, ID36,
ID30, ID37
ID36
Quantity 26 ID1, ID2, ID3, ID3, ID11, ID2, ID11, ID7 ID39 ID16,
ID7, ID11, ID12, ID12, ID13, ID17,
ID12, ID15, ID15, ID14, ID16, ID18
ID16, ID17, ID16, ID18, ID19,
ID18, ID19, ID17, ID26, ID33,
ID21, ID23, ID18, ID34, ID35,
ID26, ID28, ID23, ID36, ID37,
ID29, ID31, ID26, ID38, ID39
ID36, ID39 ID29,
ID31,
ID33,
ID36,
ID39
Time 5 ID5 ID5, ID9, ID5, ID9, ID10, ID5
ID10 ID14, ID34
F Quantity 3 ID14, ID34,
ID37
one or more activities/assignments (De-Marcos et al. 2016a, 2016b; Huang and Hew
2018; Moccozet et al. 2013; Tenório et al. 2016; Usami et al. 2015), or they posted one
or more entries within a forum (Anderson et al. 2014; Berkling 2016; Ding et al. 2018;
Huang and Hew 2018; Kyewski and Krämer 2018; Li et al. 2013; Sevilla-Pavón and
Haba-Osca 2017; Staubitz et al. 2017; Winoto and Tang 2015). Lastly, relating to time-
based metrics, authors earned rewards when they submitted deliverables on time (Hew
et al. 2016), or they posted the first entry in a forum (Anderson et al. 2014; Rizzardini
et al. 2016), or they contributed the first deliverables (Denny 2013; Walsh et al. 2018).
Only a small number of primary studies gamified arc B (view or consideration of
artifacts by reviewers). All of those used quantitative metrics to deliver rewards, for
example, by reading or considering at least one artifact (Anderson et al. 2014;
Moccozet et al. 2013; Papadopoulos et al. 2016).
The assessment of artifacts (arc C) by reviewers was gamified with respect to
quality, quantity, and time. First, based on qualitative metrics, reviewers re-
ceived rewards when their assessment was considered high quality by the system
(Hsu and Wang 2018; Rojas et al. 2017), or when they commented on a post in a
forum and received several replies (Ding et al. 2018). Other examples include
answering/collaborating in discussions/questions in a forum and receiving one or
more positive reactions (Ding et al. 2017, 2018; Kyewski and Krämer 2018;
Morales and Amado-salvatierra 2016; Rizzardini et al. 2016; Rizzardini et al.
2014; Ruipérez-Valiente et al. 2016; Staubitz et al. 2017; Winoto and Tang
2015), and when comments/answers to deliverables received positive responses
(Agapito and Rodrigo 2018; Denny 2013; Maxim et al. 2017; Rojas et al. 2017;
Usami et al. 2015; Walsh et al. 2018; Zimmerling et al. 2018). Secondly, based
on quantitative metrics, reviewers received points or badges when they
commented, answered or corrected several deliverables (Boticki et al. 2015;
De-Marcos et al. 2016a, 2016b; De-Marcos et al. 2017; Denny 2013; Gehringer
and Peddycord 2013; Khandelwal et al. 2017; Li et al. 2013; Maxim et al. 2017;
Moccozet et al. 2013; Papadopoulos et al. 2016; Tenório et al. 2016; Usami et al.
2015; Walsh et al. 2018; Wang et al. 2016; Zimmerling et al. 2018). Moreover,
reviewers received rewards when they gave contributions/replies/comments/
votes to posts in forum discussions (Anderson et al. 2014; Berkling 2016;
Hew et al. 2016; Huang and Hew 2018; Kyewski and Krämer 2018; Latulipe
et al. 2015; Ruipérez-Valiente et al. 2016; Sevilla-Pavón and Haba-Osca 2017;
Winoto and Tang 2015). Lastly, time-based metrics were used to reward re-
viewers when they posted the first comment or response in a forum (Ding et al.
2018; Morales and Amado-salvatierra 2016; Rizzardini et al. 2016) or when they
submitted the first answer or commented on a deliverable (Denny 2013; Walsh
et al. 2018).
Few studies gamified arc F (judgment of feedbacks by raters) based solely on
quantitative metrics, such as when raters grade/reply/vote on feedback submitted by
their peers (Denny 2013; Walsh et al. 2018) or when raters voted on answers/feedback
in a forum discussion (Ruipérez-Valiente et al. 2016).
Nearly all primary studies delivered rewards to students using an automated system.
However, a few studies adopted a hybrid approach, which also included rewards given
manually by a teacher. An example of this was when expert knowledge was needed to
judge students’ assignments or feedback (Ding et al. 2017, 2018).
In general, while a diverse combination of methods and game elements have been
studied for gamifying activities in peer review tasks, there exist some specific ap-
proaches that have not been explored in detail.
4.2.3 RQ1.3 – Game elements
RQ 1.3 aims to identify the game elements used in each primary study using a game
mechanics framework adapted from Zichermann and Cunningham (2011), and
Bunchball (2010). Table 4 lists, for each primary study, the specific game elements
reported. Points, badges, and leaderboards were the most popular gaming mechanics.
However, a variety of elements were explored, including progress bars, virtual gifts,
levels, missions, and prizes. Many primary studies (62%) involved the use of some
combination of elements, with just 15 studies reporting the use of a single element.
Points were one of the most commonly explored elements. Examples of common
types of points used in the primary studies include 1) scale-points, where the points are
used to differentiate the level of activities (Moccozet et al. 2013), 2) experience points
that count towards student grades (Berkling 2016; Gehringer and Peddycord 2013;
Winoto and Tang 2015), 3) points that are accumulated and are visualized in the user’s
avatar, leaderboards or ranking systems (Maxim et al. 2017; Otake et al. 2015; Tenório
et al. 2016), 4) points that are used as a virtual currency (De-Marcos et al. 2016a,
2016b, 2017).
Similar to points, badges were reported in most of the primary studies. Using the
classification of badges by Antin and Churchill (2011), we present some examples of
the use of badges as: 1) instruction badges (students received them for completing
single training objectives within the systems (Walsh et al. 2018; Denny 2013), 2) goal-
setting badges (students received them for achieving certain goals such as producing
several artifacts or delivering feedback multiple times or receiving positive reviews
from peers (Morales and Amado-salvatierra 2016; Rizzardini et al. 2016; Tenório et al.
2016). Goal-setting badges were designed in several ways, for example, 1) in the form
of various names such as Smart, Genius, or Einstein badges to differentiate the
achievement in one particular activity (Boticki et al. 2015; Denny 2013; Hew et al.
2016; Walsh et al. 2018) 2) they could also be in the form of different visualizations of
badges such as bronze, silver and gold, or ribbons or medals to reward different
achievements (Tenório et al. 2016; Usami et al. 2015) or 3) they could be converted
into experience points, with greater achievements earning more experience points
(Ding et al. 2017, 2018). Such experience points could later be transformed to virtual
currency (De-Marcos et al. 2016a, 2016b, 2017), progress bars or leaderboards (Ding
et al. 2017, 2018).
Leaderboards, progress bars, and levels were usually applied together with points.
Leaderboards were the visualization of accumulated points, usually in the form of a list
of all participants or a top-n list, allowing students to obtain a sense of their perfor-
mance relative to others in the class. Like leaderboards, progress bars and levels also
illustrate earned experience points, thus provide students with a visual presentation of
their current achievements and progress. In some cases, progress bars were associated
closely with levels (Ding et al. 2017; Hsu and Wang 2018; Li et al. 2013), but in a few
cases, these were independent (Berkling 2016; Ding et al. 2018).
A small number of studies introduced quests as a game element (Sevilla-Pavón and
Haba-Osca 2017; Tenório et al. 2016). A quest could be perceived as a mission or a
task to be accomplished by students that will lead to some reward.
Finally, a few studies reported virtual goods or gifts as one of their game elements
(De-Marcos et al. 2016a, 2016b, 2017; Ding et al. 2017; Sevilla-Pavón and Haba-Osca
Table 4 Game Elements
Study ID Points Leaderboards Badges Progress Bar Level Mission/ Quest Virtual Gift Prize
6 Y Y Y Y Y Y
30 Y Y Y Y Y
39 Y Y Y Y Y
15 Y Y Y Y Y
5 Y Y Y Y
2 Y Y Y Y
16 Y Y Y Y
18 Y Y Y Y
11 Y Y Y
12 Y Y Y
23 Y Y Y
26 Y Y Y
36 Y Y Y
7 Y Y Y
17 Y Y Y
27 Y Y Y
3 Y Y
4 Y Y
29 Y Y
31 Y Y
9 Y Y
10 Y Y
33 Y Y
19 Y Y
1 Y
21 Y
22 Y
25 Y
28 Y
32 Y
8 Y
13 Y
14 Y
20 Y
24 Y
34 Y
35 Y
37 Y
38 Y
2017) and one study provided a physical prize (Sevilla-Pavón and Haba-Osca 2017) to
reward students’ achievements. Typical forms of virtual gifts included avatars, virtual
coffee, virtual flowers, and virtual dolls, which could be purchased using points as a
virtual currency and then delivered from students to their peers.
4.3 RQ2 – The context and level
RQ2 addresses the context and level of the gamified peer review activities within the
literature. This question focuses on identifying the type of subject/course and level of
studies that have been gamified. Naturally, artifacts generated by students are related to
the context and level of the courses they are studying. Analysis of the collected data
identified the following types of artifacts:
1. Multimedia: artifacts that use a combination of various forms such as text, audio,
images, animations, video, and interactive content.
2. Source code: artifacts that contain a set of instructions and statements produced by
programmers using any programming language.
3. Text: artifacts that contain a human-readable sequence of characters that form
words. This includes short text (less than 300 words) or long text (essays).
4. Mix: artifacts that contain a combination of two or more types of artifacts.
In order to answer RQ2, each gamification experiment in every primary study was
recorded. In some cases, a single article reported two gamification experiments in
different contexts and different levels of study. Therefore, the frequency of the context
and level reported in this section exceeds the total number of primary studies.
Analysis of the collected data produced three levels of education: elementary, high
school, and university. Only a small number of experiments reported that participants
were at high school (Tenório et al. 2016) or elementary level (Boticki et al. 2015; Hsu
and Wang 2018). The majority (92%) of the gamification experiments were conducted
within traditional learning and distance learning contexts at the higher education level.
We use the National Survey of Student Engagement (2018) to categorize the various
subject areas at the university level. According to the NSSE 2018 Codebook (National
Survey of Student Engagement 2018), there are 11 major subject categories, namely:
Arts & Humanities, Biological Science, Agriculture, & Natural Resources, Physical
Science, Mathematics & Computer Science, Social Sciences, Business, Communica-
tions, Media, & Public Relations, Education, Engineering, Health Professions, Social
Service Professions, and Other majors (not categorized).
Figure 6 illustrates the frequency of the context at the university level and the kind of
artifacts produced by authors in the corresponding gamified peer review tasks. Most of
the studies are conducted in Physical Sciences, Mathematics & Computer Science,
Engineering, and Education contexts. A small number of studies could not be classified
due to a lack of information.
As seen in Fig. 6, most of the artifacts produced were text. An interesting finding
was that in Physical sciences and Mathematics & Computer Science, the artifacts
produced were not only text but a mix of types. This is because the artifacts were
assignments that were typically a combination of text and multimedia or text and source
code. In other contexts, artifacts were more commonly text based. The least popular
Fig. 6 Context and type of artifacts at the university level
artifacts were source code, mix (such as digital stories), and multimedia (music
recording).
In summary, gamified peer review tasks have not been well explored in elementary
and high school settings. Most of the work in this area has been at the university level in
mathematics, computer science, and engineering courses.
4.4 RQ3 – Evidence and impact
RQ3 summarises the evidence and impact of the gamified peer review studies. Hence,
it synthesizes the frequency of primary studies according to the research measures, the
research size (the number of participants in the experiment), and the type of impact.
According to the primary studies, the research measures for evaluating the impact of
gamification in peer review activities can be classified as follows:
1. Engagement: to measure the impact of gamification towards students’ enthusiasm

in participation/contribution (such as to create more artifacts and deliver more
feedback).
2. Contentment: to measure whether students perceive the method of gamification as
enjoyable and/or students experience pleasure and satisfaction implicit in the
activity.
3. Correlation between peer review activities and performance: to measure the impact
of gamification towards the association of activity frequency and students’
performance.
4. Correlation between the difficulty of activity and reward: to measure the impact of
gamification towards the correlation of activity difficulty and reward given (such as
badges).
5. Creation of learning communities: to measure whether the method of gamification
can stimulate the creation of learning communities.
6. Performance: to measure the impact of gamification towards students’ performance
(for example by the quality of product).
Table 5 lists each of the research measures broken down by the size of the study (the
number of participants) and the outcomes. Study size is classified into four categories
which represent quartiles, and a category for studies where the number of participants
has not been reported:
1. Small: the number of participants was less than 40 (n = 10)

2. Medium: the number of participants was between 40 and 79 (n = 11)
3. Large: the number of participants was between 80 and 299 (n = 11)
4. Extra-large: the number of participants was 300 or greater (n = 11)
5. Unknown/unspecified: the number of participants was unspecified (n = 1)
Outcomes are also divided into three types:
1. Positive: primary studies that report a positive result with respect to the research
measure.
2. Negative: primary studies that report a negative result with respect to the research
measure.
3. Neutral: primary studies that report a neutral result with respect to the research
measure.
Each publication evaluated the impact of gamification using one or more measures, and
some literature conducted more than one study or experiment. Therefore, the total
numbers shown in Table 5 add up to more than the number of primary studies.
As illustrated by Table 5, most of the primary studies evaluated the impact of
gamification with respect to students’ participation or contribution towards peer review
activities. Twenty studies reported positive evidence (across a mix of study sizes). A
variety of indicators were used to measure engagement, such as the number of essays
produced (Tenório et al. 2016), the number of posts made in a discussion forum (Ding
et al. 2018), and the number of completed activities (Huang and Hew 2018). In
addition, five studies reported neutral or no impact. For example, Rizzardini et al.
(2016) and Kyewski and Krämer (2018) report no increase in the number of products
delivered in the learning activities. Khandelwal et al. (2017) also mentioned that there
Table 5 Research measures reported by primary studies, broken down by study outcome and study size
Research measures Positive Negative Neutral
S M L XL U S M L XL U S M L XL U
engagement 6 7 3 3 1 1 1 2 1
contentment 3 5 3 2
correlation between peer review activities and 2
performance
correlation between the difficulty of activity and 1
reward
creation of learning communities 1
performance 3 4 3 2 1 1 2
was no impact of gamification on the code review process in terms of the number of
comments produced and the number of bugs found.
Regarding performance, 12 studies reported positive evidence (three in small, four in
medium, three in large classes, and two in extra-large classes). Different indicators were
used to measure performance, for example, Tenório et al. (2016) used the number of
essays corrected, Huang and Hew (2018) reported the quality of the artifacts produced
in the activity, and De-Marcos et al. (2016a) based their results on a post-test assess-
ment. Four studies reported a neutral impact on student performance. For example, De-
Marcos et al. (2017) found no impact on final examination scores, Kyewski and
Krämer (2018) concluded that badges did not influence grades or quiz results, and
Agapito and Rodrigo (2018) reported no significant differences in the quiz scores of
gamification and non-gamification groups.
Concerning the perception of the tools, 13 studies reported positive results (three in
small, five in medium, and three in large and two extra-large classes). For example,
Wang et al. (2016) reported that students considered the use of the Edventure tool
interesting compared to a previous mathematics course in which it was not used. Ding
et al. (2017) found that the majority of participants were emotionally engaged by the
gamification approach, and report that it generated more fun and less intimidating
online discussions. Five studies reported a neutral impact of gamification. For example,
Morales and Amado-salvatierra (2016) and Rizzardini et al. (2016) found that earning
badges in their gamified forums was unimportant to many students. A similar result
was also reported by Kyewski and Krämer (2018), who found that students did not
perceive badges as fun and reported no impact on students’ intrinsic motivation.
Two studies reported a positive association between student engagement in the
learning activity and their course performance. For example, Walsh et al. (2018) stated
that a higher frequency of question writing was correlated with better performance,
while Ruipérez-Valiente et al. (2016) indicated that the quantity and the quality of
contributions made by students, as well as the feedback they received, correlated with
final-year examination scores.
Finally, only one study discussed the effect of gamification on the creation of
learning communities. De-Marcos et al. (2016b) conducted a social network analysis
of their activity and concluded that social gamification tools encourage the creation of
meaningful learning communities.
5 Discussion
Interest in the gamification of peer review activities appears to be growing, increasing

from four publications in 2013 to 11 publications in 2016. As peer review tools become
more widely available for use by teachers (e.g., tools such as PeerMark supported by
Turnitin (Turnitin 2019), there is increased opportunity to adapt these systems by
incorporating game elements. However, there are still relatively few studies that have
investigated the gamification of peer review systems. Given the potential for
gamification to improve student engagement, gamification may become more common
in peer review systems. Relatively few game elements have been the focus of study,
suggesting opportunity for future studies to explore the impact of a wider range of
game elements, and to confirm the generalizability of previous findings by conducting
studies in a wide range of contents. More innovative implementations of gamification

in peer review systems may also be the focus of future work.
Our model of peer review interactions (shown in Fig. 3) identifies seven user actions
that could potentially be gamified, yet, most studies of gamification focus on just two of
these interactions – artifact creation (arc A), and the assessment/feedback on the artifact
(arc C). This result is not surprising because these interactions are the essential
components in peer review. However, it is worth noting that there are very few studies
that explore gamification applied to other actions that peer review systems commonly
support (such as rating and reading feedback). Teachers may adopt existing software
that was used in the primary studies we reviewed, such as PeerWise (Denny 2013),
Khan Academy (Ruipérez-Valiente et al. 2016), Github (Khandelwal et al. 2017), Elgg
(De-Marcos et al. 2016a), Moodle (Kyewski and Krämer 2018), or they may develop
entirely new systems. New systems that support the full range of interactions in the peer
review model may be suitable for future research that looks to explore the application
of gamification to the set of actions that have not yet been studied.
The analysis of primary studies indicated that the quantity and quality of the artifacts
and feedback were the main dimensions used to gamify the peer review activities.
Quantitative metrics are objective and can be measured automatically by the system.
For example, the number of artifacts and feedback submissions can be summed to
calculate the quantity of work that authors, reviewers, or raters have completed. On the
other hand, qualitative metrics are predominantly based on more subjective measures,
such as the opinions of peers, because judgement about quality is difficult to automate.
However, systems such as Perusall (Miller et al. 2018) have demonstrated that machine
learning techniques may be sufficient to provide automated grading of qualitative
comments, suggesting that it may be possible to use such approaches to gamify
qualitative feedback in the future.
The gamification of both quality and quantity has been explored by multiple primary
studies. However, the dimension of time which can be automatically obtained through
system timestamps seems to have been less well explored by the primary studies.
Several empirical studies (Britton and Tesser 1991; Macan et al. 1990) suggest that
effective time management is correlated with improved work output and MacCann
et al. (2012) claim that students who are conscientiously organizing their learning
process by managing their time more effectively may improve their academic achieve-
ment. Although a small number of studies have used gamification to reward students
for the timeliness of their activities, the effectiveness of the time dimension as a method
to gamify peer review activities needs to be further investigated.
Both automated and manual (instructor-based) game mechanics were used to reward
students. For example, an instructor may send badges to students due to their posts in a
discussion forum that are aligned with theories from the course content. The qualitative
and quantitative results from Ding et al. (2018) and Ding et al. (2017) suggest that the
instructor-based system might increase students’ motivation to produce higher quality
artifacts because students are more influenced by the opinion of their instructors than
their peers. Although such a manual system might increase student motivation and
contentment, it is more time-consuming for the instructor, whereas the (potentially) less
effective automated allocation of rewards is less costly. Further work that investigates
the impact and trade-offs between automated and manual reward allocation would be
beneficial.
Although peer review at pre-tertiary levels has the advantage of lower administrative
costs (Tenório et al. 2016) and the promotion of collaborative learning (Boticki et al.
2015), most studies investigating the gamification of peer review did so at the univer-
sity level. Indeed, peer review could be implemented with students at any level of
education, from elementary (Boticki et al. 2015; Hsu and Wang 2018; Tenório et al.
2016) to university (Sunikova et al. 2017). Some schools may face difficulties relating
to adequate computing facilities, devices, or network connectivity required to make
effective use of online peer review. However, there is a growing trend of BYOD at
many schools (Hopkins et al. 2017), which may help to improve accessibility.
Expanding the use of peer review more broadly introduces age as a factor, and research
has shown that students at different ages are motivated by different things (Handelsman
et al. 2005; Marks 2000; Shernoff et al. 2014; Skinner and Belmont 1993; Wang and
Eccles 2013). This suggests a need for studying the gamification of peer review across
a wide range of classrooms and year levels to determine the transferability of findings.
A final aspect to highlight is the context in which studies have been conducted. Most
of the studies in our review investigated the gamification of peer review tasks in
computer science and software engineering domains. The most obvious explanation
for this bias is that many of the studies involve the development or modification of peer
review tools, requiring some knowledge of software development, and researchers
typically find it easiest to experiment with their own classrooms (or classrooms within
their own subject discipline). Another possible explanation might be that, if an online
system is used for peer review, then the artifacts need to be easily shared and reviewed
in a digital form. This may be more difficult for handwritten material that includes
diagrams or symbols that are difficult to enter using a keyboard (such as in music,
mathematics, and many other STEM disciplines). Interestingly, although 19 studies
were conducted in the area of STEM, only five of them reviewed source code (pure
source code and mixed artifacts), despite review of source code being standard practice
in industry (Li 2006; Sripada et al. 2015; Turner et al. 2008). Further research on the
gamification of tasks involving peer review of source code is therefore warranted.
6 Threats to validity
There are several potential threats to the validity of this study. The first aspect to
address is the search coverage. This threat relates to the integrity of the search
process and the preventive steps to ensure all relevant studies are included. There
were several steps undertaken to ensure thorough coverage of the literature. The
selection of the search string was as comprehensive as possible, including a
systematic investigation of synonyms for major search terms, as described in
section 2.2. The search process was conducted on the most common and largest
repositories of scientific literature, including the ACM Digital Library, IEEE
Xplore, ScienceDirect, Springer Link, Scopus, Web of Science Core Collection,
and ERIC. The search method in each database was chosen carefully; not only
searching based on the title, keywords, and abstracts but also full text and
reference lists; in order to capture as extensive a range of studies as possible.
To include relevant literature that was not indexed by the databases, the reference
list of each of the existing papers was examined via backward snowballing.
The second aspect to address is the study criteria. This threat relates to the possibility
of author disagreement on exclusion and inclusion criteria and author subjectivity. The
initial inclusion-exclusion criteria and selection process was predominantly performed
by the first author; then, all authors discussed each criterion in order to reach a common
understanding. The inclusion-exclusion criteria were applied by the first author, but any
studies that were considered difficult to categorize were discussed by all authors until
consensus was reached. A similar process was undertaken during data extraction, with
any difficult cases discussed until consensus was reached.
The last matter to address is the limited information provided by individual papers.
Many authors did not clearly describe all of the relevant information, such as the details
around how game mechanics were implemented, or the context of the study. It is
therefore possible that the results reported in this study do not precisely reflect the use
of gamification where the corresponding literature provides inadequate detail. We are
only able to report objective information as provided by the authors of published work.
7 Conclusion
In this study, we report the results of a large-scale systematic literature review of the use
of gamification in educational peer review systems. The goal of this work was to
explore how gamification has been applied to peer review systems, including the
specific activities that have been gamified, the game mechanics that have been used,
the context and level of the gamified peer review tasks, and the evidence of the impact
of gamification in these studies.
We identified 39 primary studies after searching the ACM Digital Library,
IEEE Xplore, ScienceDirect, Springer Link, Scopus, Web of Science Core
Collection, and ERIC databases, and applying backward snowballing.
Reviewing artifacts and creating artifacts, the primary components of peer
review, were the two most commonly gamified activities in the literature.
Nevertheless, there are some aspects of peer review tasks that are largely
unexplored. These include reflective activities such as authors critically
reviewing their own work as a result of the feedback they receive, and
assessing the quality and accuracy of such feedback. Similarly, although time-
based metrics for rewarding student actions have been used, their impact has
not been empirically measured as with other reward mechanics. These findings
suggest opportunities for future research.
The most popular game elements reported in the primary studies were points,
badges, and leaderboards, mirroring the results of previous large-scale reviews of
gamification. Most studies involved multiple game mechanics used in combination,
and most were conducted in traditional or distance learning contexts in higher educa-
tion settings. Interestingly, although most studies were conducted in computer science
or software engineering courses, very few involved the review of source code. Future
work on the gamification of peer review activities should look to target the gaps
identified by this systematic review.
Acknowledgments This work has been funded by Indonesian Endowment Fund for Education (LPDP) as
part of the Ph.D. scholarship.
Appedix
Table 6 Data Extraction Form
Data item Value RQ
Study ID The ID of the paper

Article Title Title of the article
Citation Citation information of the article
Year Year of publication
Venue Type of the venue either journal, conference, or book chapter
Gamified actions in peer review What actions have been gamified and what methods have RQ1.1 and
and method to gamify been used to gamify the actions? RQ1.2
Game elements What gamification mechanics have been used in primary RQ 1.3
studies?
Artifacts What kind of artifacts have been produced? RQ 2
Context What was the subject of the course or the discipline? RQ 2
Level What was the level of education (elementary school, RQ 2
intermediate school, high school, or university)?
Sample size Was the sample size small, medium, large, extra-large, or RQ 3
unknown?
Research measure What was the measure to evaluate the impact of gamification? RQ 3
Impact Was the impact of applying gamification in peer review either RQ 3
positive, negative, or neutral?
Table 7 List of primary studies
Study Title Authors

ID
ID1 Gamification-based assessment of group work (Moccozet et al. 2013)

ID2 A gamified peer assessment model for on-line learning environments in a (Tenório et al. 2016)
competitive context
ID3 Edventure: Gamification for collaborative problem design and solving (Wang et al. 2016)
ID4 A proposal of an SNS to support individual practices in a voluntary (Otake et al. 2015)
community
ID5 An exploratory study of student engagement in gamified online (Ding et al. 2018)
discussions
ID6 Studies of student engagement in gamified online discussions (Ding et al. 2017)
ID7 Gamification behind the scenes (Berkling 2016)
ID8 MOOC in Latin America: Implementation and lessons learned (Rizzardini et al. 2014)
ID9 A practical experience on the use of gamification in MOOC courses as a (Morales and
strategy to increase motivation Amado-salvatierra
2016)
ID10 An attrition model for MOOCs: Evaluating the learning strategies of (Rizzardini et al. 2016)
gamification
Table 7 (continued)
Study Title Authors

ID
ID11 Exploring the influence of common game elements on ideation output (Zimmerling et al. 2018)
and motivation
ID12 Implementing a theory-driven gamification model in higher education (Huang and Hew 2018)
flipped courses: Effects on out-of-class activity completion and quality
of artifacts
ID13 To gamify or not to gamify? An experimental field study of the influence (Kyewski and Krämer
of badges on motivation, activity, and performance in an online 2018)
learning course
ID14 Formative student-authored question bank: Perceptions, question quality (Walsh et al. 2018)
and association with summative performance
ID15 Learning from real life and not books: A gamified approach to business (Sevilla-Pavón and
english task design in transatlantic telecollaboration Haba-Osca 2017)
ID16 On the effectiveness of game-like and social approaches in learning: (De-Marcos et al. 2016a)
Comparing educational gaming, gamification & social networking
ID17 Social network analysis of a gamified e-learning course: Small-world (De-Marcos et al. 2016b)
phenomenon and network metrics as predictors of academic perfor-
mance
ID18 Towards the social gamification of e-learning: A practical experiment (De-Marcos et al. 2017)
ID19 The gamification of a MOOC platform (Staubitz et al. 2017)
ID20 Peer assessment augmented with digital badges: A first experience report (Sunikova et al. 2017)
ID21 Grading by experience points: An example from computer ethics (Gehringer and
Peddycord 2013)
ID22 Code Defenders: Crowdsourcing effective tests and subtle mutants with a (Rojas et al. 2017)
mutation testing game
ID23 Use of role-play and gamification in a software project course (Maxim et al. 2017)
ID24 New Challenges for the motivation and learning in engineering education (Borras-Gene et al. 2016)
using gamification in MOOC
ID25 Investigating the impact of a meaningful gamification-based intervention (Agapito and Rodrigo
on novice programmers’ achievement 2018)
ID26 Development of web learning support system using “My Dictionary” in (Usami et al. 2015)
English study
ID27 Designing massive open online learning processes: The sMOOC (Brouns et al. 2017)
pedagogical framework
ID28 From market place to collusion detection: Case studies of gamification in (Winoto and Tang 2015)
education
ID29 Impact of gamification on code review process: An experimental study (Khandelwal et al. 2017)
ID30 Applying game mechanics and student-generated questions to an online (Hsu and Wang 2018)
puzzle-based game learning system to promote algorithmic thinking
skills
ID31 How revealing rankings affects student attitude and performance in a peer (Papadopoulos et al.
review learning environment 2016)
ID32 Betting system for formative code review in educational competitions (Palomo-Duarte et al.
2014)
ID33 Structuring flipped classes with lightweight teams and gamification (Latulipe et al. 2015)
ID34 The effect of virtual achievements on student engagement (Denny 2013)
Table 7 (continued)
Study Title Authors

ID
ID35 Engaging with massive online courses (Anderson et al. 2014)

ID36 Engaging Asian students through game mechanics: Findings from two (Hew et al. 2016)
experiment studies.
ID37 An analysis of the use of badges in an educational experiment (Ruipérez-Valiente et al.
2016)
ID38 Usage of a mobile social learning platform with virtual badges in a (Boticki et al. 2015)
primary school
ID39 Engaging computer science students through gamification in an online (Li et al. 2013)
social network based collaborative learning environment
References
Adaji, I., & Vassileva, J. (2017). A gamified system for influencing healthy E-commerce shopping habits.
WPPG: Fifty Shades of Personalization - Workshop on Personalization in Serious and Persuasive Games
and Gameful Interaction, 398–401. https://doi.org/10.1145/3099023.3099110.
Agapito, J. L., & Rodrigo, M. M. T. (2018). Investigating the impact of a meaningful gamification-based
intervention on novice programmers’ achievement. International conference on artificial intelligence in
education (AIED), 3–16. https://doi.org/10.1007/978-3-319-93843-1_1.
Alhammad, M. M., & Moreno, A. M. (2018). Gamification in software engineering education: A systematic
mapping. Journal of Systems and Software, 141, 131–150. https://doi.org/10.1016/j.jss.2018.03.065.
Anderson, A., Huttenlocher, D., Kleinberg, J., & Leskovec, J. (2014). Engaging with massive online courses.
International World Wide Web Conference Committee (IW3C2), 687–698. https://doi.org/10.1111
/jcal.12165.
Antin, J., & Churchill, E. F. (2011). Badges in social media: A social psychological perspective. CHI 2011
Gamification Workshop Proceedings, 1–4.
Anvari, F., Richards, D., Hitchens, M., & Tran, H. M. T. (2019). Teaching user centered conceptual design
using cross-cultural personas and peer reviews for a large cohort of students. Proceedings of the 41st
international conference on software engineering: Software engineering education and training, 62–73.
https://doi.org/10.1109/ICSE-SEET.2019.00015.
Berkling, K. (2016). Gamification behind the scenes. Computer supported education: 7th international
conference, CSEDU 2015 Lisbon, Portugal, May 23–25, 274–292. https://doi.org/10.1007/978-3-319-
29585-5_16.
Borras-Gene, O., Martiñez-nunez, M., & Fidalgo-Blanco, Á. (2016). New challenges for the motivation and
learning in engineering education using gamification in MOOC. International Journal of Engineering
Education, 32(1), 501–512 https://repositorio.grial.eu/bitstream/grial/560/1/19_ijee3155ns.pdf.
Boticki, I., Baksa, J., Seow, P., & Looi, C. K. (2015). Usage of a mobile social learning platform with virtual
badges in a primary school. Computers & Education, 86, 120–136. https://doi.org/10.1016/j.
compedu.2015.02.015.
Britton, B. K., & Tesser, A. (1991). Effects of time-management practices on college grades. Journal of
Educational Psychology, 83(3), 405–410.
Brouns, F., Morgado, L., & Fano, S. (2017). Designing massive open online learning processes: The sMOOC
pedagogical framework. In Open Education: from OERs to MOOCs (pp. 315–336). Springer-Verlag
Berlin Heidelberg. https://doi.org/10.1007/978-3-662-52925-6.
Bunchball, I. (2010). Gamification 101: An introduction to the use of game dynamics to influence behavior.
White Paper, 9.
Conaway, R., & Garay, M. C. (2014). Gamification and service marketing. SpringerPlus, 3(1), 653.
De-Marcos, L., García-López, E., & García-Cabot, A. (2016a). On the effectiveness of game-like and social
approaches in learning: Comparing educational gaming, gamification & social networking. Computers &
Education, 95, 99–113. https://doi.org/10.1016/j.compedu.2015.12.008.
De-Marcos, L., García-López, E., García-Cabot, A., Medina-Merodio, J.-A., Domínguez, A., Martínez-
Herráiz, J.-J., & Diez-Folledo, T. (2016b). Social network analysis of a gamified e-learning course:
Small-world phenomenon and network metrics as predictors of academic performance. Computers in
Human Behavior, 60, 312–321. https://doi.org/10.1016/j.chb.2016.02.052.
De-Marcos, L., García-Cabot, A., & García-López, E. (2017). Towards the social gamification of e-learning:
A practical experiment. International Journal of Engineering Education, 33(1), 66–73.
Denny, P. (2013). The effect of virtual achievements on student engagement. Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems - CHI ‘13, 763. https://doi.org/10.1145
/2470654.2470763.
Denny, P., McDonald, F., Empson, R., Kelly, P., & Petersen, A. (2018). Empirical support for a causal
relationship between gamification and learning outcomes. Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems, 311:1–311:13. https://doi.org/10.1145/3173574.3173885.
Deterding, S., Sicart, M., Nacke, L., O’Hara, K., & Dixon, D. (2011). Gamification. Using game-design
elements in non-gaming contexts. CHI ‘11 Extended Abstracts on Human Factors in Computing Systems,
2425–2428. https://doi.org/10.1145/1979742.1979575.
Dicheva, D., Dichev, C., Agre, G., & Angelova, G. (2015). Gamification in education: A systematic mapping
study. Journal of Educational Technology & Society, 18(3), 75–88 http://www.jstor.
org/stable/jeductechsoci.18.3.75.
Dickson, H., Harvey, J., & Blackwood, N. (2019). Feedback, feedforward: Evaluating the effectiveness of an
oral peer review exercise amongst postgraduate students. Assessment & Evaluation in Higher Education,
44(5), 692–704. https://doi.org/10.1080/02602938.2018.1528341.
Ding, L., Kim, C., & Orey, M. (2017). Studies of student engagement in gamified online discussions.
Computers & Education, 115, 126–142. https://doi.org/10.1016/j.compedu.2017.06.016.
Ding, L., Er, E., & Orey, M. (2018). An exploratory study of student engagement in gamified online
discussions. Computers & Education, 120(January), 213–226. https://doi.org/10.1016/j.
compedu.2018.02.007.
Dochy, F. J. R. C., & McDowell, L. (1997). Assessment as a tool for learning. Studies in Educational
Evaluation, 23(4), 279–298. https://doi.org/10.1016/S0191-491X(97)86211-6.
Falchikov, N. (1995). Peer feedback marking: Developing peer assessment. Innovations in Education and
Training International, 32(2), 175–187. https://doi.org/10.1080/1355800950320212.
Fu, Q.-K., Lin, C.-J., & Hwang, G.-J. (2019). Research trends and applications of technology-supported peer
assessment: A review of selected journal publications from 2007 to 2016. Journal of Computers in
Education, 6(2), 191–213. https://doi.org/10.1007/s40692-019-00131-x.
Gehringer, E., & Peddycord, B. (2013). Grading by experience points: An example from computer ethics.
Proceedings - Frontiers in Education Conference, FIE, 1545–1550. https://doi.org/10.1109
/FIE.2013.6685097.
Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work? - a literature review of empirical studies
on gamification. Proceedings of the annual Hawaii international conference on system sciences, 3025–
3034. https://doi.org/10.1109/HICSS.2014.377.
Hamer, J., Ma, K. T. K., & Kwong, H. H. F. (2005). A method of automatic grade calibration in peer
assessment. Proceedings of the 7th Australasian Conference on Computing Education-Volume 42, 67–
72. http://dl.acm.org.ezproxy.auckland.ac.nz/citation.cfm?id=1082424.1082433
Hamer, J., Kell, C., & Spence, F. (2007). Peer assessment using Aropä. Proceedings of the Ninth Australasian
Conference on Computing Education-Volume 66, 43–54. http://dl.acm.org.ezproxy.auckland.ac.
nz/citation.cfm?id=1273672.1273678
Hamnett, H. J., & McKie, A. E. (2019). Developing a procedure for learning and assessing peer review in a
forensic science programme. Assessment & Evaluation in Higher Education, 44(5), 787–798. https://doi.
org/10.1080/02602938.2018.1536924.
Handelsman, M. M., Briggs, W. L., Sullivan, N., & Towler, A. (2005). A measure of college student course
engagement. The Journal of Educational Research, 98(3), 184–192.
Hew, K. F., Huang, B., Chu, K. W. S., & Chiu, D. K. W. (2016). Engaging Asian students through game
mechanics: Findings from two experiment studies. Computers & Education, 92–93, 221–236. https://doi.
org/10.1016/j.compedu.2015.10.010.
Heylighen, F., Kostov, I., & Kiemen, M. (2013). Mobilization systems: Technologies for motivating and
coordinating human action. The New Development Paradigm: Education, Knowledge Economy and
Digital Futures. Routledge. Retrieved from http://Pcp.Vub.Ac.Be/Papers/MobilizationSystems.Pdf/.
Hopkins, N., Tate, M., Sylvester, A., & Johnstone, D. (2017). Motivations for 21st century school children to
bring their own device to school. Information Systems Frontiers, 19(5), 1191–1203. https://doi.
org/10.1007/s10796-016-9644-z.
Hsu, C. C., & Wang, T. I. (2018). Applying game mechanics and student-generated questions to an online
puzzle-based game learning system to promote algorithmic thinking skills. Computers & Education, 121,
73–88. https://doi.org/10.1016/j.compedu.2018.02.002.
Huang, B., & Hew, K. F. (2018). Implementing a theory-driven gamification model in higher education
flipped courses: Effects on out-of-class activity completion and quality of artifacts. Computers in
Education, 125(June), 254–272. https://doi.org/10.1016/j.compedu.2018.06.018.
Johnson, D., Deterding, S., Kuhn, K.-A., Staneva, A., Stoyanov, S., & Hides, L. (2016). Gamification for
health and wellbeing: A systematic review of the literature. Internet Interventions, 6, 89–106. https://doi.
org/10.1016/j.invent.2016.10.002.
Khandelwal, S., Sripada, S. K., & Reddy, Y. R. (2017). Impact of gamification on code review process.
Proceedings of the 10th Innovations in Software Engineering Conference on - ISEC ‘17, 122–126.
https://doi.org/10.1145/3021460.3021474.
Kitchenham, B. (2004). Procedures for performing systematic reviews. Keele, UK, Keele University, 33, 1–26
http://www.it.hiof.no/~haraldh/misc/2016-08-22-smat/Kitchenham-Systematic-Review-2004.pdf.
Koivisto, J., & Hamari, J. (2019). The rise of motivational information systems: A review of gamification
research. International Journal of Information Management, 45, 191–210. https://doi.org/10.1016/j.
ijinfomgt.2018.10.013.
Kumar, J. (2013). Gamification at work: Designing engaging business software. International conference of
design, user experience, and usability, 528–537.
Kyewski, E., & Krämer, N. C. (2018). To gamify or not to gamify? An experimental field study of the
influence of badges on motivation, activity, and performance in an online learning course. Computers &
Education, 118, 25–37. https://doi.org/10.1016/j.compedu.2017.11.006.
Landers, R. N., Collmus, A. B., & Williams, H. (2019). The greatest battle is within ourselves: An experiment
on the effects of competition alone on task performance. International Journal of Human-Computer
Studies, 127, 51–61. https://doi.org/10.1016/j.ijhcs.2018.09.011.
Latulipe, C., Long, N. B., & Seminario, C. E. (2015). Structuring flipped classes with lightweight teams and
gamification. Proceedings of the 46th ACM Technical Symposium on Computer Science Education -
SIGCSE ‘15, 392–397. https://doi.org/10.1145/2676723.2677240.
Li, X. (2006). Using peer review to assess coding standards-a case study. Proceedings. Frontiers in Education.
36th Annual Conference, 9–14. https://doi.org/10.1109/FIE.2006.322572.
Li, C., Dong, Z., Untch, R. H., & Chasteen, M. (2013). Engaging computer science students through
gamification in an online social network based collaborative learning environment. International
Journal of Information and Education Technology, 3(1), 72–77. https://doi.org/10.7763/IJIET.2013.
V3.237.
Luxton-Reilly, A. (2009). A systematic review of tools that support peer assessment. Computer Science
Education, 19(4), 209–232. https://doi.org/10.1080/08993400903384844.
Macan, T. H., Shahani, C., Dipboye, R. L., & Phillips, A. P. (1990). College students’ time management:
Correlations with academic performance and stress. Journal of Educational Psychology, 82(4), 760–768.
MacCann, C., Fogarty, G. J., & Roberts, R. D. (2012). Strategies for success in education: Time management
is more important for part-time than full-time community college students. Learning and Individual
Differences, 22(5), 618–623.
Marks, H. M. (2000). Student engagement in instructional activity: Patterns in the elementary, middle, and
high school years. American Educational Research Journal, 37(1), 153–184.
Maxim, B. R., Brunvand, S., & Decker, A. (2017). Use of role-play and gamification in a software project
course. 2017 IEEE Frontiers in Education Conference (FIE), 1–5. https://doi.org/10.1109
/FIE.2017.8190501.
Miller, K., Lukoff, B., King, G., & Mazur, E. (2018). Use of a social annotation platform for pre-class reading
assignments in a flipped introductory physics class. Frontiers in Education, 3, 8. https://doi.org/10.3389
/feduc.2018.00008.
Moccozet, L., Tardy, C., Opprecht, W., & Leonard, M. (2013). Gamification-based assessment of group work.
2013 international conference on interactive collaborative learning, ICL 2013, September, 171–179.
https://doi.org/10.1109/ICL.2013.6644565.
Morales, M., & Amado-salvatierra, H. R. (2016). A practical experience on the use of gamification in MOOC
courses as a strategy to increase motivation. International Workshop on Learning Technology for
Education in Cloud, LTEC 2016 Springer, Cham., CCIS 620, 139–149. https://doi.org/10.1007/978-3-
319-42147-6_12.
Morschheuser, B., Hamari, J., & Koivisto, J. (2016). Gamification in crowdsourcing: A review. 2016 49th
Hawaii international conference on system sciences (HICSS), 4375–4384.
Nacke, L. E., & Deterding, S. (2017). The maturing of gamification research. Computers in Human Behavior,
71, 450–454. https://doi.org/10.1016/j.chb.2016.11.062.
National Survey of Student Engagement. (2018). NSSE 2018 Codebook U.S. Version. http://nsse.indiana.
edu/2018_Institutional_Report/data_codebooks/NSSE 2018 Codebook.pdf.
Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education : A peer
review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122.
Ortiz-Rojas, M., Chiluiza, K., & Valcke, M. (2019). Gamification through leaderboards: An empirical study in
engineering education. Computer Applications in Engineering Education, 27(4), 777–788. https://doi.
org/10.1002/cae.12116.
Otake, K., Komuro, M., & Shinozawa, Y. (2015). A proposal of an SNS to support individual practices in a
voluntary community. International conference on human-computer interaction. Springer, Cham., CCIS
529, 107–112. https://doi.org/10.1007/978-3-319-21383-5_18.
Palomo-Duarte, M., Manuel Dodero, J., & García-Domínguez, A. (2014). Betting system for formative code
review in educational competitions. Expert Systems with Applications, 41(5), 2222–2230. https://doi.
org/10.1016/j.eswa.2013.09.020.
Papadopoulos, P. M., Lagkas, T., & Demetriadis, S. N. (2016). How revealing rankings affects student attitude
and performance in a peer review learning environment. International conference on computer supported
education (CSEDU 2015), CCIS 583, 225–240. https://doi.org/10.1007/978-3-319-29585-5_13.
Pond, K., Ul-Haq, R., & Wade, W. (1995). Peer review: A precursor to peer assessment. Innovations in
Education and Training International, 32(4), 314–323. https://doi.org/10.1080/1355800950320403.
Rizzardini, R. H., Gütl, C., Chang, V., & Morales, M. (2014). MOOC in Latin America: Implementation and
lessons learned. In L. Uden, Y.-H. Tao, H.-C. Yang, & I.-H. Ting (Eds.), The 2nd International Workshop
on Learning Technology for Education in Cloud (pp. 147–158). Springer Netherlands. https://doi.
org/10.1007/978-94-007-7308-0_16.
Rizzardini, R. H., Chan, M. M., & Guetl, C. (2016). An attrition model for MOOCs: Evaluating the learning
strategies of gamification. In S. Caballé & R. Clarisó (Eds.), Formative assessment, Learning Data
Analytics and Gamification. Academic Press. https://doi.org/10.1016/B978-0-12-803637-2.00014-2.
Rojas, J. M., White, T. D., Clegg, B. S., & Fraser, G. (2017). Code defenders: Crowdsourcing effective tests
and subtle mutants with a mutation testing game. Proceedings - 2017 IEEE/ACM 39th international
conference on software engineering, ICSE 2017, 677–688. https://doi.org/10.1109/ICSE.2017.68.
Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., & Kloos, C. D. (2016). An analysis of the use of badges in an
educational experiment. Proceedings - Frontiers in Education Conference, FIE, 1–8. https://doi.
org/10.1109/FIE.2016.7757424.
Seifert, T., & Feliks, O. (2019). Online self-assessment and peer-assessment as a tool to enhance student-
teachers’ assessment skills. Assessment & Evaluation in Higher Education, 44(2), 169–185. https://doi.
org/10.1080/02602938.2018.1487023.
Sevilla-Pavón, A., & Haba-Osca, J. (2017). Learning from real life and not books: A gamified approach to
business english task design in transatlantic telecollaboration. Iberica, 33, 235–260 https://www.redalyc.
org/pdf/2870/287050576010.pdf.
Shernoff, D. J., Csikszentmihalyi, M., Schneider, B., & Shernoff, E. S. (2014). Student engagement in high
school classrooms from the perspective of flow theory. In Applications of flow in human development and
education (pp. 475–494). Springer.
Sitthiworachart, J., & Joy, M. (2004). Effective peer assessment for learning computer programming.
Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer
Science Education, 122–126. https://doi.org/10.1145/1007996.1008030.
Skinner, E. A., & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of teacher behavior
and student engagement across the school year. Journal of Educational Psychology, 85(4), 571–581.
Søndergaard, H., & Mulder, R. A. (2012). Collaborative learning through formative peer review: Pedagogy,
programs and potential. Computer Science Education, 22(4), 343–367.
Sripada, S., Reddy, Y. R., & Sureka, A. (2015). In support of peer code review and inspection in an
undergraduate software engineering course. 2015 IEEE 28th conference on software engineering educa-
tion and training, 3–6.
Staubitz, T., Willems, C., Hagedorn, C., & Meinel, C. (2017). The gamification of a MOOC platform. IEEE
Global Engineering Education Conference, EDUCON, April, 883–892. https://doi.org/10.1109
/EDUCON.2017.7942952.
Subhash, S., & Cudney, E. A. (2018). Gamified learning in higher education : A systematic review of the
literature. Computers in Human Behavior, 87, 192–206. https://doi.org/10.1016/j.chb.2018.05.028.
Sunikova, D., Kubincova, Z., Simionescu, S., & Popescu, E. (2017). Peer assessment augmented with digital
badges: A first experience report. 2017 16th international conference on information technology based
higher education and training, ITHET 2017, 1–5. https://doi.org/10.1109/ITHET.2017.8067799.
Tenório, T., Bittencourt, I. I., Isotani, S., Pedro, A., & Ospina, P. (2016). A gamified peer assessment model
for on-line learning environments in a competitive context. Computers in Human Behavior, 64, 247–263.
https://doi.org/10.1016/j.chb.2016.06.049.
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational
Research, 68(3), 249–276.
Turner, S. A., Quintana-Castillo, R., Pérez-Quiñones, M. A., & Edwards, S. H. (2008). Misunderstandings
about object-oriented design: Experiences using code reviews. ACM SIGCSE Bulletin, 40(1), 97–101.
Turner, S., Pérez-Quiñones, M. A., Edwards, S., & Chase, J. (2011). Student attitudes and motivation for peer
review in CS2. Proceedings of the 42nd ACM Technical Symposium on Computer Science Education,
347–352.
Turnitin. (2019). About PeerMark. https://help.turnitin.com/feedback-studio/turnitin-
website/student/peermark/about-peermark.htm?Highlight=peermark
Usami, H., Eguchi, H., Ozaki, M., & Adachi, Y. (2015). Development of web learning support system using
“my dictionary” in English study. Procedia Computer Science, 60(1), 944–951. https://doi.org/10.1016/j.
procs.2015.08.258.
Walsh, J. L., Harris, B. H. L., Denny, P., & Smith, P. (2018). Formative student-authored question bank:
Perceptions, question quality and association with summative performance. Postgraduate Medical
Journal, 94(1108), 97–103. https://doi.org/10.1136/postgradmedj-2017-135018.
Wang, M.-T., & Eccles, J. S. (2013). School context, achievement motivation, and academic engagement: A
longitudinal study of school engagement using a multidimensional perspective. Learning and Instruction,
28, 12–23.
Wang, C. H., Hsu, Y. C., Yeh, P. C., Lin, C. Y., & Lai, I. W. (2016). Edventure: Gamification for
collaborative problem design and solving. 2016 15th international conference on information technology
based higher education and training, ITHET 2016, 8–12. https://doi.org/10.1109/ITHET.2016.7760719.
Winoto, P., & Tang, T. Y. (2015). From market place to collusion detection: Case studies of gamification in
education. In T. Reiners & L. C. Wood (Eds.), Gamification in Education and Business (pp. 1–710).
Springer International Publishing. https://doi.org/10.1007/978-3-319-10208-5_14.
Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software
engineering. Proceedings of the 18th international conference on evaluation and assessment in software
engineering, 38.
Xu, F., Buhalis, D., & Weber, J. (2017). Serious games and the gamification of tourism. Tourism
Management, 60, 244–256. https://doi.org/10.1016/j.tourman.2016.11.020.
Zichermann, G., & Cunningham, C. (2011). Gamification by design: Implementing game mechanics in web
and mobile apps. “ O’Reilly Media, Inc.”
Zimmerling, E., Höllig, C. E., Sandner, P. G., & Welpe, I. M. (2018). Exploring the influence of common
game elements on ideation output and motivation. Journal of Business Research, 1–11. https://doi.
org/10.1016/j.jbusres.2018.02.030.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and
institutional affiliations.

Indriasari2020 Article GamificationOfStudentPeerRevie

Uploaded by

Copyright:

Available Formats

Indriasari2020 Article GamificationOfStudentPeerRevie

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Indriasari2020 Article GamificationOfStudentPeerRevie

Uploaded by

Copyright:

Available Formats

Education and Information Technologies (2020) 25:5205–5234

Gamification of student peer review in education:

Keywords Gamification . Education . Peer review . Peer assessment . Systematic literature

* Theresia Devi Indriasari

Within education, peer review is a collaborative activity in which students consider,

In conducting this systematic literature review, we follow the guidelines proposed by

2.1 Research questions

2.2 Search process

Table 1 Research questions

RQ1 How have peer review activities been gamified?

Fig. 1 The seven-step search process

“gameification”, “gameified”, “gameify”, “gameifies”, and “gameifying”. Synonyms

1. ACM Digital Library: search in the full text

help” OR “code review” OR crowdsource) AND (education OR learning OR student

2.3 Selection process

Table 2 Inclusion and exclusion criteria

Inclusion Criteria Exclusion Criteria

2.4 Data extraction and analysis process

Fig. 2 Selection process

Student Engagement (NSSE) 2018 Codebook (National Survey of Student

a. Arc A (Author develops, creates, or improves artifact)

2. Game elements. This field records the gamification mechanics used.

A peer review process involves several activities or actions, such as creating an

Fig. 3 Gamified Peer Review Model

3.1 Gamified peer review model

3.2 Example scenarios

4.1 General results

4.2 RQ 1 – The implementation of gamified peer review activities

RQ 1 addresses the implementation of gamification in peer review activities. As

4.2.1 RQ1.1 – Gamified peer review activities

Fig. 4 Distribution of primary studies by year

Fig. 5 Gamified peer review actions

4.2.2 RQ1.2 – Method or technique to gamify an activity

1. Quality: methods that measure the grade or excellence of artifacts produced by

Table 3 Method or technique to gamify an action and game element received

Arc Method N Game Elements

Points Leaderboards Badges Progress Level V i r t u a l Prize

4.2.3 RQ1.3 – Game elements

Table 4 Game Elements

4.3 RQ2 – The context and level

Fig. 6 Context and type of artifacts at the university level

4.4 RQ3 – Evidence and impact

1. Engagement: to measure the impact of gamification towards students’ enthusiasm

1. Small: the number of participants was less than 40 (n = 10)

Outcomes are also divided into three types:

Research measures Positive Negative Neutral

Interest in the gamification of peer review activities appears to be growing, increasing

studies in a wide range of contents. More innovative implementations of gamification

Table 6 Data Extraction Form

Data item Value RQ

Study ID The ID of the paper

Table 7 List of primary studies

Study Title Authors

ID1 Gamification-based assessment of group work (Moccozet et al. 2013)

Study Title Authors

Study Title Authors

ID35 Engaging with massive online courses (Anderson et al. 2014)

You might also like