101 | ELT-Echo, Volume 2, Number 2, November 2017

ISSN: 2579-8170 e-ISSN: 2549-5089

DEVELOPING SCORING RUBRIC: DO THE STUDENTS NEED IT?

Aninda Nidhommil Hima

Universitas Muhammadiyah Malang
[email protected]

Teguh Hadi Saputro

Universitas Muhammadiyah Malang
[email protected]

Abstract: For many reasons, the use of scoring rubrics has acquired
more attention. However, the number of studies investigating the
perception and the writing performance as well as the relationship
between those two variables in the use of rubric is limited. Therefore, by
involving 27 students in Writing -class of an English Language
Education Department at a particular university in Indonesia, the present
study aimed to explore the students’ perceptions through surveys and
their writing performance in the three different situations in which they
had no scoring rubric, generate their own scoring rubric and had teacher-
made scoring rubric prior to the writing tasks. Also, the study was
intended to explore how the students perceived the three situations
related to their writing performance. After analyzing the data, it was
found that the most helpful situation for the students was the second
situation. It also helped the students to achieve the higher score compared
to the other two situations. However, there was a contradictory finding in
which the students’ perception and their performance were negatively
correlated.

Keywords: developing scoring rubrics, writing performance, students’

perception

BACKGROUND
A scoring rubric is generally defined as an instrument that a teacher needs
to assess students’ performance. It is an assessment guide that describes evaluation
criteria (or grading standards) based on the expected outcomes and performances of
the students. Naturally, rubrics are utilized to advance the reliable use of learning
outlook, learning goals, or learning models in the classroom, or to quantify their
accomplishment against a steady arrangement of criteria (Hidden Curriculum, 2014).
Hima,Saputro Develpoping Scoring Rubric| 102
DOI: 10.24235/eltecho.v%vi%i.2170

In addition, rubrics are commonly used in scoring or grading written assignments or

oral presentations and also provide a source for doing self-assessment, reflection,
and peer review. As confirmed by Office of Graduate Studies Teaching at UNL
(2017) scoring rubrics are especially appropriate for evaluating complex tasks or
assignments such as: written work (e.g., assignments, essay tests, papers, portfolios);
presentations (e.g., debates, role plays); group work; or other types of work products
or performances (e.g., artistic works, portfolios).
Scoring rubrics are usually presented in a table which contains some aspects
to be rated, quality descriptions for each aspect and a rating scale for those
descriptions. Each rubric comprises of an arrangement of scoring criteria and point
esteems related with these criteria. In many rubrics, the criteria are assembled into
classes so the teacher and the students can separate among the classifications by the
level of execution (Hricko & Howell, 2006). A rubric isolates the assigned work
into categories and gives clear portrayals of the work related to every aspect, at
different levels of performance (Karkehabadi, 2013).
Rubrics can be classified as either holistic or analytical. Holistic rubrics can
be defined as scoring rubrics that provide a single score based on an overall
impression of a student’s performance on a task. A holistic scoring rubric comprises
of a single scale with all criteria to be incorporated into the assessment being viewed
together (e.g., content, organization, language use, and mechanics). By using this
rubric, the rater doles out a single score (as a rule on a 1 to 4 or 1 to 6 point scale) in
a view of a general judgment of the students’ work (Jackson, 2017). It benefits the
raters as it provides an overview of students’ achievement (Lane, 1996). From a
practicality viewpoint, it is faster (Alaei, Ahmadi, & Zadeh, 2014) and absolutely
saves time by decreasing the number of judgment the raters make (Jackson, 2017).
In contrast, an analytical rubric measures various categories of the work
independently in which its process of assessment tends to be more closed, restricted
and detailed. Consequently, it possibly occurs a phenomenon which is one
dimension of the work could be excellent, but on one or more other dimensions of
the work might be poor to average. As noticed by (Çetin, 2011) and Lane (1996),
analytical rubrics benefits both the raters and the students by giving more detailed
feedback and access to self‐assess in evaluating their strengths and weaknesses of
their performance. However, it is very time-consuming to score compared to
holistic. Additionally, based on the research conducted by Rezaei & Lovorn (2010)
raters using an analytic rubric for writing assessment tends to concern on the
mechanical characteristics of students’ writing rather than the content itself.
The use of rubrics as the core instrument in assessment has been very
popular today as they promote objectivity. However, the existence of the rubric itself
in evaluating writing proficiency has been a challenge and a controversy among
103 | ELT-Echo, Volume 2, Number 2, November 2017
ISSN: 2579-8170 e-ISSN: 2549-5089

scholars (East, 2009; Rakedzon & Baram-Tsabari, 2017; & Sundeen, 2014). A
number of previous studies suggest that scoring rubrics are essential since they bring
a number of benefits for both students and teachers. Ene & Kosobucki (2016) state
that criteria shared with the students in advance becomes integrated into instruction
as a formative teaching tool. They can assist the evaluator to achieve intra-rater and
inter-rater reliability in assessing students' writing as well. Further, rubrics are
important since they make it clear for students about the qualities their work should
be. It is often conveyed in terms of students' understanding of the learning target and
criteria for success. As confirmed by Allen & Tanner (2006), rubrics are not only
intended to formulate guidelines on what to achieve in a certain performance but
additionally, they can be used to make these guidelines clear and specific to the
students. For this reason, rubrics help teachers teach and manage instruction and
assessment, and they support students to learn as well.
On the contrary, some argue that scoring rubrics are not crucial for students’
success in terms of their writing. As stated by Matsuda & Jeffery (2012), rubrics
cannot capture or encourage the development of voice. The rubrics can also restrict
the student's mind power since they will feel that they need to complete the
assignment strictly to the rubric instead of taking the initiative to explore their
learning. Additionally, developing a rubric can be time intensive because there is a
need to ensure that the students are aware of the expectations and to guarantee the
intra-rater and inter-rater reliability(Labs, 2017).
While many studies have been completed to show the benefits and weaknesses
of rubrics use in the instructional context especially in writing classes, very few
investigated the effect of involving the students in developing scoring rubric prior to
writing tasks. In the era in which student-centered approach is highly promoted, the
students are not only expected to evaluate their performance, but they also need to
decide the criteria as well as the tools to determine the score of their work. Andrade
& Du (2005)investigate the students’ perspectives regarding the regular use of
rubrics, including co-creating rubrics in class, formal rubric-referenced self-
assessment, and teacher feedback. The students stated that using rubrics helped them
focus on their efforts, produce better quality work, get a better grade, and feel less
anxious about an assignment. Additionally, Neill (2007) and Yoshina & Harada
(2007) supported that students who are involved in the process of creating a rubric
have a better understanding of what must be done to reach expectations. Similarly,
Becker (2016)examines the effectiveness of developing and/or applying a rubric on
the writing performance of adult English as second language learners. Using a
pretest-posttest, control group design, his study resulted in a way that the holistic
scores on the post-test summary writing task were significantly higher for those who
participated in the development and/or application of the scoring rubric.
Hima,Saputro Develpoping Scoring Rubric| 104
DOI: 10.24235/eltecho.v%vi%i.2170

Regarding this shift of attention and the emphasis placed on the writing
assessment, the research in the area of involving students in developing scoring
rubrics compared to not using as well as using scoring rubrics in the instructional
context is still relevant to be conducted. Given the findings mentioned previously,
further research on students’ perception and students’ performance on the presence
of scoring rubric in writing class is warranted. Therefore, this study incorporated
three situations prior to the students’ writing tasks: 1. having no rubric, 2. generating
scoring rubric, 3. doing peer assessment by using a teacher-made scoring rubric.
Furthermore, the study attempted to reveal the students’ perception as well as their
writing performance in those settings. Finally, the correlation between the students’
perceptions and performances would be explored.
Accordingly, the present study sought to answer the following research
questions:
1. How do the students perceive the situation in which they have no rubric?
2. How do the students perceive the situation in which they are involved in
developing a scoring rubric?
3. How do the students perceive the condition in which they are involved in using
a rubric to do peer-assessment?
4. What are the effects of: (a) having no rubric, (b) generating scoring rubric, and
(c) participating in peer-assessing by using a rubric on the students’ writing
scores?
5. How do the students’ perceptions of the three situations correlate with their
writing scores?

METHOD

The 27 students (5 males and 22 females) of Writing II in Class A

participated in this current study. They are the 4thsemester students of English
Language Education Department at Universitas Muhammadiyah Malang who joined
Writing II class twice a week (on Wednesday and Friday
Data collection took 11 days in total. The study employed surveys and a task
that assigned the students to write a paragraph on three different situations that were
(a) having no rubric, (b) generating a scoring rubric, and (c) participating in peer-
assessing by using a rubric from the lecturer. The survey was done by distributing a
questionnaire with five-5-point Likert scale-type questions. The questionnaires were
to reveal the students’ perceptions associated with three different situation
treatments prior to the writing task. The questions in the questionnaire covered the
aspects of writing quality, expectations on the writing ability, writing process,
feedback, and motivation.
105 | ELT-Echo, Volume 2, Number 2, November 2017
ISSN: 2579-8170 e-ISSN: 2549-5089

In addition, to get the data of the students’ writing score, the students were
assigned to write a paragraph on each situation created by the writing lecturer. The
paragraph was assessed by using scoring rubric made by the lecturer.
After all the data from the questionnaire and students’ score had been
collected, they were inputted to the spreadsheet. To answer the first, second and
third research questions which were asking the students’ perceptions of each
situation, the study combined the five-5-point Likert-type scale questionnaire scores
for a maximum of 25 points and displayed the data using bar charts. The survey data
were also used to rank the three situations in order of preference. Later, the data
were analyzed statistically by using Statistical Package for the Social Sciences
Version 21.0 (SPSS). To determine the differences and to test all pairwise
comparisons (situation 1&2; 1&3; 2&3), the study used a within-subjects ANOVA
with Bonferroni’s post hoc.
The study also used an experimental design for research question four as it
was deemed necessary whether those three situations affected the students’writing
performance. The statistical analysis by using ANOVA with Bonferroni’s post hoc
was used to examine the differences among the students’ score on those different
situations. Meanwhile, to obtain the answer for the research question five in terms of
whether or not there was a relationship between students’ perceptions and their
writing performance, the study used Spearman’s Rho correlation coefficient. Later,
it determined a positive, negative, or zero correlation between those variables.

FINDINGS

a. Research Questions 1, 2 and 3

To answer the research questions one to three, the descriptive statistics was
presented as it provides the measurement of the students’ perceptions in the
situations of having no rubric, being participated in designing a rubric and using a
rubric from the lecturer to peer-assessing prior to their writing tasks. The students’
score was calculated with maximum points of 25 and the result was displayed using
bar charts (figure 1,2 and3). To interpret the data, the score range of 1-5 is defined as
strongly disagreement, 6-10 as disagreement, 11-15 as a neutral response, 16-20 as
agreement and 21-25 as strong agreement.
Figure 1 illustratesthat17 students agreed and 5 students strongly agreed that
although they did not know how their work would be evaluated, they could present
how much they knew about writing a good paragraph. Only 5 outof 27 students
positioned themselves in a neutral area. Furthermore, there were no students had a
disagreement or strong disagreement with this condition.
Hima,Saputro Develpoping Scoring Rubric| 106
DOI: 10.24235/eltecho.v%vi%i.2170

20

15 17

10

5
5 5
0 0
0
1 to 5 6 to 10 11 to 15 16 to 20 21 to 25

Figure 1.The Students’ Perception of Having No Rubric.

Figure 2 indicates that after being participated in designing rubric, the

number of students who agreed that they can compose a better writing was
increasing to be 22 students and 4 students gave strongly agree responses.
Additionally, only one participant gave a neutral response and there was no
respondent who felt disagreed and strongly disagreed.
Number of Students

22

0 0 1 4

1 to 5 6 to 10 11 to 15 16 to 20 21 to 25
Total Score

Figure 2. The Students’ Perception of Generating Rubric

In Figure3, it can be found the students’ response toward the third situation
in which the students were peer-assessed by using scoring rubric from the teacher. It
shows that there was still one participant in the neutral position. 18 out of 27
students agreed and 8 students strongly agreed that this setting helped them to
perform better. Overall, the response of disagreement and strong disagreement could
not be found in this setting.
107 | ELT-Echo, Volume 2, Number 2, November 2017
ISSN: 2579-8170 e-ISSN: 2549-5089

Number of Students
18

8
0 0 1

1 to 5 6 to 10 11 to 15 16 to 20 21 to 25
Total Score

Figure 3. The Students’ Perception of Using Rubric

To ensure the students’ preference on those settings in rubrics uses, the

statistical analysis using within-subject ANOVA was used. Then, the means of the
students’ perceptions in relation to having no rubric, generating rubric and using the
teacher-made rubric to do peer-assessment were compared. Using Bonferonni’s post
hoc, it was found that the students’ perceptions of being participated in designing
rubric differed significantly from having no rubric in prior to the writing task
(p=.027). However, when comparing the students’ perceptions of having no rubric to
using the teacher-made rubric, there was no statistically significant (p=.330).
Similarly, the students’ perceptions of the condition of having no rubric did not
differ greatly from using the rubric in peer-assessing (p=.176).

b. Research Question 4
The purpose of research question 4 was to reveal the impact of those
conditions in using the rubric to the students’ writing performance. All the students’
writings in those settings were assessed by using the scoring rubric from the teacher
by two raters by a maximum point of 6. In order to guarantee the reliability of the
students’ writing score, Cronbach’s alpha coefficient was calculated. The resulting
scores were all at .984 showing that the reliability level of this research was high.
Further, the means of the students’ score of the three different situations were
calculated. The highest mean score was for the second situation in which the
students were involved in generating scoring rubric before they were assigned to
write that was 4.07. Meanwhile, the mean score of the students’ writing when they
did not know how their work would be evaluated was 3.59. This information is
visually illustrated in figure 4.
Hima,Saputro Develpoping Scoring Rubric| 108
DOI: 10.24235/eltecho.v%vi%i.2170

5,00

The mean score

4,00
4,07
3,00 3,59
3,22
2,00
1,00
0,00
1 2 3
Conditions

Figure 4. The Comparison of the Students’ Mean Score

Although it was found that the highest score obtained by the students was
from the second setting, a process of statistical analysis was still needed. The SPSS
calculation using ANOVA Bonferroni’s post hoc when investigating the students’
perception was done similarly to analyze the students’ performance to ensure
whether or not those three conditions affect their writing performance. The result
showed that there was no significant different (p=.341) in the score of the students
when they had no rubric and participated in designing rubric. In the same way, the
comparison between the students’ score in having the absence of rubric and the
rubric from the teacher before they composed a paragraph also showed no
meaningful difference (p=.191). The other insignificant difference (p=.790) was also
indicated from the students’ score in the second setting which they were involved to
make rubric compared to the third setting which they were given a rubric to peer-
assessing. These results suggest that the students had comparable writing proficiency
although they were on different situations.

c. Research Questions 5
This research question was about the correlation between the students’
perceptions of each alternative use of rubrics and their writing performance on each
situation. In order to reveal the possible relationships, a Spearman’s Rho correlation
coefficient was calculated to assess. The visual illustrations on the students’
perception and their writing score of a particular condition later were presented in
figures 5,6, and 7.
After being analyzed, in the first situation in which the students did not know
what considerations used by the lecturer to evaluate their writing performance, there
was a positive correlation between the students’ perception and their scores, p=.095
with a medium effect size of .328. As the score increased, positive perceptions of
having no rubric also increased and so did the reverse. The result is displayed in
Figure 5 with a scatter plot.
109 | ELT-Echo, Volume 2, Number 2, November 2017
ISSN: 2579-8170 e-ISSN: 2549-5089

25

Students' Perceptions
20
15
10
5
0
0 2 4 6 8
Students' Score

Figure 5. The Correlation between the Students’ Score and the Students’
Perceptions of Having no Scoring Rubric

With regard to the students’ involvement in designing a scoring rubric prior

to their writing task, there was a weak negative correlation between the students’
score and their perceptions with a small effect size -.147. Some students positively
perceived their participation in generating a rubric but had average or low scores
while the others viewed it negatively but had high scores. The following, Figure 6,
summarizes the results with a scatter plot.

30
Students' Perceptions

25
20
15
10
5
0
0 2 4 6 8
Students' Score

Figure 6.The Correlation between the Students’ Score and the Students’
Perceptions of Developing a Scoring Rubric

In the last setting in which the students’ were involved in doing peer
assessment using a rubric designed by the lecturer, it showed a weak positive
relationship of the students’ writing score and their performance with a small effect
size of .274. Figure 6 indicates the relationship existed between those aspects.
Hima,Saputro Develpoping Scoring Rubric| 110
DOI: 10.24235/eltecho.v%vi%i.2170

Students' Perceptions
6

0
0 5 10 15 20 25
Students' Score

Figure 6.The Correlation between the Students’ Score and the Students’
Perceptions of Using a Scoring Rubric

From the illustration through the scatter plot, it can be inferred that some students
positively perceived the use of a rubric from the lecturer and achieved high scores,
while the others felt the same way but had average or low scores.

DISCUSSION
The purpose of the current study was to investigate the students’ perceptions
and performance as well as the correlations between those variables about three
different conditions prior to the writing tasks. Those conditions were having no
rubric, being involved in generating a rubric and using a lecturer made rubric in
doing peer- assessment.
After answering the research question 1, 2 and 3, it was found that there was
a significant difference between the students’ perception of the first and second
situation. The result also showed that there was no great difference between the first
and the third condition as well as the second and the third. It indicated that among
those situations made by the lecturer before assigning the students to write a
paragraph, the best-perceived setting was when the students participated in
developing scoring rubric to evaluate their writing performance.
Furthermore, it was also found that although there was no significant
difference among the students’ writing performance in those three different
situations. Despite the fact that there was no statistically significance, the students’
writing performance in the second setting obtained the highest score. When inviting
students to create rubrics at the beginning of an activity, it helps them plan to write
better(Yoshina & Harada, 2007 and Neill, 2007) as well as enhance their motivation
and interest(Vision, 2017). When students are involved in rubric construction, the
assignment itself becomes more meaningful to the students(Lane, 1996). In addition,
Becker (2016)also revealed that there is a positive effect on the writing performance
of ESL learners when they are involved to design and apply a scoring rubric.
111 | ELT-Echo, Volume 2, Number 2, November 2017
ISSN: 2579-8170 e-ISSN: 2549-5089

When comparing the students’ perceptions about their performances on each

situation, a moderate relationship existed between the students’ perception of having
no rubric and their performance. As positive perceptions increased, so did the
students’ scores. Meanwhile, a negative correlation between the students’
perceptions and their scores occurred in the situation of the students’ involvement in
designing a rubric. It means that the students positively perceived this condition but
they had low scores. The correlation analysis also revealed that the relationship
between the students’ perceptions and their score in the third condition had a weak
positive relationship.
One thing that becomes a concern in this study is that the situation in which
the lecturer invited the students to design the rubric was the best-perceived situation
for the students. Also, this condition showed that it enhanced the students to achieve
the higher score compared to the other two conditions. However, there was a
contradictory fact in terms of their correlation. A negative relationship between the
students’ perceptions and the students’ writing performance occurred. It happened as
there were some students positively perceived their participation in designing a
rubric but they had average or low scores. Additionally, the other students thought
that this situation could not help them to write better but they could achieve high
scores.

CONCLUSION
The findings led the conclusion that the best-perceived situation for the
students was when the students were involved in designing a rubric before the
lecturer assigned them to write a paragraph. Although there was no significant
number of difference in terms of students’ performance in those three different
situations, the highest score the students achieved were in the second condition when
they experienced themselves in deciding criteria to evaluate their performance.
However, there was a fact that both variables (the students’ performance and their
score) are negatively connected. It occurs as those who positively believed that this
activity can enhance them to make a better writing had low or average scores.
Moreover, the students who could achieve the high scores were those who were not
confident that they could improve their writing after they participated in generating a
rubric.

ACKNOWLEDGEMENT
The authors express their gratitude to Directorate of Research and
Community Service Center, Universitas Muhammdiyah Malang for the funding
support.
Hima,Saputro Develpoping Scoring Rubric| 112
DOI: 10.24235/eltecho.v%vi%i.2170

REFERENCES
Alaei, M. M., Ahmadi, M., & Zadeh, N. S. (2014). The Impact of Rater’s
Personality Traits on Holistic and Analytic Scores: Does Genre Make any
Difference too? Procedia - Social and Behavioral Sciences, 98, 1240–1248.
https://doi.org/10.1016/j.sbspro.2014.03.539
Allen, D., & Tanner, K. (2006). Rubrics: tools for making learning goals and
evaluation criteria explicit for both teachers and learners. CBE Life Sciences
Education, 5(3), 197–203. https://doi.org/10.1187/cbe.06-06-0168
Andrade, H., & Du, Y. (2005). Student perspectives on rubric-referenced
assessment. Practical Assessment, Research & Evaluation, 10(3), 1–11.
https://doi.org/10.1080/02602930801955986
Becker, A. (2016). Student-generated scoring rubrics: Examining their formative
value for improving ESL students’ writing performance. Assessing Writing, 29,
15–24. https://doi.org/10.1016/j.asw.2016.05.002
Çetin, Y. (2011). Reliability of raters for writing assessment: Analytic - Holistic,
Analytic - Analytic, Holistic - Holistic. Mustafa Kemal University Journal of
Social Sciences Institute, 8(16), 471–486. Retrieved from
http://sbed.mku.edu.tr/article/view/1038000298/1038000247
East, M. (2009). Evaluating the reliability of a detailed analytic scoring rubric for
foreign language writing. Assessing Writing, 14(2), 88–115.
https://doi.org/10.1016/j.asw.2009.04.001
Ene, E., & Kosobucki, V. (2016). Rubrics and corrective feedback in ESL writing: A
longitudinal case study of an L2 writer. Assessing Writing, 30, 3–20.
https://doi.org/10.1016/j.asw.2016.06.003
Hidden Curriculum. (2014). Rubric Definition - The Glossary of Education Reform.
Retrieved November 13, 2017, from http://edglossary.org/rubric/
Hricko, M., & Howell, S. L. (2006). Online assessments and measurement :
foundations and challenges. Hershey Penns.: Information Science Pub.
Jackson, E. (2017). Types of Rubrics | Rubrics | Feedback & Grading |
Teaching Guides | DePaul University Teaching Commons. Retrieved
November 13, 2017, from https://resources.depaul.edu/teaching-
commons/teaching-guides/feedback-grading/rubrics/Pages/types-of-
rubrics.aspx
Karkehabadi, S. (2013). Using Rubrics to Measure and Enhance Student
Performance. Retrieved from
https://www.nvcc.edu/assessment/_docs/FTW5.usingrubricsmeasurestuperf-
spr13.pdf
Labs, C. (2017). Overview of Rubrics: Advantages and Disadvantages – Baseline
Help Center. Retrieved August 31, 2017, from
http://baselinesupport.campuslabs.com/hc/en-us/articles/204305625-Overview-
of-Rubrics-Advantages-and-Disadvantages
Lane, J. L. (1996). The Basic of Rubrics. Retrieved from http://ctl.clayton.edu/cid
Matsuda, P. K., & Jeffery, J. V. (2012). Voice in Student Essays. In Stance and
Voice in Written Academic Genres (pp. 151–165). London: Palgrave Macmillan
113 | ELT-Echo, Volume 2, Number 2, November 2017
ISSN: 2579-8170 e-ISSN: 2549-5089

UK. https://doi.org/10.1057/9781137030825_10
Neill, L. C. O. (2007). Student perceptions regarding the use of rubrics in writing
assignments.
Office of graduate Studies Teaching at UNL. (2017). Scoring Rubrics | Teaching at
UNL | University of Nebraska–Lincoln. Retrieved November 13, 2017, from
https://www.unl.edu/gtahandbook/scoring-rubrics
Rakedzon, T., & Baram-Tsabari, A. (2017). To make a long story short: A rubric for
assessing graduate students’ academic and popular science writing skills.
Assessing Writing, 32, 28–42. https://doi.org/10.1016/j.asw.2016.12.004
Rezaei, A. R., & Lovorn, M. (2010). Reliability and validity of rubrics for
assessment through writing. Assessing Writing, 15(1), 18–39.
https://doi.org/10.1016/j.asw.2010.01.003
Sundeen, T. H. (2014). Instructional rubrics: Effects of presentation options on
writing quality. Assessing Writing, 21, 74–88.
https://doi.org/10.1016/j.asw.2014.03.003
Vision, T. (2017). Student-Generated Rubrics: Part Five in a Five-Part Series -
TeacherVision. Retrieved November 20, 2017, from
https://www.teachervision.com/student-generated-rubrics-part-five-five-part-
series
Yoshina, J. M., & Harada, V. H. (2007). Involving Students in Learning through
Rubrics. Library Media Connection, 25(5), 10–14.
https://doi.org/http://dx.doi.org/Article