Key of Language Assessment: Rubrics and Rubric Design: Ülkü Ayhan
Key of Language Assessment: Rubrics and Rubric Design: Ülkü Ayhan
Key of Language Assessment: Rubrics and Rubric Design: Ülkü Ayhan
2; June 2015
Ülkü Ayhan
International Burch University
M. Uğur Türkyılmaz
The University of South
East Europe Lumina
Abstract
The purpose of this paper is to provide an overview of “rubric” in language testing and assessment and to
highlight the parts of a rubric through various dimensions. It sets forth objectives, types of the rubric use. As the
paper will illustrate, language testing and assessment in SLA demands many facets to take the learners to
“successful learner point”. Being aware of the effective use of rubrics reflect robustness of this critical
assessment process.
Keywords: Rubric, assessment, dimensions of a rubric, measurement
The word “rubric” comes from the Latin word for “red.” It was once used to signify the highlights of a legal
decision as well as the directions for conducting religious services, found in the margins of liturgical books—both
written in red.
In a broad sense, rubric refers to a term which has existed in English for more than 600 years and during that time,
mostly it has meant a set of “printed rules or instructions” (Encarta Encyclopedia, 2004). However, in educational
sense, it refers to different categories such as criteria foe assessment, evaluation of learning, gradients of learning
of a set of instructions etc. (Brown, 2012).
Construction of a Rubric
A good activity never guarantees the accurate determination of a student’s competency at a given task. At this
point, rubrics stand for this main requirement. Since it specifies the skill being examined and what constitutes
various levels of performance success. In order to construct a good rubric focus on “what to measure exactly, how
to measure performance and decision on what a passing level of performance competency is” plays the key role.
Even though based on the general guidelines a general rubric design may be organized and be used multiple
times. Here is the process in detail:
1. Defining the Behavior to Be Assessed
Expected student outcomes, what they should accomplish at the end of each unit and end of each term should be
clarified. For this, some questions should be asked:
- What concept, skill or knowledge am I trying to assess?
- What should my students know?
- At what level should my students be performing?
- What type of knowledge is being assessed: reasoning, memory or process.(Stiggins, 1994)
2. Choosing the Activity
After the determination of the purpose of the assessment, you should decide an activity and consider issues
regarding time constraints, resources, and how much data is required. (Airasian, 1991; Popham, 1995; Stiggins,
1994)
3. Defining the Criteria
Third step after the decision of activity and tasks to be used, definition of which elements of the project/task will
be used to find the success of the students’ performance.
82
ISSN 2374-8850 (Print), 2374-8869 (Online) © Center for Promoting Ideas, USA www.ijllnet.com
83
International Journal of Language and Linguistics Vol. 2, No. 2; June 2015
84
ISSN 2374-8850 (Print), 2374-8869 (Online) © Center for Promoting Ideas, USA www.ijllnet.com
thesis, which shows a highly analytical structure and a analytical structure and no
developed awareness of central thesis, but the central thesis.
historiographic or social issues and a analysis is not always fully
30%
thesis.
The presentation is imaginative and Presentation techniques The presentation fails to
Communication
effective in conveying ideas to the used are effective in capture the interest of
audience. The presenter responds conveying main ideas, but the audience and/or is
effectively to audience reactions and they are a bit unimaginative. confusing in what is to be
questions. Some questions from the communicated.
20%
audience remain
unanswered.
The presentation includes The presentation includes The presentation includes no
appropriate and easily understood appropriate visual aids, but visual aids or includes visual
Use of visual aids
visual aids, which the presenter these are too few, are in a aids that are inappropriate or
refers to and explains at appropriate format that makes the too small or messy to be
moments in the presentation. difficult to use or understood. The presenter
understand, or the presenter makes no mention of them in
20%
body language to engage the fails to use eye contact, understood. There is no
audience. gestures, and body language attempt to engage the
skills
85
International Journal of Language and Linguistics Vol. 2, No. 2; June 2015
86
ISSN 2374-8850 (Print), 2374-8869 (Online) © Center for Promoting Ideas, USA www.ijllnet.com
Before giving the decision which one to use in class, it is important to decide the key criteria. Blaz (2001) compile
in her study most common tasks used in assessments. Her study is based on performance tasks: written and oral
tasks. (See table:6)
Table-6
Some activities for oral performance Some activities for written performance
assessments assessments
anecdote advertisement lyrics
ballad/rap/song biography magazine/news
book report book report article
campaign speech booklet/brochure metaphor
choral reading/speech business letter movie review
debate cartoon newsletter
demonstration celebrity profile new story ending
dialogue checklist notes
discussion comic book observation sheet
documentary commercial script outline
dramatization comparison pamphlet
explanation creative writing parody
fairy tale critique petition
free verse description play
interview dialogue poem
jingle/ad campaign diary/journal poster
joke e-mail message prediction
lecture editorial essay puppet show
lesson fairy tale/myth questionnaire
mock interview glossary quiz
monologue guidebook recipe
narration handbook report
newscast handout review
recipe headline riddle
riddle interview script script
role-play job description short story
seminar joke slogan
skit journal story problem
weather report lesson plan survey
letter telegram
list travel log
log yearbook entry
a. Holistic Rubric:
Holistic rubrics contain different levels of performance that describes the quality, quantity, quantity/quality of a
task. (See sample holistic rubric: Appendix 1 and 2)
87
International Journal of Language and Linguistics Vol. 2, No. 2; June 2015
Holistic rubric
Proficien The student`s project has a hypothesis, a procedure, collected data, and analyzed results.
A score
The project is thorough and the findings are in agreement with the data collected. There are
of 3 –
The student`s project may have a hypothesis, a procedure, collected data and analyzed
Adequat
A score
results. The project is not as thorough as it could be; there are a few overlooked areas. The
of 2 –
project has a few inaccuracies that affect the quality of the project.
e
The student`s project may have a hypothesis, a procedure, collected data and analyzed
Limi
scor
e of
ted
1-
A
results. The project has several inaccuracies that affect the quality of the project.
In that type of evaluation, raters judge by forming an overall impression of learners` performance and matching it
to best fitting column on the scale. Each scale describes performance according to several criteria such as range of
vocabulary, grammatical accuracy, fluency etc. Mostly, 4 or 5 performance levels are available in holistic rubrics.
Generally, teachers find it efficient and easy to use for classroom assessments.
Advantages and Disadvantages of Holistic Rubrics
Advantages
- They are often written generically and can be used with many tasks.
- They emphasize what learners can do, rather than what they cannot do.
- They save time by minimizing the number of decisions raters must make.
- Trained raters tend to apply them consistently, resulting in more reliable measurement.
- They are usually less detailed than analytic rubrics and may be more easily understood by younger learners.
Disadvantages
- They do not provide specific feedback to test takers about the strengths and weaknesses of their performance.
- Performances may meet criteria in two or more categories, making it difficult to select the one best description.
(If this occurs frequently, the rubric may be poorly written.)
- Criteria cannot be differentially weighted.(Teddick, 2002; TeacherVision.com, 2000-2002)
b. Analytic Rubric
According to Taggart, analytic scales are the types which tend to focus on broad dimensions of writing or
speaking performance. These dimensions may be similar with those found in a holistic scale, but they are
presented in separate categories and rated individually. Points may be assigned for performance on each of the
dimensions and a total score calculated. (See sample analytic rubrics: Appendix 3)
In general sense, analytic rubrics are associated with large-scale assessment of general dimensions of language
performance. However, analytic rubrics certainly can be created or adapted for use in classroom settings and with
particular tasks (1998).
In practice, the names "analytic rubric" and "multiple trait rubric" may be used interchangeably.
Advantages
According to Moskal, analytic rubrics have those advantages;
- They provide useful feedback to learners on areas of strength and weakness.
- Their dimensions can be weighted to reflect relative importance.
- They can show learners that they have made progress over time in some or all dimensions when the same rubric
categories are used repeatedly (2000).
Disadvantages
- According to Teddick, For different aspects of students` writing or speaking performance, separate scores are
considered more artificial, since learner can`t get a good assessment of the whole of performance (2002).
- They take more time to create and use.
- It is hard to reach inter and intra reliability on all the dimensions in comparison to a single scored holistic
rubric.
- Raters tend to evaluate grammar related categories more strictly than other categories. In other words, they
overemphasize the role of accuracy. (McNamara, T. (1996)
88
ISSN 2374-8850 (Print), 2374-8869 (Online) © Center for Promoting Ideas, USA www.ijllnet.com
- Many rubrics of this type have been developed by teachers who are willing to share them online, at conferences,
and in materials available for purchase.
Disadvantages
- Information provided by primary trait rubrics is limited and may not easily translate into grades.
- Task-specific rubrics cannot be applied to other tasks without adaptation of at least one or more dimensions.
(CARLA, 2012)
Conclusion
It is very significant to know the procedure of assessment and use of efficient tools to do it. Rubrics play a key
role in evaluation of proficiency of students especially by providing validity and reliability. Getting right
feedback, learners feel more comfortable for the next step to be taken to improve; Therefore, assessors` role is not
just giving feedback, but encouraging for further development by indicating the learners` weakness without
discouraging in an objective way.
Appendices
Appendix 1 - Advanced Holistic Rubric
Role Play
Exceeds Student accurately and compellingly articulates the needs of his/her character and responds
Expectations meaningfully to others' comments using well-connected sentences. Student uses a variety of
sentence structures and grammatical forms that allow discourse to flow. Vocabulary is specific
and appropriate, and there are minimal errors in grammar and word choice that do not impact
comprehension.
Meets Student accurately articulates the needs of his/her character and responds to others' comments
expectations- with complex sentences. Student uses a variety of sentence structures and grammatical forms
strong that allow discourse to flow most of the time. Vocabulary is mostly appropriate, and several
minor errors in grammar and word choice may have a slight impact on comprehension.
Meets Student articulates most of the needs of his/her character and responds to 1-2 comments with
expectations- complete sentences. Student uses at least 3 different sentence structures and several grammatical
weak forms that allow discourse to flow most of the time. Available vocabulary somewhat limits
conversation; circumlocution or other strategies may be used. Errors in grammar and word
choice lead to some errors in comprehension that are clarified in the target language.
Does not meet Student fails to articulate his/her character's needs and/or doesn't respond to other comments.
expectations Speech consists mostly of short sentences and uses limited sentence structures. Many errors in
grammar and vocabulary lead to significant errors in comprehension.
http://www.carla.umn.edU/assessment/vac/evaluation/e 1 .html
Appendix 2 – Holistic Rubric
Who am I? Intermediate
3 Student asks a variety of questions related to both physical features and other traits. Student answers
partner's questions with complete sentences and accurate information. Questions and answers use a range of
appropriate vocabulary and grammar features taught in class. Minor grammatical errors do not impact
comprehension.
2 Student asks several questions related to physical features and other traits. Student answers partner's
questions with mostly complete sentences and accurate information. Questions and answers use appropriate
vocabulary and grammar features taught in class. Several grammatical errors may slightly impact
comprehension, but misunderstandings are clarified in the target language.
1 Student asks minimal questions related to physical features or other traits. Student answers partner's
questions with some accuracy. Vocabulary and grammar are simple. Significant grammatical errors lead to
multiple misunderstandings.
90
ISSN 2374-8850 (Print), 2374-8869 (Online) © Center for Promoting Ideas, USA www.ijllnet.com
91
International Journal of Language and Linguistics Vol. 2, No. 2; June 2015
References
American Council on the Teaching of Foreign Languages. (1999).ACTFL performance guidelines for K-12
learners. Yonkers, NY: ACTFL.
Applebee, A.N. (2000).Alternative models of writing development.
Airasian, P.W.. (1991). Classroom Assessment. New York:McGraw-Hill
Lim, H-Y,&Griffith, W.I. (2011). Practice doesn’t make perfect
Blaz, Deborah. (2001). A collection of performance tasks and rubrics: Foreign languages. Larchmont, NY: Eye on
Education.
Brown, J. D.(Ed.)(2012). Developing, using and Analyzing Rubrics in Language Assessment with Case Studies in
Asian and Pacific Languages. Honolulu, HI: National Foreign Languages Resource Center.
Center for Advanced Research on Language Acquisition (CARLA),
http://www.carla.umn.edu/assessment/vac/Evaluation/res_1.html
Fiderer, A. (1999).40 rubrics & checklists to assess reading and writing. New York: Scholastic
Goodrich Andrade, H. (1997).Understanding rubrics. Originally published in Educational Leadership
http://www.ascd.org/publications/educationalleadership/dec96/vol54/num04/Understanding-Rubrics.aspx
KasmanValenza, J. (2000).Students and teachers alike can benefit from rubrics: Selections from an interview with
Dr. AndStix.
http://www.interactiveclassroom.com/pdf/2000_Students_and_Teachers_Alike_Can_Benefit_from_Rubri
cs_Philly.pdf
Lloyd-Jones, R. (1977). Primary trait scoring. In C.R. Cooper & L. Odell (Eds.), Evaluating writing: Describing,
measuring, judging (pp. 33-66). Urbana, IL: National Council of Teachers of English.
Luft, Julie. Design Your Own Rubric.
http://plato.acadiau.ca/courses/educ/GMacKinnon/EDUC4143/graphics/rubrics.pdf
McNamara, T. (1996).Measuring second language performance. London: Longman.
Moskal, B.M. (2000). Scoring rubrics: What, when and how? Practical Assessment, Research & Evaluation, 7(3)
Popham, W.J. (1995). Classroom Assessment: What Teachers Need to Know. Needham Heights, MA: Allyn and
Bacon.
SRI International: Center for Technology in Learning. (1997-2002). PALS Guide
http://pals.sri.com/guide/index.html
Stevens, D.& Levi, A. An Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote
Student Learning.2005
Stiggins, R.J. (1994). Student-Centered Classroom Assessment. New York: Macmillan Publishing Company.
Taggart, G.L., S.J. Phifer, J.A. Nixon, and M. Wood. (1998).Rubrics: A handbook for construction and use.
Lancaster, PA: Technomic Publishing Company
TeacherVision.com. (2000-2002).The advantages of rubrics: Part one in a five-part
serieshttp://www.teachervision.fen.com/teaching-methods-and-management/rubrics/4522.html
Tedick, D.J. (2002).Proficiency-oriented language instruction and assessment: Standards, philosophies, and
considerations for assessment. In Minnesota Articulation Project, D. J. Tedick (Ed.), Proficiency-oriented
language instruction and assessment: A curriculum handbook for teachers (Rev Ed.). CARLA Working
Paper Series. Minneapolis, MN: University of Minnesota, The Center for Advanced Research on
Language Acquisition.
92