Wafa Alotaibi

Medical Education Department in Qassim University

Research Proposal for PhD Project

Reliability and validity of a peer assessment tool and its impact on

students’ behaviour and achievement in problem-based learning

This is my research proposal for the application for a PhD programme in medical
education; the purpose of choosing this particular topic area is that assessment in medical
education is my main interest. In this proposal, I will follow the proposal protocol guideline
outlined by Swanwick (2014). The proposal will take the following order: title and aim,
research question, background literature, methods and methodology, plans for the dissertation
findings, timeline, and references. Ethical considerations must be taken into account in order
to complete the proposal; this will be done later.

Title and aim

The impact of a reliable and valid peer assessment tool in first-year medical students’
behaviour and academic achievement in the problem-based learning environment at Qassim
University, Saudi Arabia.

Research questions

To what extent can a newly developed peer assessment instrument with good psychometric
measures be used as an assessment and development tool at once?

To what extent can a valid and reliable instrument decrease the main drawbacks of peer
assessment, such as, low student engagement, students’ frustration at filling in a peer
assessment tool, and the need of anonymise in many situations?

To what extent can a reliable and valid developmental peer assessment change both students’
assessment behaviours and professional behaviours?

To what extent do students change their behaviour before and after receiving peer feedback
related to problem-based learning behaviour and academic performance?

PHD PROPOSAL "1

This research contains the following aims, which show its importance:

Firstly, I conducted previous research during my master’s degree about students’ learning and
study strategies in the first year; the result was quite disappointing as many students had a
defect in multiple essential learning strategies. In addition, there was a high dropout rate that
was reported among first-year medical students at Qassim university, which is probably
caused by default in adapting to the university learning style with quite a low level of learning
strategy. Faculty members at Qassim Medical College assume that students who are unable to
continue through medical school are in need of special support, but this is not currently
specified. Thus, peer assessment could provide students with the proper support that they
need. Especially as the main goals of peer assessment, as mentioned by Gielen (2007), are: to
prepare students to practice self-monitoring and self-regulation strategies in lifelong learning,
to install social control into the learning process by reaching desired behaviour and avoiding
undesired behaviour, to stimulate students to actively participate by supporting their
autonomy, and the most popular goals of peer assessment, as an assessment tool, to judge
peers on their learning attitude, and as a learning tool, to adjust students’ professional
behaviour. So, by the end of this research, I hope to have outlined a plan to use peer
assessment to determine if students’ professional behaviour and academic achievement in a
problem-based setting will change after receiving peer feedback. Secondly, Qassim students
come from an entirely traditional learning background to a problem-based learning style, so it
is worthwhile assessing if they can determine assessment behaviours and professional
behaviours or not.

Relevant background literature

It is important to outline the exact meaning and definition of peer assessment and its
concepts and principles in education, which is well defined in the literature. Peer assessment is
an important factor in the design of learning environments applying a more participatory
culture of learning (Kollar and Fischer, 2010). Additional definitions by Topping (1998) and
Van den Berg (2006) state that peer assessment is considered as a process of evaluating the
quality or success of peer outcomes, which is followed by the provision of feedback. Research
into peer assessments has found that peer assessment can be described in many different
terminologies, such as, peer revision by Cho and MacArthur (2010) and peer feedback by
Strijbos (2010). To clarify the significance of and need for peer assessment in medical
education, it is well documented in research that peer assessment is fundamentally a
cooperative activity that occurs between at least two peers (Kollar and Fischer, 2010). The
literature on peer assessment could be used to improve students’ professional behaviour,
which is currently an increasingly important part of the medical school curriculum (Roberts,
2010). Thus, peer assessment is considered an important factor in developing medical
students’ professional behaviour and could lead to further educational improvement.

PHD PROPOSAL "2

An overview of the mainstream research literature in the field, using peer assessment in a
problem-based learning (PBL) system, illustrates that some colleges have used peer assessment
within PBL groups with the expectation that scores will inform applicable action by faculty
staff, with reference to challenging students (Roberts, 2017). Although there is minimal
literature that discusses the efficacy of introducing peer assessment in PBL classrooms, peer
assessment holds potential for assessing professionalism, as well as refusal and reluctance to
participate, which have been noted among practising physicians and medical students
(Arnold, 2005; Dent, 2013). In addition, Dent has stated that the use of peer assessment in
PBL could be considered a unique way to assess professionalism, by providing students with
other student perceptions. Interactions between students typically happens during their daily
academic activities, such as, during teaching sessions, on the wards, and in study groups. As
peer assessment could significantly enhance students’ professionalism, it is important to make
sure that it could also improve problem-solving skills among medical students, as these skills
signify an important shift in the learner, from novice to expert (Scott, 2017). Scott’s (2017)
study examined the effect of simulated peer assessment in significantly influencing student
performance; students thought that a simulated approach could be used to avoid the negative
aspects of direct peer assessment. In contrast, Kritikos’ (2011) study was the first to examine
the use of peer assessment in the PBL tutorial setting for small groups of pharmacy students
who graded their peers’ presentations and provided feedback, in combination with facilitator
feedback and assessment. In his study, Kritikos illustrates that peer assessment is an
appropriate method to assess PBL skills and is accepted by students as appropriate. Thus, peer
assessment can improve student learning by providing them with other students’ perceptions,
to assess their academic performance and learning process.

In the context of group work, both peer assessment and PBL focus on group collaboration.
Peer assessment offers a powerful way for students to obtain feedback on their learning
(Ballantyne, 2002). For example, weak students who obtain low scores in academic activities
can learn from colleagues who know how to learn and who obtain higher scores in the
university academic activities, including in PBL sessions. Despite the fact that PBL adopts a
deep learning approach, and encourages collaboration and active learning among students to
promote internal motivation through the provision of pragmatic goals (Kelson and
Distlehorst, 2000), the peer assessment process could support students in achieving these
goals. In addition, due to challenges in assessing students during all the learning hours of the
PBL system, peer assessment is considered an advanced method of assessing higher order
learning in these tutorials. These hours could amount to 18 of the 48 credit hours that
students must attain, which is what occurs in the Bachelor of Medicine programmes at the
University of Qassim.

PHD PROPOSAL "3

In the context of research conclusions about peer assessments and students’ professional
behaviours, they are considered by some to have a sufficient advantage to warrant summative
use. This is despite student uncertainty towards judgements of these behaviours (Rees and
Shepherd, 2005). According to Roberts (2017), a variety of methodological approaches have
been used to afford valid evidence in support of the use of peer assessment in measuring
students’ professional behaviours used in PBL. Some studies have established the internal
arrangement of a particular peer assessment instrument to be used in the local context.
Kamp and others (2011) validated and developed the Maastricht Peer Activity Rating Scale
(M-PARS), which is a tool that measures collaborative, motivational and constructive factors
in PBL classes. In contrast, Papinczak and others (2007) developed a peer assessment
instrument in which Cronbach's alpha of peers across all PBL groups ranged from 0.77 to
0.66. In Papinczak’s instrument, a correlation between faculty-based and self-based
assessments in the PBL tutorial was found; the correlation was moderate, initially it was (r =
0.40) and it improved over time to (r = 0.60). Reiter and others (2002) developed an
instrument that enables PBL students, in their groups, to rank the professional behaviours of
their peers. Since ratings varied from one week to the next, this instrument proved an
unreliable measure of tutorial performance.

Methods and methodology

Data collection instrument

The peer assessment instrument that will be used in this research will be designed to stimulate
reflection to improve students’ academic achievement in problem-based sessions, and to
improve students’ professional behaviour, with two purposes: (i) to change behaviours
according to students’ own performance, and (ii) to change behaviours according to students’
peer performance, from the student’s perspective. The instrument that will be used is derived
from Roberts and others’ (2017) study. In addition, this instrument will be modified to include
students’ academic achievements assessment by adding the academic assessment scale used by
Qassim University. The instrument developed by Roberts is a modified version of a
previously validated peer assessment instrument for use in PBL groups. The original measure
was developed by Papinczak and others (2007), using qualitative and quantitative data
collected from tutor assessments of students’ PBL performance. The original instrument
included 17 items across five areas: self-awareness, critical analysis, responsibility and respect,
information processing, and communication. Students are asked to rate the strength of their
agreement or disagreement with statements about their peers’ performance in that week of
PBL classes using a 5-point Likert-type scale (1 = totally disagree; 5 = totally agree).
Therefore, there is time for each student to express and display their communication skills and
educational leadership abilities. For each student, the mean score of the instrument is
calculated across the nine items and the ratings of students within that student's PBL group (n

PHD PROPOSAL "4

= 9–14). In addition, I will carry out a group discussion interview to understand the factors
beyond students’ perception of their peers. This will be used after the selected student has
presented a summary to the group.

Sample size and data analysis

In this research, I will use the quasi-experimental method of a pre-/post-test design for a
cohort of students in year one; this will take the form of giving PBL peer assessment, with the
intervention of receiving a summary of peer feedback. The sampling method will be simple
random sampling, which has a respectable range of validity and reliability. The number of
first year medical students recruited will be 120 male and female students. The Papinczak
instrument can be distributed on paper or electronically, using the blackboard university
website. So, for this research, I will prepare 120 electronic versions and distribute the link to
the students, to be completed by them after the PBL session. Thus, as the total number of
students is around 120, the results will be generalizable (Golafshani, 2003), which was
recommended in Roberts’ (2017) study.

Since there are five domains in the research instrument (self-awareness, critical analysis,
responsibility and respect, information processing, and communication) that will assess
students’ peer performance, the analysis of this data will be difficult. Data analysis in
qualitative and quantitative research is known to be very important and can be challenging.
One of the biggest advantages of using the web-based instrument tool is that the analysis will
be done easily, as the coding part will be carried out by the website. The tool will yield
statistical analyses using SPSS in schemes, such as, a statistical analysis in relation to age,
gender, and college status, item statistics for each instrument scale, individual scale statistics,
and correlation among scales. There is an additional significant analysis that could also be
carried out by the blackboard website. In relation to these statistics, I will correlate them with
the earlier students’ scores and performance results, general percentages of the student
dropout rate, and percentages of student success in academic courses.

Plans for the dissertation findings

In general, plans for the research findings will depend on the student analysis results of
Papinczak’s instrument. If a student does well in the Papinczak scores and shows evidence of
a good range of professional behaviour abilities, in relation to the students' pre- and post-
professional behaviours and academic performance, then there will be special data
interpretation of the findings . This could indicate another possible cause for the
improvement of their professional behaviours and decreases the percentage of dropouts from
medical school. In this case, further research will be recommended to assess other factors that

PHD PROPOSAL "5

could contribute to students continuing through medical school. Conversely, if the Papinczak
instrument results indicate a defect in a student’s professional behaviour abilities after
receiving peer feedback, this could indicate that they are having difficulty in adapting to the
PBL environment. In this case, suggestions for a special support programme will need to be
introduced. It is equally important to mention that whether the first-year students’ assessment
results are positive or negative with regard to professional behaviours abilities, it is worthwhile
that all students keep assessing their abilities to maintain an awareness of their progress.
Therefore, it is recommended that Qassim Medical College continuously offers access to such
a programme to all first year medical students, so that they know if they lack any professional
behaviour strategies early in the year and can keep monitoring this themselves. It will also be
worthwhile for students to assess themselves at the beginning and end of each module to
compare their progress in a valid way; especially since there is no such assessment programme
offered to first year medical students. Finally, the exact detailed plan for the research findings
will be clearer once the research has taken place.

Proposed timeline

The timeline schedule will be based on the full-time duration of PhD study (three years). The
schedule in Table 1 illustrates each month’s duty, outlining the exact nature of the work that
will be involved. As advised by Hart (2005), one or two months each year have been kept free
in case of any unexpected circumstances.

Year Task

Year 1 Month 1: detailed systematic literature review

Month 2: start writing the systematic literature review

Month 3: continue systematic literature review + prepare the full research project checklist

Month 4: finalise the writing of the systematic literature review

Month 5: develop the research instrument with good psychometric characteristic measures

Month 6: pilot testing of the research instrument + checking the validity and reliability

Month 7: create a semi-structured group interview

Month 8: pilot testing of the interview

Month 9: prepare the 9 month report

Month 10: re-write the systematic literature review as a thesis literature review

Month 11: write the thesis methodology

Month 12: vacation

PHD PROPOSAL "6

Year Task

Year 2 Month 1:
1) prepare the pre-test to assess the students’ level of professional behaviour and academic
achievement before the introduction of the modified peer assessment tool
2) distribute to the students (online distribution or paper distribution in case of low respondent
rate)

Month 2: distribute pre-test 1

Month 3: gather data and arrange proper analysis

Month 4: code the data for SPSS analysis

Month 5: SPSS analysis

Month 6: SPSS analysis

Month 7: conclude the analysis results

Month 8: write the results part of the pre-test

Month 9: write the discussion part of the pre-test

Month 10: pre-test group discussion interview + introduce the modified peer assessment tool to
be used for a duration of 1 month = 4 problem-based sessions at least

Month 11:
1) prepare the post-test to assess students’ professional behaviour development and academic
achievement improvements
2) distribute to the students (online distribution or paper distribution in case of low respondent
rate)

Month 12: vacation

Year 3 Month 1: NVIVO analysis for the pre-test group discussion interview

Month 2: NVIVO + SPSS analysis for the modified peer assessment tool results

Month 3: NVIVO + SPSS analysis for the post test result

Month 4: post-test groups discussion interview to discuss the finding results and possible
explanations

Month 5: NVIVO analysis for the post-test group discussion interview

Month 6: write thesis results

Month 7: write thesis discussion

Month 8: write thesis conclusion and recommendation

Month 9: write thesis introduction

Month 10 & 11: revision of all the work and check any missing data

Table 1: Timeline for the PhD research

PHD PROPOSAL "7

Conclusion

In conclusion, this proposal is for a mixed methods research study and its main aim is to assess
professional behaviour, improving abilities using peer assessment among first year medical
students at Qassim Medical School, in Saudi Arabia. The selection of this exact research field
was mainly due to the observed student dropout rate in some local medical schools in the first
year. The second important reason for choosing this research question is that all the students
come from traditional learning backgrounds. Thus, the result of testing their abilities to assess
their peers according to their professional learning behaviour will be valuable in determining
their needs in their first academic year. The literature suggests a variety of assessment
methods, but the data collection in this research will be performed using Papinczak’s
instrument. This instrument is preferred because it can concentrate on many important areas,
such as, self-awareness, critical analysis, and communication. Additionally, the sampling
method will be simple random sampling, which has a respectable range of validity and
reliability consideration in my research. The data will be analysed using SPSS and NVIVO,
to analyse the numerical data retrieved using Papinczak’s instrument and the group discussion
interview. The results of this study will be important to the Qassim Medical College faculty, as
they could reveal the significance of introducing peer assessment to the PBL system. Also, the
results will be important in discovering whether first year medical students are able to develop
professional behaviours in a PBL environment. It is important that this particular issue is
considered, as students they have come from a completely traditional learning environment.
Recommendations will be generated after this study has been completed, depending on the
results of students’ assessment. These results will define if the student positively developed
professional behaviours with the use of peer assessment. Consequently, recommendations will
be developed based on these considerations.

PHD PROPOSAL "8

References

Arnold, L., Shue, C. K., Kritt, B., Ginsburg, S. and Stern, D. T. (2005). Medical students'
views on peer assessment of professionalism. Journal of General Internal Medicine, 20, pp. 819–
824.

Ballantyne R., Hughes K. and Mylonas A. (2002). Developing procedures for implementing
peer assessment in large classes using an action research process. Educ., 27(5), pp. 427–441.

Dent, J.A., and Ronald, M.H. (2013). A practical guide for medical teachers. Churchill Livingstone.

Golafshani, N. (2003). Understanding reliability and validity in qualitative research. NSU

Works, 8(4), pp. 597-606.

Gielen S. (2007). Peer assessment as a tool for learning [dissertation]. Leuven, Belgium: Catholic
University of Leuven.

Hart, C. (2005). Doing your master’s dissertation. London: SAGE Publications.


Kamp R. J. A., Dolmans D. H. J. M., van Berkel H. J. M. and Schmidt H. G. (2011). Can
students adequately evaluate the activities of their peers in PBL? Med Teach, 33(2), pp.145–
50. Problem-Based Learning: A Research Perspective on Learning Interactions. Associates.

Kollar, I. and Fischer, F. (2010). Peer assessment as collaborative learning: A cognitive

perspective. Learning and Instruction, 20, pp. 344- 348.

Papinczak T., Young L., Groves M. and Haynes M. (2007). An analysis of peer, self, and tutor
assessment in problem-based learning tutorials. Med Teach, 29(5), pp. e122–32.

Rees C. and Shepherd M. (2005). The acceptability of 360-degree judgements as a method

of assessing undergraduate medical students’ personal and professional behaviours. Med Educ,
39 (1), pp. 49–57.

Reiter H. I., Eva K. W., Hatala R. M. and Norman G. R. (2002). Self and peer assessment in
tutorials: application of a relative-ranking model. Acad Med, 77(11), pp. 1134–9.

Kelson A. C. and Distlehorst L. H. (2000). Groups in problem-based learning (PBL):

essential elements in theory and practice. In: D. H. Everson and C. E. Hmelo, ed., Mahwah,
NJ: Lawrence Erlbaum.

Roberts, C., Jorm, C., Gentilcore, S. and Crossley, J. (2017). Peer assessment of professional
behaviours in problem-based learning groups. Med Educ, 51, pp. 390–400.

PHD PROPOSAL "9

Strijbos, J. W., Narciss, S. and Du ̈nnebier, K. (2010). Peer feedback content and sender’s
competence level in academic writing revision tasks: are they critical for feedback perceptions
and efficiency? Learning and Instruction, 20(4), pp. 291-303.

Scott, F. J. (2017). A simulated peer-assessment approach to improve students’ performance in

numerical problem-solving questions in high school biology. Journal of Biological Education,
51(2), pp. 107-122.

Topping, K. J. (2005). Trends in Peer Learning. Educational Psychology, 25(6), pp. 631–645.

Van den Berg, I., Admiraal, W. and Pilot, A. (2006). Peer assessment in university teaching:
evaluating seven course designs. Assessment and Evaluation in Higher Education, 31(1) pp. 19–36.

PHD PROPOSAL "10