OPTIMISING POSTGRADUATE OUTCOMES FOR CLINICAL PSYCHOLOGISTS

Lisa van Seters

Student ID: 43191503

PSYC3020

Semester 2, 2015

School of Psychology, The University of Queensland

Tutor: Jo Brown P05

Word Count (not including references): 2057

Executive Summary

Currently to be considered as an applicant for a mainstream Australian

postgraduate training program for clinical psychology, one requires the following: a

face-to-face interview, high academic achievement, and a reference from their

honours supervisor, an academic staff member and a third person who ‘knows them

well’ (however, not a relative or friend). Even though these elements are important in

the evaluative process, they have major limitations such as (1) face-to-face

interviewers have interviewer bias, (2) academic achievement has a modest

predictability over job performance, and (3) references are not an accurate measure of

character due to possible bias also. Due to these limitations, the characteristics

applicants are judged upon lack validity and reliability. Therefore, it follows that

applicants may not possess essential traits that would enable them to work in the field

of clinical psychology.

I propose two measures that would increase dependability in the selection of

applicants for clinical postgraduate degrees. These measures assess interpersonal

skills and psychological fitness via the self-report SIB/FIB performance task and MHI

self-report scale, respectively. Both these measures are important qualities for the

practising clinical psychologist.

To evaluate the effectiveness of the proposed battery, the inclusion of these

new measures will be contrasted against the current model as well as evaluated

collectively with the current model to test for optimal postgraduate outcomes.
OPTIMISING POSTGRADUATE OUTCOMES FOR CLINICAL PSYCHOLOGISTS

Optimising Postgraduate Outcomes for Clinical Psychology Students via

Selection of the Finest Applicants

A postgraduate clinical psychology degree remains one of the most pursued

degrees within the graduate population. As such, the need for a proficient selection

process that selects the finest students is undeniably necessary, especially because

their role in society can be equivalently productive and devastating (Johnson &

Campbell, 2002).

The current model incorporates three measures to test suitability for

postgraduate programs. These are face-to-face interviews, a high academic

achievement, as well as three references. The current measures will be evaluated in

regards to reliability and validity.

Face-to-Face Interviews

The use of face-to-face interviews to measure admissions into postgraduate

degrees is quite controversial due to an inherent lack of validity. The biggest

limitation with interviews is the prospect of interview bias, which can facilitate

subjective evaluation of an applicant. As well as this, verbal reinforcement as a

product of social desirability can often sway expression and self-report from the

applicant to appear as a more favourable applicant (Hildum & Brown, 1956).

Furthermore, interviews use explicit self-report measures, which have low

correlations to implicit attitudes and actual behaviour (Hofmann, Gawronski,

Gschwendner, Le & Schmitt, 2005). This lack of consistency diminishes reliability of

the measure.

Academic Achievement

Grade point average (GPA) cannot produce completely valid contrasts in a

student population because course demands and supervisor standards are not constant
OPTIMISING POSTGRADUATE OUTCOMES FOR CLINICAL PSYCHOLOGISTS

as they are subjective (Bailey, Rosenthal & Yoon, 2014). Specifically, Bailey and

colleagues (2014) found that students who took easier classes, and therefore attained a

higher GPA, had a higher chance of being accepted into a postgraduate degree.

However, they also found supporting evidence that students who had a higher GPA

continued to get good grades. Also, a meta-analysis conducted by Roth, BeVier,

Switzer and Schippnann (1996) found that there was a correlation (.30 when

corrected). Therefore, we conclude that GPA should be considered as a measure

however, should be taken with a grain of salt due to lack of consistent/salient results

in the literature.

Letters of Recommendation

Letters of Recommendation are regarded as the most important method

determining post-graduate admissions (Fauber, 2006). However, there is inconsistent

research regarding job performance and letters of recommendation, some suggesting a

positive correlation (Aamodt, Bryan & Whitcomb, 1993), some suggesting a weak

positive relation (Kuncel, Kochevar & Aamodt Ones, 2014), and some even

suggesting a gender bias where female references were viewed in a less positive way

(Madera, Hebl, & Martin, 2009). This lack of consistency in results undermines the

reliability of having references as such an important measure in postgraduate

programs.

These current three measures, despite their lack in validity and reliability,

remain important in the selection process and are therefore included in the battery

constructed. However, to optimise selection processes, I propose that interpersonal

skills as well as psychological stability are considered in the application process.

Interpersonal Skills

Anderson and colleagues (2009) research provides supporting evidence that

interpersonal skills directly affect therapeutic success. 25 therapists (16 male; 9

female) participated, collectively with 1141 clients. After each session, they used The

Outcome Questionaire-45 (OQ-45; Lambert et al., 2005), a 45 item self-report scale,

to measure cliental outcomes. This measure has good internal-consistency as well as

test-retest reliability.

To measure the interpersonal skills of the therapist, they used The Social

Skills Inventory (SSI; Riggio, 1986) which is a 90-item self report scale that uses a 5-

point Likert scale. The SSI has high test-retest reliability as well as internal

consistency. As well as the self-report scale, the Facilitative Interpersonal Skills

Performance Task (FIS) (Anderson et al., 2009) was used. For the sake of the

experiment, a video performance task was created to purposefully elicit challenging

interpersonal situations in a clinical psychology context and responses were

evaluated. Performance tasks are more valid as they have less self-serving bias and

more face validity than self-report measures.

Results indicated that the therapists who were better at handling

interpersonally challenging encounters and who had better social skills were had

clients with a more positive outcome questionnaire result. This suggests that

interpersonal skills are and essential quality of being a successful psychologist and

should be included as a measure on the current admissions process.

Psychological Stability

The estimated prevalence of depression and anxiety amongst graduate students

is 13% (Eisenberg et. al. 2007). On top of this, individuals who are drawn specifically

to study a degree in psychology are increasingly likely to be suffering from mental

illness before they begin training (Thorne 2000). Psychological stability underpins a

clinical psychologists ability to reliably provide therapy for others (Johnson et. al.,

2004). As such the need for a measure of psychological stability needs a greater

emphasis in applications for postgraduate degrees. To test for psychological stability

the use of the Mental Health Inventory (MHI) should be optimised. The MHI has

good internal consistency as well high test-retest reliability (Heubeck & Neill, 2000).

Proposed Research

I propose the development of an assessment battery to measure proficiency of

applicants applying for postgraduate training program for clinical psychology. Current

tests entail the following: a face-to-face interview, high academic achievement, and

three letters of recommendation. Even though these elements are important in the

evaluative process they have major limitations such as (1) face-to-face interviewers

have interviewer bias, (2) academic achievement has a modest predictability over job

performance, and (3) references are not an accurate measure of character due to

possible bias and a lack of reliability. Due to these limitations, the overall test lacks

validity and reliability. To improve this issue, it will be necessary to include

assessments of interpersonal skills which will be measured via the Scale of

Interpersonal Behaviour (SIB).

The SIB is a self-report scale that has high discriminatory power, high internal

consistency as well as test-retest reliability (Bouvard et. al, 1999; Arrindell et. al.,

2004) for testing both a general population and a clinical sample. The SIB measures

(1) display of negative assertion, (2) expression of and dealing with personal

limitations, (3) assertiveness and (4) positive assertion. On top of this, and as

previously discussed, the FIB test creates a simulation of a challenging interpersonal

context and should be administered during the face-to-face interview.

To measure psychological stability, I also propose that the MHI self-report

scale should be administered.

Test Validation and Evaluation

Design

A longitudinal 3(MHI, SIB and FIB test included in current process vs. MHI,

SIB and FIB tests not included in current process vs. only MHI, SIB and FIB tested) x

1(Postgraduate outcomes) in a between-groups design.

Participants

At least 100 psychology graduates applying for a post-graduate degree will be

assessed for all measurements. They will then be randomised into 3 groups: (1) MHI,

FIB and SIB results ONLY; (2) MHI, FIB and SIB results in conjunction with current

measures; (3) assessment based on current measures ONLY. They will then be

assessed by the board of governors based only on the measures respectively and either

be accepted or declined (fake simulation). These participants will then be scrutinised

as per usual as to not interfere ethically with current selection processes.

Measures and Procedure

Before attending their face-to-face interview they will follow the usual

guidelines and provide essential paperwork including academic transcripts (GPA

proof) and letters of recommendation, which will be assessed as per usual. However,

upon arrival to their face-to-face interview, they will need to fill out a MHI and SIB

self-report scale. On top of this, they will view video-recordings of challenging client-

therapist interaction and rate the therapists response (FIB performance task). This

video will be standardised and have the same therapist communicate on five differing

levels of intrapersonal skills (poor, satisfactory, good, excellent, perfect) with a client.

The applicant will then rate these 5 therapists from greatest to poorest interpersonal
OPTIMISING POSTGRADUATE OUTCOMES FOR CLINICAL PSYCHOLOGISTS

skills. They will then have a face-to-face interview in which the interviewer will ask

standardised questions.

These 5 measures’ results will then be assessed by the board of governors.

Participants will be randomised into 3 groups of even sex and age ratio. However, sex

and age will not be revealed to the board (to eliminate age and gender bias). The

measures for the randomised groups will consist of the following: (1) MHI, FIB and

SIB results ONLY; (2) MHI, FIB and SIB results in conjunction with current

measures; (3) assessment based on current measures ONLY.

‘Fictitious’ admission success based on the three different groups will be

recorded and then the board will be free to choose admissions based on their current

processes.

The postgraduate success of the accepted applicants will be monitored over a 3

year period. During this period, outcome evaluation of patients will be assessed via

the OQ-45 scale in likeness of the Anderson (2009) study.

Reliability Criteria

If the test is reliable, I predict internal consistency (all of the measures inter-

correlate and have predictive values over future postgraduate outcomes) as well as

test-retest reliability (correlation between participants with similar scores on a test

will have similar predictive value to another participant with the same score) will both

be positive and high. Also, the results of the FIB and SIB should predict similar levels

of interpersonal skills of the therapist (to indicate parallel-forms reliability). Finally if

this battery is reliable, individuals in the board of governors should rate applicants at

the same level of approval (high inter-rater reliability). Also, replication of this

proposal must find analogous results to indicate stable and consistent results.
OPTIMISING POSTGRADUATE OUTCOMES FOR CLINICAL PSYCHOLOGISTS

Validity Criteria

If the test is valid, it will have high criterion-related validity. If there is high

criterion-related validity, then there should be a high correlation between the current

application process and the new measures. Also, if valid, it is expected that there will

be a high correlation between interpersonal skills and the SIB self-report test, the FIB

performance test as well as the face-to-face interview (as increased interpersonal

skills tends to lead to a more positive evaluation by interviewer). This will establish

the validity of the scales administered. Also, a valid study should predict that: (1) the

application process that involves all measures, including current measures, will

produce applicants with the most successful postgraduate outcomes; (2) the current

process will produce applicants with the second highest postgraduate outcomes; (3)

applicants that were only tested for interpersonal skills and psychological stability

would have the lowest predictive value over postgraduate outcomes.

Conclusions

A postgraduate clinical psychology degree remains one of the most pursued

degrees within the graduate population. As such, the need for a proficient selection

process that promotes the best future psychologists is undeniably necessary,

particularly, because a psychologist’s role in society is so crucial. Hence, a reliable

and valid measure has been proposed to include more personal assessment of the

applicants. The proposed battery will transform selection processes into a more valid

and reliable overall measure to conclusively increase the amount of proficient clinical

psychologists and hence increase stable mental health for the betterment of society.
OPTIMISING POSTGRADUATE OUTCOMES FOR CLINICAL PSYCHOLOGISTS

References

Aamodt, M., Bryan, D., & Whitcomb, A. (1993). Predicting Performance with Letters

of Recommendation. Public Personnel Management, 22(1), 81-90.

Anderson, T., Ogles,, B., Patterson, C., Lambert, M., & Vermeersch, D. (2009).

Therapist effects: facilitative interpersonal skills as a predictor of therapist

success. Journal Of Clinical Psychology,65(7), 755-768.

Arce-Ferrer, A., & Castillo, I. (2007). Investigating Postgraduate College Admission

Interviews: Generalizability Theory Reliability and Incremental Predictive

Validity. Journal Of Hispanic Higher Education, 6(2), 118-134.

Arrindell, W., Akkerman, A., van der Ende, J., Schreurs, P., Brugman, A., & Stewart,

R. et al. (2005). Normative studies with the Scale for Interpersonal Behaviour

(SIB): III. Psychiatric [Link] And Individual Differences, 38(4),

941-952.

Arrindell, W., Bridges, K., van der Ende, J., St. Lawrence, J., Gray-Shellberg, L., &

Harnish, R. et al. (2001). Normative studies with the Scale for Interpersonal

Behaviour (SIB): II. US [Link] Research And Therapy, 39(12),

1461-1479.

Bailey, M., Rosenthal, J., & Yoon, A. (2014). Grades and incentives: assessing

competing grade point average measures and postgraduate outcomes. Studies In

Higher Education, 1-15.

Bouvard, M., Arrindell, W., Guérin, J., Bouchard, C., Rion, A., & Ducottet, E. et al.

(1999). Psychometric appraisal of the scale for interpersonal behavior (SIB) in

France. Behaviour Research And Therapy, 37(8), 741-762.

Eisenberg, D., Gollust, S., Golberstein, E., & Hefner, J. (2007). Prevalence and

correlates of depression, anxiety, and suicidality among university

students. American Journal Of Orthopsychiatry, 77(4), 534-542.

Fauber, R. L. (2006). Graduate admissions in clinical psychology: Observations on

the present and thoughts on the future. Clinical Psychology: Science and

Practice, 13(3), 227-234.

Heubeck, B., & Neill, J. (2000). Confirmatory factor analysis and reliability of the

mental health inventory for Australian adolescents. Psychological Reports, 87(2),

431-440.

Hildum, D. C., & Brown, R. W. (1956). Verbal reinforcement and interviewer

bias. The Journal of Abnormal and Social Psychology, 53(1), 108.

Hofmann, W. (2005). A Meta-Analysis on the Correlation Between the Implicit

Association Test and Explicit Self-Report Measures. Personality And Social

Psychology Bulletin, 31(10), 1369-1385.

Johnson, W., & Campbell, C. (2002). Character and fitness requirements for

professional psychologists: Are there any?. Professional Psychology: Research

And Practice, 33(1), 46-53.

Johnson, W., & Campbell, C. (2004). Character and Fitness Requirements for

Professional Psychologists: Training Directors' Perspectives. Professional

Psychology: Research And Practice,35(4), 405-411.

Kuncel, N., Kochevar, R., & Ones, D. (2014). A Meta-analysis of Letters of

Recommendation in College and Graduate Admissions: Reasons for

hope. International Journal Of Selection And Assessment, 22(1), 101-107.

Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005).

Providing feedback to psychotherapists on their patients' progress: clinical results

and practice suggestions. Journal of clinical psychology, 61(2), 165-174.

Madera, J., Hebl, M., & Martin, R. (2009). Gender and letters of recommendation for

academia: Agentic and communal differences. Journal Of Applied

Psychology, 94(6), 1591-1599.

Riggio, R. E. (1986). Assessment of basic social skills. Journal of Personality and

social Psychology, 51(3), 649.

Rim, Y. (1976). How Reliable are Letters of Recommendation?. The Journal Of

Higher Education,47(4), 437

Robbins, S., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do

Psychosocial and Study Skill Factors Predict College Outcomes? A Meta-

Analysis. Psychological Bulletin, 130(2), 261-288.

Roth, P., BeVier, C., Switzer, F., & Schippmann, J. (1996). Meta-analyzing the

relationship between grades and job performance. Journal Of Applied

Psychology, 81(5), 548-556

Thorne, F. (2000). The field of clinical psychology: Past, present, and future. Journal

Of Clinical Psychology, 56(3), 257-274.