0

Calculating Inter-Rater Rank Order Correlation

Sun 1 Sep, 2013 11:40 am
I'm trying to calculate an inter-rater reliability statistic for the results of a rank ordering exercise that was already completed previously. I thought I knew how to do this myself, but now - maybe not so much.

These were the raw number results from the rank ordering, along with the rank order averages, which would indicate the preferred rank order of the options.

So, while the rank order seems pretty clear, I would like to calculate an inter-rater rank order correlation coefficient - especially since it appears as though there are some pretty high inter-rater reliabilities. However, I'm not sure how to calculate this.

I'm not entirely a novice when it comes to statistics, so I do have some background and capability. However, I've researched what I can, and I think that I should be using the Spearman rank order correlation coefficient, but beyond that - I'm lost.

I've tried to provide as much of the raw data as possible, in case someone is highly-motivated and wants to take the raw data to the actual answer. However, I'll be equally happy with just guidance/formula (and maybe the adaptation in Excel), and I'll take it the final step myself. Thanks!
• Topic Stats
• Top Replies
Type: Question • Score: 0 • Views: 2,345 • Replies: 3
No top replies

JPB

1
Sun 1 Sep, 2013 12:45 pm
Take a look at Kendall's W statistic. I think this may be more in line with what you need.

http://en.wikipedia.org/wiki/Kendall's_W
CDobyns

1
Sun 20 Oct, 2013 04:42 pm
@JPB,
I thought I would revisit this inquiry, especially since the suggestion to use the Kendall W test of concordance between rank (for more than two raters) appeared to be the correct statistical treatment, to obtain the correlation coefficient. This was the original rater input, which was made easier through the use of this hugely helpful on-line Kendall W program.

The Kendallâ€™s W for agreement in rank ordering resulted in an overall correlation coefficient of: 0.9813 and with p<0.01, and a r2 value of 0.9629.

This was good advice, which I'm not sure I would have gotten to on my own. Thanks!
JPB

1
Sun 20 Oct, 2013 08:59 pm
@CDobyns,
You're very welcome. Happy to help.
0 Replies

Related Topics

1 in 5 women get raped? - Discussion by maxdancona
How will you die? - Discussion by Robert Gentel
Statistics - probability - Question by jrl06
Sampling formula - Question by Moses mbeya

1. Forums
2. » Calculating Inter-Rater Rank Order Correlation