Calculating Inter-Rater Rank Order Correlation

Reply Sun 1 Sep, 2013 11:40 am
I'm trying to calculate an inter-rater reliability statistic for the results of a rank ordering exercise that was already completed previously. I thought I knew how to do this myself, but now - maybe not so much.

These were the raw number results from the rank ordering, along with the rank order averages, which would indicate the preferred rank order of the options.



So, while the rank order seems pretty clear, I would like to calculate an inter-rater rank order correlation coefficient - especially since it appears as though there are some pretty high inter-rater reliabilities. However, I'm not sure how to calculate this.

I'm not entirely a novice when it comes to statistics, so I do have some background and capability. However, I've researched what I can, and I think that I should be using the Spearman rank order correlation coefficient, but beyond that - I'm lost.

I've tried to provide as much of the raw data as possible, in case someone is highly-motivated and wants to take the raw data to the actual answer. However, I'll be equally happy with just guidance/formula (and maybe the adaptation in Excel), and I'll take it the final step myself. Thanks!
  • Topic Stats
  • Top Replies
  • Link to this Topic
Type: Question • Score: 0 • Views: 2,197 • Replies: 3
No top replies

Reply Sun 1 Sep, 2013 12:45 pm
Take a look at Kendall's W statistic. I think this may be more in line with what you need.

Reply Sun 20 Oct, 2013 04:42 pm
I thought I would revisit this inquiry, especially since the suggestion to use the Kendall W test of concordance between rank (for more than two raters) appeared to be the correct statistical treatment, to obtain the correlation coefficient. This was the original rater input, which was made easier through the use of this hugely helpful on-line Kendall W program.


The Kendall’s W for agreement in rank ordering resulted in an overall correlation coefficient of: 0.9813 and with p<0.01, and a r2 value of 0.9629.

This was good advice, which I'm not sure I would have gotten to on my own. Thanks!
Reply Sun 20 Oct, 2013 08:59 pm
You're very welcome. Happy to help.
0 Replies

Related Topics

  1. Forums
  2. » Calculating Inter-Rater Rank Order Correlation
Copyright © 2023 MadLab, LLC :: Terms of Service :: Privacy Policy :: Page generated in 0.03 seconds on 10/03/2023 at 05:45:19