Hostname: page-component-cb9f654ff-d5ftd Total loading time: 0 Render date: 2025-08-21T07:35:42.507Z Has data issue: false hasContentIssue false
  • English
  • Français

A Note on the Linearly and Quadratically Weighted Kappa Coefficients

Published online by Cambridge University Press:  01 January 2025

Pingke Li Open the ORCID record for Pingke Li [Opens in a new window]
Pingke Li*
Affiliation:
Tsinghua University
*
Correspondence should be made to Pingke Li, Department of Industrial Engineering, Tsinghua University, Beijing, 100084, China. Email: pkli@tsinghua.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

The linearly and quadratically weighted kappa coefficients are popular statistics in measuring inter-rater agreement on an ordinal scale. It has been recently demonstrated that the linearly weighted kappa is a weighted average of the kappa coefficients of the embedded 2 by 2 agreement matrices, while the quadratically weighted kappa is insensitive to the agreement matrices that are row or column reflection symmetric. A rank-one matrix decomposition approach to the weighting schemes is presented in this note such that these phenomena can be demonstrated in a concise manner.

Keywords

inter-rater agreementweighted kapparank-one representationcomplete positivity

Information

Type
Original Paper
Information
Psychometrika , Volume 81 , Issue 3 , September 2016 , pp. 795 - 801
Copyright
Copyright 2016 © The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Berman, A., & Shaked-Monderer, N. (2003). Completely positive matrices, Singapore: World ScientificCrossRefGoogle Scholar
Brenner, H., & Kliebsch, U. (1996). Dependence of weighed kappa coefficients on the number of categories. Epidemiology, 7, 199–202.CrossRefGoogle Scholar
Cantoni, A., & Butler, P. (1976). Eigenvalues and eigenvectors of symmetric centrosymmetric matrices. Linear Algebra and its Applications, 13, 275–288.CrossRefGoogle Scholar
Cicchetti, D., & Allison, T. (1971). A new procedure for assessing reliability of scoring EEG sleep recordings. American Journal EEG Technology, 11, 101–109.CrossRefGoogle Scholar
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 213–220.CrossRefGoogle Scholar
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220.CrossRefGoogle ScholarPubMed
Fleiss, J. L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measure of reliability. Educational and Psychological Measurement, 33, 613–619.CrossRefGoogle Scholar
Kolda, T. G., & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51, 455–500.CrossRefGoogle Scholar
Schuster, C. (2004). A note on the interpretation of weighted kappa and its relation to other rater agreement statistics for metric scales. Educational and Psychological Measurement, 64, 243–253.CrossRefGoogle Scholar
Vanbelle, S. (2014). A new interpretation of the weighted kappa coefficients. Psychometrika. doi:10.1007/s11336-014-9439.CrossRefGoogle Scholar
Vanbelle, S., & Albert, A. (2009). A note on the linearly weighted kappa coefficient for ordinal scales. Statistical Methodology, 6, 157–163.CrossRefGoogle Scholar
Warrens, M. J. (2011). Cohen’s linearly weighted kappa is a weighted average of 2×2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2\times 2$$\end{document} kappas. Psychometrika, 76, 471–486.CrossRefGoogle Scholar
Warrens, M. J. (2012). Cohen’s linearly weighted kappa is a weighted average. Advances in Data Analysis and Classification, 6, 67–79.CrossRefGoogle Scholar
Warrens, M. J. (2012). Some paradoxical results for the quadratically weighted kappa. Psychometrika, 77, 315–323.CrossRefGoogle Scholar
Warrens, M. J. (2013). The Cicchetti–Allison weighting matrix is positive definite. Computational Statistics and Data Analysis, 59, 180–182.CrossRefGoogle Scholar
Warrens, M. J. (2013). Cohen’s weighted kappa with additive weights. Advances in Data Analysis and Classification, 7, 41–55.CrossRefGoogle Scholar
Warrens, M. J. (2015). Additive kappa can be increased by combining adjacent categories. International Mathematical Forum, 10, 323–328.CrossRefGoogle Scholar
Yang, J., Chinchilli, V. M. (2011). Fixed-effects modeling of Cohen’s weighted kappa for bivariate multinomial data. Computational Statistics and Data Analysis, 55, 1061–1070.CrossRefGoogle Scholar