Gwet's AC2 - weighted chance-corrected agreement for ordinal ratings

Computes Gwet's AC2 coefficient (Gwet, 2008, 2014) for ordinal ratings, which generalizes AC1 (see gwet_ac1()) to the case where rating categories are ordered and partial agreement between adjacent categories should count. Where AC1 dichotomizes ratings before computing chance- corrected agreement, AC2 preserves the full ordinal information through a weight matrix that assigns higher weights to pairs of ratings that are close together (e.g., a rating of 3 and 4) and lower weights to pairs that are far apart (e.g., 1 and 4).

Usage

gwet_ac2(
  ratings,
  weights = c("quadratic", "linear", "identity"),
  categories = NULL,
  na.rm = FALSE,
  ci = FALSE,
  n_boot = 2000,
  ci_method = c("percentile", "bca"),
  conf_level = 0.95,
  seed = NULL
)

Arguments

ratings: A numeric matrix or data frame of expert ratings (rows = experts, columns = items). A numeric vector is also accepted, treated as a single item.
weights: One of "quadratic" (default), "linear", "identity", or a custom $q \times q$ numeric weight matrix. Quadratic weights emphasize closeness between rating categories more strongly than linear weights. Identity weights reduce AC2 to AC1 on the raw (non-dichotomized) categories.
categories: Numeric vector of all possible rating values. Strongly recommended for content-validity work, where some categories may not appear in a given dataset. If NULL (the default), categories are inferred from the observed ratings, which can silently produce incorrect AC2 values when extreme categories are unused. See Details.
na.rm: Logical. If TRUE, missing ratings are excluded when counting experts on a per-item basis. Defaults to FALSE.
ci: Logical. If TRUE, returns a data frame with bootstrap confidence intervals alongside the point estimate. Defaults to FALSE.
n_boot: Integer. Number of bootstrap replicates when ci = TRUE. Defaults to 2000 (Davison & Hinkley, 1997; Hesterberg, 2015).
ci_method: Character. One of "percentile" (default; Efron & Tibshirani, 1993) or "bca" (bias-corrected and accelerated; DiCiccio & Efron, 1996).
conf_level: Numeric. Confidence level between 0 and 1. Defaults to 0.95.
seed: Integer or NULL. If supplied, passed to set.seed() for reproducible bootstrap samples. Defaults to NULL.

Value

When ci = FALSE (default), a named numeric vector of AC2 values, one per item (or a single numeric value if ratings is a vector). When ci = TRUE, a data frame with columns item, gwet_ac2, ci_lower, ci_upper, ci_method, conf_level, n_boot.

Details

Optional bootstrap confidence intervals are available via ci = TRUE. Resampling is performed at the expert (row) level, matching the standard inferential frame for inter-rater reliability analyses (Gwet, 2014).

For a single item with N experts whose ratings populate the q-category counts $n_k$ ($k = 1, \ldots, q$) and weight matrix $W = (w_{kl})$:

$$p_a = \sum_k n_k (n_k^W - 1) / [N (N - 1)]$$

where $n_k^W = \sum_l w_{kl} n_l$ is the weighted count for category k. Chance agreement uses Gwet's marginal-adjusted null:

$$p_e = T_w \sum_k \pi_k (1 - \pi_k)$$

with $T_w = \sum_{k,l} w_{kl} / [q (q - 1)]$ and $\pi_k = n_k / N$. The coefficient is $\mathrm{AC2} = (p_a - p_e) / (1 - p_e)$.

This implementation reproduces the formulas used by the irrCAC package (by Kilem Gwet, the original author of AC1/AC2) so that AC2 values from this function are bit-for-bit equivalent to those from gwet.ac1.raw() from irrCAC on the same data with the same weight matrix and category list.

Quadratic and linear weights are computed as in Gwet (2014):

$$w^{quad}_{kl} = 1 - (c_k - c_l)^2 / (c_q - c_1)^2$$ $$w^{lin}_{kl} = 1 - |c_k - c_l| / |c_q - c_1|$$

where $c_1, \ldots, c_q$ are the (sorted) category values.

Important: the categories argument should typically be set explicitly to the full theoretical rating scale (e.g., categories = 1:4 for a standard relevance scale), not left at NULL. If a particular item's ratings happen to use only a subset of categories (e.g., all experts rated 3 or 4), the default category-inference logic will produce a smaller weight matrix and substantially different AC2 values. This caveat matches the documented behavior of gwet.ac1.raw() from the irrCAC package.

References

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. doi:10.1348/000711006X126600

Gwet, K. L. (2014). Handbook of inter-rater reliability (4th ed.). Advanced Analytics, LLC.

Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13(1), 61. doi:10.1186/1471-2288-13-61

Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge University Press. doi:10.1017/CBO9780511802843

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189-228. doi:10.1214/ss/1032280214

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman and Hall. doi:10.1201/9780429246593

Hesterberg, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. The American Statistician, 69(4), 371-386. doi:10.1080/00031305.2015.1089789

Examples

# Standard 4-point relevance scale, 5 experts on 4 items
ratings <- matrix(
  c(4, 4, 3, 4, 4,
    3, 4, 4, 4, 3,
    2, 3, 3, 4, 3,
    1, 2, 3, 2, 3),
  nrow = 5,
  dimnames = list(NULL, paste0("item", 1:4))
)

# Quadratic weights are the default and most common choice for
# ordinal data. Pass the full rating scale explicitly.
gwet_ac2(ratings, categories = 1:4)
#>     item1     item2     item3     item4 
#> 0.9357602 0.8760331 0.7588424 0.5945946 

# Linear weights are an alternative
gwet_ac2(ratings, weights = "linear", categories = 1:4)
#>     item1     item2     item3     item4 
#> 0.8224852 0.6808511 0.5275591 0.3362832 

# With bootstrap confidence intervals
gwet_ac2(ratings, categories = 1:4, ci = TRUE,
         n_boot = 1000, seed = 1)
#>    item  gwet_ac2  ci_lower ci_upper  ci_method conf_level n_boot
#> 1 item1 0.9357602 0.8760331        1 percentile       0.95   1000
#> 2 item2 0.8760331 0.8760331        1 percentile       0.95   1000
#> 3 item3 0.7588424 0.5020502        1 percentile       0.95   1000
#> 4 item4 0.5945946 0.4208494        1 percentile       0.95   1000