Skip to contents

Gwet's AC1 - chance-corrected agreement

Source: R/gwet_ac1.R
gwet_ac1.Rd

Computes Gwet's AC1 coefficient (Gwet, 2008) for each item rated by an expert panel on a relevance scale. AC1 is a chance-corrected agreement index that uses a marginal-adjusted null model: chance agreement is computed under the assumption that each expert rates "relevant" with probability equal to the observed marginal proportion. This is methodologically distinct from the modified kappa of Polit, Beck, and Owen (2007), which uses a fixed null (each expert independently rates relevant with probability 0.5). The two indices can therefore yield substantively different answers for the same data, particularly when the prevalence of "relevant" ratings is far from 0.5 (the typical case in content-validity work). Reporting both – alongside I-CVI – gives a more complete picture of inter-rater agreement than any single index. Wongpakaran et al. (2013, BMC Medical Research Methodology) recommended AC1 over Cohen's traditional kappa for high-prevalence rating contexts.

Usage

gwet_ac1(
  ratings,
  relevant_threshold = 3,
  na.rm = FALSE,
  ci = FALSE,
  n_boot = 2000,
  ci_method = c("percentile", "bca"),
  conf_level = 0.95,
  seed = NULL
)

Arguments

ratings

A numeric matrix or data frame of expert ratings (rows = experts, columns = items). A numeric vector is also accepted, treated as a single item.

relevant_threshold

Integer. Minimum rating considered "relevant". Ratings are dichotomized at this threshold before AC1 is computed, following standard practice in content-validity work (Polit, Beck, & Owen, 2007). Defaults to 3.

na.rm

Logical. If TRUE, missing ratings are excluded when counting experts. Defaults to FALSE.

ci

Logical. If TRUE, returns a data frame with bootstrap confidence intervals alongside the point estimate. Defaults to FALSE.

n_boot

Integer. Number of bootstrap replicates when ci = TRUE. Defaults to 2000 (Davison & Hinkley, 1997; Hesterberg, 2015).

ci_method

Character. One of "percentile" (default; Efron & Tibshirani, 1993) or "bca" (bias-corrected and accelerated; DiCiccio & Efron, 1996).

conf_level

Numeric. Confidence level between 0 and 1. Defaults to 0.95.

seed

Integer or NULL. If supplied, passed to set.seed() for reproducible bootstrap samples. Defaults to NULL.

Value

When ci = FALSE (default), a named numeric vector of AC1 values, one per item (or a single numeric value if ratings is a vector). When ci = TRUE, a data frame with columns item, gwet_ac1, ci_lower, ci_upper, ci_method, conf_level, n_boot.

Details

Optional bootstrap confidence intervals are available via ci = TRUE. Resampling is performed at the expert (row) level, matching the standard inferential frame for inter-rater reliability analyses (Gwet, 2014).

The formula is:

$$\mathrm{AC1} = (p_a - p_e) / (1 - p_e)$$

For a single item with N experts of whom \(n_R\) rate as relevant:

$$p_a = [n_R(n_R - 1) + (N - n_R)(N - n_R - 1)] / [N(N - 1)]$$ $$p_e = 2 \pi (1 - \pi), \quad \pi = n_R / N$$

This is Gwet's binary-rating form (Gwet, 2008, equation 5). The chance agreement term \(p_e = 2\pi(1-\pi)\) is maximised at 0.5 when \(\pi = 0.5\) and approaches zero as \(\pi\) approaches either extreme.

Note that the "kappa paradox" (Feinstein & Cicchetti, 1990) and the Wongpakaran et al. (2013) comparison both refer to Cohen's kappa, whose chance-agreement term \(\pi^2 + (1 - \pi)^2\) approaches 1 at the prevalence extremes. The modified kappa of Polit et al. (2007), implemented in this package as mod_kappa(), uses a different chance-correction (\(C(N, A) \times 0.5^N\), a fixed binomial null) and does not behave like Cohen's kappa under high prevalence. The practical consequence is that mod_kappa and AC1 typically diverge when prevalence is far from 0.5 – modified kappa approaches I-CVI while AC1 discounts more of the observed agreement as prevalence-driven. Both are defensible; they answer different questions about chance.

Common interpretation cutoffs follow Altman (1991), as adapted to AC1 by Wongpakaran et al. (2013):

  • AC1 < 0.20: poor

  • AC1 0.20-0.39: fair

  • AC1 0.40-0.59: moderate

  • AC1 0.60-0.80: good

  • AC1 > 0.80: very good

(Boundary values fall in the higher tier, matching the classifier used by apa_table() with interpretation_index = "gwet_ac1".)

References

Altman, D. G. (1991). Practical statistics for medical research. Chapman and Hall.

Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543-549. doi:10.1016/0895-4356(90)90158-L

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. doi:10.1348/000711006X126600

Gwet, K. L. (2014). Handbook of inter-rater reliability (4th ed.). Advanced Analytics, LLC.

Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459-467. doi:10.1002/nur.20199

Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13(1), 61. doi:10.1186/1471-2288-13-61

Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge University Press. doi:10.1017/CBO9780511802843

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189-228. doi:10.1214/ss/1032280214

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman and Hall. doi:10.1201/9780429246593

Hesterberg, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. The American Statistician, 69(4), 371-386. doi:10.1080/00031305.2015.1089789

See also

mod_kappa(), icvi()

Examples

ratings <- matrix(
  c(4, 4, 3, 4, 4,    # 5 of 5 relevant
    3, 4, 4, 4, 3,    # 5 of 5 relevant
    2, 3, 3, 4, 3,    # 4 of 5 relevant
    1, 2, 3, 2, 3),   # 2 of 5 relevant
  nrow = 5,
  dimnames = list(NULL, paste0("item", 1:4))
)
gwet_ac1(ratings)
#>      item1      item2      item3      item4 
#>  1.0000000  1.0000000  0.4117647 -0.1538462 

# Compare with modified kappa to see Gwet's advantage at extremes
mod_kappa(ratings)
#>     item1     item2     item3     item4 
#> 1.0000000 1.0000000 0.7629630 0.1272727 

# With bootstrap confidence intervals
gwet_ac1(ratings, ci = TRUE, n_boot = 1000, seed = 1)
#>    item   gwet_ac1   ci_lower ci_upper  ci_method conf_level n_boot
#> 1 item1  1.0000000  1.0000000        1 percentile       0.95   1000
#> 2 item2  1.0000000  1.0000000        1 percentile       0.95   1000
#> 3 item3  0.4117647 -0.1538462        1 percentile       0.95   1000
#> 4 item4 -0.1538462 -0.1538462        1 percentile       0.95   1000