Skip to contents

Scale-level Content Validity Index, Universal Agreement method (S-CVI/UA)

Source: R/scvi_ua.R
scvi_ua.Rd

Computes the Scale-level Content Validity Index using the universal agreement method, defined as the proportion of items where all experts rate the item as relevant.

Usage

scvi_ua(
  ratings,
  relevant_threshold = 3,
  na.rm = FALSE,
  ci = FALSE,
  n_boot = 2000,
  ci_method = c("percentile", "bca"),
  conf_level = 0.95,
  seed = NULL
)

Arguments

ratings

A numeric matrix or data frame of expert ratings (rows = experts, columns = items) on a relevance scale.

relevant_threshold

Integer. Minimum rating considered "relevant". Defaults to 3.

na.rm

Logical. If TRUE, missing ratings are ignored when checking universal agreement. Defaults to FALSE.

ci

Logical. If TRUE, returns a data frame with a bootstrap confidence interval alongside the point estimate. Defaults to FALSE (returns a single numeric value, identical to the package's pre-0.2.0 behaviour).

n_boot

Integer. Number of bootstrap replicates when ci = TRUE. Defaults to 2000 (Davison & Hinkley, 1997; Hesterberg, 2015).

ci_method

Character. One of "percentile" (default; Efron & Tibshirani, 1993) or "bca" (bias-corrected and accelerated; DiCiccio & Efron, 1996).

conf_level

Numeric. Confidence level between 0 and 1. Defaults to 0.95.

seed

Integer or NULL. If supplied, passed to set.seed() for reproducible bootstrap samples. Defaults to NULL.

Value

When ci = FALSE (default), a single numeric value: the proportion of items with universal agreement. When ci = TRUE, a one-row data frame with columns item (set to "scale"), scvi_ua, ci_lower, ci_upper, ci_method, conf_level, n_boot.

Details

Optional bootstrap confidence intervals are available via ci = TRUE. Resampling is performed at the expert (row) level, matching the standard inferential frame for inter-rater reliability analyses (Gwet, 2014).

S-CVI/UA is a stricter criterion than S-CVI/Ave and tends to produce lower values, especially with larger expert panels. Polit and Beck (2006) recommend reporting both indices together. With small panels of 3-5 experts, S-CVI/UA >= 0.80 is often considered acceptable.

References

Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489-497. doi:10.1002/nur.20147

Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge University Press. doi:10.1017/CBO9780511802843

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189-228. doi:10.1214/ss/1032280214

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman and Hall. doi:10.1201/9780429246593

Gwet, K. L. (2014). Handbook of inter-rater reliability (4th ed.). Advanced Analytics, LLC.

Hesterberg, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. The American Statistician, 69(4), 371-386. doi:10.1080/00031305.2015.1089789

See also

icvi(), scvi_ave()

Examples

ratings <- matrix(
  c(4, 4, 3, 4, 4,
    3, 4, 4, 4, 3,
    2, 3, 3, 4, 3,
    1, 2, 3, 2, 3),
  nrow = 5
)
scvi_ua(ratings)
#> [1] 0.5

# With bootstrap confidence interval (new in v0.2.0)
scvi_ua(ratings, ci = TRUE, n_boot = 1000, seed = 1)
#>    item scvi_ua ci_lower ci_upper  ci_method conf_level n_boot
#> 1 scale     0.5      0.5     0.75 percentile       0.95   1000