Item-level Content Validity Index (I-CVI)

Computes the Item-level Content Validity Index (I-CVI) for one or more items rated by a panel of experts on a relevance scale. Following Lynn (1986) and Polit & Beck (2006), I-CVI is calculated as the proportion of experts who rate an item as 3 (relevant) or 4 (highly relevant) on a 4-point relevance scale.

Usage

icvi(
  ratings,
  relevant_threshold = 3,
  na.rm = FALSE,
  ci = FALSE,
  n_boot = 2000,
  ci_method = c("percentile", "bca"),
  conf_level = 0.95,
  seed = NULL
)

Arguments

ratings: A numeric matrix or data frame of expert ratings, where rows represent experts and columns represent items. Values are typically on a 1-4 relevance scale. A numeric vector is also accepted, treated as a single item.
relevant_threshold: Integer. The minimum rating considered "relevant". Defaults to 3 (i.e., ratings of 3 or 4 count as relevant on a 4-point scale).
na.rm: Logical. If TRUE, missing ratings are excluded from the calculation. Defaults to FALSE, in which case any NA produces NA for the affected item.
ci: Logical. If TRUE, returns a data frame with bootstrap confidence intervals in addition to the point estimate. Defaults to FALSE (returns a numeric vector, identical to the package's pre-0.2.0 behaviour).
n_boot: Integer. Number of bootstrap replicates when ci = TRUE. Defaults to 2000, following Davison and Hinkley (1997, ch. 5), who recommend at least 1000 replicates for stable percentile intervals, and Hesterberg (2015), who notes that 1000 is sufficient and 10,000 is ideal on modern hardware. 2000 balances stability against compute time.
ci_method: Character. One of "percentile" (default) or "bca" (bias-corrected and accelerated). Percentile (Efron & Tibshirani, 1993) respects the [0, 1] bounds of I-CVI naturally. BCa (DiCiccio & Efron, 1996) is preferred when the bootstrap distribution is skewed, which is common for I-CVI values near 1.0.
conf_level: Numeric. Confidence level between 0 and 1. Defaults to 0.95.
seed: Integer or NULL. If supplied, passed to set.seed() for reproducible bootstrap samples. Defaults to NULL.

Value

When ci = FALSE (default), a named numeric vector of I-CVI values, one per item (or a single numeric value if ratings is a vector). When ci = TRUE, a data frame with one row per item and columns item, icvi, ci_lower, ci_upper, ci_method, conf_level, n_boot.

Details

Optional bootstrap confidence intervals are available via ci = TRUE. When requested, the function resamples experts (rows) with replacement and recomputes I-CVI on each replicate. Resampling experts (rather than items) matches the standard inferential frame for inter-rater reliability analyses: experts are the random sample from a population of potential raters, while items are fixed by the study design (Gwet, 2014).

Common interpretation guidelines (Polit & Beck, 2006):

I-CVI >= 0.78: excellent content validity (with 6 or more experts).
I-CVI 0.70-0.78: acceptable, item may need revision.
I-CVI < 0.70: item should be revised or eliminated.

With fewer than six experts, Lynn (1986) recommends a stricter cutoff of I-CVI = 1.00 for unanimous agreement.

References

Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382-385. doi:10.1097/00006199-198611000-00017

Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Critique and recommendations. Research in Nursing & Health, 29(5), 489-497. doi:10.1002/nur.20147

Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their application. Cambridge University Press. doi:10.1017/CBO9780511802843

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals. Statistical Science, 11(3), 189-228. doi:10.1214/ss/1032280214

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman and Hall. doi:10.1201/9780429246593

Gwet, K. L. (2014). Handbook of inter-rater reliability (4th ed.). Advanced Analytics, LLC.

Hesterberg, T. C. (2015). What teachers should know about the bootstrap: Resampling in the undergraduate statistics curriculum. The American Statistician, 69(4), 371-386. doi:10.1080/00031305.2015.1089789

Examples

# Five experts rating four items on a 1-4 relevance scale
ratings <- matrix(
  c(4, 4, 3, 4, 4,    # Item 1
    3, 4, 4, 4, 3,    # Item 2
    2, 3, 3, 4, 3,    # Item 3
    1, 2, 3, 2, 3),   # Item 4
  nrow = 5,
  dimnames = list(NULL, paste0("item", 1:4))
)
icvi(ratings)
#> item1 item2 item3 item4 
#>   1.0   1.0   0.8   0.4 

# Single item supplied as a vector
icvi(c(4, 4, 3, 3, 4))
#> [1] 1

# Stricter threshold (only highest rating counts as relevant)
icvi(ratings, relevant_threshold = 4)
#> item1 item2 item3 item4 
#>   0.8   0.6   0.2   0.0 

# With bootstrap confidence intervals (new in v0.2.0)
set.seed(1)
icvi(ratings, ci = TRUE, n_boot = 1000)
#>    item icvi ci_lower ci_upper  ci_method conf_level n_boot
#> 1 item1  1.0      1.0      1.0 percentile       0.95   1000
#> 2 item2  1.0      1.0      1.0 percentile       0.95   1000
#> 3 item3  0.8      0.4      1.0 percentile       0.95   1000
#> 4 item4  0.4      0.0      0.8 percentile       0.95   1000

# BCa intervals, recommended when I-CVI values cluster near 1.0
icvi(ratings, ci = TRUE, ci_method = "bca", n_boot = 1000, seed = 1)
#>    item icvi ci_lower ci_upper ci_method conf_level n_boot
#> 1 item1  1.0      1.0      1.0       bca       0.95   1000
#> 2 item2  1.0      1.0      1.0       bca       0.95   1000
#> 3 item3  0.8      0.2      1.0       bca       0.95   1000
#> 4 item4  0.4      0.0      0.6       bca       0.95   1000