Add AUROC quartet

DESCRIPTION

@@ -12,8 +12,9 @@ Description: In the spirit of Anscombe's quartet, this package includes datasets
    assumptions about the data generating mechanism are needed when estimating 
    causal effects. The package includes "Anscombe's Quartet" (Anscombe 1973) <doi:10.1080/00031305.1973.10478966>,
    D'Agostino McGowan & Barrett (2023) "Causal Quartet" <doi:10.1080/26939169.2023.2276446>, 
-   "Datasaurus Dozen" (Matejka & Fitzmaurice 2017), "Interaction Triptych" (Rohrer & Arslan 2021) <doi:10.1177/25152459211007368>, "Rashomon Quartet" (Biecek et al. 2023) <doi:10.48550/arXiv.2302.13356>, and
-   Gelman "Variation and Heterogeneity Causal Quartets" (Gelman et al. 2023) <doi:10.48550/arXiv.2302.12878>.
+   "Datasaurus Dozen" (Matejka & Fitzmaurice 2017), "Interaction Triptych" (Rohrer & Arslan 2021) <doi:10.1177/25152459211007368>, "Rashomon Quartet" (Biecek et al. 2023) <doi:10.48550/arXiv.2302.13356>,
+   Gelman "Variation and Heterogeneity Causal Quartets" (Gelman et al. 2023) <doi:10.48550/arXiv.2302.12878>, and 
+   an AUROC quartet for binary classification problems.
 License: MIT + file LICENSE
 URL: https://github.com/r-causal/quartets,
     https://r-causal.github.io/quartets/

R/auroc_quartet.R

@@ -0,0 +1,20 @@
+#' Area under the ROC Curve (AUROC) Quartet Data
+#'
+#' This dataset contains 4 datasets, each with equivalent
+#' values for the AUROC, a popular model evaluation metric for
+#' classification problems. Each curve features a substantially
+#' different AUROC curve shape with implications for other evaluation
+#' methods such as lift, precision, and recall. 
+#' 
+#' AUROC curves are parameterized as one of two functional forms
+#' defined by `method` (piecewise linear or the Beta distribution CDF). 
+#'
+#' @references Riederer E (2026). _roctet_: . Python package version 0.1.0. 
+#'
+#' @format A dataframe with 4,000 rows and 4 variables:
+#'
+#' * `id`: The data id number within each method
+#' * `method`: The data generating mechanism ("beta" or "piecewise")
+#' * `score`: The numerical prediction score
+#' * `target`: The true binary outcome being predicted
+"auroc_quartet"

README.qmd

@@ -30,6 +30,7 @@ The quartets package is a collection of datasets aimed to help data analysis pra
 * [Interaction Triptych](#interaction-triptych)
 * [Rashomon Quartet](#rashomon-quartet)
 * [Gelman Variation and Heterogeneity Causal Quartets](#gelman-variation-and-heterogenity-causal-quartets)
+* [AUROC Quartet](#auroc-quartet)
 
 ## Installation 
 
@@ -284,6 +285,50 @@ heterogeneous_causal_quartet |>
   knitr::kable()
 ```
 
+## AUROC Quartet 
+
+Similar to the Rashomon Quartet, this dataset demonstrates that the popular AUROC model evaluation metric for predictive classification models does not tell the full story. The `auroc_quartet` dataset contains four datasets consisting of a predictive score and true target. Each score-target pair has the same AUC but the shapes of the ROC plots are materially different, leading to large differences in other fit-for-purpose evaluation metrics like precision and recall. 
+
+Data is generated from the [`roctet`](https://github.com/emilyriederer/roctet) python package.
+
+## Example
+
+```{r}
+#| message: false
+#| warning: false
+#| cache: false
+library(tidyverse)
+library(tidymodels)
+library(patchwork)
+library(quartets)
+
+auroc_quartet_prep <-
+  auroc_quartet |>
+  group_by(id) |>
+  mutate(
+    target = factor(target, levels = c(0,1), labels = c(0,1)),
+    pred_at_10 = factor(as.integer(score > quantile(score, 0.9)))
+    )
+
+gg_auroc <- autoplot(
+  roc_curve(auroc_quartet_prep, target, score, event_level = "second")
+) + labs(title = "AUROC")
+
+gg_gain <- autoplot(
+  gain_curve(auroc_quartet_prep, target, score, event_level = "second")
+) + labs(title = "Gain Curve")
+
+(gg_auroc + gg_gain)
+
+auroc_quartet_prep |>
+  summarize(
+    auroc = roc_auc_vec(target, score,event_level = "second"),
+    precision_at_10 = precision_vec(target, pred_at_10, event_level = "second"),
+    recall_at_10 = recall_vec(target, pred_at_10, event_level = "second")
+  ) |>
+  knitr::kable(digits = 2)
+```
+
 
 ## References