Skip to content

docs+fix: ecarts doc/code resolus, read_rules parle a l'editeur de YAML #55

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
VincentGuyader merged 1 commit into from
Jul 3, 2026
Merged

docs+fix: ecarts doc/code resolus, read_rules parle a l'editeur de YAML #55

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,26 @@
projection (`paste0(character(0), "__ok")` yields `"__ok"`) that `any_of()`
had been eating since 0.4.8 (issue #17).

## Documentation and input friction

* Doc-vs-code drift resolved (issue #32): `@return` of
`compare_datasets_from_yaml()` now lists all 8 elements including
`all_passed` (the first one) and describes `agent` truthfully (configured,
NOT interrogated - `reponse` is); the README Quick Start shows the
zero-configuration mode first and passes `key` explicitly everywhere; the
README "Dev part" no longer embeds check/coverage outputs that go stale
(it froze results from 0.4.2); the vignette documents the local-only
restriction of the `__absdiff`/`__thresh` extract diagnostics and the
DuckDB-only NaN caveat of the lazy booleans.

* `read_rules()` speaks to the person who edits the YAML by hand: an
unsupported `version` gets an explicit error naming the file, the declared
and the supported versions (was a raw `stopifnot` output), and unknown
top-level or `defaults` fields (typos like `by_nmae:` or `no_equal:`) get a
warning naming them and the known fields, instead of being silently
ignored. `write_rules_template()` rejects a `version` other than 1 upfront
(it happily wrote version 2 templates that `read_rules()` then refused).

## Bug fixes

* Three NEWS-vs-code contradictions introduced by the "Perf (#1)" commit after
Expand Down
61 changes: 59 additions & 2 deletions R/compare_datasets_from_yaml.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@ write_rules_template <- function(data_reference,
datetime_equal_mode = "exact",
logical_equal_mode = "exact") {

if (!identical(suppressWarnings(as.numeric(version)), 1)) {
stop(sprintf(
"Parameter 'version' must be 1 (the only rules version read_rules() accepts), got %s.",
version
), call. = FALSE)
}

# Validate key if provided
# Use a 0-row collect to retrieve column names and types: works for both local
# data.frames and lazy tables (tbl_lazy) without loading all rows.
Expand Down Expand Up @@ -122,7 +129,53 @@ write_rules_template <- function(data_reference,
#' @export
read_rules <- function(path) {
r <- read_yaml(path)
stopifnot(is.list(r), !is.null(r$version), r$version == 1)
if (!is.list(r)) {
stop(sprintf("'%s' does not contain a YAML mapping of rules.", path),
call. = FALSE)
}
version_ok <- is.atomic(r$version) &&
length(r$version) == 1 &&
identical(suppressWarnings(as.numeric(r$version)), 1)
if (!version_ok) {
shown <- if (is.atomic(r$version) && length(r$version) == 1) {
as.character(r$version)
} else if (is.null(r$version)) {
"<missing>"
} else {
# lists / vectors from YAML like [1, 2] or {v: 1}: printable summary
paste(class(r$version)[1], "of length", length(r$version))
}
stop(sprintf(
"'%s' declares rules version %s; this version of {datadiff} supports version 1 only.",
path, shown
), call. = FALSE)
}
# Hand-edited YAML again: a scalar where a mapping is expected
# (defaults: yes) would crash far away on a $ access; refuse it here
for (section in c("defaults", "by_type", "by_name", "row_validation")) {
if (!is.null(r[[section]]) && !is.list(r[[section]])) {
stop(sprintf(
"'%s': the '%s' section must be a YAML mapping (key: value lines), not a single value.",
path, section
), call. = FALSE)
}
}
# A hand-edited YAML is the main input of this package: flag the fields the
# comparison will silently ignore (typos like by_nmae or no_equal)
known_top <- c("version", "defaults", "by_type", "by_name", "row_validation")
unknown_top <- setdiff(names(r), known_top)
known_defaults <- c("na_equal", "ignore_columns", "keys", "key", "label")
unknown_defaults <- setdiff(names(r$defaults %||% list()), known_defaults)
unknown <- c(unknown_top, unknown_defaults)
Comment on lines +165 to +169
if (length(unknown) > 0) {
warning(sprintf(
"'%s' contains field(s) the comparison will ignore: %s. Check for typos (known top-level fields: %s; known defaults fields: %s).",
path,
paste(sprintf("'%s'", unknown), collapse = ", "),
paste(known_top, collapse = ", "),
paste(known_defaults, collapse = ", ")
), call. = FALSE)
}
r$defaults <- r$defaults %||% list()
r$by_type <- r$by_type %||% list()
r$by_name <- r$by_name %||% list()
Expand Down Expand Up @@ -292,7 +345,11 @@ validate_comparison_key <- function(key, ref_cols, cand_cols) {
#' effect when Arrow datasets are used: plain `data.frame`s or `tbl_lazy`
#' inputs ignore a valid value.
#' @return A list containing:
#' \item{agent}{Configured pointblank agent with validation results}
#' \item{all_passed}{Logical; \code{TRUE} when every check passed. The
#' first element of the returned list and the single verdict consumers
#' should read.}
#' \item{agent}{The configured pointblank agent as built (NOT interrogated:
#' the verdict lives in \code{reponse}, which is the interrogated one)}
#' \item{reponse}{Interrogated pointblank agent (class \code{datadiff_report}):
#' usable by \code{pointblank::all_passed()} / \code{get_data_extracts()};
#' printing it lazily renders a full pointblank-style report from
Expand Down
37 changes: 14 additions & 23 deletions README.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,20 @@ candidate <- data.frame(
category = c("a", "b", "c") # Different case
)

# Generate a rules template
# Zero configuration: near-exact rules generated from the reference structure
result <- compare_datasets_from_yaml(reference, candidate, key = "id")
result$all_passed # FALSE here: exact rules reject the differences above

# To tune tolerances, generate a rules template and edit it
rules_path <- tempfile(fileext = ".yaml")
write_rules_template(reference, key = "id",
path = "validation_rules.yaml",
path = rules_path,
numeric_abs = 0.01,
character_case_insensitive = TRUE)

# Edit the rules file to configure validation (e.g., set tolerances, case sensitivity)

# Compare datasets
result <- compare_datasets_from_yaml(reference, candidate, path = "validation_rules.yaml")
# Compare datasets with the tuned rules (key stays explicit)
result <- compare_datasets_from_yaml(reference, candidate,
key = "id", path = rules_path)

result$all_passed # overall verdict (TRUE/FALSE)
result$coverage # one row per check performed: column, type, n, n_failed, status
Expand Down Expand Up @@ -401,24 +405,11 @@ MIT License

## Dev part

This `README` has been compiled on the

```{r}
Sys.time()
```

Here are the test & coverage results:
Run the checks and the coverage locally (embedded outputs went stale as the
package evolved, so this README no longer freezes them at compile time):

```{r error = TRUE}
devtools::check(quiet = TRUE)
```

```{r echo = FALSE}
unloadNamespace("datadiff")
```

```{r error = TRUE}
Sys.setenv("NOT_CRAN" = TRUE)
```r
devtools::check()
covr::package_coverage()
```

Expand Down
105 changes: 42 additions & 63 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,20 @@ candidate <- data.frame(
category = c("a", "b", "c") # Different case
)

# Generate a rules template
# Zero configuration: near-exact rules generated from the reference structure
result <- compare_datasets_from_yaml(reference, candidate, key = "id")
result$all_passed # FALSE here: exact rules reject the differences above

# To tune tolerances, generate a rules template and edit it
rules_path <- tempfile(fileext = ".yaml")
write_rules_template(reference, key = "id",
path = "validation_rules.yaml",
path = rules_path,
numeric_abs = 0.01,
character_case_insensitive = TRUE)

# Edit the rules file to configure validation (e.g., set tolerances, case sensitivity)

# Compare datasets
result <- compare_datasets_from_yaml(reference, candidate, path = "validation_rules.yaml")
# Compare datasets with the tuned rules (key stays explicit)
result <- compare_datasets_from_yaml(reference, candidate,
key = "id", path = rules_path)

result$all_passed # overall verdict (TRUE/FALSE)
result$coverage # one row per check performed: column, type, n, n_failed, status
Expand Down Expand Up @@ -142,13 +146,13 @@ by_name:
`by_name` taking precedence. A column not listed in `by_name` uses its
`by_type` defaults unchanged.

| Column | Effective rule | Source |
|----|----|----|
| `id` | `abs: 0` | `by_type.integer` |
| `amount` | `abs: 0.01, rel: 0` | `by_name.amount` overrides `by_type.numeric` |
| `unit_price` | `abs: 0.001, rel: 0.0001` | `by_name.unit_price` overrides `by_type.numeric` |
| `category` | `case_insensitive: yes, trim: yes` | `by_name.category` overrides `by_type.character` |
| `created_at` | `equal_mode: exact` | `by_type.date` (by_name is redundant here) |
| Column | Effective rule | Source |
|--------------|------------------------------------|--------------------------------------------------|
| `id` | `abs: 0` | `by_type.integer` |
| `amount` | `abs: 0.01, rel: 0` | `by_name.amount` overrides `by_type.numeric` |
| `unit_price` | `abs: 0.001, rel: 0.0001` | `by_name.unit_price` overrides `by_type.numeric` |
| `category` | `case_insensitive: yes, trim: yes` | `by_name.category` overrides `by_type.character` |
| `created_at` | `equal_mode: exact` | `by_type.date` (by_name is redundant here) |

### Numeric tolerance: formula, edge effects, and best practices

Expand All @@ -162,10 +166,10 @@ threshold**:

The two parameters **add up**: they are not two independent guards.

| Parameter | Role | Default value |
|----|----|----|
| `abs` | Absolute tolerance: fixed floor, independent of the magnitude of values | `1e-9` |
| `rel` | Relative tolerance: fraction of the reference value added to the threshold | `0` |
| Parameter | Role | Default value |
|-----------|----------------------------------------------------------------------------|---------------|
| `abs` | Absolute tolerance: fixed floor, independent of the magnitude of values | `1e-9` |
| `rel` | Relative tolerance: fraction of the reference value added to the threshold | `0` |

#### Pure absolute mode (recommended by default)

Expand Down Expand Up @@ -201,11 +205,11 @@ by_type:
rel: 0.01 # tolerance of 1% of the reference value
```

| Reference | Candidate | Difference | Threshold (1%) | Result |
|----|---:|---:|---:|----|
| 100.00 | 100.50 | 0.50 | 1.00 | OK |
| 1 000 000.00 | 1 005 000.00 | 5 000 | 10 000 | OK |
| 0.00 | 0.001 | 0.001 | **0** | **ERROR** (implicit division by zero) |
| Reference | Candidate | Difference | Threshold (1%) | Result |
|--------------|-------------:|-----------:|---------------:|---------------------------------------|
| 100.00 | 100.50 | 0.50 | 1.00 | OK |
| 1 000 000.00 | 1 005 000.00 | 5 000 | 10 000 | OK |
| 0.00 | 0.001 | 0.001 | **0** | **ERROR** (implicit division by zero) |

> **Warning**: if the reference value is `0`, the relative threshold is
> `0`, so any difference, even tiny, will be detected as an error.
Expand Down Expand Up @@ -286,15 +290,15 @@ row_validation:
This validates that the candidate dataset has between 950 and 1050 rows
(1000 +/- 50).

If `expected_count` is not specified, the reference dataset's row count
If `expected_count` is not specified, the reference dataset’s row count
is used as the expected value.

## Comparing Parquet files (large datasets)

`datadiff` can compare Parquet files that are too large to fit in RAM.
The recommended approach uses `arrow::open_dataset()` - **do not call
`arrow::to_duckdb()` yourself** before passing to `datadiff`; the
package handles the Arrow -> DuckDB conversion internally with a single
package handles the Arrow -\> DuckDB conversion internally with a single
connection.

### Recommended strategy: Arrow Dataset (lazy, out-of-core)
Expand Down Expand Up @@ -322,12 +326,12 @@ Internally, `compare_datasets_from_yaml()`:

1. Opens a private DuckDB connection (`fresh_con`).
2. Materialises each Parquet dataset as a DuckDB physical temp table
via `read_parquet()` - all memory is managed by DuckDB's buffer
via `read_parquet()` - all memory is managed by DuckDB’s buffer
pool, so disk spilling works correctly.
3. Runs the full join + 125 boolean expressions as a single lazy SQL
query.
4. `dplyr::compute()` materialises only the slim boolean result table
(~125 logical columns * N rows) - the wide source data is never
(~125 logical columns \* N rows) - the wide source data is never
loaded into R.
5. `dplyr::collect()` brings the slim boolean table (~few GB) into R
and passes a plain `data.frame` to pointblank - no live DuckDB
Expand All @@ -336,7 +340,7 @@ Internally, `compare_datasets_from_yaml()`:

### Memory tuning: `duckdb_memory_limit`

DuckDB's default memory cap (80 % of total RAM) can leave insufficient
DuckDB’s default memory cap (80 % of total RAM) can leave insufficient
headroom when R, Arrow, and the OS are already using significant memory.
The `duckdb_memory_limit` parameter controls how much RAM DuckDB may use
before spilling intermediate results to `tempdir()`:
Expand Down Expand Up @@ -365,12 +369,12 @@ limit only applies when Arrow datasets are used - it has no effect for

### Strategy comparison

| Strategy | Input type | RAM usage | Requires |
|----|----|----|----|
| **Arrow Dataset** ✅ recommended | `arrow::open_dataset()` | Slim boolean table only (~few GB) | arrow, duckdb |
| Arrow Table | `arrow::read_parquet(as_data_frame=FALSE)` | Same as Arrow Dataset | arrow, duckdb |
| Lazy table (dbplyr) | `tbl(con, "table_name")` | Slim boolean table only | DBI, dbplyr |
| `data.frame` | `read.csv()`, `readr::read_csv()`, etc. | Full data in RAM | - |
| Strategy | Input type | RAM usage | Requires |
|----------------------------------|--------------------------------------------|-----------------------------------|---------------|
| **Arrow Dataset** ✅ recommended | `arrow::open_dataset()` | Slim boolean table only (~few GB) | arrow, duckdb |
| Arrow Table | `arrow::read_parquet(as_data_frame=FALSE)` | Same as Arrow Dataset | arrow, duckdb |
| Lazy table (dbplyr) | `tbl(con, "table_name")` | Slim boolean table only | DBI, dbplyr |
| `data.frame` | `read.csv()`, `readr::read_csv()`, etc. | Full data in RAM | \- |

### What NOT to do

Expand All @@ -389,7 +393,7 @@ ref_df <- dplyr::collect(ds_ref) # loads full 4 GB into R RAM

DuckDB spills to `tempdir()` when the memory limit is reached. On
Windows this is typically `C:\Users\<user>\AppData\Local\Temp`. Ensure
that directory has sufficient free disk space (up to ~2-3* the size of
that directory has sufficient free disk space (up to ~2-3\* the size of
your Parquet files in the worst case).

## Main Functions
Expand Down Expand Up @@ -421,36 +425,11 @@ MIT License

## Dev part

This `README` has been compiled on the

``` r
Sys.time()
#> [1] "2026-03-11 18:07:04 CET"
```

Here are the test & coverage results:

``` r
devtools::check(quiet = TRUE)
#> ℹ Loading datadiff
#> ── R CMD check results ───────────────────────────────────── datadiff 0.4.2 ────
#> Duration: 4m 47.4s
#>
#> ❯ checking for future file timestamps ... NOTE
#> unable to verify current time
#>
#> 0 errors ✔ | 0 warnings ✔ | 1 note ✖
```
Run the checks and the coverage locally (embedded outputs went stale as
the package evolved, so this README no longer freezes them at compile
time):

``` r
Sys.setenv("NOT_CRAN" = TRUE)
devtools::check()
covr::package_coverage()
#> datadiff Coverage: 98.98%
#> R/preprocessing.R: 97.22%
#> R/compare_datasets_from_yaml.R: 98.58%
#> R/data_types.R: 100.00%
#> R/pointblank_setup.R: 100.00%
#> R/tolerance.R: 100.00%
#> R/utils.R: 100.00%
#> R/validation.R: 100.00%
```
6 changes: 5 additions & 1 deletion man/compare_datasets_from_yaml.Rd
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading