Snapshot Failures Found During R CMD Check Due to New ggplot2 Version

[Andrew-Acker125]

Hello,

We've been running R CMD check for packages on our package registry manager PRISM and noticed that cowplot is failing with the following error:

checking tests Running testthat.R
  Running the tests in tests/testthat.R failed.
  Last 13 lines of output:
    1. cowplot:::expect_doppelganger(...) at test_themes.R:106:3
    2. vdiffr::expect_doppelganger(title, fig, ...) at ./helper-vdiffr.R:3:3
    3. base::withCallingHandlers(...)
    4. testthat::expect_snapshot_file(...)

    Snapshots
    To review and process snapshots locally:
    * Locate check directory.
    * Copy 'tests/testthat/_snaps' to local package.
    * Run testthat::snapshot_accept() to accept all changes.
    * Run testthat::snapshot_review() to review all changes.

    [ FAIL 14 | WARN 0 | SKIP 0 | PASS 84 ]
    Error: ! Test failures.
    Execution halted

Here are the tests that are failing:

In test_themes.R:

test_that("themes look right", {
  p <- ggplot(mtcars, aes(hp, disp, color = mpg, shape = factor(cyl))) +
    geom_point(size = 2) +
    guides(
      shape = guide_legend(order = 1),
      color = guide_colorbar(order = 2)
    )

  expect_doppelganger("theme half open",
    p + theme_half_open()
  )

  expect_doppelganger("theme minimal grid",
    p + theme_minimal_grid()
  )

  expect_doppelganger("theme minimal hgrid",
    p + theme_minimal_hgrid()
  )

  expect_doppelganger("theme minimal vgrid",
    p + theme_minimal_vgrid()
  )

  expect_doppelganger("theme half open tiny",
    p + theme_half_open(7)
  )

  expect_doppelganger("theme minimal grid tiny",
    p + theme_minimal_grid(7)
  )

  expect_doppelganger("theme minimal hgrid tiny",
    p + theme_minimal_hgrid(7)
  )

  expect_doppelganger("theme minimal vgrid tiny",
    p + theme_minimal_vgrid(7)
  )

  expect_doppelganger("theme half open huge",
    p + theme_half_open(22)
  )

  expect_doppelganger("theme minimal grid huge",
    p + theme_minimal_grid(22)
  )

  expect_doppelganger("theme minimal hgrid huge",
    p + theme_minimal_hgrid(22)
  )

  expect_doppelganger("theme minimal vgrid huge",
    p + theme_minimal_vgrid(22)
  )

})

and in test_key_glyph.R:

test_that("key glyphs", {
  set.seed(1233)
  df <- data.frame(
    x = sample(letters[1:2], 10, TRUE),
    y = rnorm(10)
  )
  p1 <- ggplot(df, aes(x, y, color = x)) +
    geom_point(
      key_glyph = rectangle_key_glyph(fill = color, padding = margin(3, 3, 3, 3))
    )
  expect_doppelganger("rectangle key glyph, color used as fill",
    p1 + theme_map()
  )
  p2 <- ggplot(df, aes(x, y, color = x)) +
    geom_boxplot(
      key_glyph = circle_key_glyph(
        fill = color,
        color = "black", linetype = 3, size = 0.3,
        padding = margin(2, 2, 2, 2)
      )
    )
  expect_doppelganger("circle key glyph, color used as fill",
    p2 + theme_map()
  )
})

The expected vs actual SVGs can be found here:
cowplot.zip

Worth noting that there is very little visual difference causing the snapshots to fail.

We found the root cause to be new versions of ggplot2 causing differences in the visual snapshots — the tests pass when downgrading to ggplot2 3.5.2. We've seen the same issue across several other packages. Happy to provide further details if needed.

[clauswilke]

I would recommend that you turn off checking for snapshots. They're turned off on CRAN also (I believe). We have seen so many false positives with snapshots that we've found it's generally the better strategy to not be overly concerned about snapshot failures.

[dpastoor]

thanks @clauswilke - we're trying to piece this together. One thing we've found is it seems there are a number of tests that suppressing does hide actual broken behavior. Some help with system specific gaps, but others are being missed on CRAN, thus why we've been starting to run direct checks on the system for compliance and risk assessment diligence. We don't intend on turning off snapshots for now since there are snapshot that do reveal problematic behavior, particularly packages where ggplot 3.5/4.0 transition caused breakages that some real regressions are unfortunately mixed in with a bunch of snapshot failure noise.

Hopefully we can devote some energy in figuring out how to potentially make the snapshot feature more robust/controlled across systems or at least more visible so less false positives.