testthat 3.1.0

We’re stoked to announce the release of testthat 3.1.0. testthat makes it easy to turn your existing informal tests into formal, automated tests that you can rerun quickly and easily. testthat is the most popular unit-testing package for R, and is used by over 6,000 CRAN and Bioconductor packages. You can learn more about unit testing at https://r-pkgs.org/tests.html.

You can install testthat from CRAN with:

install.packages("testthat")

This release of testthat includes a bunch of minor improvements to snapshotting, as well as one breaking change (which only applies if you’re using the 3rd edition). You can see a full list of changes in the release notes.

library(testthat)

Snapshot tests

Most of the effort in this release has gone into snapshot tests, a new feature in testthat 3.0.0. snapshot tests (also known as golden tests) record expected output in a separate human-readable file instead of using code to describe what the expected output looks like. Since the release of testthat 3.0.0, we’ve started using snapshot tests across a bunch of tidyverse packages and they’ve been working out really well. I don’t anticipate any major changes (although we may continue to add new features), so the snapshot functions have changed lifecycle stages from experimental to stable.

This release also includes two new features that help you use snapshot tests in more places:

expect_snapshot() gains a transform argument, which should be a function that takes a character vector of lines and returns a modified character vector of lines. This makes it easy to remove sensitive (e.g. API keys) or stochastic (e.g. random temporary directory names) data from snapshot output.

get_info <- function() {
  list(
    name = "Hadley", 
    password = "sssh-its-a-secret"
  )
}

hide_password <- function(x) {
  is_password <- grepl("password", x)
  ifelse(is_password, "<REDACTED>", x)
}

test_that("info retruns name and password", {
  local_edition(3)
  expect_snapshot(str(get_info()), transform = hide_password)
})
#> Can't compare snapshot to reference when testing interactively
#> ℹ Run `devtools::test()` or `testthat::test_file()` to see changes
#> ℹ Current value:
#> Code
#>   str(get_info())
#> Output
#>   List of 2
#>    $ name    : chr "Hadley"
#>   <REDACTED>
#> ── Skip (test-that.R:50:3): info retruns name and password ─────────────────────
#> Reason: empty test

If you need transform, I recommend designing your printing methods so the output can be easily manipulated with regexps.

expect_snapshot() and friends get an experimental new variant argument which causes the snapshot to be saved in _snaps/{variant}/{test}.md instead of _snaps/{test}.md. This allows you to generate (and compare) unique snapshots for different scenarios where the output is otherwise out of your control, like differences across operating systems or R versions.
```
r_version <- function() paste0("R", getRversion()[, 1:2])

test_that("can capture version nickname", {
  expect_snapshot(version$nickname, variant = r_version())
})
```

Remember that snapshot tests are not run on CRAN by default because they require a human to confirm whether or not a change is a breakage, so you shouldn’t rely only on snapshot tests to ensure that your code is correct. While it is possible to set cran = TRUE, to force snapshot tests to run on CRAN, I don’t generally recommend it as snapshots are often vulnerable to minor changes that don’t merit breaking your released package.

Breaking changes

We made one breaking change that affects the third edition. Previously, expect_message() and expect_warning() returned the value of the first argument, unless that was NULL, when they instead returned the condition object. This meant you could write code like this:

expect_equal(expect_warning(f(), "warning"), "value")

Now expect_message() and expect_warning() always return the condition object so you need to flip the order of the expectations:

expect_warning(expect_equal(f(), "value"), "warning")

This (IMO) is a little easier to read with the pipe:

# Or use the pipe:
f() %>% 
  expect_equal("value") %>% 
  expect_warning("warning")

Or with an intermediate object:

expect_warning(value <- f(), "warning")
expect_equal(value, "value")

As with any breaking change, we made this change with great care. Fortunately it only affects the 3rd edition, which relatively few packages use, and we submitted PRs to all affected packages on CRAN.

We made this change because it makes expect_message() and expect_warning() more consistent with expect_error(), while make it easier to inspect both the value and the message/warning. This is important because it makes it easier to test functions that produce custom condition objects that themselves contain meaningful data:

library(rlang)
informative_error <- function() {
  abort(
    "An error with extra info",
    name = "patrice",
    number = 17
  )
}

err <- expect_error(my_function(), class = "package_error_class")
expect_equal(err$name, "patrice")
expect_equal(err$number, 17)

Richer conditions are a tool that we use within the tidyverse to provide more context in errors, warnings, and messages. You’re unlikely to see them directly, but we’re using them as part of our general effort to make more actionable error messages.

Acknowledgements

A big thanks to all 104 contributors who filed issues and contributed code to this and the last few patch releases: @Aariq, @Aehmlo, @aguynamedryan, @ahjota, @akersting, @Akirathan, @arnaud-feldmann, @aronatkins, @Bisaloo, @boennecd, @BS1125, @byoung, @cboettig, @clauswilke, @ColinFay, @CorradoLanera, @dagola, @DanChaltiel, @david-barnett, @david-cortes, @DavisVaughan, @dmenne, @dpprdan, @egonulates, @espinielli, @eveyp, @FBartos, @federicomarini, @flying-sheep, @franzbischoff, @gaborcsardi, @hadley, @hamstr147, @harell, @harsh9898, @helix123, @hongooi73, @hsloot, @jeffreyhanson, @jennybc, @jeroen, @jesterxd, @jimhester, @kevinushey, @krlmlr, @lcougnaud, @linusheinz, @lionel-, @llrs, @lutzgruber-quantco, @maelle, @maia-sh, @malcolmbarrett, @MarkEdmondson1234, @marko-stojovic, @mattfidler, @maxachis, @maxheld83, @mbojan, @mcol, @MechantRouquin, @mem48, @mgirlich, @MichaelChirico, @michaelquinn32, @mihaiconstantin, @mikemahoney218, @MilesMcBain, @mjskay, @mllg, @Mosk915, @ms609, @multimeric, @nbenn, @neonira, @nicholasproietti, @njtierney, @nkehrein, @pat-s, @pbarber, @pkrog, @przmv, @r2evans, @raphael-lorenzdelaigue, @rfaelens, @rjnb50, @romainfrancois, @rwhetten, @salim-b, @schloerke, @sigmafelix, @srfall, @strengejacke, @SubieG, @thebioengineer, @thisisnic, @tiQu, @torbjorn, @ttimbers, @tzakharko, @vspinu, @wch, @weiyaw, and @yasushm.