testthat 3.0.0

We’re tickled pink to announce the release of testthat 3.0.0. testthat makes it easy to turn your existing informal tests into formal, automated tests that you can rerun quickly and easily. testthat is the most popular unit-testing package for R, and is used by over 5,000 CRAN and Bioconductor packages. You can learn more about unit testing at https://r-pkgs.org/tests.html.

You can install it from CRAN with:

install.packages("testthat")

In this blog post, I want to discuss the two biggest changes: the introduction of the “edition”, which allows us to make breaking changes that you have to specifically opt-in to, and a new family of “snapshot” tests. I’ll also give a quick round up of the other major changes, but you might want to look at the release notes for the full details.

library(testthat)

3rd edition

testthat 3.0.0 introduces the idea of an edition. An edition is a bundle of behaviours that you have to explicitly choose to use, allowing us to make backward incompatible changes. This is particularly important for testthat since it’s used by a very large number of packages, and the old edition includes a few design infelicities that we wanted want to fix. Choosing to use the 3rd edition allows you to use our latest recommendations for ongoing and new work, while historical packages continue to use the old behaviour. To use it, and this line to your DESCRIPTION:

Config/testthat/edition: 3

The second edition will never go away, but new features will only appear in the 3rd edition. We don’t anticipate creating new editions very often (maybe once every 5 years at most), and they’ll always be paired with a new major version of testthat, i.e. if there’s another edition, it’ll be the 4th edition and will come with testthat 4.0.0.

The full details of the 3rd edition are described in vignette("third-edition", package = "testthat"). If you’re a heavy testthat user, I strongly recommend reading the vignette. Here are the most important changes:

• context() is now deprecated. It hasn’t been recommended for some time since it usually duplicates data also present in the file name.

• expect_identical() and expect_equal() use waldo::compare() to compare actual and expected results. This should give considerably more informative output for test failures, but waldo::compare() is not 100% compatible with all.equal() (which previously powered expect_equal()). There are a few differences due to a bug in testthat’s comparison of floating point values and subtle differences when comparing environments stored in attributes (common in modelling functions); see the vignette for full details.

  expect_equivalent() is now deprecated because it’s equivalent to expect_equal(ignore_attr = TRUE) (which is generally not recommended, anyway, because it’s such a blunt tool).

• expect_error(), expect_warning(), expect_message(), and expect_condition() now all use the same underlying logic: they capture the first condition that matches class/regexp and allow anything else to bubble up. This behaviour pairs particularly well with upcoming changes to the pipe in magrittr 2.0.0.

• Messages are no longer automatically silenced. You’ll now need to use suppressMessages() to hide unimportant messages or expect_messsage() to catch important messages.

• test_that() now sets a number of options and env vars to make output as reproducible as possible. Many of these options were previously set in various places (including devtools::test(), test_dir(), test_file(), and verify_output()) but they have now been centralised in to local_test_context().

Snapshot testing

The goal of a unit test is to record the expected output of a function using code. This is a powerful technique because because not only does it ensure that code doesn’t change unexpectedly, it also expresses the desired behaviour in a way that a human can understand. However, it’s not always convenient to record the expected behaviour with code. Some challenges include:

• Text output that includes many characters like quotes and newlines that require special handling in a string.

• Output that is large, making it painful to define the reference output, and bloating the size of the test file and making it hard to navigate.

• Cases where you want to record a mix of printed output, messages, warnings, or errors.

• Binary formats, like plots or images, which are very difficult to describe in code: i.e. the plot looks right, the error message is useful to a human, the print method uses colour effectively.

For these situations, testthat provides an alternative mechanism: snapshot tests. Instead of using code to describe expected output, snapshot tests (also known as golden tests) record results in a separate human-readable file. Snapshot tests in testthat are inspired primarily by Jest, thanks to a number of very useful discussions with Joe Cheng.

If snapshot tests sound intriguing to you, please read the details in vignette("snapshotting", package = "testthat").

Other improvements

• testthat has preliminary support for running tests in parallel. See details in vignette("parallel", package = "testthat")

• We now recommend using test fixtures rather than setup() and teardown() code. See details in vignette("test-fixtures", package = "testthat").

• We have revamped the non-interactive reporters to do a better job of displaying skipped tests, and have written a vignette, vignette("skipping", package = "testthat"), describing best practices (including how to test your own skip functions).

• Made a number of tweaks to test reporting. Most importantly, the reporter used by devtools::test() now displays stack traces of warnings and errors that occur outside of tests, making these problems substantially easier to track down. (It also gets a number of new random praise options that use emoji). There’s also a new reporter for use by devtools::test_file() which displays results more compactly.

Acknowledgements

A big thanks to all 64 contributors who helped with this release by discussing problems, proposing features and contributing code: @aalucaci, @andrewmarx, @Anirban166, @benxiahu, @Bisaloo, @boerjames, @brodieG, @c0k3b0y, @Christoph999, @daattali, @damianooldoni, @DanChaltiel, @davidchall, @DavisVaughan, @EdwardJGillian, @emiliesecherre, @gaborcsardi, @gavinsimpson, @gergness, @hadley, @hbaniecki, @hongooi73, @HughParsonage, @jarauh, @jcheng5, @jennybc, @jimhester, @jonkeane, @kevinushey, @khaeru, @kjytay, @kos59125, @krlmlr, @ksvanhorn, @lionel-, @maelle, @manumart, @mardam, @md0u80c9, @melvidoni, @mgirlich, @MichaelChirico, @msberends, @mtkerbeR, @MyKo101, @nbenn, @Nelson-Gon, @njtierney, @noeliarico, @omsai, @pat-s, @psteinb, @salim-b, @SKalt, @stefanoborini, @stufield, @sumedh10, @tanho63, @tbrown122387, @torfason, @torockel, @xiaodaigh, @yitao-li, and @yutannihilation.