Skip test on M1mac in R

Question

According to Carl Boettiger in this thread, "...when tests fail. Like Solaris, some of these failures can occur when an upstream dependency installs on the platform but does not actually run." My code fails numerically on M1mac, but not on other platforms, while using stats::integrate on functions returning very small values.

Should I skip the test on M1mac (arm64)?

require(testthat)
test_that("correct numeric solution", {
     skip_on_os("mac", arch = "aarch64")

     # code of the test using expect_equal()
})

Alternatively, can adjusting the tolerance argument in expect_equal() help resolve the specific issue on the one system? If yes, to what value should tolerance be changed, if my results is at 1e-9 (using tolerance = 1e-6 now, and the test fails)?

In more general terms, what is the best coding practice (or a long term solution) for R packages on CRAN to solve problems with specific tests failing on one OS?

Answer 1

Speaking as a package developer (but not a CRAN expert by any means), my advice would be to reformulate (not skip) the test. You can use the test as an opportunity to document the issue. Here are the steps I would take:

• Define platform-specific assertions. On M1 Mac, assert that the approximation is not equal to the exact value of the integral at the tolerance that you are using for other platforms, by wrapping the failing expect_equal call inside of expect_error. Then assert that the two are equal at some minimal, greater tolerance with a less strict expect_equal call. Your test block would contain something like this:

  x <- approximate_integral_value
  y <- exact_integral_value
  
  ## If testing on M1 Mac
  if (tolower(Sys.info()[["sysname"]]) == "darwin" && R.version[["arch"]] == "aarch64") {
    ## Expect strict test to fail
    expect_error(expect_equal(x, y, tolerance = 1e-9)) 
    ## Expect less strict test to pass
    expect_equal(x, y, tolerance = 1e-4)
  ## Otherwise
  } else {
    ## Expect strict test to pass
    expect_equal(x, y, tolerance = 1e-9)
  }
  

  This way, you will detect if the problem resolves itself (the first expect_equal will pass, causing the expect_error to fail) and you will detect if the problem gets worse (the second expect_equal will fail). In both cases, you would update the test code.

• Leave a comment near the expect_error call explaining why the integral approximation is inaccurate on M1 Macs and describe any attempts you've made to work around the underlying numerical issue (or why it cannot be resolved).

• If you think that functions in your package might not work on M1 Macs because this test is failing, then use warning or stop inside of those functions to let users know. If those functions are central to your package's functionality, then add a platform note somewhere more visible (e.g., on the landing page of your package website).

As for your numerical issue:

• Try to find a more stable algorithm for computing the integrand. For example, it might help to replace prod(x) with exp(sum(log(x))).

• Experiment with the optional arguments of integrate, namely rel.tol and abs.tol.

FWIW, there are dedicated forums for these kinds of questions, namely the R-package-devel and R-SIG-Mac mailing lists:

• R-package-devel would be appropriate for the question, "What is the best practice for handling this package development problem?" There, you are more likely to get an answer directly from a CRAN maintainer.

• R-SIG-Mac would be appropriate for the question, "Why am I encountering this behaviour only on M1 Mac?" There, you are more likely to get an answer from an R Core Team member who develops R for Macs. [Edit: @Roland points out in the comments that native R builds for ARM-based platforms do not support extended precision arithmetic.]