How can a test script inform R CMD check that it should emit a custom message?

You should be able to alter the doSvUnit.R script to at least emit warnings and errors as you describe. What you want to do is to run the tests and then inspect the return value of the test runner and have R code that calls, warning() or stop().

For an example of how this was done using RUnit, take a look at the codetoolsBioC package in the Bioconductor repository. The relevant code is in inst/templates and copied below:

.test <- function(dir, pattern = ".*_test\\.R$"){    .failure_details <- function(result) {        res <- result[[1L]]        if (res$nFail > 0 || res$nErr > 0) {            Filter(function(x) length(x) > 0,                   lapply(res$sourceFileResults,                          function(fileRes) {                              names(Filter(function(x) x$kind != "success",                                           fileRes))                          }))        } else list()    }    if (missing(dir)) {        dir <- system.file("unitTests", package="@PKG@")        if (!nzchar(dir)) {            dir <- system.file("UnitTests", package="@PKG@")            if (!nzchar(dir))                stop("unable to find unit tests, no 'unitTests' dir")        }    }    ## Run unit tests from the directory containing the test files.    ## This allows tests to refer to data files with relative paths    cwd <- getwd()    on.exit(setwd(cwd))    setwd(dir)    require("RUnit", quietly=TRUE) || stop("RUnit package not found")    RUnit_opts <- getOption("RUnit", list())    RUnit_opts$verbose <- 0L    RUnit_opts$silent <- TRUE    RUnit_opts$verbose_fail_msg <- TRUE    options(RUnit = RUnit_opts)    suite <- defineTestSuite(name="@PKG@ RUnit Tests", dirs=getwd(),                             testFileRegexp=pattern,                             rngKind="default",                             rngNormalKind="default")    result <- runTestSuite(suite)    cat("\n\n")    printTextProtocol(result, showDetails=FALSE)    if (length(details <- .failure_details(result)) >0) {        cat("\nTest files with failing tests\n")        for (i in seq_along(details)) {            cat("\n  ", basename(names(details)[[i]]), "\n")            for (j in seq_along(details[[i]])) {                cat("    ", details[[i]][[j]], "\n")            }        }        cat("\n\n")        stop("unit tests failed for package @PKG@")    }    result}

I contacted two of the maintainers at r-forge and CRAN and they pointed me to the sources of R, in particular the check.R script.

if I understand it correctly:

• scripts in the tests directory are called using a system call,
• output (stdout and stderr) go to one single file,
• there are two possible outcomes: ok or not ok,
• to answer this question we need patching the library/tools package.

I opened a change request on R, my first guess is something like bit-coding the return status, bit-0 for ERROR (as it is now), bit-1 for WARNING, bit-2 for NOTE.

from doSvUnit.R, I would quit with status 2 in case of failures and 4 in case of skipped tests.

the patch would look like this:

Index: src/library/tools/R/testing.R===================================================================--- src/library/tools/R/testing.R   (revision 57214)+++ src/library/tools/R/testing.R   (working copy)@@ -352,10 +352,16 @@         } else             cmd <- paste("LANGUAGE=C", "R_TESTS=startup.Rs", cmd)         res <- system(cmd)-        if (res) {+        if (res%/%4 %% 2) {+            message("NOTE")+        }+        if (res%/%2 %% 2) {+            message("WARNING")+        }+        if (res %% 2) {             file.rename(outfile, paste(outfile, "fail", sep="."))             return(1L)         }         savefile <- paste(outfile, "save", sep = "." )         if (file.exists(savefile)) {             message("  Comparing ", sQuote(outfile), " to ",

unpatched R sees anything different from 0 as ERROR.