diff options
Diffstat (limited to 'doc/testament.md')
| -rw-r--r-- | doc/testament.md | 111 |
1 files changed, 85 insertions, 26 deletions
diff --git a/doc/testament.md b/doc/testament.md index 812b10984..0ff3591ac 100644 --- a/doc/testament.md +++ b/doc/testament.md @@ -17,7 +17,7 @@ so can be useful to run your tests, even the most complex ones. Test files location =================== -By default, Testament looks for test files on ``"./tests/*.nim"``. +By default, Testament looks for test files on ``"./tests/category/*.nim"``. You can overwrite this pattern glob using `pattern <glob>`:option:. The default working directory path can be changed using `--directory:"folder/subfolder/"`:option:. @@ -27,24 +27,31 @@ You can change that using `--nim:"folder/subfolder/nim"`:option:. Running JavaScript tests with `--targets:"js"`:option: requires a working NodeJS on `PATH`. +Commands +======== + +p|pat|pattern <glob> run all the tests matching the given pattern +all run all tests inside of category folders +c|cat|category <category> run all the tests of a certain category +r|run <test> run single test file +html generate testresults.html from the database + Options ======= ---print Also print results to the console ---simulate See what tests would be run but don't run them - (for debugging) ---failing Only show failing/ignored tests ---targets:"c cpp js objc" - Run tests for specified targets (default: c) ---nim:path Use a particular nim executable (default: $PATH/nim) ---directory:dir Change to directory dir before reading the tests - or doing anything else. +--print print results to the console +--verbose print commands (compiling and running tests) +--simulate see what tests would be run but don't run them (for debugging) +--failing only show failing/ignored tests +--targets:"c cpp js objc" run tests for specified targets (default: c) +--nim:path use a particular nim executable (default: $PATH/nim) +--directory:dir Change to directory dir before reading the tests or doing anything else. --colors:on|off Turn messages coloring on|off. ---backendLogging:on|off Disable or enable backend logging. - By default, turned on. ---skipFrom:file Read tests to skip from ``file`` - one test per - line, # comments ignored +--backendLogging:on|off Disable or enable backend logging. By default turned on. +--megatest:on|off Enable or disable megatest. Default is on. +--valgrind:on|off Enable or disable valgrind support. Default is on. +--skipFrom:file Read tests to skip from `file` - one test per line, # comments ignored Running a single test @@ -54,18 +61,21 @@ This is a minimal example to understand the basics, not very useful for production, but easy to understand: ```console - $ mkdir tests - $ echo "assert 42 == 42" > tests/test0.nim - $ testament run test0.nim - PASS: tests/test0.nim C ( 0.2 sec) - $ testament r test0 - PASS: tests/test0.nim C ( 0.2 sec) + $ mkdir -p tests/category + $ echo "assert 42 == 42" > tests/category/test0.nim + $ testament run tests/category/test0.nim + PASS: tests/category/test0.nim c ( 0.2 sec) + $ testament r tests/category/test0 + PASS: tests/category/test0.nim C ( 0.2 sec) ``` Running all tests from a directory ================================== + This will run all tests in the top level tests/ directory. NOTE: these + tests are skipped by `testament all`. + ```console $ testament pattern "tests/*.nim" ``` @@ -102,8 +112,21 @@ Example "template" **to edit** and write a Testament unittest: # "run": expect successful compilation and execution # "reject": expect failed compilation. The "reject" action can catch # {.error.} pragmas but not {.fatal.} pragmas because + # {.error.} calls are expected to originate from the test-file, + # and can explicitly be specified using the "file", "line" and + # "column" options. # {.fatal.} pragmas guarantee that compilation will be aborted. action: "run" + + # For testing failed compilations you can specify the expected origin of the + # compilation error. + # With the "file", "line" and "column" options you can define the file, + # line and column that a compilation-error should have originated from. + # Use only with action: "reject" as it expects a failed compilation. + # Requires errormsg or msg to be defined. + # file: "" + # line: "" + # column: "" # The exit code that the test is expected to return. Typically, the default # value of 0 is fine. Note that if the test will be run by valgrind, then @@ -117,13 +140,14 @@ Example "template" **to edit** and write a Testament unittest: output: "" outputsub: "" - # Whether to sort the output lines before comparing them to the desired - # output. + # Whether to sort the compiler output lines before comparing them to the + # expected output. sortoutput: true - # Each line in the string given here appears in the same order in the - # compiler output, but there may be more lines that appear before, after, or - # in between them. + # Provide a `nimout` string to assert that the compiler during compilation + # prints the defined lines to the standard out. + # The lines must match in order, but there may be more lines that appear + # before, after, or in between them. nimout: ''' a very long, multi-line @@ -150,6 +174,9 @@ Example "template" **to edit** and write a Testament unittest: # "leaks": run the test with Valgrind, but do not check for memory leaks valgrind: false # Can use Valgrind to check for memory leaks, or not (Linux 64Bit only). + # Checks that the specified piece of C-code is within the generated C-code. + ccodecheck: "'Assert error message'" + # Command the test should use to run. If left out or an empty string is # provided, the command is taken to be: # "nim $target --hints:on -d:testing --nimblePath:build/deps/pkgs $options $file" @@ -186,7 +213,7 @@ Example "template" **to edit** and write a Testament unittest: * As you can see the "Spec" is just a `discard """ """`. * Spec has sane defaults, so you don't need to provide them all, any simple assert will work just fine. * This is not the full spec of Testament, check [the Testament Spec on GitHub, - see parseSpec()](https://github.com/nim-lang/Nim/blob/devel/testament/specs.nim#L315). + see parseSpec()](https://github.com/nim-lang/Nim/blob/devel/testament/specs.nim#L317). * Nim itself uses Testament, so [there are plenty of test examples]( https://github.com/nim-lang/Nim/tree/devel/tests). * Has some built-in CI compatibility, like Azure Pipelines, etc. @@ -273,6 +300,38 @@ Expected to fail: assert not_defined == "not_defined", "not_defined is not defined" ``` +Expected to fail with error thrown from another file: +```nim +# test.nim +discard """ + action: "reject" + errorMsg: "I break" + file: "breakPragma.nim" +""" +import ./breakPragma + +proc x() {.justDo.} = discard + +# breakPragma.nim +import std/macros + +macro justDo*(procDef: typed): untyped = + error("I break") + return procDef +``` + +Expecting generated C to contain a given piece of code: + + ```nim + discard """ + # Checks that the string "Assert error message" is in the generated + # C code. + ccodecheck: "'Assert error message'" + """ + assert 42 == 42, "Assert error message" + ``` + + Non-Zero exit code: ```nim |