From 6289b002b6a1ac8dcf5a97075f4db550df080191 Mon Sep 17 00:00:00 2001 From: quantimnot <54247259+quantimnot@users.noreply.github.com> Date: Thu, 1 Sep 2022 11:52:13 -0400 Subject: [Testament] Extend and document message testing aids (#19996) * [Testament] Extend and document message testing aids * Enable inline msgs when not reject action. Eliminates the pain of changing the line and column numbers in `nimout` or `output` while making changes to the test. * Enable using inline msgs and nimout together. Allows ease of inline msgs for the test as well as testing msgs from other modules. * Add path separator and test filename variable interpolation in msgs. Eases handling path separators in the msgs. * Add some documentation. * Fixed lots of broken tests * Fixed more broken tests * Support multiple inline messages per a line * Fix a broken test * Revert variable substitution in `output` * Remove uneeded params * Update doc/testament.md Co-authored-by: Clay Sweetser * Update testament/specs.nim Co-authored-by: Clay Sweetser * Update testament/specs.nim Co-authored-by: Clay Sweetser * Fix indentation Co-authored-by: quantimnot Co-authored-by: Clay Sweetser --- doc/testament.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 72 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/testament.md b/doc/testament.md index 89e3da44d..6d0b3a69c 100644 --- a/doc/testament.md +++ b/doc/testament.md @@ -113,7 +113,7 @@ Example "template" **to edit** and write a Testament unittest: # Provide an `output` string to assert that the test prints to standard out # exactly the expected string. Provide an `outputsub` string to assert that # the string given here is a substring of the standard out output of the - # test. + # test (the output includes both the compiler and test execution output). output: "" outputsub: "" @@ -153,8 +153,7 @@ Example "template" **to edit** and write a Testament unittest: # Command the test should use to run. If left out or an empty string is # provided, the command is taken to be: # "nim $target --hints:on -d:testing --nimblePath:build/deps/pkgs $options $file" - # You can use the $target, $options, and $file placeholders in your own - # command, too. + # Subject to variable interpolation. cmd: "nim c -r $file" # Maximum generated temporary intermediate code file size for the test. @@ -189,7 +188,76 @@ Example "template" **to edit** and write a Testament unittest: * `This is not the full spec of Testament, check the Testament Spec on GitHub, see parseSpec(). `_ * `Nim itself uses Testament, so there are plenty of test examples. `_ * Has some built-in CI compatibility, like Azure Pipelines, etc. -* `Testament supports inlined error messages on Unittests, basically comments with the expected error directly on the code. `_ + + +Inline hints, warnings and errors (notes) +----------------------------------------- + +Testing the line, column, kind and message of hints, warnings and errors can +be written inline like so: + +.. code-block:: nim + + {.warning: "warning!!"} #[tt.Warning + ^ warning!! [User] ]# + +The opening `#[tt.` marks the message line. +The `^` marks the message column. + +Inline messages can be combined with `nimout` when `nimoutFull` is false (default). +This allows testing for expected messages from other modules: + +.. code-block:: nim + + discard """ + nimout: "config.nims(1, 1) Hint: some hint message [User]" + """ + {.warning: "warning!!"} #[tt.Warning + ^ warning!! [User] ]# + +Multiple messages for a line can be checked by delimiting messages with ';': + +.. code-block:: nim + + discard """ + matrix: "--errorMax:0 --styleCheck:error" + """ + + proc generic_proc*[T](a_a: int) = #[tt.Error + ^ 'generic_proc' should be: 'genericProc'; tt.Error + ^ 'a_a' should be: 'aA' ]# + discard + +Use `--errorMax:0` in `matrix`, or `cmd: "nim check $file"` when testing +for multiple 'Error' messages. + +Output message variable interpolation +------------------------------------- + +`errormsg`, `nimout`, and inline messages are subject to these variable interpolations: + +* `${/}` - platform's directory separator +* `$file` - the filename (without directory) of the test + +All other `$` characters need escaped as `$$`. + +Cmd variable interpolation +-------------------------- + +The `cmd` option is subject to these variable interpolations: + +* `$target` - the compilation target, e.g. `c`. +* `$options` - the options for the compiler. +* `$file` - the file path of the test. +* `$filedir` - the directory of the test file. + +.. code-block:: nim + + discard """ + cmd: "nim $target --nimblePath:./nimbleDir/simplePkgs $options $file" + """ + +All other `$` characters need escaped as `$$`. Unit test Examples -- cgit 1.4.1-2-gfad0