[Testament] Extend and document message testing aids (#19996)

* [Testament] Extend and document message testing aids

* Enable inline msgs when not reject action.
Eliminates the pain of changing the line and column numbers in `nimout`
or `output` while making changes to the test.
* Enable using inline msgs and nimout together.
Allows ease of inline msgs for the test as well as testing msgs from
other modules.
* Add path separator and test filename variable interpolation in
msgs.
Eases handling path separators in the msgs.
* Add some documentation.

* Fixed lots of broken tests

* Fixed more broken tests

* Support multiple inline messages per a line

* Fix a broken test

* Revert variable substitution in `output`

* Remove uneeded params

* Update doc/testament.md

Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>

* Update testament/specs.nim

Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>

* Update testament/specs.nim

Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>

* Fix indentation

Co-authored-by: quantimnot <quantimnot@users.noreply.github.com>
Co-authored-by: Clay Sweetser <Varriount@users.noreply.github.com>

1 files changed, 72 insertions, 4 deletions

diff --git a/doc/testament.md b/doc/testament.md
index 89e3da44d..6d0b3a69c 100644
--- a/doc/testament.md
+++ b/doc/testament.md
@@ -113,7 +113,7 @@ Example "template" **to edit** and write a Testament unittest:
     # Provide an `output` string to assert that the test prints to standard out
     # exactly the expected string. Provide an `outputsub` string to assert that
     # the string given here is a substring of the standard out output of the
-    # test.
+    # test (the output includes both the compiler and test execution output).
     output: ""
     outputsub: ""
 
@@ -153,8 +153,7 @@ Example "template" **to edit** and write a Testament unittest:
     # Command the test should use to run. If left out or an empty string is
     # provided, the command is taken to be:
     # "nim $target --hints:on -d:testing --nimblePath:build/deps/pkgs $options $file"
-    # You can use the $target, $options, and $file placeholders in your own
-    # command, too.
+    # Subject to variable interpolation.
     cmd: "nim c -r $file"
 
     # Maximum generated temporary intermediate code file size for the test.
@@ -189,7 +188,76 @@ Example "template" **to edit** and write a Testament unittest:
 * `This is not the full spec of Testament, check the Testament Spec on GitHub, see parseSpec(). <https://github.com/nim-lang/Nim/blob/devel/testament/specs.nim#L238>`_
 * `Nim itself uses Testament, so there are plenty of test examples. <https://github.com/nim-lang/Nim/tree/devel/tests>`_
 * Has some built-in CI compatibility, like Azure Pipelines, etc.
-* `Testament supports inlined error messages on Unittests, basically comments with the expected error directly on the code. <https://github.com/nim-lang/Nim/blob/9a110047cbe2826b1d4afe63e3a1f5a08422b73f/tests/effects/teffects1.nim>`_
+
+
+Inline hints, warnings and errors (notes)
+-----------------------------------------
+
+Testing the line, column, kind and message of hints, warnings and errors can
+be written inline like so:
+
+.. code-block:: nim
+
+  {.warning: "warning!!"} #[tt.Warning
+           ^ warning!! [User] ]#
+
+The opening `#[tt.` marks the message line.
+The `^` marks the message column.
+
+Inline messages can be combined with `nimout` when `nimoutFull` is false (default).
+This allows testing for expected messages from other modules:
+
+.. code-block:: nim
+
+  discard """
+    nimout: "config.nims(1, 1) Hint: some hint message [User]"
+  """
+  {.warning: "warning!!"} #[tt.Warning
+           ^ warning!! [User] ]#
+
+Multiple messages for a line can be checked by delimiting messages with ';':
+
+.. code-block:: nim
+
+  discard """
+    matrix: "--errorMax:0 --styleCheck:error"
+  """
+
+  proc generic_proc*[T](a_a: int) = #[tt.Error
+       ^ 'generic_proc' should be: 'genericProc'; tt.Error
+                        ^ 'a_a' should be: 'aA' ]#
+    discard
+
+Use `--errorMax:0` in `matrix`, or `cmd: "nim check $file"` when testing
+for multiple 'Error' messages.
+
+Output message variable interpolation
+-------------------------------------
+
+`errormsg`, `nimout`, and inline messages are subject to these variable interpolations:
+
+* `${/}` - platform's directory separator
+* `$file` - the filename (without directory) of the test
+
+All other `$` characters need escaped as `$$`.
+
+Cmd variable interpolation
+--------------------------
+
+The `cmd` option is subject to these variable interpolations:
+
+* `$target` - the compilation target, e.g. `c`.
+* `$options` - the options for the compiler.
+* `$file` - the file path of the test.
+* `$filedir` - the directory of the test file.
+
+.. code-block:: nim
+
+  discard """
+    cmd: "nim $target --nimblePath:./nimbleDir/simplePkgs $options $file"
+  """
+
+All other `$` characters need escaped as `$$`.
 
 
 Unit test Examples