Moves caasdriver comments into idetools document.

1 files changed, 95 insertions, 2 deletions

diff --git a/doc/idetools.txt b/doc/idetools.txt
index ce6062e63..59e2ee49e 100644
--- a/doc/idetools.txt
+++ b/doc/idetools.txt
@@ -17,7 +17,7 @@ definition of symbols or suggestions for completion.
 
 This document will guide you through the available options. If you
 want to look at practical examples of idetools support you can look
-at the test files found in ``tests/caas/*.txt`` or `various editor
+at the test files found in the `Test suite`_ or `various editor
 integrations <https://github.com/Araq/Nimrod/wiki/Editor-Support>`_
 already available.
 
@@ -190,7 +190,7 @@ of text it thinks necessary plus an empty line to indicate the end
 of the answer.
 
 You can find examples of client/server communication in the idetools
-tests found in ``tests/caas/*.txt``.
+tests found in the `Test suite`_.
 
 
 Parsing idetools output
@@ -467,3 +467,96 @@ skVar
     --> col 2: $MODULE.writeTempFile.output
         col 3: TFile
         col 7: ""
+
+
+Test suite
+==========
+
+To verify that idetools is working properly there are files in the
+``tests/caas/`` directory which provide unit testing. If you find
+odd idetools behaviour and are able to reproduce it, you are welcome
+to report it as a bug and add a test to the suite to avoid future
+regressions.
+
+
+Running the test suite
+----------------------
+
+At the moment idetools support is still in development so the test
+suite is not integrated with the main test suite and you have to
+run it manually. First you have to compile the tester::
+
+	$ cd my/nimrod/checkout
+	$ nimrod c tests/tester.nim
+
+Running the tester without parameters will display some options.
+To run the caas test suite (and other special tests) you need to
+use the `special` command. You need to run this command from the
+root of the checkout or it won't be able to open the required files::
+
+	$ ./tests/tester special
+
+However this is a roundabout way of running the test suite. You can
+also compile and run ``tests/caasdriver.nim`` manually. In fact,
+running it manually will allow you to specify special parameters
+too. Example::
+
+	$ cd my/nimrod/checkout/tests
+	$ nimrod c caasdriver.nim
+
+Running the ``caasdriver`` without parameters will attempt to process
+all the test cases in all three operation modes. If a test succeeds
+nothing will be printed and the process will exit with zero. If any
+test fails, the specific line of the test preceeding the failure
+and the failure itself will be dumped to stdout, along with a final
+indicator of the success state and operation mode. You can pass the
+parameter ``verbose`` to force all output even on successfull tests.
+
+The normal operation mode is called ``ProcRun`` and it involves
+starting a process for each command or query, similar to running
+manually the Nimrod compiler from the commandline. The ``CaasRun``
+mode starts a server process to answer all queries. The ``SymbolProcRun``
+mode is used by compiler developers.  This means that running all
+tests involves processing all ``*.txt`` files three times, which
+can be quite time consuming.
+
+If you don't want to run all the test case files you can pass any
+substring as a parameter to ``caasdriver``. Only files matching the
+passed substring will be run. The filtering doesn't use any globbing
+metacharacters, it's a plain match. For example, to run only
+``*-compile*.txt`` tests in verbose mode::
+
+	./caasdriver verbose -compile
+
+
+Test case file format
+---------------------
+
+All the ``tests/caas/*.txt`` files encode a session with the compiler:
+
+* The first line indicates the main project file.
+
+* Lines starting with ``>`` indicate a command to be sent to the
+  compiler and the lines following a command include checks for
+  expected or forbidden output (``!`` for forbidden).
+
+* If a line starts with ``#`` it will be ignored completely, so you
+  can use that for comments.
+
+* Since some cases are specific to either ``ProcRun`` or ``CaasRun``
+  modes, you can prefix a line with the mode and the line will be
+  processed only in that mode.
+
+* The rest of the line is treated as a `regular expression <re.html>`_,
+  so be careful escaping metacharacters like parenthesis.
+
+Before the line is processed as a regular expression, some basic
+variables are searched for and replaced in the tests. The variables
+which will be replaced are:
+
+* **$TESTNIM**: filename specified in the first line of the script.
+* **$MODULE**: like $TESTNIM but without extension, useful for
+  expected output.
+
+When adding a test case to the suite it is a good idea to write a
+few comments about what the test is meant to verify.