Starts idetools specific documentation.

1 files changed, 467 insertions, 0 deletions

diff --git a/doc/idetools.txt b/doc/idetools.txt
new file mode 100644
index 000000000..0ae29e824
--- /dev/null
+++ b/doc/idetools.txt
@@ -0,0 +1,467 @@
+================================
+  Nimrod IDE Integration Guide
+================================
+
+:Author: Britney Spears
+:Version: |nimrodversion|
+
+.. contents::
+
+
+Nimrod differs from many other compilers in that it is really fast,
+and being so fast makes it suited to provide external queries for
+text editors about the source code being written. Through the
+``idetools`` command of `the compiler <nimrodc.html>`_, any IDE can
+query a ``.nim`` source file and obtain useful information like
+definition of symbols or suggestions for completion.
+
+This document will guide you through the available options. If you
+want to look at practical examples of idetools support you can look
+at the test files found in ``tests/caas/*.txt`` or `various editor
+integrations <https://github.com/Araq/Nimrod/wiki/Editor-Support>`_
+already available.
+
+
+Idetools invokation
+===================
+
+Specifying the location of the query
+------------------------------------
+
+All of the available idetools commands require you to specify a
+query location through the ``--track`` or ``--trackDirty`` switches.
+The general idetools invokations are::
+
+    nimrod idetools --track:FILE,LINE,COL <switches> proj.nim
+
+Or::
+
+    nimrod idetools --trackDirty:DIRTY_FILE,FILE,LINE,COL <switches> proj.nim
+
+``proj.nim``
+    This is the main *project* filename. Most of the time you will
+    pass in the same as **FILE**, but for bigger projects this is
+    the file which is used as main entry point for the program, the
+    one which users compile to generate a final binary.
+
+``<switches>``
+    This would be any of the other idetools available options, like
+    ``--def`` or ``--suggest`` explained in the following sections.
+
+``COL``
+    An integer with the column you are going to query. For the
+    compiler columns start at zero, so the first column will be
+    **0** and the last in an 80 column terminal will be **79**.
+
+``LINE``
+    An integer with the line you are going to query. For the compiler
+    lines start at **1**.
+
+``FILE``
+    The file you want to perform the query on. Usually you will
+    pass in the same value as **proj.nim**.
+
+``DIRTY_FILE``
+    The **FILE** paramater is enough for static analysis, but IDEs
+    tend to have *unsaved buffers* where the user may still be in
+    the middle of typing a line. In such situations the IDE can
+    save the current contents to a temporary file and then use the
+    ``--trackDirty`` switch.
+
+    Dirty files are likely to contain errors and they are usually
+    compiled partially only to the point needed to service the
+    idetool request. The compiler discriminates them to ensure that
+    **a)** they won't be cached and **b)** they won't invalidate
+    the cached contents of the original module.
+
+    The other reason is that the dirty file can appear anywhere on
+    disk (e.g. in tmpfs), but it must be treated as having a path
+    matching the original module when it comes to usage of relative
+    paths, etc. Queries, however, will refer to the dirty module
+    name in their answers instead of the normal filename.
+
+
+Definitions
+-----------
+
+The ``--def`` idetools switch performs a query about the definition
+of a specific symbol. If available, idetools will answer with the
+type, source file, line/column information and other accessory data
+if available like a docstring. With this information an IDE can
+provide the typical *Jump to definition* where a user puts the
+cursor on a symbol or uses the mouse to select it and is redirected
+to the place where the symbol is located.
+
+Since Nimrod is implemented in Nimrod, one of the nice things of
+this feature is that any user with an IDE supporting it can quickly
+jump around the standard library implementation and see exactly
+what a proc does, learning about the language and seeing real life
+examples of how to write/implement specific features.
+
+Idetools will always answer with a single definition or none if it
+can't find any valid symbol matching the position of the query.
+
+
+Suggestions
+-----------
+
+The ``--suggest`` idetools switch performs a query about possible
+completion symbols at some point in the file. IDEs can easily provide
+an autocompletion feature where the IDE scans the current file (and
+related ones, if it knows about the language being edited and follows
+includes/imports) and when the user starts typing something a
+completion box with different options appears.
+
+However such features are not context sensitive and work simply on
+string matching, which can be problematic in Nimrod especially due
+to the case insensitiveness of the language (plus underscores as
+separators!).
+
+The typical usage scenario for this option is to call it after the
+user has typed the dot character for `the object oriented call
+syntax <tut2.html#method-call-syntax>`_.
+
+
+Invokation context
+------------------
+
+The ``--context`` idetools switch is very similar to the suggestions
+switch, but instead of being used after the user has typed a dot
+character, this one is meant to be used after the user has typed
+an opening brace to start typing parameters.
+
+
+Symbol usages
+-------------
+
+The ``--usages`` idetools switch lists all usages of the symbol at
+a position. IDEs can use this to find all the places in the file
+where the symbol is used and offer the user to rename it in all
+places at the same time. Again, a pure string based search and
+replace may catch symbols out of the scope of a funcion/loop.
+
+For this kind of query the IDE will most likely ignore all the
+type/signature info provided by idetools and concentrate on the
+filename, line and column position of the multiple returned answers.
+
+
+Expression evaluation
+---------------------
+
+This feature is still under development. In the future it will allow
+an IDE to evaluate an expression in the context of the currently
+running/debugged user project.
+
+
+Compiler as a service (CAAS)
+============================
+
+The ocasional use of idetools is acceptable for things like
+definitions, where the user puts the cursor on a symbol or double
+clicks it and after a second or two the IDE displays where that
+symbol is defined. Such latencies would be terrible for features
+like symbol suggestion, plus why wait at all if we can avoid it?
+
+The idetools command can be run as a compiler service, where you
+first launch the compiler and it will stay online as a server,
+accepting queries in a telnet like fashion. The advantage of staying
+on is that for many queries the compiler can cache the results of
+the compilation, and subsequent queries should be fast in the
+millisecond range, thus being responsive enough for IDEs.
+
+If you want to start the server using stdin/stdout as communication
+you need to type::
+
+    nimrod serve --server.type:stdin proj.nim
+
+If you want to start the server using tcp and a port, you need to type::
+
+    nimrod serve --server.type:tcp --server.port:6000 \
+      --server.address:hostname proj.nim
+
+In both cases the server will start up and await further commands.
+The syntax of the commands you can now send to the server is
+practically the same as running the nimrod compiler on the commandline,
+you only need to remove the name of the compiler since you are
+already talking to it. The server will answer with as many lines
+of text it thinks necessary plus an empty line to indicate the end
+of the answer.
+
+You can find examples of client/server communication in the idetools
+tests found in ``tests/caas/*.txt``.
+
+
+Parsing idetools output
+=======================
+
+Idetools outputs is always returned on single lines separated by
+tab characters (``\t``). The values of each column are:
+
+1. Three characters indicating the type of returned answer (e.g.
+   def for definition, ``sug`` for suggestion, etc).
+2. Type of the symbol. This can be ``skProc``, ``skLet``, and just
+   about any of the enums defined in the module ``compiler/ast.nim``.
+3. Full qualitifed path of the symbol. If you are querying a symbol
+   defined in the ``proj.nim`` file, this would have the form
+   ``proj.symbolName``.
+4. Type/signature. For variables and enums this will contain the
+   type of the symbol, for procs, methods and templates this will
+   contain the full unique signature (e.g. ``proc (TFile)``).
+5. Full path to the file containing the symbol.
+6. Line where the symbol is located in the file. Lines start to
+   count at **1**.
+7. Column where the symbol is located in the file. Columns start
+   to count at **0**.
+8. Docstring for the symbol if available or the empty string. To
+   differentiate the docstring from end of answer in server mode,
+   the docstring is always provided enclosed in double quotes, and
+   if the docstring spans multiple lines, all following lines of the
+   docstring will start with a blank space to align visually with
+   the starting quote.
+
+   Also, you won't find raw ``\n`` characters breaking the one
+   answer per line format. Instead you will need to parse sequences
+   in the form ``\xHH``, where *HH* is a hexadecimal value (e.g.
+   newlines generate the sequence ``\x0A``).
+
+The following sections define the expected output for each kind of
+symbol for which idetools returns valid output.
+
+
+skConst
+-------
+
+| **Third column**: module + [n scope nesting] + const name.
+| **Fourth column**: the type of the const value.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    const SOME_SEQUENCE = @[1, 2]
+    --> col 2: $MODULE.SOME_SEQUENCE
+        col 3: seq[int]
+        col 7: ""
+
+
+skEnumField
+-----------
+
+| **Third column**: module + [n scope nesting] + enum type + enum field name.
+| **Fourth column**: enum type grouping other enum fields.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    Open(filename, fmWrite)
+    --> col 2: system.TFileMode.fmWrite
+        col 3: TFileMode
+        col 7: ""
+
+
+skForVar
+--------
+
+| **Third column**: module + [n scope nesting] + var name.
+| **Fourth column**: type of the var.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    proc looper(filename = "tests.nim") =
+      for letter in filename:
+        echo letter
+    --> col 2: $MODULE.looper.letter
+        col 3: char
+        col 7: ""
+
+
+skIterator
+----------
+
+The fourth column will be the empty string if the iterator is being
+defined, since at that point in the file the parser hasn't processed
+the full line yet. The signature will be returned complete in
+posterior instances of the iterator.
+
+| **Third column**: module + [n scope nesting] + iterator name.
+| **Fourth column**: signature of the iterator including return type.
+| **Docstring**: docstring if available.
+
+.. code-block:: nimrod
+    let
+      text = "some text"
+      letters = toSeq(runes(text))
+    --> col 2: unicode.runes
+        col 3: iterator (string): TRune
+        col 7: "iterates over any unicode character of the string `s`."
+
+
+skLet
+-----
+
+| **Third column**: module + [n scope nesting] + let name.
+| **Fourth column**: the type of the let variable.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    let
+      text = "some text"
+    --> col 2: $MODULE.text
+        col 3: TaintedString
+        col 7: ""
+
+
+skMethod
+--------
+
+The fourth column will be the empty string if the method is being
+defined, since at that point in the file the parser hasn't processed
+the full line yet. The signature will be returned complete in
+posterior instances of the method.
+
+Methods imply `dynamic dispatch <tut2.html#dynamic-dispatch>`_ and
+idetools performs a static analysis on the code. For this reason
+idetools may not return the definition of the correct method you
+are querying because it may be impossible to know until the code
+is executed. It will try to return the method which covers the most
+possible cases (i.e. for variations of different classes in a
+hierarchy it will prefer methods using the base class).
+
+While at the language level a method is differentiated from others
+by the parameters and return value, the signature of the method
+returned by idetools returns also the pragmas for the method.
+
+Note that at the moment the word ``proc`` is returned for the
+signature of the found method instead of the expected ``method``.
+This may change in the future.
+
+| **Third column**: module + [n scope nesting] + method name.
+| **Fourth column**: signature of the method including return type.
+| **Docstring**: docstring if available.
+
+.. code-block:: nimrod
+    method eval(e: PExpr): int = quit "to override!"
+    method eval(e: PLiteral): int = e.x
+    method eval(e: PPlusExpr): int = eval(e.a) + eval(e.b)
+    echo eval(newPlus(newPlus(newLit(1), newLit(2)), newLit(4)))
+    --> col 2: $MODULE.eval
+        col 3: proc (PPlusExpr): int
+        col 7: ""
+
+
+skParam
+-------
+
+| **Third column**: module + [n scope nesting] + param name.
+| **Fourth column**: the type of the parameter.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    proc reader(filename = "tests.nim") =
+      let text = readFile(filename)
+    --> col 2: $MODULE.reader.filename
+        col 3: string
+        col 7: ""
+
+
+skProc
+------
+
+The fourth column will be the empty string if the proc is being
+defined, since at that point in the file the parser hasn't processed
+the full line yet. The signature will be returned complete in
+posterior instances of the proc.
+
+While at the language level a proc is differentiated from others
+by the parameters and return value, the signature of the proc
+returned by idetools returns also the pragmas for the proc.
+
+| **Third column**: module + [n scope nesting] + proc name.
+| **Fourth column**: signature of the proc including return type.
+| **Docstring**: docstring if available.
+
+.. code-block:: nimrod
+    Open(filename, fmWrite)
+    --> col 2: system.Open
+        col 3: proc (var TFile, string, TFileMode, int): bool
+        col 7:
+    "Opens a file named `filename` with given `mode`.
+
+     Default mode is readonly. Returns true iff the file could be opened.
+     This throws no exception if the file could not be opened."
+
+
+skResult
+--------
+
+| **Third column**: module + [n scope nesting] + result.
+| **Fourth column**: the type of the result.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    proc getRandomValue() : int =
+      return 4
+    --> col 2: $MODULE.getRandomValue.result
+        col 3: int
+        col 7: ""
+
+
+skTemplate
+----------
+
+The fourth column will be the empty string if the template is being
+defined, since at that point in the file the parser hasn't processed
+the full line yet. The signature will be returned complete in
+posterior instances of the template.
+
+| **Third column**: module + [n scope nesting] + template name.
+| **Fourth column**: signature of the template including return type.
+| **Docstring**: docstring if available.
+
+.. code-block:: nimrod
+    let
+      text = "some text"
+      letters = toSeq(runes(text))
+    --> col 2: sequtils.toSeq
+        col 3: proc (expr): expr
+        col 7:
+    "Transforms any iterator into a sequence.
+
+     Example:
+
+     .. code-block:: nimrod
+       let
+         numeric = @[1, 2, 3, 4, 5, 6, 7, 8, 9]
+         odd_numbers = toSeq(filter(numeric) do (x: int) -> bool:
+           if x mod 2 == 1:
+             result = true)
+       assert odd_numbers == @[1, 3, 5, 7, 9]"
+
+
+skType
+------
+
+| **Third column**: module + [n scope nesting] + type name.
+| **Fourth column**: the type.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    proc writeTempFile() =
+      var output: TFile
+    --> col 2: system.TFile
+        col 3: TFile
+        col 7: ""
+
+
+skVar
+-----
+
+| **Third column**: module + [n scope nesting] + var name.
+| **Fourth column**: the type of the var.
+| **Docstring**: always the empty string.
+
+.. code-block:: nimrod
+    proc writeTempFile() =
+      var output: TFile
+      output.open("/tmp/somefile", fmWrite)
+      output.write("test")
+    --> col 2: $MODULE.writeTempFile.output
+        col 3: TFile
+        col 7: ""