manual split up into multiple files; documented the new concurrency system

1 files changed, 404 insertions, 0 deletions

diff --git a/doc/manual/templates.txt b/doc/manual/templates.txt
new file mode 100644
index 000000000..d63f61f54
--- /dev/null
+++ b/doc/manual/templates.txt
@@ -0,0 +1,404 @@
+Templates
+=========
+
+A template is a simple form of a macro: It is a simple substitution
+mechanism that operates on Nim's abstract syntax trees. It is processed in
+the semantic pass of the compiler.
+
+The syntax to *invoke* a template is the same as calling a procedure.
+
+Example:
+
+.. code-block:: nim
+  template `!=` (a, b: expr): expr =
+    # this definition exists in the System module
+    not (a == b)
+
+  assert(5 != 6) # the compiler rewrites that to: assert(not (5 == 6))
+
+The ``!=``, ``>``, ``>=``, ``in``, ``notin``, ``isnot`` operators are in fact 
+templates:
+
+| ``a > b`` is transformed into ``b < a``.
+| ``a in b`` is transformed into ``contains(b, a)``. 
+| ``notin`` and ``isnot`` have the obvious meanings.
+
+The "types" of templates can be the symbols ``expr`` (stands for *expression*), 
+``stmt`` (stands for *statement*) or ``typedesc`` (stands for *type 
+description*). These are "meta types", they can only be used in certain 
+contexts. Real types can be used too; this implies that expressions are 
+expected.
+
+
+Ordinary vs immediate templates
+-------------------------------
+
+There are two different kinds of templates: immediate templates and
+ordinary templates. Ordinary templates take part in overloading resolution. As
+such their arguments need to be type checked before the template is invoked.
+So ordinary templates cannot receive undeclared identifiers:
+
+.. code-block:: nim
+
+  template declareInt(x: expr) = 
+    var x: int
+
+  declareInt(x) # error: unknown identifier: 'x'
+
+An ``immediate`` template does not participate in overload resolution and so
+its arguments are not checked for semantics before invocation. So they can
+receive undeclared identifiers:
+
+.. code-block:: nim
+
+  template declareInt(x: expr) {.immediate.} = 
+    var x: int
+
+  declareInt(x) # valid
+
+
+Passing a code block to a template
+----------------------------------
+
+If there is a ``stmt`` parameter it should be the last in the template
+declaration, because statements are passed to a template via a
+special ``:`` syntax:
+
+.. code-block:: nim
+
+  template withFile(f, fn, mode: expr, actions: stmt): stmt {.immediate.} =
+    var f: TFile
+    if open(f, fn, mode):
+      try:
+        actions
+      finally:
+        close(f)
+    else:
+      quit("cannot open: " & fn)
+      
+  withFile(txt, "ttempl3.txt", fmWrite):
+    txt.writeln("line 1")
+    txt.writeln("line 2")
+  
+In the example the two ``writeln`` statements are bound to the ``actions``
+parameter.
+
+
+Symbol binding in templates
+---------------------------
+
+A template is a `hygienic`:idx: macro and so opens a new scope. Most symbols are
+bound from the definition scope of the template:
+
+.. code-block:: nim
+  # Module A
+  var 
+    lastId = 0
+  
+  template genId*: expr =
+    inc(lastId)
+    lastId
+
+.. code-block:: nim
+  # Module B
+  import A
+  
+  echo genId() # Works as 'lastId' has been bound in 'genId's defining scope
+
+As in generics symbol binding can be influenced via ``mixin`` or ``bind`` 
+statements.
+
+
+
+Identifier construction
+-----------------------
+
+In templates identifiers can be constructed with the backticks notation:
+
+.. code-block:: nim
+
+  template typedef(name: expr, typ: typedesc) {.immediate.} = 
+    type
+      `T name`* {.inject.} = typ
+      `P name`* {.inject.} = ref `T name`
+      
+  typedef(myint, int)
+  var x: PMyInt
+
+In the example ``name`` is instantiated with ``myint``, so \`T name\` becomes
+``Tmyint``.
+
+
+Lookup rules for template parameters
+------------------------------------
+
+A parameter ``p`` in a template is even substituted in the expression ``x.p``.
+Thus template arguments can be used as field names and a global symbol can be
+shadowed by the same argument name even when fully qualified:
+
+.. code-block:: nim
+  # module 'm'
+
+  type
+    TLev = enum
+      levA, levB
+
+  var abclev = levB
+
+  template tstLev(abclev: TLev) =
+    echo abclev, " ", m.abclev
+
+  tstLev(levA)
+  # produces: 'levA levA'
+  
+But the global symbol can properly be captured by a ``bind`` statement:
+
+.. code-block:: nim
+  # module 'm'
+
+  type
+    TLev = enum
+      levA, levB
+
+  var abclev = levB
+
+  template tstLev(abclev: TLev) =
+    bind m.abclev
+    echo abclev, " ", m.abclev
+
+  tstLev(levA)
+  # produces: 'levA levB'
+
+
+Hygiene in templates
+--------------------
+
+Per default templates are `hygienic`:idx:\: Local identifiers declared in a
+template cannot be accessed in the instantiation context:
+
+.. code-block:: nim
+  
+  template newException*(exceptn: typedesc, message: string): expr =
+    var
+      e: ref exceptn  # e is implicitly gensym'ed here
+    new(e)
+    e.msg = message
+    e
+    
+  # so this works:
+  let e = "message"
+  raise newException(EIO, e)
+
+
+Whether a symbol that is declared in a template is exposed to the instantiation
+scope is controlled by the `inject`:idx: and `gensym`:idx: pragmas: gensym'ed
+symbols are not exposed but inject'ed are.
+
+The default for symbols of entity ``type``, ``var``, ``let`` and ``const``
+is ``gensym`` and for ``proc``, ``iterator``, ``converter``, ``template``,
+``macro`` is ``inject``. However, if the name of the entity is passed as a 
+template parameter, it is an inject'ed symbol:
+
+.. code-block:: nim
+  template withFile(f, fn, mode: expr, actions: stmt): stmt {.immediate.} =
+    block:
+      var f: TFile  # since 'f' is a template param, it's injected implicitly
+      ...
+      
+  withFile(txt, "ttempl3.txt", fmWrite):
+    txt.writeln("line 1")
+    txt.writeln("line 2")
+
+
+The ``inject`` and ``gensym`` pragmas are second class annotations; they have
+no semantics outside of a template definition and cannot be abstracted over:
+
+.. code-block:: nim
+  {.pragma myInject: inject.}
+  
+  template t() =
+    var x {.myInject.}: int # does NOT work
+
+
+To get rid of hygiene in templates, one can use the `dirty`:idx: pragma for
+a template. ``inject`` and ``gensym`` have no effect in ``dirty`` templates.
+
+
+
+Macros
+======
+
+A macro is a special kind of low level template. Macros can be used
+to implement `domain specific languages`:idx:. Like templates, macros come in
+the 2 flavors *immediate* and *ordinary*.
+
+While macros enable advanced compile-time code transformations, they
+cannot change Nim's syntax. However, this is no real restriction because
+Nim's syntax is flexible enough anyway.
+
+To write macros, one needs to know how the Nim concrete syntax is converted
+to an abstract syntax tree.
+
+There are two ways to invoke a macro:
+(1) invoking a macro like a procedure call (`expression macros`)
+(2) invoking a macro with the special ``macrostmt`` syntax (`statement macros`)
+
+
+Expression Macros
+-----------------
+
+The following example implements a powerful ``debug`` command that accepts a
+variable number of arguments:
+
+.. code-block:: nim
+  # to work with Nim syntax trees, we need an API that is defined in the
+  # ``macros`` module:
+  import macros
+
+  macro debug(n: varargs[expr]): stmt =
+    # `n` is a Nim AST that contains the whole macro invocation
+    # this macro returns a list of statements:
+    result = newNimNode(nnkStmtList, n)
+    # iterate over any argument that is passed to this macro:
+    for i in 0..n.len-1:
+      # add a call to the statement list that writes the expression;
+      # `toStrLit` converts an AST to its string representation:
+      add(result, newCall("write", newIdentNode("stdout"), toStrLit(n[i])))
+      # add a call to the statement list that writes ": "
+      add(result, newCall("write", newIdentNode("stdout"), newStrLitNode(": ")))
+      # add a call to the statement list that writes the expressions value:
+      add(result, newCall("writeln", newIdentNode("stdout"), n[i]))
+
+  var
+    a: array [0..10, int]
+    x = "some string"
+  a[0] = 42
+  a[1] = 45
+
+  debug(a[0], a[1], x)
+
+The macro call expands to:
+
+.. code-block:: nim
+  write(stdout, "a[0]")
+  write(stdout, ": ")
+  writeln(stdout, a[0])
+
+  write(stdout, "a[1]")
+  write(stdout, ": ")
+  writeln(stdout, a[1])
+
+  write(stdout, "x")
+  write(stdout, ": ")
+  writeln(stdout, x)
+
+
+Arguments that are passed to a ``varargs`` parameter are wrapped in an array
+constructor expression. This is why ``debug`` iterates over all of ``n``'s
+children.
+
+
+BindSym
+-------
+
+The above ``debug`` macro relies on the fact that ``write``, ``writeln`` and
+``stdout`` are declared in the system module and thus visible in the 
+instantiating context. There is a way to use bound identifiers
+(aka `symbols`:idx:) instead of using unbound identifiers. The ``bindSym`` 
+builtin can be used for that:
+
+.. code-block:: nim
+  import macros
+
+  macro debug(n: varargs[expr]): stmt =
+    result = newNimNode(nnkStmtList, n)
+    for i in 0..n.len-1:
+      # we can bind symbols in scope via 'bindSym':
+      add(result, newCall(bindSym"write", bindSym"stdout", toStrLit(n[i])))
+      add(result, newCall(bindSym"write", bindSym"stdout", newStrLitNode(": ")))
+      add(result, newCall(bindSym"writeln", bindSym"stdout", n[i]))
+
+  var
+    a: array [0..10, int]
+    x = "some string"
+  a[0] = 42
+  a[1] = 45
+
+  debug(a[0], a[1], x)
+
+The macro call expands to:
+
+.. code-block:: nim
+  write(stdout, "a[0]")
+  write(stdout, ": ")
+  writeln(stdout, a[0])
+
+  write(stdout, "a[1]")
+  write(stdout, ": ")
+  writeln(stdout, a[1])
+
+  write(stdout, "x")
+  write(stdout, ": ")
+  writeln(stdout, x)
+
+However, the symbols ``write``, ``writeln`` and ``stdout`` are already bound
+and are not looked up again. As the example shows, ``bindSym`` does work with
+overloaded symbols implicitly.
+
+
+Statement Macros
+----------------
+
+Statement macros are defined just as expression macros. However, they are
+invoked by an expression following a colon.
+
+The following example outlines a macro that generates a lexical analyzer from
+regular expressions:
+
+.. code-block:: nim
+  import macros
+
+  macro case_token(n: stmt): stmt =
+    # creates a lexical analyzer from regular expressions
+    # ... (implementation is an exercise for the reader :-)
+    discard
+
+  case_token: # this colon tells the parser it is a macro statement
+  of r"[A-Za-z_]+[A-Za-z_0-9]*":
+    return tkIdentifier
+  of r"0-9+":
+    return tkInteger
+  of r"[\+\-\*\?]+":
+    return tkOperator
+  else:
+    return tkUnknown
+
+
+**Style note**: For code readability, it is the best idea to use the least
+powerful programming construct that still suffices. So the "check list" is:
+
+(1) Use an ordinary proc/iterator, if possible.
+(2) Else: Use a generic proc/iterator, if possible.
+(3) Else: Use a template, if possible.
+(4) Else: Use a macro.
+
+
+Macros as pragmas
+-----------------
+
+Whole routines (procs, iterators etc.) can also be passed to a template or 
+a macro via the pragma notation: 
+
+.. code-block:: nim
+  template m(s: stmt) = discard
+
+  proc p() {.m.} = discard
+
+This is a simple syntactic transformation into:
+
+.. code-block:: nim
+  template m(s: stmt) = discard
+
+  m:
+    proc p() = discard
+