Merge pull request #1299 from gradha/pr_documents_interfacing

Documents backend interfacing

4 files changed, 443 insertions, 47 deletions

diff --git a/doc/advopt.txt b/doc/advopt.txt
index f5ff90791..e3a62c31b 100644
--- a/doc/advopt.txt
+++ b/doc/advopt.txt
@@ -1,7 +1,8 @@
 Advanced commands:
-  //compileToC, cc          compile project with C code generator
-  //compileToCpp, cpp       compile project to C++ code
-  //compileToOC, objc       compile project to Objective C code
+  //compileToC, cc          compile project with C code generator, see `Backend language options`_
+  //compileToCpp, cpp       compile project to C++ code, see `Backend language options`_
+  //compileToOC, objc       compile project to Objective C code, see `Backend language options`_
+  //js                      compile project to Javascript, see `Backend language options`_
   //rst2html                convert a reStructuredText file to HTML
   //rst2tex                 convert a reStructuredText file to TeX
   //jsondoc                 extract the documentation to a json file
diff --git a/doc/backends.txt b/doc/backends.txt
new file mode 100644
index 000000000..0ab682c29
--- /dev/null
+++ b/doc/backends.txt
@@ -0,0 +1,413 @@
+================================
+   Nimrod Backend Integration
+================================
+
+:Author: Puppet Master
+:Version: |nimrodversion|
+
+.. contents::
+
+  "If we all reacted the same way, we'd be predictable, and there's
+  always more than one way to view a situation. What's true for the
+  group is also true for the individual. It's simple: overspecialize,
+  and you breed in weakness.  It's slow death." -- Major Motoko
+  Kusanagi
+
+
+Introduction
+============
+
+The `Nimrod Compiler User Guide <nimrodc.html>`_ documents the typical
+compiler invocation, using the ``compile`` or ``c`` command to transform a
+``.nim`` file into one or more ``.c`` files which are then compiled with the
+platform's C compiler into a static binary. However there are other commands
+to compile to C++, Objective-C or JavaScript. This document tries to
+concentrate in a single place all the backend and interfacing options.
+
+The Nimrod compiler supports mainly two backend families: the C, C++ and
+Objective-C targets and the JavaScript target. `The C like targets`_ creates
+source files which can be compiled into a library or a final executable. `The
+JavaScript target`_ can generate a ``.js`` file which you reference from an
+HTML file or create a `standalone nodejs program <http://nodejs.org>`_.
+
+On top of generating libraries or standalone applications, Nimrod offers
+bidirectional interfacing with the backend targets through generic and
+specific pragmas.
+
+
+Backends
+========
+
+The C like targets
+------------------
+
+The commands to compile to either C, C++ or Objective-C are:
+
+  //compileToC, cc          compile project with C code generator
+  //compileToCpp, cpp       compile project to C++ code
+  //compileToOC, objc       compile project to Objective C code
+
+The most significant difference between these commands is that if you look
+into the ``nimcache`` directory you will find ``.c``, ``.cpp`` or ``.m``
+files, other than that all of them will produce a native binary for your
+project.  This allows you to take the generated code and place it directly
+into a project using any of these languages. Here are some typical command
+line invocations::
+
+    $ nimrod c hallo.nim
+    $ nimrod cpp hallo.nim
+    $ nimrod objc hallo.nim
+
+The compiler commands select the target backend, but if needed you can
+`specify additional switches for cross compilation
+<nimrodc.html#cross-compilation>`_ to select the target CPU, operative system
+or compiler/linker commands.
+
+
+The JavaScript target
+---------------------
+
+Nimrod can also generate `JavaScript`:idx: code through the ``js`` command.
+However, the JavaScript code generator is experimental!
+
+Nimrod targets JavaScript 1.5 which is supported by any widely used browser.
+Since JavaScript does not have a portable means to include another module,
+Nimrod just generates a long ``.js`` file.
+
+Features or modules that the JavaScript platform does not support are not
+available. This includes:
+
+* manual memory management (``alloc``, etc.)
+* casting and other unsafe operations (``cast`` operator, ``zeroMem``, etc.)
+* file management
+* most modules of the Standard library
+* proper 64 bit integer arithmetic
+* unsigned integer arithmetic
+
+However, the modules `strutils <strutils.html>`_, `math <math.html>`_, and
+`times <times.html>`_ are available! To access the DOM, use the `dom
+<dom.html>`_ module that is only available for the JavaScript platform.
+
+To compile a Nimrod module into a ``.js`` file use the ``js`` command; the
+default is a ``.js`` file that is supposed to be referenced in an ``.html``
+file. However, you can also run the code with `nodejs`:idx:, a `software
+platform for easily building fast, scalable network applications
+<http://nodejs.org>`_::
+
+  nimrod js -d:nodejs -r examples/hallo.nim
+
+
+Interfacing
+===========
+
+Nimrod offers bidirectional interfacing with the target backend. This means
+that you can call backend code from Nimrod and Nimrod code can be called by
+the backend code. Usually the direction of which calls which depends on your
+software architecture (is Nimrod your main program or is Nimrod providing a
+component?).
+
+
+Nimrod code calling the backend
+--------------------------------
+
+Nimrod code can interface with the backend through the `Foreign function
+interface <manual.html#foreign-function-interface>`_ mainly through the
+`importc pragma <manual.html#importc-pragma>`_. The ``importc`` pragma is the
+*generic* way of making backend symbols available in Nimrod and is available
+in all the target backends (JavaScript too).  The C++ or Objective-C backends
+have their respective `ImportCpp <nimrodc.html#importcpp-pragma>`_ and
+`ImportObjC <nimrodc.html#importobjc-pragma>`_ pragmas to call methods from
+classes.
+
+Whenever you use any of these pragmas you need to integrate native code into
+your final binary. In the case of JavaScript this is no problem at all, the
+same html file which hosts the generated JavaScript will likely provide other
+JavaScript functions which you are importing with ``importc``.
+
+However, for the C like targets you need to link external code either
+statically or dynamically. The preferred way of integrating native code is to
+use dynamic linking because it allows you to compile Nimrod programs without
+the need for having the related development libraries installed. This is done
+through the `dynlib pragma for import
+<manual.html#dynlib-pragma-for-import>`_, though more specific control can be
+gained using the `dynlib module <dynlib.html>`_.
+
+The `dynlibOverride <nimrodc.html#dynliboverride>`_ command line switch allows
+to avoid dynamic linking if you need to statically link something instead.
+Nimrod wrappers designed to statically link source files can use the `compile
+pragma <nimrodc.html#compile-pragma>`_ if there are few sources or providing
+them along the Nimrod code is easier than using a system library. Libraries
+installed on the host system can be linked in with the `PassL pragma
+<nimrodc.html#passl-pragma>`_.
+
+To wrap native code, take a look at the `c2nim tool <c2nim.html>`_ which helps
+with the process of scanning and transforming header files into a Nimrod
+interface.
+
+C invocation example
+~~~~~~~~~~~~~~~~~~~~
+
+Create a ``logic.c`` file with the following content:
+
+.. code-block:: c
+  int addTwoIntegers(int a, int b)
+  {
+    return a + b;
+  }
+
+Create a ``calculator.nim`` file with the following content:
+
+.. code-block:: nimrod
+
+  {.compile: "logic.c".}
+  proc addTwoIntegers(a, b: int): int {.importc.}
+
+  when isMainModule:
+    echo addTwoIntegers(3, 7)
+
+With these two files in place, you can run ``nimrod c -r calculator.nim`` and
+the Nimrod compiler will compile the ``logic.c`` file in addition to
+``calculator.nim`` and link both into an executable, which outputs ``10`` when
+run. Another way to link the C file statically and get the same effect would
+be remove the line with the ``compile`` pragma and run the following typical
+Unix commands::
+
+    $ gcc -c logic.c
+    $ ar rvs mylib.a logic.o
+    $ nimrod c --passL:mylib.a -r calculator.nim
+
+Just like in this example we pass the path to the ``mylib.a`` library (and we
+could as well pass ``logic.o``) we could be passing switches to link any other
+static C library.
+
+
+JavaScript invocation example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create a ``host.html`` file with the following content:
+
+.. code-block::
+
+  <html><body>
+  <script type="text/javascript">
+  function addTwoIntegers(a, b)
+  {
+    return a + b;
+  }
+  </script>
+  <script type="text/javascript" src="calculator.js"></script>
+  </body></html>
+
+Create a ``calculator.nim`` file with the following content (or reuse the one
+from the previous section):
+
+.. code-block:: nimrod
+
+  proc addTwoIntegers(a, b: int): int {.importc.}
+
+  when isMainModule:
+    echo addTwoIntegers(3, 7)
+
+Compile the Nimrod code to JavaScript with ``nimrod js -o:calculator.js
+calculator.nim`` and open ``host.html`` in a browser. If the browser supports
+javascript, you should see the value ``10``. In JavaScript the `echo proc
+<system.html#echo>`_ will modify the HTML DOM and append the string. Use the
+`dom module <dom.html>`_ for specific DOM querying and modification procs.
+
+
+Backend code calling Nimrod
+---------------------------
+
+Backend code can interface with Nimrod code exposed through the `exportc
+pragma <manual.html#exportc-pragma>`_. The ``exportc`` pragma is the *generic*
+way of making Nimrod symbols available to the backends. By default the Nimrod
+compiler will mangle all the Nimrod symbols to avoid any name collision, so
+the most significant thing the ``exportc`` pragma does is maintain the Nimrod
+symbol name, or if specified, use an alternative symbol for the backend in
+case the symbol rules don't match.
+
+The JavaScript target doesn't have any further interfacing considerations
+since it also has garbage collection, but the C targets require you to
+initialize Nimrod's internals, which is done calling a ``NimMain`` function.
+Also, C code requires you to specify a forward declaration for functions or
+the compiler will asume certain types for the return value and parameters
+which will likely make your program crash at runtime.
+
+The Nimrod compiler can generate a C interface header through the ``--header``
+command line switch. The generated header will contain all the exported
+symbols and the ``NimMain`` proc which you need to call before any other
+Nimrod code.
+
+
+Nimrod invocation example from C
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create a ``fib.nim`` file with the following content:
+
+.. code-block:: nimrod
+
+  proc fib(a: cint): cint {.exportc.} =
+    if a <= 2:
+      result = 1
+    else:
+      result = fib(a - 1) + fib(a - 2)
+
+Create a ``maths.c`` file with the following content:
+
+.. code-block:: c
+
+  #include "fib.h"
+  #include <stdio.h>
+
+  int main(void)
+  {
+    NimMain();
+    for (int f = 0; f < 10; f++)
+      printf("Fib of %d is %d\n", f, fib(f));
+    return 0;
+  }
+
+Now you can run the following Unix like commands to first generate C sources
+form the Nimrod code, then link them into a static binary along your main C
+program::
+
+  $ nimrod c --noMain --noLinking --header:fib.h fib.nim
+  $ gcc -o m -Inimcache -Ipath/to/nimrod/lib nimcache/*.c maths.c
+
+The first command runs the Nimrod compiler with three special options to avoid
+generating a ``main()`` function in the generated files, avoid linking the
+object files into a final binary, and explicitly generate a header file for C
+integration. All the generated files are placed into the ``nimcache``
+directory. That's why the next command compiles the ``maths.c`` source plus
+all the ``.c`` files form ``nimcache``. In addition to this path, you also
+have to tell the C compiler where to find Nimrod's ``nimbase.h`` header file.
+
+Instead of depending on the generation of the individual ``.c`` files you can
+also ask the Nimrod compiler to generate a statically linked library::
+
+  $ nimrod c --app:staticLib --noMain --header fib.nim
+  $ gcc -o m -Inimcache -Ipath/to/nimrod/lib libfib.nim.a maths.c
+
+The Nimrod compiler will handle linking the source files generated in the
+``nimcache`` directory into the ``libfib.nim.a`` static library, which you can
+then link into your C program.  Note that these commands are generic and will
+vary for each system. For instance, on Linux systems you will likely need to
+use ``-ldl`` too to link in required dlopen functionality.
+
+
+Nimrod invocation example from JavaScript
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Create a ``mhost.html`` file with the following content:
+
+.. code-block::
+
+  <html><body>
+  <script type="text/javascript" src="fib.js"></script>
+  <script type="text/javascript">
+  alert("Fib for 9 is " + fib(9));
+  </script>
+  </body></html>
+
+Create a ``fib.nim`` file with the following content (or reuse the one
+from the previous section):
+
+.. code-block:: nimrod
+
+  proc fib(a: cint): cint {.exportc.} =
+    if a <= 2:
+      result = 1
+    else:
+      result = fib(a - 1) + fib(a - 2)
+
+Compile the Nimrod code to JavaScript with ``nimrod js -o:fib.js fib.nim`` and
+open ``mhost.html`` in a browser. If the browser supports javascript, you
+should see an alert box displaying the text ``Fib for 9 is 34``. As mentioned
+earlier, JavaScript doesn't require an initialisation call to ``NimMain`` or
+similar function and you can call the exported Nimrod proc directly.
+
+
+Memory management
+=================
+
+In the previous sections the ``NimMain()`` function reared its head. Since
+JavaScript already provides automatic memory management, you can freely pass
+objects between the two language without problems. In C and derivate languages
+you need to be careful about what you do and how you share memory. The
+previous examples only dealt with simple scalar values, but passing a Nimrod
+string to C, or reading back a C string in Nimrod already requires you to be
+aware of who controls what to avoid crashing.
+
+
+Strings and C strings
+---------------------
+
+The manual mentions that `Nimrod strings are implicitly convertible to
+cstrings <manual.html#cstring-type>`_ which makes interaction usually
+painless. Most C functions accepting a Nimrod string converted to a
+``cstring`` will likely not need to keep this string around and by the time
+they return the string won't be needed any more. However, for the rare cases
+where a Nimrod string has to be preserved and made available to the C backend
+as a ``cstring``, you will need to manually prevent the string data from being
+freed with `GC_ref <system.html#GC_ref>`_ and `GC_unref
+<system.html#GC_unref>`_.
+
+A similar thing happens with C code invoking Nimrod code which returns a
+``cstring``. Consider the following proc:
+
+.. code-block:: nimrod
+
+  proc gimme(): cstring {.exportc.} =
+    result = "Hey there C code! " & $random(100)
+
+Since Nimrod's garbage collector is not aware of the C code, once the
+``gimme`` proc has finished it can reclaim the memory of the ``cstring``.
+However, from a practical standpoint, the C code invoking the ``gimme``
+function directly will be able to use it since Nimrod's garbage collector has
+not had a chance to run *yet*. This gives you enough time to make a copy for
+the C side of the program, as calling any further Nimrod procs *might* trigger
+garbage collection making the previously returned string garbage. Or maybe you
+are `triggering yourself the collection <gc.html>`_.
+
+
+Custom data types
+-----------------
+
+Just like strings, custom data types that are to be shared between Nimrod and
+the backend will need careful consideration of who controlls who. If you want
+to hand a Nimrod reference to C code, you will need to use `GC_ref
+<system.html#GC_ref>`_ to mark the reference as used, so it does not get
+freed. And for the C backend you will need to expose the `GC_unref
+<system.html#GC_unref>`_ proc to clean up this memory when it is not required
+any more.
+
+Again, if you are wrapping a library which *mallocs* and *frees* data
+structures, you need to expose the appropriate *free* function to Nimrod so
+you can clean it up. And of course, once cleaned you should avoid accessing it
+from Nimrod (or C for that matter). Typically C data structures have their own
+``malloc_structure`` and ``free_structure`` specific functions, so wrapping
+these for the Nimrod side should be enough.
+
+
+Thread coordination
+-------------------
+
+When the ``NimMain()`` function is called Nimrod initializes the garbage
+collector to the current thread, which is usually the main thread of your
+application. If your C code later spawns a different thread and calls Nimrod
+code, the garbage collector will fail to work properly and you will crash.
+
+As long as you don't use the threadvar emulation Nimrod uses native thread
+variables, of which you get a fresh version whenever you create a thread. You
+can then attach a GC to this thread via
+
+.. code-block:: nimrod
+
+  setStackBottom(addr(someLocal))
+  initGC()
+
+At the moment this support is still experimental so you need to expose these
+functions yourself or submit patches to request a public API. If the Nimrod
+code you are calling is short lived, another possible solution is to disable
+the garbage collector and enable it after the call from your background
+thread.
diff --git a/doc/manual.txt b/doc/manual.txt
index e96e50999..8a7cfba54 100644
--- a/doc/manual.txt
+++ b/doc/manual.txt
@@ -5488,7 +5488,10 @@ spelled*:
   proc printf(formatstr: cstring) {.header: "<stdio.h>", importc: "printf", varargs.}
 
 Note that this pragma is somewhat of a misnomer: Other backends will provide
-the same feature under the same name.
+the same feature under the same name. Also, if you are interfacing with C++
+you can use the `ImportCpp pragma <nimrodc.html#importcpp-pragma>`_ and
+interfacing with Objective-C the `ImportObjC pragma
+<nimrodc.html#importobjc-pragma>`_.
 
 
 Exportc pragma
diff --git a/doc/nimrodc.txt b/doc/nimrodc.txt
index fea1037da..428c42f39 100644
--- a/doc/nimrodc.txt
+++ b/doc/nimrodc.txt
@@ -384,10 +384,11 @@ Example:
 

 ImportCpp pragma

 ----------------

-The ``importcpp`` pragma can be used to import `C++`:idx: methods. The

-generated code then uses the C++ method calling syntax: ``obj->method(arg)``.

-In addition with the ``header`` and ``emit`` pragmas this allows *sloppy*

-interfacing with libraries written in C++:

+Similar to the `importc pragma for C <manual.html#importc-pragma>`_, the
+``importcpp`` pragma can be used to import `C++`:idx: methods. The generated
+code then uses the C++ method calling syntax: ``obj->method(arg)``.  In
+addition with the ``header`` and ``emit`` pragmas this allows *sloppy*
+interfacing with libraries written in C++:
 

 .. code-block:: Nimrod

   # Horrible example of how to interface with a C++ engine ... ;-)

@@ -422,11 +423,11 @@ emits C++ code.
 

 ImportObjC pragma

 -----------------

-The ``importobjc`` pragma can be used to import `Objective C`:idx: methods. 

-The generated code then uses the Objective C method calling 

-syntax: ``[obj method param1: arg]``.

-In addition with the ``header`` and ``emit`` pragmas this allows *sloppy*

-interfacing with libraries written in Objective C:

+Similar to the `importc pragma for C <manual.html#importc-pragma>`_, the
+``importobjc`` pragma can be used to import `Objective C`:idx: methods.  The
+generated code then uses the Objective C method calling syntax: ``[obj method
+param1: arg]``.  In addition with the ``header`` and ``emit`` pragmas this
+allows *sloppy* interfacing with libraries written in Objective C:
 

 .. code-block:: Nimrod

   # horrible example of how to interface with GNUStep ...

@@ -550,8 +551,18 @@ against. For instance, to link statically against Lua this command might work
 on Linux::
 
   nimrod c --dynlibOverride:lua --passL:liblua.lib program.nim
-

-

+
+
+Backend language options
+========================
+
+The typical compiler usage involves using the ``compile`` or ``c`` command to
+transform a ``.nim`` file into one or more ``.c`` files which are then
+compiled with the platform's C compiler into a static binary. However there
+are other commands to compile to C++, Objective-C or Javascript. More details
+can be read in the `Nimrod Backend Integration document <backends.html>`_.
+
+
 Nimrod documentation tools
 ==========================
 
@@ -695,35 +706,3 @@ efficient:
     else: quit(errorStr(p, "expected: console or gui"))

   of "license": c.license = UnixToNativePath(k.value)

   else: quit(errorStr(p, "unknown variable: " & k.key))

-
-
-The JavaScript target
-=====================
-
-Nimrod can also generate `JavaScript`:idx: code. However, the
-JavaScript code generator is experimental!
-
-Nimrod targets JavaScript 1.5 which is supported by any widely used browser.
-Since JavaScript does not have a portable means to include another module,
-Nimrod just generates a long ``.js`` file.
-
-Features or modules that the JavaScript platform does not support are not
-available. This includes:
-
-* manual memory management (``alloc``, etc.)
-* casting and other unsafe operations (``cast`` operator, ``zeroMem``, etc.)
-* file management
-* most modules of the Standard library
-* proper 64 bit integer arithmetic
-* unsigned integer arithmetic
-
-However, the modules `strutils`:idx:, `math`:idx:, and `times`:idx: are
-available! To access the DOM, use the `dom`:idx: module that is only
-available for the JavaScript platform.
-
-To compile a Nimrod module into a ``.js`` file use the ``js`` command; the
-default is a  ``.js`` file that is supposed to be referenced in an ``.html``
-file. However, you can also run the code with `nodejs`:idx:\:  
-
-  nimrod js -d:nodejs -r examples/hallo.nim
-