diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/advopt.txt | 4 | ||||
| -rw-r--r-- | doc/codeowners.rst | 2 | ||||
| -rw-r--r-- | doc/contributing.rst | 8 | ||||
| -rw-r--r-- | doc/filters.rst | 9 | ||||
| -rw-r--r-- | doc/grammar.txt | 9 | ||||
| -rw-r--r-- | doc/intern.rst | 2 | ||||
| -rw-r--r-- | doc/lib.rst | 5 | ||||
| -rw-r--r-- | doc/manual.rst | 71 | ||||
| -rw-r--r-- | doc/nims.rst | 2 | ||||
| -rw-r--r-- | doc/tut2.rst | 402 | ||||
| -rw-r--r-- | doc/tut3.rst | 354 |
11 files changed, 424 insertions, 444 deletions
diff --git a/doc/advopt.txt b/doc/advopt.txt index 252067129..7cd72f6c3 100644 --- a/doc/advopt.txt +++ b/doc/advopt.txt @@ -29,6 +29,9 @@ Advanced options: produce hints or errors for Nim identifiers that do not adhere to Nim's official style guide https://nim-lang.org/docs/nep1.html + --showAllMismatches:on|off + show all mismatching candidates in overloading + resolution --lib:PATH set the system library path --import:PATH add an automatically imported module --include:PATH add an automatically included module @@ -98,6 +101,7 @@ Advanced options: --listCmd list the commands used to execute external programs --parallelBuild:0|1|... perform a parallel build value = number of processors (0 for auto-detect) + --incremental:on|off only recompile the changed modules (experimental!) --verbosity:0|1|2|3 set Nim's verbosity level (1 is default) --experimental:$1 enable experimental language feature diff --git a/doc/codeowners.rst b/doc/codeowners.rst index 0758e3ffb..129c47cfc 100644 --- a/doc/codeowners.rst +++ b/doc/codeowners.rst @@ -51,7 +51,7 @@ async dom96 strutils araq sequtils dom96, araq times GULPF -os, ospaths dom96, araq +os dom96, araq re araq nre flaviu math, fenv krux02, cooldome diff --git a/doc/contributing.rst b/doc/contributing.rst index 84b15f419..a2c95db74 100644 --- a/doc/contributing.rst +++ b/doc/contributing.rst @@ -22,7 +22,7 @@ There are 3 types of tests: 2. tests in ``when isMainModule:`` block, ran by ``nim c mymod.nim`` ``nimble test`` also typially runs these in external nimble packages. -3. testament tests, eg: tests/stdlib/tospaths.nim (only used for Nim repo). +3. testament tests, eg: tests/stdlib/tos.nim (only used for Nim repo). Not all the tests follow the convention here, feel free to change the ones that don't. Always leave the code cleaner than you found it. @@ -30,7 +30,7 @@ that don't. Always leave the code cleaner than you found it. Stdlib ------ -If you change the stdlib (anything under ``lib/``, eg ``lib/pure/ospaths.nim``), +If you change the stdlib (anything under ``lib/``, eg ``lib/pure/os.nim``), put a test in the file you changed. Add the tests under a ``when isMainModule:`` condition so they only get executed when the tester is building the file. Each test should be in a separate ``block:`` statement, such that @@ -53,7 +53,7 @@ Sample test: doAssert: not 1 == 2 Newer tests tend to be run via ``testament`` rather than via ``when isMainModule:``, -eg ``tests/stdlib/tospaths.nim``; this allows additional features such as custom +eg ``tests/stdlib/tos.nim``; this allows additional features such as custom compiler flags; for more details see below. Compiler @@ -197,7 +197,7 @@ as well as ``testament`` and guarantee they stay in sync. result = a & "Bar" -See `parentDir <https://nim-lang.github.io/Nim/ospaths.html#parentDir%2Cstring>`_ +See `parentDir <https://nim-lang.github.io/Nim/os.html#parentDir%2Cstring>`_ example. The RestructuredText Nim uses has a special syntax for including code snippets diff --git a/doc/filters.rst b/doc/filters.rst index e8106749e..40346ecaf 100644 --- a/doc/filters.rst +++ b/doc/filters.rst @@ -31,13 +31,16 @@ Usage ===== First, put your SCF code in a separate file with filters specified in the first line. -**Note:** You can name your SCF file with any file extension you want, but the conventional extension is ``.tmpl``. +**Note:** You can name your SCF file with any file extension you want, but the +conventional extension is ``.nimf`` +(it used to be ``.tmpl`` but that was too generic, for example preventing github to +recognize it as Nim source file). -If we use `generateXML` code shown above and call the SCF file `xmlGen.tmpl` +If we use `generateXML` code shown above and call the SCF file `xmlGen.nimf` In your `main.nim`: .. code-block:: nim - include "xmlGen.tmpl" + include "xmlGen.nimf" echo generateXML("John Smith","42") diff --git a/doc/grammar.txt b/doc/grammar.txt index e06ebd5d9..b5aa6fd5a 100644 --- a/doc/grammar.txt +++ b/doc/grammar.txt @@ -9,7 +9,7 @@ operator = OP0 | OP1 | OP2 | OP3 | OP4 | OP5 | OP6 | OP7 | OP8 | OP9 | 'is' | 'isnot' | 'in' | 'notin' | 'of' | 'div' | 'mod' | 'shl' | 'shr' | 'not' | 'static' | '..' prefixOperator = operator -optInd = COMMENT? +optInd = COMMENT? IND? optPar = (IND{>} | IND{=})? simpleExpr = arrowExpr (OP0 optInd arrowExpr)* pragma? arrowExpr = assignExpr (OP1 optInd assignExpr)* @@ -85,17 +85,19 @@ paramListColon = paramList? (':' optInd typeDesc)? doBlock = 'do' paramListArrow pragmas? colcom stmt procExpr = 'proc' paramListColon pragmas? ('=' COMMENT? stmt)? distinct = 'distinct' optInd typeDesc +forStmt = 'for' (identWithPragma ^+ comma) 'in' expr colcom stmt +forExpr = forStmt expr = (blockExpr | ifExpr | whenExpr | caseExpr + | forExpr | tryExpr) / simpleExpr typeKeyw = 'var' | 'out' | 'ref' | 'ptr' | 'shared' | 'tuple' | 'proc' | 'iterator' | 'distinct' | 'object' | 'enum' primary = typeKeyw typeDescK / prefixOperator* identOrLiteral primarySuffix* - / 'static' primary / 'bind' primary typeDesc = simpleExpr typeDefAux = simpleExpr @@ -142,12 +144,11 @@ tryExpr = 'try' colcom stmt &(optInd 'except'|'finally') (optInd 'except' exprList colcom stmt)* (optInd 'finally' colcom stmt)? exceptBlock = 'except' colcom stmt -forStmt = 'for' (identWithPragma ^+ comma) 'in' expr colcom stmt blockStmt = 'block' symbol? colcom stmt blockExpr = 'block' symbol? colcom stmt staticStmt = 'static' colcom stmt deferStmt = 'defer' colcom stmt -asmStmt = 'asm' pragma? (STR_LIT | RSTR_LIT | TRIPLE_STR_LIT) +asmStmt = 'asm' pragma? (STR_LIT | RSTR_LIT | TRIPLESTR_LIT) genericParam = symbol (comma symbol)* (colon expr)? ('=' optInd expr)? genericParamList = '[' optInd genericParam ^* (comma/semicolon) optPar ']' diff --git a/doc/intern.rst b/doc/intern.rst index 0fa4fd7b6..b71ad592f 100644 --- a/doc/intern.rst +++ b/doc/intern.rst @@ -31,7 +31,7 @@ Path Purpose reStructuredText files ``lib`` the Nim library ``web`` website of Nim; generated by ``nimweb`` - from the ``*.txt`` and ``*.tmpl`` files + from the ``*.txt`` and ``*.nimf`` files ============ =================================================== diff --git a/doc/lib.rst b/doc/lib.rst index 89ab26d81..46e1f9d19 100644 --- a/doc/lib.rst +++ b/doc/lib.rst @@ -147,10 +147,13 @@ String handling * `subexes <subexes.html>`_ This module implements advanced string substitution operations. -* `editdistance <editdistance>`_ +* `std/editdistance <editdistance.html>`_ This module contains an algorithm to compute the edit distance between two Unicode strings. +* `std/wordwrap <wordwrap.html>`_ + This module contains an algorithm to wordwrap a Unicode string. + Generic Operating System Services --------------------------------- diff --git a/doc/manual.rst b/doc/manual.rst index 98943619d..b39711dfb 100644 --- a/doc/manual.rst +++ b/doc/manual.rst @@ -514,6 +514,9 @@ are used for other notational purposes. ``*:`` is as a special case treated as the two tokens `*`:tok: and `:`:tok: (to support ``var v*: T``). +The ``not`` keyword is always a unary operator, ``a not b`` is parsed +as ``a(not b)``, not as ``(a) not (b)``. + Other tokens ------------ @@ -1231,6 +1234,37 @@ so that the builtin ``echo`` proc does what is expected: # prints "@[1, 2, 3]" and not "123" +Unchecked arrays +---------------- +The ``UncheckedArray[T]`` type is a special kind of ``array`` where its bounds +are not checked. This is often useful to implement customized flexibly sized +arrays. Additionally an unchecked array is translated into a C array of +undetermined size: + +.. code-block:: nim + type + MySeq = object + len, cap: int + data: UncheckedArray[int] + +Produces roughly this C code: + +.. code-block:: C + typedef struct { + NI len; + NI cap; + NI data[]; + } MySeq; + +The base type of the unchecked array may not contain any GC'ed memory but this +is currently not checked. + +**Future directions**: GC'ed memory should be allowed in unchecked arrays and +there should be an explicit annotation of how the GC is to determine the +runtime size of the array. + + + Tuples and object types ----------------------- A variable of a tuple or object type is a heterogeneous storage @@ -2349,7 +2383,8 @@ Automatic self insertions Starting with version 0.14 of the language, Nim supports ``field`` as a shortcut for ``self.field`` comparable to the `this`:idx: keyword in Java or C++. This feature has to be explicitly enabled via a ``{.this: self.}`` -statement pragma. This pragma is active for the rest of the module: +statement pragma (instead of ``self`` any other identifier can be used too). +This pragma is active for the rest of the module: .. code-block:: nim type @@ -2364,10 +2399,6 @@ statement pragma. This pragma is active for the rest of the module: # is rewritten to: # result = self.parentField + self.childField -Instead of ``self`` any other identifier can be used too, but -``{.this: self.}`` will become the default directive for the whole language -eventually. - In addition to fields, routine applications are also rewritten, but only if no other interpretation of the call is possible: @@ -7897,36 +7928,6 @@ defined, and it should not be used with GC'ed memory (ref's). **Future directions**: Using GC'ed memory in packed pragma will result in compile-time error. Usage with inheritance should be defined and documented. -Unchecked pragma ----------------- -The ``unchecked`` pragma can be used to mark a named array as ``unchecked`` -meaning its bounds are not checked. This is often useful to -implement customized flexibly sized arrays. Additionally an unchecked array is -translated into a C array of undetermined size: - -.. code-block:: nim - type - ArrayPart{.unchecked.} = array[0, int] - MySeq = object - len, cap: int - data: ArrayPart - -Produces roughly this C code: - -.. code-block:: C - typedef struct { - NI len; - NI cap; - NI data[]; - } MySeq; - -The base type of the unchecked array may not contain any GC'ed memory but this -is currently not checked. - -**Future directions**: GC'ed memory should be allowed in unchecked arrays and -there should be an explicit annotation of how the GC is to determine the -runtime size of the array. - Dynlib pragma for import ------------------------ diff --git a/doc/nims.rst b/doc/nims.rst index dea09e1e8..034ad1fda 100644 --- a/doc/nims.rst +++ b/doc/nims.rst @@ -31,7 +31,7 @@ available. So the stdlib modules using ``importc`` cannot be used with Nim's VM. However, at least the following modules are available: * `macros <macros.html>`_ -* `ospaths <ospaths.html>`_ +* `os <os.html>`_ * `strutils <strutils.html>`_ * `math <math.html>`_ * `distros <distros.html>`_ diff --git a/doc/tut2.rst b/doc/tut2.rst index 39e3bd89a..d0c6e7247 100644 --- a/doc/tut2.rst +++ b/doc/tut2.rst @@ -13,7 +13,6 @@ Introduction "Repetition renders the ridiculous reasonable." -- Norman Wildberger - This document is a tutorial for the advanced constructs of the *Nim* programming language. **Note that this document is somewhat obsolete as the** `manual <manual.html>`_ **contains many more examples of the advanced language @@ -652,369 +651,8 @@ avoid a common bug: to forget to close the file. Note how the ``let fn = filename`` statement ensures that ``filename`` is evaluated only once. -Macros -====== - -Macros enable advanced compile-time code transformations, but they cannot -change Nim's syntax. However, this is no real restriction because Nim's -syntax is flexible enough anyway. Macros have to be implemented in pure Nim -code if the `foreign function interface (FFI) -<manual.html#foreign-function-interface>`_ is not enabled in the compiler, but -other than that restriction (which at some point in the future will go away) -you can write any kind of Nim code and the compiler will run it at compile -time. - -There are two ways to write a macro, either *generating* Nim source code and -letting the compiler parse it, or creating manually an abstract syntax tree -(AST) which you feed to the compiler. In order to build the AST one needs to -know how the Nim concrete syntax is converted to an abstract syntax tree -(AST). The AST is documented in the `macros <macros.html>`_ module. - -Once your macro is finished, there are two ways to invoke it: -(1) invoking a macro like a procedure call (expression macros) -(2) invoking a macro with the special ``macrostmt`` - syntax (statement macros) - - -Expression Macros ------------------ - -The following example implements a powerful ``debug`` command that accepts a -variable number of arguments: - -.. code-block:: nim - :test: "nim c $1" - # to work with Nim syntax trees, we need an API that is defined in the - # ``macros`` module: - import macros - - macro debug(n: varargs[untyped]): typed = - # `n` is a Nim AST that contains a list of expressions; - # this macro returns a list of statements (n is passed for proper line - # information): - result = newNimNode(nnkStmtList, n) - # iterate over any argument that is passed to this macro: - for x in n: - # add a call to the statement list that writes the expression; - # `toStrLit` converts an AST to its string representation: - result.add(newCall("write", newIdentNode("stdout"), toStrLit(x))) - # add a call to the statement list that writes ": " - result.add(newCall("write", newIdentNode("stdout"), newStrLitNode(": "))) - # add a call to the statement list that writes the expressions value: - result.add(newCall("writeLine", newIdentNode("stdout"), x)) - - var - a: array[0..10, int] - x = "some string" - a[0] = 42 - a[1] = 45 - - debug(a[0], a[1], x) - -The macro call expands to: - -.. code-block:: nim - write(stdout, "a[0]") - write(stdout, ": ") - writeLine(stdout, a[0]) - - write(stdout, "a[1]") - write(stdout, ": ") - writeLine(stdout, a[1]) - - write(stdout, "x") - write(stdout, ": ") - writeLine(stdout, x) - - - -Statement Macros ----------------- - -Statement macros are defined just as expression macros. However, they are -invoked by an expression following a colon. - -The following example outlines a macro that generates a lexical analyzer from -regular expressions: - -.. code-block:: nim - - macro case_token(n: varargs[untyped]): typed = - # creates a lexical analyzer from regular expressions - # ... (implementation is an exercise for the reader :-) - discard - - case_token: # this colon tells the parser it is a macro statement - of r"[A-Za-z_]+[A-Za-z_0-9]*": - return tkIdentifier - of r"0-9+": - return tkInteger - of r"[\+\-\*\?]+": - return tkOperator - else: - return tkUnknown - - -Building your first macro -------------------------- - -To give a footstart to writing macros we will show now how to turn your typical -dynamic code into something that compiles statically. For the exercise we will -use the following snippet of code as the starting point: - -.. code-block:: nim - :test: "nim c $1" - - import strutils, tables - - proc readCfgAtRuntime(cfgFilename: string): Table[string, string] = - let - inputString = readFile(cfgFilename) - var - source = "" - - result = initTable[string, string]() - for line in inputString.splitLines: - # Ignore empty lines - if line.len < 1: continue - var chunks = split(line, ',') - if chunks.len != 2: - quit("Input needs comma split values, got: " & line) - result[chunks[0]] = chunks[1] - - if result.len < 1: quit("Input file empty!") - - let info = readCfgAtRuntime("data.cfg") - - when isMainModule: - echo info["licenseOwner"] - echo info["licenseKey"] - echo info["version"] - -Presumably this snippet of code could be used in a commercial software, reading -a configuration file to display information about the person who bought the -software. This external file would be generated by an online web shopping cart -to be included along the program containing the license information:: - - version,1.1 - licenseOwner,Hyori Lee - licenseKey,M1Tl3PjBWO2CC48m - -The ``readCfgAtRuntime`` proc will open the given filename and return a -``Table`` from the `tables module <tables.html>`_. The parsing of the file is -done (without much care for handling invalid data or corner cases) using the -`splitLines proc from the strutils module <strutils.html#splitLines>`_. There -are many things which can fail; mind the purpose is explaining how to make -this run at compile time, not how to properly implement a DRM scheme. - -The reimplementation of this code as a compile time proc will allow us to get -rid of the ``data.cfg`` file we would need to distribute along the binary, plus -if the information is really constant, it doesn't make from a logical point of -view to have it *mutable* in a global variable, it would be better if it was a -constant. Finally, and likely the most valuable feature, we can implement some -verification at compile time. You could think of this as a *better unit -testing*, since it is impossible to obtain a binary unless everything is -correct, preventing you to ship to users a broken program which won't start -because a small critical file is missing or its contents changed by mistake to -something invalid. - - -Generating source code -++++++++++++++++++++++ - -Our first attempt will start by modifying the program to generate a compile -time string with the *generated source code*, which we then pass to the -``parseStmt`` proc from the `macros module <macros.html>`_. Here is the -modified source code implementing the macro: - -.. code-block:: nim - :number-lines: - - import macros, strutils - - macro readCfgAndBuildSource(cfgFilename: string): typed = - let - inputString = slurp(cfgFilename.strVal) - var - source = "" - - for line in inputString.splitLines: - # Ignore empty lines - if line.len < 1: continue - var chunks = split(line, ',') - if chunks.len != 2: - error("Input needs comma split values, got: " & line) - source &= "const cfg" & chunks[0] & "= \"" & chunks[1] & "\"\n" - - if source.len < 1: error("Input file empty!") - result = parseStmt(source) - - readCfgAndBuildSource("data.cfg") - - when isMainModule: - echo cfglicenseOwner - echo cfglicenseKey - echo cfgversion - -The good news is not much has changed! First, we need to change the handling -of the input parameter (line 3). In the dynamic version the -``readCfgAtRuntime`` proc receives a string parameter. However, in the macro -version it is also declared as string, but this is the *outside* interface of -the macro. When the macro is run, it actually gets a ``PNimNode`` object -instead of a string, and we have to call the `strVal proc -<macros.html#strVal>`_ (line 5) from the `macros module <macros.html>`_ to -obtain the string being passed in to the macro. - -Second, we cannot use the `readFile proc <system.html#readFile>`_ from the -`system module <system.html>`_ due to FFI restriction at compile time. If we -try to use this proc, or any other which depends on FFI, the compiler will -error with the message ``cannot evaluate`` and a dump of the macro's source -code, along with a stack trace where the compiler reached before bailing out. -We can get around this limitation by using the `slurp proc -<system.html#slurp>`_ from the `system module <system.html>`_, which was -precisely made for compilation time (just like `gorge <system.html#gorge>`_ -which executes an external program and captures its output). - -The interesting thing is that our macro does not return a runtime `Table -<tables.html#Table>`_ object. Instead, it builds up Nim source code into -the ``source`` variable. For each line of the configuration file a ``const`` -variable will be generated (line 15). To avoid conflicts we prefix these -variables with ``cfg``. In essence, what the compiler is doing is replacing -the line calling the macro with the following snippet of code: - -.. code-block:: nim - const cfgversion = "1.1" - const cfglicenseOwner = "Hyori Lee" - const cfglicenseKey = "M1Tl3PjBWO2CC48m" - -You can verify this yourself adding the line ``echo source`` somewhere at the -end of the macro and compiling the program. Another difference is that instead -of calling the usual `quit proc <system.html#quit>`_ to abort (which we could -still call) this version calls the `error proc <macros.html#error>`_ (line -14). The ``error`` proc has the same behavior as ``quit`` but will dump also -the source and file line information where the error happened, making it -easier for the programmer to find where compilation failed. In this situation -it would point to the line invoking the macro, but **not** the line of -``data.cfg`` we are processing, that's something the macro itself would need -to control. - - -Generating AST by hand -++++++++++++++++++++++ - -To generate an AST we would need to intimately know the structures used by the -Nim compiler exposed in the `macros module <macros.html>`_, which at first -look seems a daunting task. But we can use as helper shortcut the `dumpTree -macro <macros.html#dumpTree>`_, which is used as a statement macro instead of -an expression macro. Since we know that we want to generate a bunch of -``const`` symbols we can create the following source file and compile it to -see what the compiler *expects* from us: - -.. code-block:: nim - :test: "nim c $1" - import macros - - dumpTree: - const cfgversion: string = "1.1" - const cfglicenseOwner = "Hyori Lee" - const cfglicenseKey = "M1Tl3PjBWO2CC48m" - -During compilation of the source code we should see the following lines in the -output (again, since this is a macro, compilation is enough, you don't have to -run any binary):: - - StmtList - ConstSection - ConstDef - Ident !"cfgversion" - Ident !"string" - StrLit 1.1 - ConstSection - ConstDef - Ident !"cfglicenseOwner" - Empty - StrLit Hyori Lee - ConstSection - ConstDef - Ident !"cfglicenseKey" - Empty - StrLit M1Tl3PjBWO2CC48m - -With this output we have a better idea of what kind of input the compiler -expects. We need to generate a list of statements. For each constant the source -code generates a ``ConstSection`` and a ``ConstDef``. If we were to move all -the constants to a single ``const`` block we would see only a single -``ConstSection`` with three children. - -Maybe you didn't notice, but in the ``dumpTree`` example the first constant -explicitly specifies the type of the constant. That's why in the tree output -the two last constants have their second child ``Empty`` but the first has a -string identifier. So basically a ``const`` definition is made up from an -identifier, optionally a type (can be an *empty* node) and the value. Armed -with this knowledge, let's look at the finished version of the AST building -macro: - -.. code-block:: nim - :number-lines: - - import macros, strutils - - macro readCfgAndBuildAST(cfgFilename: string): typed = - let - inputString = slurp(cfgFilename.strVal) - - result = newNimNode(nnkStmtList) - for line in inputString.splitLines: - # Ignore empty lines - if line.len < 1: continue - var chunks = split(line, ',') - if chunks.len != 2: - error("Input needs comma split values, got: " & line) - var - section = newNimNode(nnkConstSection) - constDef = newNimNode(nnkConstDef) - constDef.add(newIdentNode("cfg" & chunks[0])) - constDef.add(newEmptyNode()) - constDef.add(newStrLitNode(chunks[1])) - section.add(constDef) - result.add(section) - - if result.len < 1: error("Input file empty!") - - readCfgAndBuildAST("data.cfg") - - when isMainModule: - echo cfglicenseOwner - echo cfglicenseKey - echo cfgversion - -Since we are building on the previous example generating source code, we will -only mention the differences to it. Instead of creating a temporary ``string`` -variable and writing into it source code as if it were written *by hand*, we -use the ``result`` variable directly and create a statement list node -(``nnkStmtList``) which will hold our children (line 7). - -For each input line we have to create a constant definition (``nnkConstDef``) -and wrap it inside a constant section (``nnkConstSection``). Once these -variables are created, we fill them hierarchichally (line 17) like the -previous AST dump tree showed: the constant definition is a child of the -section definition, and the constant definition has an identifier node, an -empty node (we let the compiler figure out the type), and a string literal -with the value. - -A last tip when writing a macro: if you are not sure the AST you are building -looks ok, you may be tempted to use the ``dumpTree`` macro. But you can't use -it *inside* the macro you are writting/debugging. Instead ``echo`` the string -generated by `treeRepr <macros.html#treeRepr>`_. If at the end of the this -example you add ``echo treeRepr(result)`` you should get the same output as -using the ``dumpTree`` macro, but of course you can call that at any point of -the macro where you might be having troubles. - -Example Templates and Macros -============================ - -Lifting Procs -+++++++++++++ +Example: Lifting Procs +---------------------- .. code-block:: nim :test: "nim c $1" @@ -1039,36 +677,6 @@ Lifting Procs liftScalarProc(sqrt) # make sqrt() work for sequences echo sqrt(@[4.0, 16.0, 25.0, 36.0]) # => @[2.0, 4.0, 5.0, 6.0] -Identifier Mangling -+++++++++++++++++++ - -.. code-block:: nim - proc echoHW() = - echo "Hello world" - proc echoHW0() = - echo "Hello world 0" - proc echoHW1() = - echo "Hello world 1" - - template joinSymbols(a, b: untyped): untyped = - `a b`() - - joinSymbols(echo, HW) - - macro str2Call(s1, s2): typed = - result = newNimNode(nnkStmtList) - for i in 0..1: - # combines s1, s2 and an integer into an proc identifier - # that is called in a statement list - result.add(newCall(!($s1 & $s2 & $i))) - - str2Call("echo", "HW") - - # Output: - # Hello world - # Hello world 0 - # Hello world 1 - Compilation to JavaScript ========================= @@ -1083,3 +691,9 @@ JavaScript-compatible code you should remember the following: - ``cstring`` in JavaScript means JavaScript string. It is a good practice to use ``cstring`` only when it is semantically appropriate. E.g. don't use ``cstring`` as a binary data buffer. + + +Part 3 +====== + +Next part will be entirely about metaprogramming via macros: `Part III <tut3.html>`_ diff --git a/doc/tut3.rst b/doc/tut3.rst new file mode 100644 index 000000000..5590db8fe --- /dev/null +++ b/doc/tut3.rst @@ -0,0 +1,354 @@ +======================= +Nim Tutorial (Part III) +======================= + +:Author: Arne Döring +:Version: |nimversion| + +.. contents:: + + +Introduction +============ + + "With Great Power Comes Great Responsibility." -- Spider Man's Uncle + +This document is a tutorial about Nim's macro system. +A macro is a function that is executed at compile time and transforms +a Nim syntax tree into a different tree. + +Examples of things that can be implemented in macros: + + * An assert macro that prints both sides of a comparison operator, if +the assertion fails. ``myAssert(a == b)`` is converted to +``if a != b: quit($a " != " $b)`` + + * A debug macro that prints the value and the name of the symbol. +``myDebugEcho(a)`` is converted to ``echo "a: ", a`` + + * Symbolic differentiation of an expression. +``diff(a*pow(x,3) + b*pow(x,2) + c*x + d, x)`` is converted to +``3*a*pow(x,2) + 2*a*x + c`` + + +Macro Arguments +--------------- + +The types of macro arguments have two faces. One face is used for +the overload resolution, and the other face is used within the macro +body. For example, if ``macro foo(arg: int)`` is called in an +expression ``foo(x)``, ``x`` has to be of a type compatible to int, but +*within* the macro's body ``arg`` has the type ``NimNode``, not ``int``! +Why it is done this way will become obvious later, when we have seen +concrete examples. + +There are two ways to pass arguments to a macro, an argument can be +either ``typed`` or ``untyped``. + + +Untyped Arguments +----------------- + +Untyped macro arguments are passed to the macro before they are +semantically checked. This means the syntax tree that is passed down +to the macro does not need to make sense for Nim yet, the only +limitation is that it needs to be parseable. Usually the macro does +not check the argument either but uses it in the transformation's +result somehow. The result of a macro expansion is always checked +by the compiler, so apart from weird error messages nothing bad +can happen. + +The downside for an ``untyped`` argument is that these do not play +well with Nim's overloading resolution. + +The upside for untyped arguments is that the syntax tree is +quite predictable and less complex compared to its ``typed`` +counterpart. + + +Typed Arguments +--------------- + +For typed arguments, the semantic checker runs on the argument and +does transformations on it, before it is passed to the macro. Here +identifier nodes are resolved as symbols, implicit type +conversions are visible in the tree as calls, templates are +expanded and probably most importantly, nodes have type information. +Typed arguments can have the type ``typed`` in the arguments list. +But all other types, such as ``int``, ``float`` or ``MyObjectType`` +are typed arguments as well, and they are passed to the macro as a +syntax tree. + + +Static Arguments +---------------- + +Static arguments are a way to pass values as values and not as syntax +tree nodes to a macro. For example for ``macro foo(arg: static[int])`` +in the expression ``foo(x)``, ``x`` needs to be an integer constant, +but in the macro body ``arg`` is just like a normal parameter of type +``int``. + +.. code-block:: nim + + import macros + + macro myMacro(arg: static[int]): untyped = + echo arg # just an int (7), not ``NimNode`` + + myMacro(1 + 2 * 3) + + +Code blocks as arguments +------------------------ + +It is possible to pass the last argument of a call expression in a +separate code block with indentation. For example the following code +example is a valid (but not a recommended) way to call ``echo``: + +.. code-block:: nim + + echo "Hello ": + let a = "Wor" + let b = "ld!" + a & b + +For macros this way of calling is very useful; syntax trees of arbitrary +complexity can be passed to macros with this notation. + + +The Syntax Tree +--------------- + +In order to build a Nim syntax tree one needs to know how Nim source +code is represented as a syntax tree, and how such a tree needs to +look like so that the Nim compiler will understand it. The nodes of the +Nim syntax tree are documented in the `macros <macros.html>`_ module. +But a more interactive way to explore the Nim +syntax tree is with ``macros.treeRepr``, it converts a syntax tree +into a multi line string for printing on the console. It can be used +to explore how the argument expressions are represented in tree form +and for debug printing of generated syntax tree. ``dumpTree`` is a +predefined macro that just prints its argument in tree representation, +but does nothing else. Here is an example of such a tree representation: + +.. code-block:: nim + + dumpTree: + var mt: MyType = MyType(a:123.456, b:"abcdef") + + # output: + # StmtList + # VarSection + # IdentDefs + # Ident "mt" + # Ident "MyType" + # ObjConstr + # Ident "MyType" + # ExprColonExpr + # Ident "a" + # FloatLit 123.456 + # ExprColonExpr + # Ident "b" + # StrLit "abcdef" + + +Custom sematic checking +----------------------- + +The first thing that a macro should do with its arguments is to check +if the argument is in the correct form. Not every type of wrong input +needs to be caught here, but anything that could cause a crash during +macro evaluation should be caught and create a nice error message. +``macros.expectKind`` and ``macros.expectLen`` are a good start. If +the checks need to be more complex, arbitrary error messages can +be created with the ``macros.error`` proc. + +.. code-block:: nim + + macro myAssert(arg: untyped): untyped = + arg.expectKind nnkInfix + + +Generating Code +--------------- + +There are two ways to generate the code. Either by creating the syntax +tree with expressions that contain a lot of calls to ``newTree`` and +``newLit``, or with ``quote do:`` expressions. The first option offers +the best low level control for the syntax tree generation, but the +second option is much less verbose. If you choose to create the syntax +tree with calls to ``newTree`` and ``newLit`` the macro +``marcos.dumpAstGen`` can help you with the verbosity. ``quote do:`` +allows you to write the code that you want to generate literally, +backticks are used to insert code from ``NimNode`` symbols into the +generated expression. This means that you can't use backticks within +``quote do:`` for anything else than injecting symbols. Make sure to +inject only symbols of type ``NimNode`` into the generated syntax +tree. You can use ``newLit`` to convert arbitrary values into +expressions trees of type ``NimNode`` so that it is safe to inject +them into the tree. + + +.. code-block:: nim + :test: "nim c $1" + + import macros + + type + MyType = object + a: float + b: string + + macro myMacro(arg: untyped): untyped = + var mt: MyType = MyType(a:123.456, b:"abcdef") + + # ... + + let mtLit = newLit(mt) + + result = quote do: + echo `arg` + echo `mtLit` + + myMacro("Hallo") + +The call to ``myMacro`` will generate the following code: + +.. code-block:: nim + echo "Hallo" + echo MyType(a: 123.456'f64, b: "abcdef") + + +Building your first macro +------------------------- + +To give a footstart to writing macros we will show now how to +implement the ``myDebug`` macro mentioned earlier. The first thing to +do is to build a simple example of the macro usage, and then just +print the argument. This way it is possible to get an idea of a +correct argument should be look like. + +.. code-block:: nim + :test: "nim c $1" + + import macros + + macro myAssert(arg: untyped): untyped = + echo arg.treeRepr + + let a = 1 + let b = 2 + + myAssert(a != b) + +.. code-block:: + + Infix + Ident "!=" + Ident "a" + Ident "b" + + +From the output it is possible to see that the information that the +argument is an infix operator (node kind is "Infix"), as well as that the two +operands are at index 1 and 2. With this information the actual +macro can be written. + +.. code-block:: nim + :test: "nim c $1" + + import macros + + macro myAssert(arg: untyped): untyped = + # all node kind identifiers are prefixed with "nnk" + arg.expectKind nnkInfix + arg.expectLen 3 + # operator as string literal + let op = newLit(" " & arg[0].repr & " ") + let lhs = arg[1] + let rhs = arg[2] + + result = quote do: + if not `arg`: + raise newException(AssertionError,$`lhs` & `op` & $`rhs`) + + let a = 1 + let b = 2 + + myAssert(a != b) + myAssert(a == b) + + +This is the code that will be generated. To debug what the macro +actually generated, the statement ``echo result.repr`` can be used, in +the last line of the macro. It is also the statement that has been +used to get this output. + +.. code-block:: nim + if not (a != b): + raise newException(AssertionError, $a & " != " & $b) + +With Power Comes Responsibility +------------------------------- + +Macros are very powerful. A good advice is to use them as little as +possible, but as much as necessary. Macros can change the semantics of +expressions, making the code incomprehensible for anybody who does not +know exactly what the macro does with it. So whenever a macro is not +necessary and the same logic can be implemented using templates or +generics, it is probably better not to use a macro. And when a macro +is used for something, the macro should better have a well written +documentation. For all the people who claim to write only perfectly +self-explanatory code: when it comes to macros, the implementation is +not enough for documentation. + +Limitations +----------- + +Since macros are evaluated in the compiler in the NimVM, macros share +all the limitations of the NimVM. They have to be implemented in pure Nim +code. Macros can start external processes on the shell, but they +cannot call C functions except from those that are built in the +compiler. + + +More Examples +============= + +This tutorial can only cover the basics of the macro system. There are +macros out there that could be an inspiration for you of what is +possible with it. + + +Strformat +--------- + +In the Nim standard library, the ``strformat`` library provides a +macro that parses a string literal at compile time. Parsing a string +in a macro like here is generally not recommended. The parsed AST +cannot have type information, and parsing implemented on the VM is +generally not very fast. Working on AST nodes is almost always the +recommended way. But still ``strformat`` is a good example for a +practical use case for a macro that is slightly more complex that the +``assert`` macro. + +`Strformat <https://github.com/nim-lang/Nim/blob/5845716df8c96157a047c2bd6bcdd795a7a2b9b1/lib/pure/strformat.nim#L280>`_ + +Ast Pattern Matching +-------------------- + +Ast Pattern Matching is a macro library to aid in writing complex +macros. This can be seen as a good example of how to repurpose the +Nim syntax tree with new semantics. + +`Ast Pattern Matching <https://github.com/krux02/ast-pattern-matching>`_ + +OpenGL Sandbox +-------------- + +This project has a working Nim to GLSL compiler written entirely in +macros. It scans recursively though all used function symbols to +compile them so that cross library functions can be executed on the GPU. + +`OpenGL Sandbox <https://github.com/krux02/opengl-sandbox>`_ |